added pca theorem proof

doc/pub/DimRed/html/DimRed.html

@@ -115,7 +115,7 @@ div { text-align: justify; text-justify: inter-word; }
               ('Towards the PCA theorem', 2, None, '___sec15'),
               ('The Algorithm before the Theorem', 2, None, '___sec16'),
               ('Classical PCA Theorem', 2, None, '___sec17'),
-              ('Prof of the PCA Theorem', 2, None, '___sec18'),
+              ('Proof of the PCA Theorem', 2, None, '___sec18'),
               ('Getting started with PCA', 2, None, '___sec19'),
               ('Principal Component Analysis', 2, None, '___sec20'),
               ('PCA and scikit-learn', 2, None, '___sec21'),
@@ -947,7 +947,7 @@ $$
 
 <ul>
 <li> Center the data by subtracting the mean value for each column. This leads to a new matrix \( \boldsymbol{X}\rightarrow \overline{\boldsymbol{X}} \).</li>
-<li> Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\boldsymbol{X}}\overline{\boldsymbol{X}}^T].</li>
+<li> Compute then the covariance/correlation matrix \( \mathbb{E}[\overline{\boldsymbol{X}}\overline{\boldsymbol{X}}^T] \).</li>
 <li> Find the eigenpairs of \( \boldsymbol{C} \) with eigenvalues \( [\lambda_0,\lambda_1,\dots,\lambda_{p-1}] \) and eigenvectors \( [\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \).</li>
 <li> Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.</li>
 <li> Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.</li>
@@ -960,10 +960,44 @@ After this we ask ourselves how do we prove the link between the maximum varianc
 
 <h2 id="___sec17">Classical PCA Theorem   </h2>
 
+<p>
+We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \)
+each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
+
+<p>
+We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \).  We define the reconstruction error (which is similar to the mean squared error we have seen before) as
+$$
+J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2,
+$$
+
+with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
+\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \).
+
+<p>
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
+
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
 
-<h2 id="___sec18">Prof of the PCA Theorem  </h2>
+<h2 id="___sec18">Proof of the PCA Theorem  </h2>
+
+<p>
+To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
+$$
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
+$$
+
+which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
+$$
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
+$$
+
+Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that
+$$
+z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i,
+$$
+
+where the vectors on the rhs are known.
 
 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>