added pca theorem proof

doc/src/DimRed/DimRed.do.txt

@@ -734,7 +734,7 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
 \]
 !et
 * Center the data by subtracting the mean value for each column. This leads to a new matrix $\bm{X}\rightarrow \overline{\bm{X}}$.
-* Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\bm{X}}\overline{\bm{X}}^T].
+* Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\bm{X}}\overline{\bm{X}}^T]$.
 * Find the eigenpairs of $\bm{C}$ with eigenvalues $[\lambda_0,\lambda_1,\dots,\lambda_{p-1}]$ and eigenvectors $[\bm{s}_0,\bm{s}_1,\dots,\bm{s}_{p-1}]$.
 * Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
 * Keep only those $l$ eigenvalues larger than a selected threshold value, discarding thus $p-l$ features since we expect small variations in the data here.
@@ -744,15 +744,44 @@ After this we ask ourselves how do we prove the link between the maximum varianc
 !split
 ===== Classical PCA Theorem   =====
 
+We assume now that we have a design matrix $\bm{X}$ which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors $[\bm{x}_0,\bm{x}_1,\dots, \bm{x}_{p-1}]$
+each with dimension $\bm{x}\in {\mathbb{R}}^{n}$.
+
+We assume also that we have an orthogonal transformation $\bm{W}\in {\mathbb{R}}^{p\times p}$.  We define the reconstruction error (which is similar to the mean squared error we have seen before) as
+!bt
+\[
+J(\bm{W},\bm{Z}) = \frac{1}{p}\sum_i (\bm{x}_i - \overline{\bm{x}_i})^2,
+\]
+!et
+with $\overline{\bm{x}_i} = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix
+$\bm{Z}\in {\mathbb{R}}^{p\times n}$.  
+
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\bm{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
+
 
 
 !split
-===== Prof of the PCA Theorem  =====
-
-
-
-
+===== Proof of the PCA Theorem  =====
 
+To show the PCA theorem let us start with the assumption that there is a vector $\bm{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
+!bt
+\[
+J(\bm{w}_0,\bm{z}_0)= \frac{1}{p}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0),
+\]
+!et
+which we can rewrite due to the orthogonality of $\bm{w}_i$ as
+!bt
+\[
+J(\bm{w}_0,\bm{z}_0)=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2).
+\]
+!et
+Minimizing $J$ with respect to the unknown parameters $z_{0i}$ we obtain that
+!bt
+\[
+z_{i0}=\bm{w}_0^T\bm{x}_i,
+\]
+!et
+where the vectors on the rhs are known. 
 
 !split
 ===== Getting started with PCA =====