updating pca

doc/src/week43/week43.do.txt

@@ -2827,7 +2827,7 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
 \]
 !et
 * Center the data by subtracting the mean value for each column. This leads to a new matrix $\bm{X}\rightarrow \overline{\bm{X}}$.
-* Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\bm{X}}\overline{\bm{X}}^T]$.
+* Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\bm{X}}^T\overline{\bm{X}}]$.
 * Find the eigenpairs of $\bm{C}$ with eigenvalues $[\lambda_0,\lambda_1,\dots,\lambda_{p-1}]$ and eigenvectors $[\bm{s}_0,\bm{s}_1,\dots,\bm{s}_{p-1}]$.
 * Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
 * Keep only those $l$ eigenvalues larger than a selected threshold value, discarding thus $p-l$ features since we expect small variations in the data here.
@@ -2997,14 +2997,7 @@ overline symbol. The matrix is defined in terms of the various column
 vectors $[\bm{x}_0,\bm{x}_1,\dots, \bm{x}_{p-1}]$ each with dimension
 $\bm{x}\in {\mathbb{R}}^{n}$.
 
-We assume also that we have an orthogonal transformation $\bm{W}\in {\mathbb{R}}^{p\times p}$.  We define the reconstruction error (which is similar to the mean squared error we have seen before) as
-!bt
-\[
-J(\bm{W},\bm{Z}) = \frac{1}{n}\sum_i (\bm{x}_i - \overline{\bm{x}}_i)^2,
-\]
-!et
-with $\overline{\bm{x}}_i = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix
-$\bm{Z}\in{\mathbb{R}}^{p\times n}$.  When doing PCA we want to reduce this dimensionality. 
+
 
 The PCA theorem states that minimizing the above reconstruction error
 corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which
@@ -3019,63 +3012,12 @@ eigenvectors of the covariance(correlations matrix).
 !split
 ===== Proof of the PCA Theorem  =====
 
-To show the PCA theorem let us start with the assumption that there is one vector $\bm{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
-!bt
-\[
-J(\bm{w}_0,\bm{z}_0)= \frac{1}{n}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{n}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0),
-\]
-!et
-which we can rewrite due to the orthogonality of $\bm{w}_i$ as
-!bt
-\[
-J(\bm{w}_0,\bm{z}_0)=\frac{1}{n}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2).
-\]
-!et
-Minimizing $J$ with respect to the unknown parameters $z_{0i}$ we obtain that
-!bt
-\[
-z_{i0}=\bm{w}_0^T\bm{x}_i,
-\]
-!et
-where the vectors on the rhs are known. 
-
+To show the PCA theorem let us start with the assumption that there is one vector $\bm{s}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
 
 !split
 ===== PCA Proof continued =====
 
-We have now found the unknown parameters $z_{i0}$. These correspond to the projected coordinates and we can write
-!bt
-\[
-J(\bm{w}_0)= \frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2.
-\]
-!et
 
-We can show that the variance of the projected coordinates defined by $\bm{w}_0^T\bm{x}_i$ are given by
-!bt
-\[
-\mathrm{var}[\bm{w}_0^T\bm{x}_i] = \frac{1}{n}\sum_i z_{i0}^2,
-\]
-!et
-since the expectation value of
-!bt
-\[
-\mathbb{E}[\bm{w}_0^T\bm{x}_i] = \mathbb{E}[z_{i0}]= \bm{w}_0^T\mathbb{E}[\bm{x}_i]=0,
-\]
-!et
-where we have used the fact that our data are centered.
-
-Recalling our definition of the covariance as 
-!bt
-\[
-\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}\bm{X}^T=\mathbb{E}[\bm{X}\bm{X}^T],
-\]
-!et
-we have thus that
-!bt
-\[
-\mathrm{var}[\bm{w}_0^T\bm{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\bm{w}_0^T\bm{C}[\bm{x}]\bm{w}_0.
-\]
-!et
 
 We are almost there, we have obtained a relation between minimizing
 the reconstruction error and the variance and the covariance