updating again

doc/src/week35/week35.do.txt

@@ -2208,17 +2208,82 @@ Multiplying from the right with $\bm{V}$ (using the orthogonality of $\bm{V}$) w
 \left(\bm{X}^T\bm{X}\right)\bm{V}=\bm{V}\tilde{\bm{\Sigma}}^2. 
 \]
 !et
-This means the vectors $\bm{v}_i$ of the orthogonal matrix $\bm{V}$ are the eigenvectors of the matrix $\bm{X}^T\bm{X}$
-with eigenvalues given by the singular values squared, that is
+
+!split
+===== What does it mean? =====
+
+This means the vectors $\bm{v}_i$ of the orthogonal matrix $\bm{V}$
+are the eigenvectors of the matrix $\bm{X}^T\bm{X}$ with eigenvalues
+given by the singular values squared, that is
+
 !bt
 \[
 \left(\bm{X}^T\bm{X}\right)\bm{v}_i=\bm{v}_i\sigma_i^2. 
 \]
 !et
 
+In other words, each non-zero singular value of $\bm{X}$ is a positive
+square root of an eigenvalue of $\bm{X}^T\bm{X}$.  It means also that
+the columns of $\bm{V}$ are the eigenvectors of
+$\bm{X}^T\bm{X}$. Since we have ordered the singular values of
+$\bm{X}$ in a descending order, it means that the column vectors
+$\bm{v}_i$ are hierarchically ordered by how much correlation they
+encode from the columns of $\bm{X}$. 
 
 
+Note that these are also the eigenvectors and eigenvalues of the
+Hessian matrix.
 
+If we now recall the definition of the covariance matrix (not using
+Bessel's correction) we have
+
+
+!bt
+\[
+\bm{C}[\bm{X}]=\frac{1}{n}\bm{X}^T\bm{X},
+\]
+!et
+
+meaning that every squared non-singular value of $\bm{X}$ divided by$n$,
+the number of samples, are the eigenvalues of the covariance
+matrix. Every singular value of $\bm{X}$ is thus a positive square
+root of an eigenvalue of $\bm{X}^T\bm{X}$. If the matrix $\bm{X}$ is
+self-adjoint, the the sinular values of $\bm{X}$ are equal to the
+absolute value of the eigenvalues of $\bm{X}$.
+
+!split
+===== And finally  $\bm{X}\bm{X}^T$ =====
+
+For $\bm{X}\bm{X}^T$ we found
+
+!bt
+\[
+$\bm{X}\bm{X}^T$=\bm{U}\bm{\Sigma}\bm{V}^T\bm{V}\bm{\Sigma}^T\bm{U}^T=\bm{U}\bm{\Sigma}^T\bm{\Sigma}\bm{U}^T. 
+\]
+!et
+Since the matrices here have dimension $n\times n$, we have
+!bt
+\[
+\bm{\Sigma}\bm{\Sigma}^T = \begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} 0 \bm{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}, 
+\]
+!et
+leading to
+!bt
+\[
+$\bm{X}\bm{X}^T$=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}\bm{U}^T. 
+\]
+!et
+
+Multiplying with $\bm{U}$ from the right gives us the eigenvalue problem
+!bt
+\[
+$\bm{X}\bm{X}^T$\bm{U}=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}. 
+\]
+!et
+
+It means that the eigenvalues of $\bm{X}\bm{X}^T$ are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of $\bm{U}$ are the eigenvectors of $\bm{X}\bm{X}^T$ and measure how much correlations are contained in the rows of $\m{X}$.
+
+Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of $\bm{V}$.
 
 
 !split