updating

doc/src/DimRed/DimRed.do.txt

@@ -337,6 +337,14 @@ applications.
 !split
 ===== Basic ideas of the Principal Component Analysis (PCA) =====
 
+The principal component analysis deals with the problem of fitting a
+low-dimensional affine subspace $S$ of dimension $d$ much smaller than
+the totaldimension $D$ of the problem at hand (our data
+set). Mathematically it can be formulated as a statistical problem or
+a geometric problem.  In our discussion of the theorem for the
+classical PCA, we will stay with a statistical approach. This is also
+what set the scene historically which for the PCA.
+
 We have a data set defined by a design/feature matrix $\bm{X}$ (see below for its definition) 
 * Each data point is determined by $p$ extrinsic (measurement) variables
 * We may want to ask the following question: Are there fewer intrinsic variables (say $d << p$) that still approximately describe the data?
@@ -347,7 +355,7 @@ We have a data set defined by a design/feature matrix $\bm{X}$ (see below for it
 ===== Introducing the Covariance and Correlation functions  =====
 
 Before we discuss the PCA theorem, we need to remind ourselves about
-the definition of the covariance and the correlation function.
+the definition of the covariance and the correlation function. These are quantities 
 
 Suppose we have defined two vectors
 $\hat{x}$ and $\hat{y}$ with $n$ elements each. The covariance matrix $\bm{C}$ is defined as 
@@ -409,7 +417,9 @@ In the above example this is the function we constructed using _pandas_.
 !split
 ===== Correlation Function and Design/Feature Matrix =====
 
-In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix $\bm{X}$ as
+In our derivation of the various regression algorithms like _Ordinary Least Squares_ or _Ridge regression_
+we defined the design/feature matrix $\bm{X}$ as
+
 !bt
 \[
 \bm{X}=\begin{bmatrix}
@@ -437,7 +447,11 @@ with a given vector
 \]
 !et
 
-With these definitions, we can now rewrite our $2\times 2$ correaltion/covariance matrix in terms of a moe general design/feature matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$ covariance matrix for the vectors $\bm{x}_i$ with $i =0,1,\dots,p-1$
+With these definitions, we can now rewrite our $2\times 2$
+correaltion/covariance matrix in terms of a moe general design/feature
+matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$
+covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$
+
 !bt
 \[
 \bm{C}[\bm{x}] = \begin{bmatrix}
@@ -624,8 +638,8 @@ columns since all matrix elements in the design matrix were set to one
 
 This means that the variance for these elements will be zero and will
 cause problems when we set up the correlation matrix.  We can simply
-drop these elements as follows and then construct the correlation
-matrix. 
+drop these elements and construct a correlation
+matrix without these elements. 
 
 
 !split
@@ -773,7 +787,7 @@ X = np.random.multivariate_normal(mean, cov, n)
 
 Make thereafter a small Python code which plots the data. Note that the function _multivariate_ returns also the covariance discussed above and that it is defined by dividing by $n-1$ instead of $n$.
 
-Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.
+Now we are going to implement the PCA algorithm. We will break it down into various substeps.
 
 === Compute the sample mean and center the data ===
 
@@ -835,8 +849,11 @@ Finally, try out your own PCA function with other data sets.
 !split
 ===== Classical PCA Theorem   =====
 
-We assume now that we have a design matrix $\bm{X}$ which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors $[\bm{x}_0,\bm{x}_1,\dots, \bm{x}_{p-1}]$
-each with dimension $\bm{x}\in {\mathbb{R}}^{n}$.
+We assume now that we have a design matrix $\bm{X}$ which has been
+centered as discussed above. For the sake of simplicity we skip the
+overline symbol. The matrix is defined in terms of the various column
+vectors $[\bm{x}_0,\bm{x}_1,\dots, \bm{x}_{p-1}]$ each with dimension
+$\bm{x}\in {\mathbb{R}}^{n}$.
 
 We assume also that we have an orthogonal transformation $\bm{W}\in {\mathbb{R}}^{p\times p}$.  We define the reconstruction error (which is similar to the mean squared error we have seen before) as
 !bt
@@ -847,7 +864,13 @@ J(\bm{W},\bm{Z}) = \frac{1}{n}\sum_i (\bm{x}_i - \overline{\bm{x}}_i)^2,
 with $\overline{\bm{x}}_i = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix
 $\bm{Z}\in{\mathbb{R}}^{p\times n}$.  When doing PCA we want to reduce this dimensionality. 
 
-The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $\bm{z}_i$ with at most $l$ vectors, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
+The PCA theorem states that minimizing the above reconstruction error
+corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which
+diagonalizes the empirical covariance(correlation) matrix. The optimal
+low-dimensional encoding of the data is then given by a set of vectors
+$\bm{z}_i$ with at most $l$ vectors, with $l << p$, defined by the
+orthogonal projection of the data onto the columns spanned by the
+eigenvectors of the covariance(correlations matrix).
 
 
 
@@ -912,7 +935,10 @@ we have thus that
 \]
 !et
 
-We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data. 
+We are almost there, we have obtained a relation between minimizing
+the reconstruction error and the variance and the covariance
+matrix. Minimizing the error is equivalent to maximizing the variance
+of the projected data.
 
 !split
 =====  The final step =====
@@ -965,9 +991,12 @@ discussion in chapter 12.2 of Murphy's text has also a nice link with
 the Singular Value Decomposition theorem. For categorical data, see
 chapter 12.4 and discussion therein.
 
+Additional part of the proof for the other eigenvectors will be added by mid January 2020.
 
+!split
+===== Geometric Interpretation and link with Singular Value Decomposition =====
 
-
+This material will be added by mid January 2020.
 
 
 !split