adding stuff to week 35

doc/src/week35/week35.do.txt

@@ -1274,8 +1274,7 @@ This serves also as a useful test of our codes.
 !bblock
 
 The examples we have looked at so far are cases where we normally can
-invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion as we
-did both for the masses and the fitting of various functions leads to
+invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion where we fit of various functions leads to
 row vectors of the design matrix which are essentially orthogonal due
 to the polynomial character of our model. Obtaining the inverse of the
 design matrix is then often done via a so-called LU, QR or Cholesky
@@ -1287,7 +1286,9 @@ this may
 however not the be case in general and a standard matrix inversion
 algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
 
-There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. 
+There is however a way to circumvent this problem and also
+gain some insights about the ordinary least squares approach, and
+later shrinkage methods like Ridge and Lasso regressions.
 
 This is given by the _Singular Value Decomposition_ (SVD) algorithm,
 perhaps the most powerful linear algebra algorithm.  The SVD provides
@@ -1295,14 +1296,17 @@ a numerically stable matrix decomposition that is used in a large
 swath oc applications and the decomposition is always stable
 numerically.
 
-In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.
-Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. 
+In machine learning it plays a central role in dealing with for
+example design matrices that may be near singular or singular.
+Furthermore, as we will see here, the singular values can be related
+to the covariance matrix (and thereby the correlation matrix) and in
+turn the variance of a given quantity. It plays also an important role
+in the principal component analysis where high-dimensional data can be
+reduced to the statistically relevant features.
+
 
 
 
-Let us look at a
-different example where we may have problems with the standard matrix
-inversion algorithm. Thereafter we dive into the math of the SVD.
 
 !eblock
 
@@ -1494,7 +1498,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
 !bc pycod
 import numpy as np
 # SVD inversion
-def SVDinv(A):
+def SVD(A):
     ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
     SVD is numerically more stable than the inversion algorithms provided by
     numpy and scipy.linalg at the cost of being slower.
@@ -1518,7 +1522,7 @@ X = np.array([ [1.0,-1.0], [1.0,-1.0]])
 #X = np.array([[1, 2], [3, 4], [5, 6]])
 
 print(X)
-C = SVDinv(X)
+C = SVD(X)
 # Print the difference between the original matrix and the SVD one
 print(C-X)
 !ec
@@ -1539,14 +1543,15 @@ in the program terminating due to a singular matrix.
 The $U$, $S$, and $V$ matrices returned from the _svd()_ function
 cannot be multiplied directly.
 
-As you can see from the code, the $S$
-vector must be converted into a diagonal matrix. This may cause a 
-as
-the size of the matrices do not fit the rules of matrix
-multiplication, where the number of columns in a matrix must match the
-number of rows in the subsequent matrix.
+As you can see from the code, the $S$ vector must be converted into a
+diagonal matrix. This may cause a problem as the size of the matrices
+do not fit the rules of matrix multiplication, where the number of
+columns in a matrix must match the number of rows in the subsequent
+matrix.
 
-If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
+If you wish to include the zero singular values, you will need to
+resize the matrices and set up a diagonal matrix as done in the above
+example
 
 
 
@@ -1560,6 +1565,108 @@ If you wish to include the zero singular values, you will need to resize the mat
 More material will be added here, see handwritten notes also.   Note that this material will be cleaned up after the lecture of Friday September 3.   See the handwritten notes from Friday's lecture at URL:"https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021".
 
 
+!split
+===== Matheamtics of the SVD and implications =====
+
+Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
+
+Our starting point is our design matrix $\bm{X}$ of dimension $n\times p$
+!bt
+\[
+\bm{X}=
+\begin{bmatrix}
+x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
+x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
+x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
+\dots& \dots &\dots& \dots & \dots &\dots\\
+x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
+\end{bmatrix}
+\]
+!et
+
+We can SVD decompose our matrix as
+!bt
+\[
+\bm{X}=\bm{U}\bm{\Sigma}\bm{V}^T,
+\]
+!et
+where $\bm{U}$ is an orthogonal matrix of dimension $n\times n$, meaning that $\bm{U}\bm{U}^T=\bm{U}^T\bm{U}=\bm{I}_n$. Here $\bm{I}_n$ is the unit matrix of dimension $n \times n$.
+
+Similarly, $\bm{V}$ is an orthogonal matrix of dimension $p\times p$, meaning that $\bm{V}\bm{V}^T=\bm{V}^T\bm{V}=\bm{I}_p$. Here $\bm{I}_n$ is the unit matrix of dimension $p \times p$.
+
+Finally $\bm{\Sigma}$ contains the singular values $\sigma_i$. This matrix has dimension $n\times p$ and the singular values $\sigma_i$ are all positive. The non-zero values are ordered in descending order, that is
+
+!bt
+\[
+\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0. 
+\]
+!et
+
+All values beyond $p-1$ are all zero.
+
+!split
+===== Example Matrix =====
+
+As an example, consider the following $3\times 2$ example for the matrix $\bm{\Sigma}$
+
+!bt
+\[
+\bm{\Sigma}=
+\begin{bmatrix}
+2& 0 \\
+0 & 1 \\
+0 & 0 \\
+\end{bmatrix}
+\]
+!et
+
+The singular values are $\sigma_0=2$ and $\sigma_1=1$. It is common to rewrite the matrix $\bm{\Sigma}$ as
+
+!bt
+\[
+\bm{\Sigma}=
+\begin{bmatrix}
+\bm{\tilde{\Sigma}}\\
+\bm{0}\\
+\end{bmatrix},
+\]
+!et
+
+where
+!bt
+\[
+\bm{\tilde{\Sigma}}=
+\begin{bmatrix}
+2& 0 \\
+0 & 1 \\
+\end{bmatrix},
+\]
+!et
+contains only the singular values.   Note also (and we will use this below) that
+
+!bt
+\[
+\bm{\Sigma}^T\bm{\sigma}=
+\begin{bmatrix}
+4& 0 \\
+0 & 1 \\
+\end{bmatrix},
+\]
+!et
+which is a $2\times 2 $ matrix while
+!bt
+\[
+\bm{\Sigma}\bm{\sigma}^T=
+\begin{bmatrix}
+4& 0 & 0\\
+0 & 1 & 0\\
+0 & 0 & 0\\
+\end{bmatrix},
+\]
+!et
+is a $3\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
+
+
 !split
 ===== Ridge and LASSO Regression =====