-
@@ -398,7 +405,7 @@ MathJax.Hub.Config({
-
@@ -1656,8 +1656,7 @@ This serves also as a useful test of our codes.
The examples we have looked at so far are cases where we normally can -invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we -did both for the masses and the fitting of various functions leads to +invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to row vectors of the design matrix which are essentially orthogonal due to the polynomial character of our model. Obtaining the inverse of the design matrix is then often done via a so-called LU, QR or Cholesky @@ -1670,7 +1669,9 @@ however not the be case in general and a standard matrix inversion algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
-There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. +There is however a way to circumvent this problem and also +gain some insights about the ordinary least squares approach, and +later shrinkage methods like Ridge and Lasso regressions.
This is given by the Singular Value Decomposition (SVD) algorithm, @@ -1680,13 +1681,13 @@ swath oc applications and the decomposition is always stable numerically.
-In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular. -Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. - -
-Let us look at a -different example where we may have problems with the standard matrix -inversion algorithm. Thereafter we dive into the math of the SVD. +In machine learning it plays a central role in dealing with for +example design matrices that may be near singular or singular. +Furthermore, as we will see here, the singular values can be related +to the covariance matrix (and thereby the correlation matrix) and in +turn the variance of a given quantity. It plays also an important role +in the principal component analysis where high-dimensional data can be +reduced to the statistically relevant features. @@ -1916,7 +1917,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
import numpy as np
# SVD inversion
-def SVDinv(A):
+def SVD(A):
''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
SVD is numerically more stable than the inversion algorithms provided by
numpy and scipy.linalg at the cost of being slower.
@@ -1940,7 +1941,7 @@ X = np.array([ [1.0,-#X = np.array([[1, 2], [3, 4], [5, 6]])
print(X)
-C = SVDinv(X)
+C = SVD(X)
# Print the difference between the original matrix and the SVD one
print(C-X)
-As you can see from the code, the \( S \) -vector must be converted into a diagonal matrix. This may cause a -as -the size of the matrices do not fit the rules of matrix -multiplication, where the number of columns in a matrix must match the -number of rows in the subsequent matrix. +As you can see from the code, the \( S \) vector must be converted into a +diagonal matrix. This may cause a problem as the size of the matrices +do not fit the rules of matrix multiplication, where the number of +columns in a matrix must match the number of rows in the subsequent +matrix.
-If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
+If you wish to include the zero singular values, you will need to
+resize the matrices and set up a diagonal matrix as done in the above
+example
@@ -1987,6 +1989,124 @@ More material will be added here, see handwritten notes also. Note that this m
+
+Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
+
+
+Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
+
+We can SVD decompose our matrix as
+
+Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).
+
+
+Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is
+
+
+All values beyond \( p-1 \) are all zero.
+
+As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)
+
+
+The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as
+
+
+where
+
-
The examples we have looked at so far are cases where we normally can
-invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we
-did both for the masses and the fitting of various functions leads to
+invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to
row vectors of the design matrix which are essentially orthogonal due
to the polynomial character of our model. Obtaining the inverse of the
design matrix is then often done via a so-called LU, QR or Cholesky
@@ -1749,7 +1753,9 @@ however not the be case in general and a standard matrix inversion
algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
-There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions.
+There is however a way to circumvent this problem and also
+gain some insights about the ordinary least squares approach, and
+later shrinkage methods like Ridge and Lasso regressions.
This is given by the Singular Value Decomposition (SVD) algorithm,
@@ -1759,13 +1765,13 @@ swath oc applications and the decomposition is always stable
numerically.
-In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.
-Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features.
-
-
-Let us look at a
-different example where we may have problems with the standard matrix
-inversion algorithm. Thereafter we dive into the math of the SVD.
+In machine learning it plays a central role in dealing with for
+example design matrices that may be near singular or singular.
+Furthermore, as we will see here, the singular values can be related
+to the covariance matrix (and thereby the correlation matrix) and in
+turn the variance of a given quantity. It plays also an important role
+in the principal component analysis where high-dimensional data can be
+reduced to the statistically relevant features.
@@ -1976,7 +1982,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
-As you can see from the code, the \( S \)
-vector must be converted into a diagonal matrix. This may cause a
-as
-the size of the matrices do not fit the rules of matrix
-multiplication, where the number of columns in a matrix must match the
-number of rows in the subsequent matrix.
+As you can see from the code, the \( S \) vector must be converted into a
+diagonal matrix. This may cause a problem as the size of the matrices
+do not fit the rules of matrix multiplication, where the number of
+columns in a matrix must match the number of rows in the subsequent
+matrix.
-If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
+If you wish to include the zero singular values, you will need to
+resize the matrices and set up a diagonal matrix as done in the above
+example
+Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
+
+
+Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
+$$
+\boldsymbol{X}=
+\begin{bmatrix}
+x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
+x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
+x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
+\dots& \dots &\dots& \dots & \dots &\dots\\
+x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
+\end{bmatrix}
+$$
+
+
+We can SVD decompose our matrix as
+$$
+\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
+$$
+
+where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).
+
+
+Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).
+
+
+Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is
+
+$$
+\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0.
+$$
+
+
+All values beyond \( p-1 \) are all zero.
+
+
+
+As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)
+
+$$
+\boldsymbol{\Sigma}=
+\begin{bmatrix}
+2& 0 \\
+0 & 1 \\
+0 & 0 \\
+\end{bmatrix}
+$$
+
+
+The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as
+
+$$
+\boldsymbol{\Sigma}=
+\begin{bmatrix}
+\boldsymbol{\tilde{\Sigma}}\\
+\boldsymbol{0}\\
+\end{bmatrix},
+$$
+
+
+where
+$$
+\boldsymbol{\tilde{\Sigma}}=
+\begin{bmatrix}
+2& 0 \\
+0 & 1 \\
+\end{bmatrix},
+$$
+
+contains only the singular values. Note also (and we will use this below) that
+
+$$
+\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
+\begin{bmatrix}
+4& 0 \\
+0 & 1 \\
+\end{bmatrix},
+$$
+
+which is a \( 2\times 2 \) matrix while
+$$
+\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
+\begin{bmatrix}
+4& 0 & 0\\
+0 & 1 & 0\\
+0 & 0 & 0\\
+\end{bmatrix},
+$$
+
+is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
+
+
+
diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html
index db79e88fa..3d05a9b45 100644
--- a/doc/pub/week35/html/week35.html
+++ b/doc/pub/week35/html/week35.html
@@ -208,6 +208,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'note-about-svd-calculations'),
('Friday September 3', 2, None, 'friday-september-3'),
+ ('Matheamtics of the SVD and implications',
+ 2,
+ None,
+ 'matheamtics-of-the-svd-and-implications'),
+ ('Example Matrix', 2, None, 'example-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -305,7 +310,7 @@ MathJax.Hub.Config({
-
The examples we have looked at so far are cases where we normally can
-invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we
-did both for the masses and the fitting of various functions leads to
+invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to
row vectors of the design matrix which are essentially orthogonal due
to the polynomial character of our model. Obtaining the inverse of the
design matrix is then often done via a so-called LU, QR or Cholesky
@@ -1754,7 +1758,9 @@ however not the be case in general and a standard matrix inversion
algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
-There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions.
+There is however a way to circumvent this problem and also
+gain some insights about the ordinary least squares approach, and
+later shrinkage methods like Ridge and Lasso regressions.
This is given by the Singular Value Decomposition (SVD) algorithm,
@@ -1764,13 +1770,13 @@ swath oc applications and the decomposition is always stable
numerically.
-In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.
-Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features.
-
-
-Let us look at a
-different example where we may have problems with the standard matrix
-inversion algorithm. Thereafter we dive into the math of the SVD.
+In machine learning it plays a central role in dealing with for
+example design matrices that may be near singular or singular.
+Furthermore, as we will see here, the singular values can be related
+to the covariance matrix (and thereby the correlation matrix) and in
+turn the variance of a given quantity. It plays also an important role
+in the principal component analysis where high-dimensional data can be
+reduced to the statistically relevant features.
@@ -1981,7 +1987,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
-As you can see from the code, the \( S \)
-vector must be converted into a diagonal matrix. This may cause a
-as
-the size of the matrices do not fit the rules of matrix
-multiplication, where the number of columns in a matrix must match the
-number of rows in the subsequent matrix.
+As you can see from the code, the \( S \) vector must be converted into a
+diagonal matrix. This may cause a problem as the size of the matrices
+do not fit the rules of matrix multiplication, where the number of
+columns in a matrix must match the number of rows in the subsequent
+matrix.
-If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
+If you wish to include the zero singular values, you will need to
+resize the matrices and set up a diagonal matrix as done in the above
+example
+Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
+
+
+Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
+$$
+\boldsymbol{X}=
+\begin{bmatrix}
+x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
+x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
+x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
+\dots& \dots &\dots& \dots & \dots &\dots\\
+x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
+\end{bmatrix}
+$$
+
+
+We can SVD decompose our matrix as
+$$
+\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
+$$
+
+where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).
+
+
+Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).
+
+
+Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is
+
+$$
+\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0.
+$$
+
+
+All values beyond \( p-1 \) are all zero.
+
+
+
+As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)
+
+$$
+\boldsymbol{\Sigma}=
+\begin{bmatrix}
+2& 0 \\
+0 & 1 \\
+0 & 0 \\
+\end{bmatrix}
+$$
+
+
+The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as
+
+$$
+\boldsymbol{\Sigma}=
+\begin{bmatrix}
+\boldsymbol{\tilde{\Sigma}}\\
+\boldsymbol{0}\\
+\end{bmatrix},
+$$
+
+
+where
+$$
+\boldsymbol{\tilde{\Sigma}}=
+\begin{bmatrix}
+2& 0 \\
+0 & 1 \\
+\end{bmatrix},
+$$
+
+contains only the singular values. Note also (and we will use this below) that
+
+$$
+\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
+\begin{bmatrix}
+4& 0 \\
+0 & 1 \\
+\end{bmatrix},
+$$
+
+which is a \( 2\times 2 \) matrix while
+$$
+\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
+\begin{bmatrix}
+4& 0 & 0\\
+0 & 1 & 0\\
+0 & 0 & 0\\
+\end{bmatrix},
+$$
+
+is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
+
+
+
diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
index 9f03671e5..8e0cda615 100644
Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ
diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb
index 6d35e29e2..31946a305 100644
--- a/doc/pub/week35/ipynb/week35.ipynb
+++ b/doc/pub/week35/ipynb/week35.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Sep 4, 2021**\n",
+ "Date: **Sep 5, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -2052,8 +2052,7 @@
"\n",
"\n",
"The examples we have looked at so far are cases where we normally can\n",
- "invert the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$. Using a polynomial expansion as we\n",
- "did both for the masses and the fitting of various functions leads to\n",
+ "invert the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$. Using a polynomial expansion where we fit of various functions leads to\n",
"row vectors of the design matrix which are essentially orthogonal due\n",
"to the polynomial character of our model. Obtaining the inverse of the\n",
"design matrix is then often done via a so-called LU, QR or Cholesky\n",
@@ -2065,7 +2064,9 @@
"however not the be case in general and a standard matrix inversion\n",
"algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.\n",
"\n",
- "There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. \n",
+ "There is however a way to circumvent this problem and also\n",
+ "gain some insights about the ordinary least squares approach, and\n",
+ "later shrinkage methods like Ridge and Lasso regressions.\n",
"\n",
"This is given by the **Singular Value Decomposition** (SVD) algorithm,\n",
"perhaps the most powerful linear algebra algorithm. The SVD provides\n",
@@ -2073,14 +2074,13 @@
"swath oc applications and the decomposition is always stable\n",
"numerically.\n",
"\n",
- "In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.\n",
- "Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. \n",
- "\n",
- "\n",
- "\n",
- "Let us look at a\n",
- "different example where we may have problems with the standard matrix\n",
- "inversion algorithm. Thereafter we dive into the math of the SVD.\n",
+ "In machine learning it plays a central role in dealing with for\n",
+ "example design matrices that may be near singular or singular.\n",
+ "Furthermore, as we will see here, the singular values can be related\n",
+ "to the covariance matrix (and thereby the correlation matrix) and in\n",
+ "turn the variance of a given quantity. It plays also an important role\n",
+ "in the principal component analysis where high-dimensional data can be\n",
+ "reduced to the statistically relevant features.\n",
"\n",
"\n",
"\n",
@@ -2380,7 +2380,7 @@
"source": [
"import numpy as np\n",
"# SVD inversion\n",
- "def SVDinv(A):\n",
+ "def SVD(A):\n",
" ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).\n",
" SVD is numerically more stable than the inversion algorithms provided by\n",
" numpy and scipy.linalg at the cost of being slower.\n",
@@ -2404,7 +2404,7 @@
"#X = np.array([[1, 2], [3, 4], [5, 6]])\n",
"\n",
"print(X)\n",
- "C = SVDinv(X)\n",
+ "C = SVD(X)\n",
"# Print the difference between the original matrix and the SVD one\n",
"print(C-X)"
]
@@ -2428,14 +2428,15 @@
"The $U$, $S$, and $V$ matrices returned from the **svd()** function\n",
"cannot be multiplied directly.\n",
"\n",
- "As you can see from the code, the $S$\n",
- "vector must be converted into a diagonal matrix. This may cause a \n",
- "as\n",
- "the size of the matrices do not fit the rules of matrix\n",
- "multiplication, where the number of columns in a matrix must match the\n",
- "number of rows in the subsequent matrix.\n",
+ "As you can see from the code, the $S$ vector must be converted into a\n",
+ "diagonal matrix. This may cause a problem as the size of the matrices\n",
+ "do not fit the rules of matrix multiplication, where the number of\n",
+ "columns in a matrix must match the number of rows in the subsequent\n",
+ "matrix.\n",
"\n",
- "If you wish to include the zero singular values, you will need to resize the matrices. More about this later.\n",
+ "If you wish to include the zero singular values, you will need to\n",
+ "resize the matrices and set up a diagonal matrix as done in the above\n",
+ "example\n",
"\n",
"\n",
"\n",
@@ -2448,6 +2449,178 @@
"More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at Matheamtics of the SVD and implications
+
+
+$$
+\boldsymbol{X}=
+\begin{bmatrix}
+x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
+x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
+x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
+\dots& \dots &\dots& \dots & \dots &\dots\\
+x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
+\end{bmatrix}
+$$
+
+
+
+$$
+\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
+$$
+
+
+where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).
+
+
+$$
+\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0.
+$$
+
+
+Example Matrix
+
+
+$$
+\boldsymbol{\Sigma}=
+\begin{bmatrix}
+2& 0 \\
+0 & 1 \\
+0 & 0 \\
+\end{bmatrix}
+$$
+
+
+
+$$
+\boldsymbol{\Sigma}=
+\begin{bmatrix}
+\boldsymbol{\tilde{\Sigma}}\\
+\boldsymbol{0}\\
+\end{bmatrix},
+$$
+
+
+
+$$
+\boldsymbol{\tilde{\Sigma}}=
+\begin{bmatrix}
+2& 0 \\
+0 & 1 \\
+\end{bmatrix},
+$$
+
+
+contains only the singular values. Note also (and we will use this below) that
+
+
+$$
+\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
+\begin{bmatrix}
+4& 0 \\
+0 & 1 \\
+\end{bmatrix},
+$$
+
+
+which is a \( 2\times 2 \) matrix while
+
+$$
+\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
+\begin{bmatrix}
+4& 0 & 0\\
+0 & 1 & 0\\
+0 & 0 & 0\\
+\end{bmatrix},
+$$
+
+
+is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
+Ridge and LASSO Regression
diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html
index d76aa761e..fc04fd44f 100644
--- a/doc/pub/week35/html/week35-solarized.html
+++ b/doc/pub/week35/html/week35-solarized.html
@@ -203,6 +203,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'note-about-svd-calculations'),
('Friday September 3', 2, None, 'friday-september-3'),
+ ('Matheamtics of the SVD and implications',
+ 2,
+ None,
+ 'matheamtics-of-the-svd-and-implications'),
+ ('Example Matrix', 2, None, 'example-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -300,7 +305,7 @@ MathJax.Hub.Config({
Sep 4, 2021
Sep 5, 2021
@@ -1735,8 +1740,7 @@ This serves also as a useful test of our codes.
import numpy as np
# SVD inversion
-def SVDinv(A):
+def SVD(A):
''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
SVD is numerically more stable than the inversion algorithms provided by
numpy and scipy.linalg at the cost of being slower.
@@ -2000,7 +2006,7 @@ X = np.array([ [1.0,-#X = np.array([[1, 2], [3, 4], [5, 6]])
print(X)
-C = SVDinv(X)
+C = SVD(X)
# Print the difference between the original matrix and the SVD one
print(C-X)
@@ -2048,6 +2055,108 @@ More material will be added here, see handwritten notes also. Note that this m
+Matheamtics of the SVD and implications
+
+
+
+Example Matrix
+
+
+
Ridge and LASSO Regression
Sep 4, 2021
Sep 5, 2021
@@ -1740,8 +1745,7 @@ This serves also as a useful test of our codes.
import numpy as np
# SVD inversion
-def SVDinv(A):
+def SVD(A):
''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
SVD is numerically more stable than the inversion algorithms provided by
numpy and scipy.linalg at the cost of being slower.
@@ -2005,7 +2011,7 @@ X = np.a
#X = np.array([[1, 2], [3, 4], [5, 6]])
print(X)
-C = SVDinv(X)
+C = SVD(X)
# Print the difference between the original matrix and the SVD one
print(C-X)
@@ -2053,6 +2060,108 @@ More material will be added here, see handwritten notes also. Note that this m
+Matheamtics of the SVD and implications
+
+
+
+Example Matrix
+
+
+
Ridge and LASSO Regression