lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

adding stuff to week 35

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-05 21:49:06 +02:00
parent d998ff24f0
commit 977be1cf3e
7 changed files with 742 additions and 117 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+26 -19
View File
@@ -183,6 +183,11 @@ Automatically generated HTML file from DocOnce source
None,
'note-about-svd-calculations'),
('Friday September 3', 2, None, 'friday-september-3'),
('Matheamtics of the SVD and implications',
2,
None,
'matheamtics-of-the-svd-and-implications'),
('Example Matrix', 2, None, 'example-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -323,23 +328,25 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs045.html#codes-for-the-svd" style="font-size: 80%;"><b>Codes for the SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs046.html#note-about-svd-calculations" style="font-size: 80%;"><b>Note about SVD Calculations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs047.html#friday-september-3" style="font-size: 80%;"><b>Friday September 3</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs048.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs049.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs050.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs048.html#matheamtics-of-the-svd-and-implications" style="font-size: 80%;"><b>Matheamtics of the SVD and implications</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs049.html#example-matrix" style="font-size: 80%;"><b>Example Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs050.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -374,7 +381,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 4, 2021</h4></center> <!-- date -->
<center><h4>Sep 5, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -398,7 +405,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs062.html">63</a></li>
<li><a href="._week35-bs064.html">65</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+140 -20
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 4, 2021</h4></center> <!-- date -->
<center><h4>Sep 5, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -1656,8 +1656,7 @@ This serves also as a useful test of our codes.
<b></b>
<p>
The examples we have looked at so far are cases where we normally can
invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we
did both for the masses and the fitting of various functions leads to
invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to
row vectors of the design matrix which are essentially orthogonal due
to the polynomial character of our model. Obtaining the inverse of the
design matrix is then often done via a so-called LU, QR or Cholesky
@@ -1670,7 +1669,9 @@ however not the be case in general and a standard matrix inversion
algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
<p>
There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions.
There is however a way to circumvent this problem and also
gain some insights about the ordinary least squares approach, and
later shrinkage methods like Ridge and Lasso regressions.
<p>
This is given by the <b>Singular Value Decomposition</b> (SVD) algorithm,
@@ -1680,13 +1681,13 @@ swath oc applications and the decomposition is always stable
numerically.
<p>
In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features.
<p>
Let us look at a
different example where we may have problems with the standard matrix
inversion algorithm. Thereafter we dive into the math of the SVD.
In machine learning it plays a central role in dealing with for
example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related
to the covariance matrix (and thereby the correlation matrix) and in
turn the variance of a given quantity. It plays also an important role
in the principal component analysis where high-dimensional data can be
reduced to the statistically relevant features.
</div>
@@ -1916,7 +1917,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># SVD inversion</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">SVDinv</span>(A):
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">SVD</span>(A):
<span style="color: #CD5555">&#39;&#39;&#39; Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).</span>
<span style="color: #CD5555"> SVD is numerically more stable than the inversion algorithms provided by</span>
<span style="color: #CD5555"> numpy and scipy.linalg at the cost of being slower.</span>
@@ -1940,7 +1941,7 @@ X = np.array([ [<span style="color: #B452CD">1.0</span>,-<span style="color: #B4
<span style="color: #228B22">#X = np.array([[1, 2], [3, 4], [5, 6]])</span>
<span style="color: #658b00">print</span>(X)
C = SVDinv(X)
C = SVD(X)
<span style="color: #228B22"># Print the difference between the original matrix and the SVD one</span>
<span style="color: #658b00">print</span>(C-X)
</pre></div>
@@ -1964,15 +1965,16 @@ The \( U \), \( S \), and \( V \) matrices returned from the <b>svd()</b> functi
cannot be multiplied directly.
<p>
As you can see from the code, the \( S \)
vector must be converted into a diagonal matrix. This may cause a
as
the size of the matrices do not fit the rules of matrix
multiplication, where the number of columns in a matrix must match the
number of rows in the subsequent matrix.
As you can see from the code, the \( S \) vector must be converted into a
diagonal matrix. This may cause a problem as the size of the matrices
do not fit the rules of matrix multiplication, where the number of
columns in a matrix must match the number of rows in the subsequent
matrix.
<p>
If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
If you wish to include the zero singular values, you will need to
resize the matrices and set up a diagonal matrix as done in the above
example
</section>
@@ -1987,6 +1989,124 @@ More material will be added here, see handwritten notes also. Note that this m
</section>
<section>
<h2 id="matheamtics-of-the-svd-and-implications">Matheamtics of the SVD and implications </h2>
<p>
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
<p>
Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
<p>&nbsp;<br>
$$
\boldsymbol{X}=
\begin{bmatrix}
x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
\end{bmatrix}
$$
<p>&nbsp;<br>
<p>
We can SVD decompose our matrix as
<p>&nbsp;<br>
$$
\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
$$
<p>&nbsp;<br>
where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).
<p>
Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).
<p>
Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is
<p>&nbsp;<br>
$$
\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0.
$$
<p>&nbsp;<br>
<p>
All values beyond \( p-1 \) are all zero.
</section>
<section>
<h2 id="example-matrix">Example Matrix </h2>
<p>
As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)
<p>&nbsp;<br>
$$
\boldsymbol{\Sigma}=
\begin{bmatrix}
2& 0 \\
0 & 1 \\
0 & 0 \\
\end{bmatrix}
$$
<p>&nbsp;<br>
<p>
The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as
<p>&nbsp;<br>
$$
\boldsymbol{\Sigma}=
\begin{bmatrix}
\boldsymbol{\tilde{\Sigma}}\\
\boldsymbol{0}\\
\end{bmatrix},
$$
<p>&nbsp;<br>
<p>
where
<p>&nbsp;<br>
$$
\boldsymbol{\tilde{\Sigma}}=
\begin{bmatrix}
2& 0 \\
0 & 1 \\
\end{bmatrix},
$$
<p>&nbsp;<br>
contains only the singular values. Note also (and we will use this below) that
<p>&nbsp;<br>
$$
\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
\end{bmatrix},
$$
<p>&nbsp;<br>
which is a \( 2\times 2 \) matrix while
<p>&nbsp;<br>
$$
\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
0 & 0 & 0\\
\end{bmatrix},
$$
<p>&nbsp;<br>
is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
</section>
<section>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
doc/pub/week35/html/week35-solarized.html
+129 -20
View File
@@ -203,6 +203,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'note-about-svd-calculations'),
('Friday September 3', 2, None, 'friday-september-3'),
('Matheamtics of the SVD and implications',
2,
None,
'matheamtics-of-the-svd-and-implications'),
('Example Matrix', 2, None, 'example-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -300,7 +305,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 4, 2021</h4></center> <!-- date -->
<center><h4>Sep 5, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1735,8 +1740,7 @@ This serves also as a useful test of our codes.
<p>
The examples we have looked at so far are cases where we normally can
invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we
did both for the masses and the fitting of various functions leads to
invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to
row vectors of the design matrix which are essentially orthogonal due
to the polynomial character of our model. Obtaining the inverse of the
design matrix is then often done via a so-called LU, QR or Cholesky
@@ -1749,7 +1753,9 @@ however not the be case in general and a standard matrix inversion
algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
<p>
There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions.
There is however a way to circumvent this problem and also
gain some insights about the ordinary least squares approach, and
later shrinkage methods like Ridge and Lasso regressions.
<p>
This is given by the <b>Singular Value Decomposition</b> (SVD) algorithm,
@@ -1759,13 +1765,13 @@ swath oc applications and the decomposition is always stable
numerically.
<p>
In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features.
<p>
Let us look at a
different example where we may have problems with the standard matrix
inversion algorithm. Thereafter we dive into the math of the SVD.
In machine learning it plays a central role in dealing with for
example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related
to the covariance matrix (and thereby the correlation matrix) and in
turn the variance of a given quantity. It plays also an important role
in the principal component analysis where high-dimensional data can be
reduced to the statistically relevant features.
</div>
@@ -1976,7 +1982,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #228B22"># SVD inversion</span>
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">SVDinv</span>(A):
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">SVD</span>(A):
<span style="color: #CD5555">&#39;&#39;&#39; Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).</span>
<span style="color: #CD5555"> SVD is numerically more stable than the inversion algorithms provided by</span>
<span style="color: #CD5555"> numpy and scipy.linalg at the cost of being slower.</span>
@@ -2000,7 +2006,7 @@ X = np.array([ [<span style="color: #B452CD">1.0</span>,-<span style="color: #B4
<span style="color: #228B22">#X = np.array([[1, 2], [3, 4], [5, 6]])</span>
<span style="color: #658b00">print</span>(X)
C = SVDinv(X)
C = SVD(X)
<span style="color: #228B22"># Print the difference between the original matrix and the SVD one</span>
<span style="color: #658b00">print</span>(C-X)
</pre></div>
@@ -2024,15 +2030,16 @@ The \( U \), \( S \), and \( V \) matrices returned from the <b>svd()</b> functi
cannot be multiplied directly.
<p>
As you can see from the code, the \( S \)
vector must be converted into a diagonal matrix. This may cause a
as
the size of the matrices do not fit the rules of matrix
multiplication, where the number of columns in a matrix must match the
number of rows in the subsequent matrix.
As you can see from the code, the \( S \) vector must be converted into a
diagonal matrix. This may cause a problem as the size of the matrices
do not fit the rules of matrix multiplication, where the number of
columns in a matrix must match the number of rows in the subsequent
matrix.
<p>
If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
If you wish to include the zero singular values, you will need to
resize the matrices and set up a diagonal matrix as done in the above
example
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2048,6 +2055,108 @@ More material will be added here, see handwritten notes also. Note that this m
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="matheamtics-of-the-svd-and-implications">Matheamtics of the SVD and implications </h2>
<p>
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
<p>
Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
$$
\boldsymbol{X}=
\begin{bmatrix}
x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
\end{bmatrix}
$$
<p>
We can SVD decompose our matrix as
$$
\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
$$
where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).
<p>
Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).
<p>
Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is
$$
\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0.
$$
<p>
All values beyond \( p-1 \) are all zero.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="example-matrix">Example Matrix </h2>
<p>
As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)
$$
\boldsymbol{\Sigma}=
\begin{bmatrix}
2& 0 \\
0 & 1 \\
0 & 0 \\
\end{bmatrix}
$$
<p>
The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as
$$
\boldsymbol{\Sigma}=
\begin{bmatrix}
\boldsymbol{\tilde{\Sigma}}\\
\boldsymbol{0}\\
\end{bmatrix},
$$
<p>
where
$$
\boldsymbol{\tilde{\Sigma}}=
\begin{bmatrix}
2& 0 \\
0 & 1 \\
\end{bmatrix},
$$
contains only the singular values. Note also (and we will use this below) that
$$
\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
\end{bmatrix},
$$
which is a \( 2\times 2 \) matrix while
$$
\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
0 & 0 & 0\\
\end{bmatrix},
$$
is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
<p>
doc/pub/week35/html/week35.html
+129 -20
View File
@@ -208,6 +208,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'note-about-svd-calculations'),
('Friday September 3', 2, None, 'friday-september-3'),
('Matheamtics of the SVD and implications',
2,
None,
'matheamtics-of-the-svd-and-implications'),
('Example Matrix', 2, None, 'example-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -305,7 +310,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 4, 2021</h4></center> <!-- date -->
<center><h4>Sep 5, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1740,8 +1745,7 @@ This serves also as a useful test of our codes.
<p>
The examples we have looked at so far are cases where we normally can
invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion as we
did both for the masses and the fitting of various functions leads to
invert the matrix \( \boldsymbol{X}^T\boldsymbol{X} \). Using a polynomial expansion where we fit of various functions leads to
row vectors of the design matrix which are essentially orthogonal due
to the polynomial character of our model. Obtaining the inverse of the
design matrix is then often done via a so-called LU, QR or Cholesky
@@ -1754,7 +1758,9 @@ however not the be case in general and a standard matrix inversion
algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
<p>
There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions.
There is however a way to circumvent this problem and also
gain some insights about the ordinary least squares approach, and
later shrinkage methods like Ridge and Lasso regressions.
<p>
This is given by the <b>Singular Value Decomposition</b> (SVD) algorithm,
@@ -1764,13 +1770,13 @@ swath oc applications and the decomposition is always stable
numerically.
<p>
In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features.
<p>
Let us look at a
different example where we may have problems with the standard matrix
inversion algorithm. Thereafter we dive into the math of the SVD.
In machine learning it plays a central role in dealing with for
example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related
to the covariance matrix (and thereby the correlation matrix) and in
turn the variance of a given quantity. It plays also an important role
in the principal component analysis where high-dimensional data can be
reduced to the statistically relevant features.
</div>
@@ -1981,7 +1987,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #408080; font-style: italic"># SVD inversion</span>
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">SVDinv</span>(A):
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">SVD</span>(A):
<span style="color: #BA2121; font-style: italic">&#39;&#39;&#39; Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).</span>
<span style="color: #BA2121; font-style: italic"> SVD is numerically more stable than the inversion algorithms provided by</span>
<span style="color: #BA2121; font-style: italic"> numpy and scipy.linalg at the cost of being slower.</span>
@@ -2005,7 +2011,7 @@ X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>a
<span style="color: #408080; font-style: italic">#X = np.array([[1, 2], [3, 4], [5, 6]])</span>
<span style="color: #008000">print</span>(X)
C <span style="color: #666666">=</span> SVDinv(X)
C <span style="color: #666666">=</span> SVD(X)
<span style="color: #408080; font-style: italic"># Print the difference between the original matrix and the SVD one</span>
<span style="color: #008000">print</span>(C<span style="color: #666666">-</span>X)
</pre></div>
@@ -2029,15 +2035,16 @@ The \( U \), \( S \), and \( V \) matrices returned from the <b>svd()</b> functi
cannot be multiplied directly.
<p>
As you can see from the code, the \( S \)
vector must be converted into a diagonal matrix. This may cause a
as
the size of the matrices do not fit the rules of matrix
multiplication, where the number of columns in a matrix must match the
number of rows in the subsequent matrix.
As you can see from the code, the \( S \) vector must be converted into a
diagonal matrix. This may cause a problem as the size of the matrices
do not fit the rules of matrix multiplication, where the number of
columns in a matrix must match the number of rows in the subsequent
matrix.
<p>
If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
If you wish to include the zero singular values, you will need to
resize the matrices and set up a diagonal matrix as done in the above
example
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2053,6 +2060,108 @@ More material will be added here, see handwritten notes also. Note that this m
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="matheamtics-of-the-svd-and-implications">Matheamtics of the SVD and implications </h2>
<p>
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
<p>
Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
$$
\boldsymbol{X}=
\begin{bmatrix}
x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
\end{bmatrix}
$$
<p>
We can SVD decompose our matrix as
$$
\boldsymbol{X}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
$$
where \( \boldsymbol{U} \) is an orthogonal matrix of dimension \( n\times n \), meaning that \( \boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{I}_n \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( n \times n \).
<p>
Similarly, \( \boldsymbol{V} \) is an orthogonal matrix of dimension \( p\times p \), meaning that \( \boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{I}_p \). Here \( \boldsymbol{I}_n \) is the unit matrix of dimension \( p \times p \).
<p>
Finally \( \boldsymbol{\Sigma} \) contains the singular values \( \sigma_i \). This matrix has dimension \( n\times p \) and the singular values \( \sigma_i \) are all positive. The non-zero values are ordered in descending order, that is
$$
\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0.
$$
<p>
All values beyond \( p-1 \) are all zero.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="example-matrix">Example Matrix </h2>
<p>
As an example, consider the following \( 3\times 2 \) example for the matrix \( \boldsymbol{\Sigma} \)
$$
\boldsymbol{\Sigma}=
\begin{bmatrix}
2& 0 \\
0 & 1 \\
0 & 0 \\
\end{bmatrix}
$$
<p>
The singular values are \( \sigma_0=2 \) and \( \sigma_1=1 \). It is common to rewrite the matrix \( \boldsymbol{\Sigma} \) as
$$
\boldsymbol{\Sigma}=
\begin{bmatrix}
\boldsymbol{\tilde{\Sigma}}\\
\boldsymbol{0}\\
\end{bmatrix},
$$
<p>
where
$$
\boldsymbol{\tilde{\Sigma}}=
\begin{bmatrix}
2& 0 \\
0 & 1 \\
\end{bmatrix},
$$
contains only the singular values. Note also (and we will use this below) that
$$
\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
\end{bmatrix},
$$
which is a \( 2\times 2 \) matrix while
$$
\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
0 & 0 & 0\\
\end{bmatrix},
$$
is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
<p>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+194 -21
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 4, 2021**\n",
"Date: **Sep 5, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -2052,8 +2052,7 @@
"\n",
"\n",
"The examples we have looked at so far are cases where we normally can\n",
"invert the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$. Using a polynomial expansion as we\n",
"did both for the masses and the fitting of various functions leads to\n",
"invert the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$. Using a polynomial expansion where we fit of various functions leads to\n",
"row vectors of the design matrix which are essentially orthogonal due\n",
"to the polynomial character of our model. Obtaining the inverse of the\n",
"design matrix is then often done via a so-called LU, QR or Cholesky\n",
@@ -2065,7 +2064,9 @@
"however not the be case in general and a standard matrix inversion\n",
"algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.\n",
"\n",
"There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions. \n",
"There is however a way to circumvent this problem and also\n",
"gain some insights about the ordinary least squares approach, and\n",
"later shrinkage methods like Ridge and Lasso regressions.\n",
"\n",
"This is given by the **Singular Value Decomposition** (SVD) algorithm,\n",
"perhaps the most powerful linear algebra algorithm. The SVD provides\n",
@@ -2073,14 +2074,13 @@
"swath oc applications and the decomposition is always stable\n",
"numerically.\n",
"\n",
"In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.\n",
"Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features. \n",
"\n",
"\n",
"\n",
"Let us look at a\n",
"different example where we may have problems with the standard matrix\n",
"inversion algorithm. Thereafter we dive into the math of the SVD.\n",
"In machine learning it plays a central role in dealing with for\n",
"example design matrices that may be near singular or singular.\n",
"Furthermore, as we will see here, the singular values can be related\n",
"to the covariance matrix (and thereby the correlation matrix) and in\n",
"turn the variance of a given quantity. It plays also an important role\n",
"in the principal component analysis where high-dimensional data can be\n",
"reduced to the statistically relevant features.\n",
"\n",
"\n",
"\n",
@@ -2380,7 +2380,7 @@
"source": [
"import numpy as np\n",
"# SVD inversion\n",
"def SVDinv(A):\n",
"def SVD(A):\n",
" ''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).\n",
" SVD is numerically more stable than the inversion algorithms provided by\n",
" numpy and scipy.linalg at the cost of being slower.\n",
@@ -2404,7 +2404,7 @@
"#X = np.array([[1, 2], [3, 4], [5, 6]])\n",
"\n",
"print(X)\n",
"C = SVDinv(X)\n",
"C = SVD(X)\n",
"# Print the difference between the original matrix and the SVD one\n",
"print(C-X)"
]
@@ -2428,14 +2428,15 @@
"The $U$, $S$, and $V$ matrices returned from the **svd()** function\n",
"cannot be multiplied directly.\n",
"\n",
"As you can see from the code, the $S$\n",
"vector must be converted into a diagonal matrix. This may cause a \n",
"as\n",
"the size of the matrices do not fit the rules of matrix\n",
"multiplication, where the number of columns in a matrix must match the\n",
"number of rows in the subsequent matrix.\n",
"As you can see from the code, the $S$ vector must be converted into a\n",
"diagonal matrix. This may cause a problem as the size of the matrices\n",
"do not fit the rules of matrix multiplication, where the number of\n",
"columns in a matrix must match the number of rows in the subsequent\n",
"matrix.\n",
"\n",
"If you wish to include the zero singular values, you will need to resize the matrices. More about this later.\n",
"If you wish to include the zero singular values, you will need to\n",
"resize the matrices and set up a diagonal matrix as done in the above\n",
"example\n",
"\n",
"\n",
"\n",
@@ -2448,6 +2449,178 @@
"More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at <https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021>.\n",
"\n",
"\n",
"## Matheamtics of the SVD and implications\n",
"\n",
"Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.\n",
"\n",
"Our starting point is our design matrix $\\boldsymbol{X}$ of dimension $n\\times p$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\n",
"\\begin{bmatrix}\n",
"x_{0,0}& x_{0,1} &x_{0,2}& \\dots & \\dots &x_{0,p-1}\\\\\n",
"x_{1,0}& x_{1,1} &x_{1,2& \\dots & \\dots &x_{1,p-1}\\\\\n",
"x_{2,0}& x_{2,1} &x_{2,2}& \\dots & \\dots &x_{2,p-1}\\\\\n",
"\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
"x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \\dots & \\dots &x_{n-1,p-1}\\\\\n",
"\\end{bmatrix}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can SVD decompose our matrix as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where $\\boldsymbol{U}$ is an orthogonal matrix of dimension $n\\times n$, meaning that $\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{I}_n$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $n \\times n$.\n",
"\n",
"Similarly, $\\boldsymbol{V}$ is an orthogonal matrix of dimension $p\\times p$, meaning that $\\boldsymbol{V}\\boldsymbol{V}^T=\\boldsymbol{V}^T\\boldsymbol{V}=\\boldsymbol{I}_p$. Here $\\boldsymbol{I}_n$ is the unit matrix of dimension $p \\times p$.\n",
"\n",
"Finally $\\boldsymbol{\\Sigma}$ contains the singular values $\\sigma_i$. This matrix has dimension $n\\times p$ and the singular values $\\sigma_i$ are all positive. The non-zero values are ordered in descending order, that is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\sigma_0 > \\sigma_1 > \\sigma_2 > \\dots > \\sigma_{p-1} > 0.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All values beyond $p-1$ are all zero.\n",
"\n",
"## Example Matrix\n",
"\n",
"As an example, consider the following $3\\times 2$ example for the matrix $\\boldsymbol{\\Sigma}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}=\n",
"\\begin{bmatrix}\n",
"2& 0 \\\\\n",
"0 & 1 \\\\\n",
"0 & 0 \\\\\n",
"\\end{bmatrix}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The singular values are $\\sigma_0=2$ and $\\sigma_1=1$. It is common to rewrite the matrix $\\boldsymbol{\\Sigma}$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}=\n",
"\\begin{bmatrix}\n",
"\\boldsymbol{\\tilde{\\Sigma}}\\\\\n",
"\\boldsymbol{0}\\\\\n",
"\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\tilde{\\Sigma}}=\n",
"\\begin{bmatrix}\n",
"2& 0 \\\\\n",
"0 & 1 \\\\\n",
"\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"contains only the singular values. Note also (and we will use this below) that"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}^T\\boldsymbol{\\sigma}=\n",
"\\begin{bmatrix}\n",
"4& 0 \\\\\n",
"0 & 1 \\\\\n",
"\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which is a $2\\times 2 $ matrix while"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}\\boldsymbol{\\sigma}^T=\n",
"\\begin{bmatrix}\n",
"4& 0 & 0\\\\\n",
"0 & 1 & 0\\\\\n",
"0 & 0 & 0\\\\\n",
"\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"is a $3\\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.\n",
"\n",
"\n",
"## Ridge and LASSO Regression\n",
"\n",
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
doc/src/week35/week35.do.txt
+124 -17
View File
@@ -1274,8 +1274,7 @@ This serves also as a useful test of our codes.
!bblock
The examples we have looked at so far are cases where we normally can
invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion as we
did both for the masses and the fitting of various functions leads to
invert the matrix $\bm{X}^T\bm{X}$. Using a polynomial expansion where we fit of various functions leads to
row vectors of the design matrix which are essentially orthogonal due
to the polynomial character of our model. Obtaining the inverse of the
design matrix is then often done via a so-called LU, QR or Cholesky
@@ -1287,7 +1286,9 @@ this may
however not the be case in general and a standard matrix inversion
algorithm based on say LU, QR or Cholesky decomposition may lead to singularities. We will see examples of this below.
There is however a way to partially circumvent this problem and also gain some insights about the ordinary least squares approach, and later shrinkage methods like Ridge and Lasso regressions.
There is however a way to circumvent this problem and also
gain some insights about the ordinary least squares approach, and
later shrinkage methods like Ridge and Lasso regressions.
This is given by the _Singular Value Decomposition_ (SVD) algorithm,
perhaps the most powerful linear algebra algorithm. The SVD provides
@@ -1295,14 +1296,17 @@ a numerically stable matrix decomposition that is used in a large
swath oc applications and the decomposition is always stable
numerically.
In machine learning it plays a central role in dealing with for example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related to the covariance matrix (and thereby the correlation matrix) and in turn the variance of a given quantity. It plays also an important role in the principal component analysis where high-dimensional data can be reduced to the statistically relevant features.
In machine learning it plays a central role in dealing with for
example design matrices that may be near singular or singular.
Furthermore, as we will see here, the singular values can be related
to the covariance matrix (and thereby the correlation matrix) and in
turn the variance of a given quantity. It plays also an important role
in the principal component analysis where high-dimensional data can be
reduced to the statistically relevant features.
Let us look at a
different example where we may have problems with the standard matrix
inversion algorithm. Thereafter we dive into the math of the SVD.
!eblock
@@ -1494,7 +1498,7 @@ In general the economy-size SVD leads to less FLOPS and still conserving the des
!bc pycod
import numpy as np
# SVD inversion
def SVDinv(A):
def SVD(A):
''' Takes as input a numpy matrix A and returns inv(A) based on singular value decomposition (SVD).
SVD is numerically more stable than the inversion algorithms provided by
numpy and scipy.linalg at the cost of being slower.
@@ -1518,7 +1522,7 @@ X = np.array([ [1.0,-1.0], [1.0,-1.0]])
#X = np.array([[1, 2], [3, 4], [5, 6]])
print(X)
C = SVDinv(X)
C = SVD(X)
# Print the difference between the original matrix and the SVD one
print(C-X)
!ec
@@ -1539,14 +1543,15 @@ in the program terminating due to a singular matrix.
The $U$, $S$, and $V$ matrices returned from the _svd()_ function
cannot be multiplied directly.
As you can see from the code, the $S$
vector must be converted into a diagonal matrix. This may cause a
as
the size of the matrices do not fit the rules of matrix
multiplication, where the number of columns in a matrix must match the
number of rows in the subsequent matrix.
As you can see from the code, the $S$ vector must be converted into a
diagonal matrix. This may cause a problem as the size of the matrices
do not fit the rules of matrix multiplication, where the number of
columns in a matrix must match the number of rows in the subsequent
matrix.
If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
If you wish to include the zero singular values, you will need to
resize the matrices and set up a diagonal matrix as done in the above
example
@@ -1560,6 +1565,108 @@ If you wish to include the zero singular values, you will need to resize the mat
More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at URL:"https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021".
!split
===== Matheamtics of the SVD and implications =====
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
Our starting point is our design matrix $\bm{X}$ of dimension $n\times p$
!bt
\[
\bm{X}=
\begin{bmatrix}
x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
\end{bmatrix}
\]
!et
We can SVD decompose our matrix as
!bt
\[
\bm{X}=\bm{U}\bm{\Sigma}\bm{V}^T,
\]
!et
where $\bm{U}$ is an orthogonal matrix of dimension $n\times n$, meaning that $\bm{U}\bm{U}^T=\bm{U}^T\bm{U}=\bm{I}_n$. Here $\bm{I}_n$ is the unit matrix of dimension $n \times n$.
Similarly, $\bm{V}$ is an orthogonal matrix of dimension $p\times p$, meaning that $\bm{V}\bm{V}^T=\bm{V}^T\bm{V}=\bm{I}_p$. Here $\bm{I}_n$ is the unit matrix of dimension $p \times p$.
Finally $\bm{\Sigma}$ contains the singular values $\sigma_i$. This matrix has dimension $n\times p$ and the singular values $\sigma_i$ are all positive. The non-zero values are ordered in descending order, that is
!bt
\[
\sigma_0 > \sigma_1 > \sigma_2 > \dots > \sigma_{p-1} > 0.
\]
!et
All values beyond $p-1$ are all zero.
!split
===== Example Matrix =====
As an example, consider the following $3\times 2$ example for the matrix $\bm{\Sigma}$
!bt
\[
\bm{\Sigma}=
\begin{bmatrix}
2& 0 \\
0 & 1 \\
0 & 0 \\
\end{bmatrix}
\]
!et
The singular values are $\sigma_0=2$ and $\sigma_1=1$. It is common to rewrite the matrix $\bm{\Sigma}$ as
!bt
\[
\bm{\Sigma}=
\begin{bmatrix}
\bm{\tilde{\Sigma}}\\
\bm{0}\\
\end{bmatrix},
\]
!et
where
!bt
\[
\bm{\tilde{\Sigma}}=
\begin{bmatrix}
2& 0 \\
0 & 1 \\
\end{bmatrix},
\]
!et
contains only the singular values. Note also (and we will use this below) that
!bt
\[
\bm{\Sigma}^T\bm{\sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
\end{bmatrix},
\]
!et
which is a $2\times 2 $ matrix while
!bt
\[
\bm{\Sigma}\bm{\sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
0 & 0 & 0\\
\end{bmatrix},
\]
!et
is a $3\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
!split
===== Ridge and LASSO Regression =====