lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating again

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-06 16:48:53 +02:00
parent 28a9f6ee81
commit 18c97ff383
7 changed files with 434 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+18 -11
View File
@@ -230,6 +230,11 @@ Automatically generated HTML file from DocOnce source
None,
'rewriting-the-covariance-and-or-correlation-matrix'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('What does it mean?', 2, None, 'what-does-it-mean'),
('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$',
2,
None,
'and-finally-boldsymbol-x-boldsymbol-x-t'),
('Ridge and LASSO Regression',
2,
None,
@@ -359,16 +364,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs059.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#linking-with-the-svd" style="font-size: 80%;"><b>Linking with the SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#what-does-it-mean" style="font-size: 80%;"><b>What does it mean?</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#and-finally-boldsymbol-x-boldsymbol-x-t" style="font-size: 80%;"><b>And finally \( \boldsymbol{X}\boldsymbol{X}^T \)</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs070.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs071.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs071.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs071.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -427,7 +434,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs069.html">70</a></li>
<li><a href="._week35-bs071.html">72</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+81 -2
View File
@@ -2726,14 +2726,93 @@ $$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="what-does-it-mean">What does it mean? </h2>
<p>
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \)
are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues
given by the singular values squared, that is
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
<p>&nbsp;<br>
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>&nbsp;<br>
<p>
In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive
square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that
the columns of \( \boldsymbol{V} \) are the eigenvectors of
\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of
\( \boldsymbol{X} \) in a descending order, it means that the column vectors
\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they
encode from the columns of \( \boldsymbol{X} \).
<p>
Note that these are also the eigenvectors and eigenvalues of the
Hessian matrix.
<p>
If we now recall the definition of the covariance matrix (not using
Bessel's correction) we have
<p>&nbsp;<br>
$$
\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X},
$$
<p>&nbsp;<br>
<p>
meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$,
the number of samples, are the eigenvalues of the covariance
matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square
root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is
self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the
absolute value of the eigenvalues of \( \boldsymbol{X} \).
</section>
<section>
<h2 id="and-finally-boldsymbol-x-boldsymbol-x-t">And finally \( \boldsymbol{X}\boldsymbol{X}^T \) </h2>
<p>
For \( \boldsymbol{X}\boldsymbol{X}^T \) we found
$$
$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T.
<p>&nbsp;<br>
$$
Since the matrices here have dimension \( n\times n \), we have
$$
<p>&nbsp;<br>
\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix},
<p>&nbsp;<br>
$$
leading to
$$
<p>&nbsp;<br>
$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T.
<p>&nbsp;<br>
$$
<p>
Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem
$$
<p>&nbsp;<br>
$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}.
$$
<p>
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \).
<p>
Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \).
</section>
doc/pub/week35/html/week35-solarized.html
+78 -2
View File
@@ -250,6 +250,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'rewriting-the-covariance-and-or-correlation-matrix'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('What does it mean?', 2, None, 'what-does-it-mean'),
('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$',
2,
None,
'and-finally-boldsymbol-x-boldsymbol-x-t'),
('Ridge and LASSO Regression',
2,
None,
@@ -2718,12 +2723,83 @@ $$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2.
$$
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="what-does-it-mean">What does it mean? </h2>
<p>
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \)
are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues
given by the singular values squared, that is
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>
In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive
square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that
the columns of \( \boldsymbol{V} \) are the eigenvectors of
\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of
\( \boldsymbol{X} \) in a descending order, it means that the column vectors
\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they
encode from the columns of \( \boldsymbol{X} \).
<p>
Note that these are also the eigenvectors and eigenvalues of the
Hessian matrix.
<p>
If we now recall the definition of the covariance matrix (not using
Bessel's correction) we have
$$
\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X},
$$
<p>
meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$,
the number of samples, are the eigenvalues of the covariance
matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square
root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is
self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the
absolute value of the eigenvalues of \( \boldsymbol{X} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="and-finally-boldsymbol-x-boldsymbol-x-t">And finally \( \boldsymbol{X}\boldsymbol{X}^T \) </h2>
<p>
For \( \boldsymbol{X}\boldsymbol{X}^T \) we found
$$
$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T.
$$
Since the matrices here have dimension \( n\times n \), we have
$$
\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix},
$$
leading to
$$
$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T.
$$
<p>
Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem
$$
$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}.
$$
<p>
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \).
<p>
Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/html/week35.html
+78 -2
View File
@@ -255,6 +255,11 @@ div { text-align: justify; text-justify: inter-word; }
None,
'rewriting-the-covariance-and-or-correlation-matrix'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('What does it mean?', 2, None, 'what-does-it-mean'),
('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$',
2,
None,
'and-finally-boldsymbol-x-boldsymbol-x-t'),
('Ridge and LASSO Regression',
2,
None,
@@ -2723,12 +2728,83 @@ $$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2.
$$
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="what-does-it-mean">What does it mean? </h2>
<p>
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \)
are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) with eigenvalues
given by the singular values squared, that is
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>
In other words, each non-zero singular value of \( \boldsymbol{X} \) is a positive
square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). It means also that
the columns of \( \boldsymbol{V} \) are the eigenvectors of
\( \boldsymbol{X}^T\boldsymbol{X} \). Since we have ordered the singular values of
\( \boldsymbol{X} \) in a descending order, it means that the column vectors
\( \boldsymbol{v}_i \) are hierarchically ordered by how much correlation they
encode from the columns of \( \boldsymbol{X} \).
<p>
Note that these are also the eigenvectors and eigenvalues of the
Hessian matrix.
<p>
If we now recall the definition of the covariance matrix (not using
Bessel's correction) we have
$$
\boldsymbol{C}[\boldsymbol{X}]=\frac{1}{n}\boldsymbol{X}^T\boldsymbol{X},
$$
<p>
meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$,
the number of samples, are the eigenvalues of the covariance
matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square
root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is
self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the
absolute value of the eigenvalues of \( \boldsymbol{X} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="and-finally-boldsymbol-x-boldsymbol-x-t">And finally \( \boldsymbol{X}\boldsymbol{X}^T \) </h2>
<p>
For \( \boldsymbol{X}\boldsymbol{X}^T \) we found
$$
$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T.
$$
Since the matrices here have dimension \( n\times n \), we have
$$
\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix},
$$
leading to
$$
$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T.
$$
<p>
Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem
$$
$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}.
$$
<p>
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \).
<p>
Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+112 -2
View File
@@ -3531,8 +3531,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$ are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$\n",
"with eigenvalues given by the singular values squared, that is"
"## What does it mean?\n",
"\n",
"This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$\n",
"are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ with eigenvalues\n",
"given by the singular values squared, that is"
]
},
{
@@ -3548,6 +3551,113 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"In other words, each non-zero singular value of $\\boldsymbol{X}$ is a positive\n",
"square root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. It means also that\n",
"the columns of $\\boldsymbol{V}$ are the eigenvectors of\n",
"$\\boldsymbol{X}^T\\boldsymbol{X}$. Since we have ordered the singular values of\n",
"$\\boldsymbol{X}$ in a descending order, it means that the column vectors\n",
"$\\boldsymbol{v}_i$ are hierarchically ordered by how much correlation they\n",
"encode from the columns of $\\boldsymbol{X}$. \n",
"\n",
"\n",
"Note that these are also the eigenvectors and eigenvalues of the\n",
"Hessian matrix.\n",
"\n",
"If we now recall the definition of the covariance matrix (not using\n",
"Bessel's correction) we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{C}[\\boldsymbol{X}]=\\frac{1}{n}\\boldsymbol{X}^T\\boldsymbol{X},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by$n$,\n",
"the number of samples, are the eigenvalues of the covariance\n",
"matrix. Every singular value of $\\boldsymbol{X}$ is thus a positive square\n",
"root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. If the matrix $\\boldsymbol{X}$ is\n",
"self-adjoint, the the sinular values of $\\boldsymbol{X}$ are equal to the\n",
"absolute value of the eigenvalues of $\\boldsymbol{X}$.\n",
"\n",
"## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n",
"\n",
"For $\\boldsymbol{X}\\boldsymbol{X}^T$ we found"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"$\\boldsymbol{X}\\boldsymbol{X}^T$=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since the matrices here have dimension $n\\times n$, we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} 0 \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"leading to"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"$\\boldsymbol{X}\\boldsymbol{X}^T$=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Multiplying with $\\boldsymbol{U}$ from the right gives us the eigenvalue problem"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"$\\boldsymbol{X}\\boldsymbol{X}^T$\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of $\\boldsymbol{U}$ are the eigenvectors of $\\boldsymbol{X}\\boldsymbol{X}^T$ and measure how much correlations are contained in the rows of $\\m{X}$.\n",
"\n",
"Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of $\\boldsymbol{V}$.\n",
"\n",
"\n",
"## Ridge and LASSO Regression\n",
"\n",
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
doc/src/week35/week35.do.txt
+67 -2
View File
@@ -2208,17 +2208,82 @@ Multiplying from the right with $\bm{V}$ (using the orthogonality of $\bm{V}$) w
\left(\bm{X}^T\bm{X}\right)\bm{V}=\bm{V}\tilde{\bm{\Sigma}}^2.
\]
!et
This means the vectors $\bm{v}_i$ of the orthogonal matrix $\bm{V}$ are the eigenvectors of the matrix $\bm{X}^T\bm{X}$
with eigenvalues given by the singular values squared, that is
!split
===== What does it mean? =====
This means the vectors $\bm{v}_i$ of the orthogonal matrix $\bm{V}$
are the eigenvectors of the matrix $\bm{X}^T\bm{X}$ with eigenvalues
given by the singular values squared, that is
!bt
\[
\left(\bm{X}^T\bm{X}\right)\bm{v}_i=\bm{v}_i\sigma_i^2.
\]
!et
In other words, each non-zero singular value of $\bm{X}$ is a positive
square root of an eigenvalue of $\bm{X}^T\bm{X}$. It means also that
the columns of $\bm{V}$ are the eigenvectors of
$\bm{X}^T\bm{X}$. Since we have ordered the singular values of
$\bm{X}$ in a descending order, it means that the column vectors
$\bm{v}_i$ are hierarchically ordered by how much correlation they
encode from the columns of $\bm{X}$.
Note that these are also the eigenvectors and eigenvalues of the
Hessian matrix.
If we now recall the definition of the covariance matrix (not using
Bessel's correction) we have
!bt
\[
\bm{C}[\bm{X}]=\frac{1}{n}\bm{X}^T\bm{X},
\]
!et
meaning that every squared non-singular value of $\bm{X}$ divided by$n$,
the number of samples, are the eigenvalues of the covariance
matrix. Every singular value of $\bm{X}$ is thus a positive square
root of an eigenvalue of $\bm{X}^T\bm{X}$. If the matrix $\bm{X}$ is
self-adjoint, the the sinular values of $\bm{X}$ are equal to the
absolute value of the eigenvalues of $\bm{X}$.
!split
===== And finally $\bm{X}\bm{X}^T$ =====
For $\bm{X}\bm{X}^T$ we found
!bt
\[
$\bm{X}\bm{X}^T$=\bm{U}\bm{\Sigma}\bm{V}^T\bm{V}\bm{\Sigma}^T\bm{U}^T=\bm{U}\bm{\Sigma}^T\bm{\Sigma}\bm{U}^T.
\]
!et
Since the matrices here have dimension $n\times n$, we have
!bt
\[
\bm{\Sigma}\bm{\Sigma}^T = \begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} 0 \bm{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix},
\]
!et
leading to
!bt
\[
$\bm{X}\bm{X}^T$=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}\bm{U}^T.
\]
!et
Multiplying with $\bm{U}$ from the right gives us the eigenvalue problem
!bt
\[
$\bm{X}\bm{X}^T$\bm{U}=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}.
\]
!et
It means that the eigenvalues of $\bm{X}\bm{X}^T$ are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of $\bm{U}$ are the eigenvectors of $\bm{X}\bm{X}^T$ and measure how much correlations are contained in the rows of $\m{X}$.
Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of $\bm{V}$.
!split