lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week 35

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-05 22:42:30 +02:00
parent da91ca8a27
commit 81e322f510
7 changed files with 314 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+23 -18
View File
@@ -192,6 +192,10 @@ Automatically generated HTML file from DocOnce source
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Further properties (important gems for our analysis)',
2,
None,
'further-properties-important-gems-for-our-analysis'),
('Ridge and LASSO Regression',
2,
None,
@@ -335,23 +339,24 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs048.html#matheamtics-of-the-svd-and-implications" style="font-size: 80%;"><b>Matheamtics of the SVD and implications</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs049.html#example-matrix" style="font-size: 80%;"><b>Example Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs050.html#setting-up-the-matrix-to-be-inverted" style="font-size: 80%;"><b>Setting up the Matrix to be inverted</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#further-properties-important-gems-for-our-analysis" style="font-size: 80%;"><b>Further properties (important gems for our analysis)</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -410,7 +415,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs065.html">66</a></li>
<li><a href="._week35-bs066.html">67</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+53
View File
@@ -2181,6 +2181,59 @@ It means that the ordinary least square model (with the optimal parameters) \( \
</section>
<section>
<h2 id="further-properties-important-gems-for-our-analysis">Further properties (important gems for our analysis) </h2>
<p>
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
<p>&nbsp;<br>
<p>
If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
<p>&nbsp;<br>
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}.
$$
<p>&nbsp;<br>
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
<p>&nbsp;<br>
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>&nbsp;<br>
<p>
Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have
<p>&nbsp;<br>
$$
\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T.
$$
<p>&nbsp;<br>
<p>
If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get
<p>&nbsp;<br>
$$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T.
$$
<p>&nbsp;<br>
This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \)
with eigenvalues given by the singular values squared, that is
<p>&nbsp;<br>
$$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
doc/pub/week35/html/week35-solarized.html
+45
View File
@@ -212,6 +212,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Further properties (important gems for our analysis)',
2,
None,
'further-properties-important-gems-for-our-analysis'),
('Ridge and LASSO Regression',
2,
None,
@@ -2221,6 +2225,47 @@ It means that the ordinary least square model (with the optimal parameters) \( \
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="further-properties-important-gems-for-our-analysis">Further properties (important gems for our analysis) </h2>
<p>
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
<p>
If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}.
$$
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>
Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have
$$
\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T.
$$
<p>
If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get
$$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T.
$$
This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \)
with eigenvalues given by the singular values squared, that is
$$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
<p>
doc/pub/week35/html/week35.html
+45
View File
@@ -217,6 +217,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Further properties (important gems for our analysis)',
2,
None,
'further-properties-important-gems-for-our-analysis'),
('Ridge and LASSO Regression',
2,
None,
@@ -2226,6 +2230,47 @@ It means that the ordinary least square model (with the optimal parameters) \( \
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="further-properties-important-gems-for-our-analysis">Further properties (important gems for our analysis) </h2>
<p>
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
<p>
If we now multiply from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}.
$$
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>
Similarly, if we use the SVD decomposition for the matrix \( \boldsymbol{X}\boldsymbol{X}^T \), we have
$$
\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T\boldsymbol{U}^T.
$$
<p>
If we now multiply from the right with \( \boldsymbol{U} \) (using the orthogonality of \( \boldsymbol{U} \)) we get
$$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{U}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T.
$$
This means the vectors \( \boldsymbol{u}_i \) of the orthogonal matrix \( \boldsymbol{U} \) are the eigenvectors of the matrix \( \boldsymbol{X}\boldsymbol{X}^T \)
with eigenvalues given by the singular values squared, that is
$$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
<p>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+100
View File
@@ -2743,6 +2743,106 @@
"\n",
"It means that the ordinary least square model (with the optimal parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\\boldsymbol{y}$ by the vectors of the matrix $\\boldsymbol{U}$.\n",
"\n",
"## Further properties (important gems for our analysis)\n",
"\n",
"Let us study again $\\boldsymbol{X}^T\\boldsymbol{X}$ in terms of our SVD,"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we now multiply from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$ are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$\n",
"with eigenvalues given by the singular values squared, that is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Similarly, if we use the SVD decomposition for the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$, we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we now multiply from the right with $\\boldsymbol{U}$ (using the orthogonality of $\\boldsymbol{U}$) we get"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This means the vectors $\\boldsymbol{u}_i$ of the orthogonal matrix $\\boldsymbol{U}$ are the eigenvectors of the matrix $\\boldsymbol{X}\\boldsymbol{X}^T$\n",
"with eigenvalues given by the singular values squared, that is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\left(\\boldsymbol{X}\\boldsymbol{X}^T\\right)\\boldsymbol{u}_i=\\boldsymbol{u}_i\\sigma_i^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Ridge and LASSO Regression\n",
"\n",
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
doc/src/week35/week35.do.txt
+48
View File
@@ -1729,6 +1729,54 @@ that belong to $i>p-1$, give all zeros when we perform the multiplications. This
It means that the ordinary least square model (with the optimal parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\bm{y}$ by the vectors of the matrix $\bm{U}$.
!split
===== Further properties (important gems for our analysis) =====
Let us study again $\bm{X}^T\bm{X}$ in terms of our SVD,
!bt
\[
\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T.
\]
!et
If we now multiply from the right with $\bm{V}$ (using the orthogonality of $\bm{V}$) we get
!bt
\[
\left(\bm{X}^T\bm{X}\right)\bm{V}=\bm{V}\bm{\Sigma}^T\bm{\Sigma}.
\]
!et
This means the vectors $\bm{v}_i$ of the orthogonal matrix $\bm{V}$ are the eigenvectors of the matrix $\bm{X}^T\bm{X}$
with eigenvalues given by the singular values squared, that is
!bt
\[
\left(\bm{X}^T\bm{X}\right)\bm{v}_i=\bm{v}_i\sigma_i^2.
\]
!et
Similarly, if we use the SVD decomposition for the matrix $\bm{X}\bm{X}^T$, we have
!bt
\[
\bm{X}\bm{X}^T=\bm{U}\bm{\Sigma}\bm{V}^T\bm{V}\bm{\Sigma}^T\bm{U}^T=\bm{U}\bm{\Sigma}\bm{\Sigma}^T\bm{U}^T.
\]
!et
If we now multiply from the right with $\bm{U}$ (using the orthogonality of $\bm{U}$) we get
!bt
\[
\left(\bm{X}\bm{X}^T\right)\bm{U}=\bm{U}\bm{\Sigma}\bm{\Sigma}^T.
\]
!et
This means the vectors $\bm{u}_i$ of the orthogonal matrix $\bm{U}$ are the eigenvectors of the matrix $\bm{X}\bm{X}^T$
with eigenvalues given by the singular values squared, that is
!bt
\[
\left(\bm{X}\bm{X}^T\right)\bm{u}_i=\bm{u}_i\sigma_i^2.
\]
!et
!split
===== Ridge and LASSO Regression =====