lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week 35

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-05 22:23:00 +02:00
parent 977be1cf3e
commit da91ca8a27
7 changed files with 414 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+23 -18
View File
@@ -188,6 +188,10 @@ Automatically generated HTML file from DocOnce source
None,
'matheamtics-of-the-svd-and-implications'),
('Example Matrix', 2, None, 'example-matrix'),
('Setting up the Matrix to be inverted',
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Ridge and LASSO Regression',
2,
None,
@@ -330,23 +334,24 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs047.html#friday-september-3" style="font-size: 80%;"><b>Friday September 3</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs048.html#matheamtics-of-the-svd-and-implications" style="font-size: 80%;"><b>Matheamtics of the SVD and implications</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs049.html#example-matrix" style="font-size: 80%;"><b>Example Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs050.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs050.html#setting-up-the-matrix-to-be-inverted" style="font-size: 80%;"><b>Setting up the Matrix to be inverted</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -405,7 +410,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs064.html">65</a></li>
<li><a href="._week35-bs065.html">66</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+75 -1
View File
@@ -2103,7 +2103,81 @@ $$
$$
<p>&nbsp;<br>
is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
<p>
is a \( 3\times 3 \) matrix. The last row and column of this last matrix
contain only zeros. This will have important consequences for our SVD
decomposition of the design matrix.
</section>
<section>
<h2 id="setting-up-the-matrix-to-be-inverted">Setting up the Matrix to be inverted </h2>
<p>
The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
$$
<p>&nbsp;<br>
and using the orthogonality of the matrix \( \boldsymbol{U} \) we have
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
<p>&nbsp;<br>
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \).
<p>
This means, using the orthogonality of \( \boldsymbol{V} \), that we get
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{D}^2.
$$
<p>&nbsp;<br>
<p>
We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where
<p>&nbsp;<br>
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
<p>&nbsp;<br>
and using our SVD decomposition of \( \boldsymbol{X} \) we have
<p>&nbsp;<br>
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{D}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y},
$$
<p>&nbsp;<br>
which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),
<p>&nbsp;<br>
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y},
$$
<p>&nbsp;<br>
<p>
Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \)
<p>&nbsp;<br>
$$
\boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}],
$$
<p>&nbsp;<br>
that belong to \( i>p-1 \), give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero).
<p>
It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).
</section>
doc/pub/week35/html/week35-solarized.html
+65 -1
View File
@@ -208,6 +208,10 @@ div { text-align: justify; text-justify: inter-word; }
None,
'matheamtics-of-the-svd-and-implications'),
('Example Matrix', 2, None, 'example-matrix'),
('Setting up the Matrix to be inverted',
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Ridge and LASSO Regression',
2,
None,
@@ -2152,7 +2156,67 @@ $$
\end{bmatrix},
$$
is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
<p>
is a \( 3\times 3 \) matrix. The last row and column of this last matrix
contain only zeros. This will have important consequences for our SVD
decomposition of the design matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-matrix-to-be-inverted">Setting up the Matrix to be inverted </h2>
<p>
The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
$$
and using the orthogonality of the matrix \( \boldsymbol{U} \) we have
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \).
<p>
This means, using the orthogonality of \( \boldsymbol{V} \), that we get
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{D}^2.
$$
<p>
We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
and using our SVD decomposition of \( \boldsymbol{X} \) we have
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{D}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y},
$$
which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y},
$$
<p>
Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \)
$$
\boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}],
$$
that belong to \( i>p-1 \), give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero).
<p>
It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/html/week35.html
+65 -1
View File
@@ -213,6 +213,10 @@ div { text-align: justify; text-justify: inter-word; }
None,
'matheamtics-of-the-svd-and-implications'),
('Example Matrix', 2, None, 'example-matrix'),
('Setting up the Matrix to be inverted',
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Ridge and LASSO Regression',
2,
None,
@@ -2157,7 +2161,67 @@ $$
\end{bmatrix},
$$
is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
<p>
is a \( 3\times 3 \) matrix. The last row and column of this last matrix
contain only zeros. This will have important consequences for our SVD
decomposition of the design matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="setting-up-the-matrix-to-be-inverted">Setting up the Matrix to be inverted </h2>
<p>
The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
$$
and using the orthogonality of the matrix \( \boldsymbol{U} \) we have
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \).
<p>
This means, using the orthogonality of \( \boldsymbol{V} \), that we get
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{D}^2.
$$
<p>
We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
and using our SVD decomposition of \( \boldsymbol{X} \) we have
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{D}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y},
$$
which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),
$$
\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y},
$$
<p>
Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \)
$$
\boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}],
$$
that belong to \( i>p-1 \), give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero).
<p>
It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+123 -1
View File
@@ -2618,9 +2618,131 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"is a $3\\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.\n",
"is a $3\\times 3 $ matrix. The last row and column of this last matrix\n",
"contain only zeros. This will have important consequences for our SVD\n",
"decomposition of the design matrix.\n",
"\n",
"\n",
"## Setting up the Matrix to be inverted\n",
"\n",
"The matrix that may cause problems for us is $\\boldsymbol{X}^T\\boldsymbol{X}$. Using the SVD we can rewrite this matrix as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and using the orthogonality of the matrix $\\boldsymbol{U}$ we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We define $\\boldsymbol{\\Sigma}^T\\boldsymbol{\\sigma}=\\boldsymbol{D}^2, where $\\boldsymbol{D}$ is a diagonal matrix containing only the singular values squared. It has dimensionality $\\p \\times p$.\n",
"\n",
"This means, using the orthogonality of $\\boldsymbol{V}$, that we get"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{D}^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can now insert the result for the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ into our equation for ordinary least squares where"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and using our SVD decomposition of $\\boldsymbol{X}$ we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{D}^{-2}\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{y},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which gives us, using the orthogonality of the matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$,"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}^T_j\\boldsymbol{y},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix $\\boldsymbol{U}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{U}=[\\boldsymbol{u}_0,\\boldsymbol{u}_1,\\dots,\\boldsymbol{u}_{n-1}],\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"that belong to $i>p-1$, give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to $i=p-1$. This corresponds also to the number of singular values (these are all non-zero).\n",
"\n",
"It means that the ordinary least square model (with the optimal parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\\boldsymbol{y}$ by the vectors of the matrix $\\boldsymbol{U}$.\n",
"\n",
"## Ridge and LASSO Regression\n",
"\n",
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
doc/src/week35/week35.do.txt
+63 -1
View File
@@ -1664,8 +1664,70 @@ which is a $2\times 2 $ matrix while
\end{bmatrix},
\]
!et
is a $3\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
is a $3\times 3 $ matrix. The last row and column of this last matrix
contain only zeros. This will have important consequences for our SVD
decomposition of the design matrix.
!split
===== Setting up the Matrix to be inverted =====
The matrix that may cause problems for us is $\bm{X}^T\bm{X}$. Using the SVD we can rewrite this matrix as
!bt
\[
\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T,
\]
!et
and using the orthogonality of the matrix $\bm{U}$ we have
!bt
\[
\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T.
\]
!et
We define $\bm{\Sigma}^T\bm{\sigma}=\bm{D}^2, where $\bm{D}$ is a diagonal matrix containing only the singular values squared. It has dimensionality $\p \times p$.
This means, using the orthogonality of $\bm{V}$, that we get
!bt
\[
\bm{X}^T\bm{X}=\bm{D}^2.
\]
!et
We can now insert the result for the matrix $\bm{X}^T\bm{X}$ into our equation for ordinary least squares where
!bt
\[
\tilde{y}_{\mathrm{OLS}}=\bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
\]
!et
and using our SVD decomposition of $\bm{X}$ we have
!bt
\[
\tilde{y}_{\mathrm{OLS}}=\bm{U}\bm{\Sigma}\bm{V}^T\bm{D}^{-2}\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{y},
\]
!et
which gives us, using the orthogonality of the matrices $\bm{U}$ and $\bm{V}$,
!bt
\[
\tilde{y}_{\mathrm{OLS}}=\bm{U}\bm{U}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_i\bm{u}^T_j\bm{y},
\]
!et
Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix $\bm{U}$
!bt
\[
\bm{U}=[\bm{u}_0,\bm{u}_1,\dots,\bm{u}_{n-1}],
\]
!et
that belong to $i>p-1$, give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to $i=p-1$. This corresponds also to the number of singular values (these are all non-zero).
It means that the ordinary least square model (with the optimal parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\bm{y}$ by the vectors of the matrix $\bm{U}$.
!split
===== Ridge and LASSO Regression =====