lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

more updates

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-06 18:13:44 +02:00
parent 35db449d6c
commit f9e7f36c5e
7 changed files with 142 additions and 710 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+9 -18
View File
@@ -239,21 +239,15 @@ Automatically generated HTML file from DocOnce source
2,
None,
'ridge-and-lasso-regression'),
('More on Ridge Regression', 2, None, 'more-on-ridge-regression'),
('Deriving the Ridge Regression Equations',
2,
None,
'deriving-the-ridge-regression-equations'),
('Interpreting the Ridge results',
2,
None,
'interpreting-the-ridge-results'),
('More interpretations', 2, None, 'more-interpretations'),
('A better understanding of regularization',
2,
None,
'a-better-understanding-of-regularization'),
('Decomposing the OLS and Ridge expressions',
2,
None,
'decomposing-the-ols-and-ridge-expressions'),
('Mathematical Properties', 2, None, 'mathematical-properties'),
('Exercises for week 36, September 6-10',
2,
None,
@@ -367,15 +361,12 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs062.html#what-does-it-mean" style="font-size: 80%;"><b>What does it mean?</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#and-finally-boldsymbol-x-boldsymbol-x-t" style="font-size: 80%;"><b>And finally \( \boldsymbol{X}\boldsymbol{X}^T \)</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#deriving-the-ridge-regression-equations" style="font-size: 80%;"><b>Deriving the Ridge Regression Equations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs070.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs071.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs071.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs071.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -434,7 +425,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs071.html">72</a></li>
<li><a href="._week35-bs068.html">69</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+25 -151
View File
@@ -2809,10 +2809,15 @@ $\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \ti
$$
<p>
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \).
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by
the non-zero singular values plus now a series of zeros. The column
vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and
measure how much correlations are contained in the rows of \( \boldsymbol{X} \).
<p>
Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \).
Since we will mainly be interested in the correlations among the features
of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular
values and the column vectors of \( \boldsymbol{V} \).
</section>
@@ -2889,7 +2894,7 @@ $$
<section>
<h2 id="more-on-ridge-regression">More on Ridge Regression </h2>
<h2 id="deriving-the-ridge-regression-equations">Deriving the Ridge Regression Equations </h2>
<p>
Using the matrix-vector expression for Ridge regression,
@@ -2904,10 +2909,10 @@ $$
by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
<p>&nbsp;<br>
$$
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
<p>&nbsp;<br>
@@ -2924,24 +2929,28 @@ $$
with \( t \) a finite positive number.
<p>
We see that Ridge regression is nothing but the standard
OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The
consequences, in particular for our discussion of the bias-variance tradeoff
are rather interesting.
<p>
Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had
When we compare this with the ordinary least squares result we have
<p>&nbsp;<br>
$$
(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}.
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
<p>&nbsp;<br>
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
<p>
We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
We see that Ridge regression is nothing but the standard OLS with a
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
<p>
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
<p>&nbsp;<br>
$$
\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}
\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
<p>&nbsp;<br>
@@ -2950,7 +2959,7 @@ For Ridge regression this becomes
<p>&nbsp;<br>
$$
\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
<p>&nbsp;<br>
@@ -3028,141 +3037,6 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
</section>
<section>
<h2 id="a-better-understanding-of-regularization">A better understanding of regularization </h2>
<p>
The parameter \( \lambda \) that we have introduced in the Ridge (and
Lasso as well) regression is often called a regularization parameter
or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?
<p>
Here we will first look at how to analyze the difference between the
standard OLS equations and the Ridge expressions in terms of a linear
algebra analysis using the SVD algorithm. Thereafter, we will link
(see the material on the bias-variance tradeoff below) these
observation to the statisical analysis of the results. In particular
we consider how the variance of the parameters \( \boldsymbol{\beta} \) is
affected by changing the parameter \( \lambda \).
</section>
<section>
<h2 id="decomposing-the-ols-and-ridge-expressions">Decomposing the OLS and Ridge expressions </h2>
<p>
We have our design matrix
\( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as
<p>&nbsp;<br>
$$
\boldsymbol{X} = \boldsymbol{U\Sigma V^T},
$$
<p>&nbsp;<br>
<p>
with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \)
and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \).
<p>
The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \).
</section>
<section>
<h2 id="mathematical-properties">Mathematical Properties </h2>
<p>
There are several interesting mathematical properties which will be
relevant when we are going to discuss the differences between say
ordinary least squares (OLS) and <b>Ridge</b> regression.
<p>
We have from OLS that the parameters of the linear approximation are given by
<p>&nbsp;<br>
$$
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>&nbsp;<br>
<p>
The matrix to invert can be rewritten in terms of our SVD decomposition as
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
<p>&nbsp;<br>
Using the orthogonality properties of \( \boldsymbol{U} \) we have
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T,
$$
<p>&nbsp;<br>
with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared.
<p>
This means that
<p>&nbsp;<br>
$$
(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D},
$$
<p>&nbsp;<br>
that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that
<p>&nbsp;<br>
$$
(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D},
$$
<p>&nbsp;<br>
that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same.
<p>
Going back to our OLS equation we have
<p>&nbsp;<br>
$$
\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
<p>&nbsp;<br>
We will come back to this expression when we discuss Ridge regression.
<p>&nbsp;<br>
$$ \tilde{y}^{OLS}={\bf X}\hat{\beta}^{OLS}=\sum_{j=1}^p {\bf u}_j{\bf u}_j^T{\bf y}$$
<p>&nbsp;<br> and for Ridge we have&#160;
<p>&nbsp;<br>
$$ \tilde{y}^{Ridge}={\bf X}\hat{\beta}^{Ridge}=\sum_{j=1}^p {\bf u}_j\frac{\sigma_j^2}{\sigma_j^2+\lambda}{\bf u}_j^T{\bf y}$$
<p>&nbsp;<br> .&#160;
<p>
It is indeed the economy-sized SVD, note the summation runs up tp <p>&nbsp;<br>
$$p$$
<p>&nbsp;<br> only and not <p>&nbsp;<br>
$$n$$
<p>&nbsp;<br>.&#160;
<p>
Here we have that <p>&nbsp;<br>
$${\bf X} = {\bf U}{\bf \Sigma}{\bf V}^T$$
<p>&nbsp;<br>, with <p>&nbsp;<br>
$$\Sigma$$
<p>&nbsp;<br> being an <p>&nbsp;<br>
$$ n\times p$$
<p>&nbsp;<br> matrix and <p>&nbsp;<br>
$${\bf V}$$
<p>&nbsp;<br> being a <p>&nbsp;<br>
$$ p\times p$$
<p>&nbsp;<br> matrix. We also have assumed here that <p>&nbsp;<br>
$$ n > p$$
<p>&nbsp;<br>.&#160;
</section>
<section>
<h2 id="exercises-for-week-36-september-6-10">Exercises for week 36, September 6-10 </h2>
doc/pub/week35/html/week35-solarized.html
+29 -127
View File
@@ -259,21 +259,15 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'ridge-and-lasso-regression'),
('More on Ridge Regression', 2, None, 'more-on-ridge-regression'),
('Deriving the Ridge Regression Equations',
2,
None,
'deriving-the-ridge-regression-equations'),
('Interpreting the Ridge results',
2,
None,
'interpreting-the-ridge-results'),
('More interpretations', 2, None, 'more-interpretations'),
('A better understanding of regularization',
2,
None,
'a-better-understanding-of-regularization'),
('Decomposing the OLS and Ridge expressions',
2,
None,
'decomposing-the-ols-and-ridge-expressions'),
('Mathematical Properties', 2, None, 'mathematical-properties'),
('Exercises for week 36, September 6-10',
2,
None,
@@ -2795,10 +2789,15 @@ $\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \ti
$$
<p>
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \).
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by
the non-zero singular values plus now a series of zeros. The column
vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and
measure how much correlations are contained in the rows of \( \boldsymbol{X} \).
<p>
Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \).
Since we will mainly be interested in the correlations among the features
of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular
values and the column vectors of \( \boldsymbol{V} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2861,7 +2860,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-ridge-regression">More on Ridge Regression </h2>
<h2 id="deriving-the-ridge-regression-equations">Deriving the Ridge Regression Equations </h2>
<p>
Using the matrix-vector expression for Ridge regression,
@@ -2874,9 +2873,9 @@ $$
by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
$$
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
<p>
@@ -2890,28 +2889,32 @@ $$
with \( t \) a finite positive number.
<p>
We see that Ridge regression is nothing but the standard
OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The
consequences, in particular for our discussion of the bias-variance tradeoff
are rather interesting.
<p>
Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had
When we compare this with the ordinary least squares result we have
$$
(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}.
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
<p>
We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
We see that Ridge regression is nothing but the standard OLS with a
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
<p>
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
$$
\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}
\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
<p>
For Ridge regression this becomes
$$
\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
<p>
@@ -2978,107 +2981,6 @@ We will come back to more interpreations after we have gone through some of the
For more discussions of Ridge and Lasso regression, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.
Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et al's article</a> is also recommended.
<p>
<!-- !split -->
<h2 id="a-better-understanding-of-regularization">A better understanding of regularization </h2>
<p>
The parameter \( \lambda \) that we have introduced in the Ridge (and
Lasso as well) regression is often called a regularization parameter
or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?
<p>
Here we will first look at how to analyze the difference between the
standard OLS equations and the Ridge expressions in terms of a linear
algebra analysis using the SVD algorithm. Thereafter, we will link
(see the material on the bias-variance tradeoff below) these
observation to the statisical analysis of the results. In particular
we consider how the variance of the parameters \( \boldsymbol{\beta} \) is
affected by changing the parameter \( \lambda \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="decomposing-the-ols-and-ridge-expressions">Decomposing the OLS and Ridge expressions </h2>
<p>
We have our design matrix
\( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as
$$
\boldsymbol{X} = \boldsymbol{U\Sigma V^T},
$$
<p>
with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \)
and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \).
<p>
The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="mathematical-properties">Mathematical Properties </h2>
<p>
There are several interesting mathematical properties which will be
relevant when we are going to discuss the differences between say
ordinary least squares (OLS) and <b>Ridge</b> regression.
<p>
We have from OLS that the parameters of the linear approximation are given by
$$
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>
The matrix to invert can be rewritten in terms of our SVD decomposition as
$$
\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
Using the orthogonality properties of \( \boldsymbol{U} \) we have
$$
\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T,
$$
with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared.
<p>
This means that
$$
(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D},
$$
that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that
$$
(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D},
$$
that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same.
<p>
Going back to our OLS equation we have
$$
\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
We will come back to this expression when we discuss Ridge regression.
$$ \tilde{y}^{OLS}={\bf X}\hat{\beta}^{OLS}=\sum_{j=1}^p {\bf u}_j{\bf u}_j^T{\bf y}$$ and for Ridge we have&#160;
$$ \tilde{y}^{Ridge}={\bf X}\hat{\beta}^{Ridge}=\sum_{j=1}^p {\bf u}_j\frac{\sigma_j^2}{\sigma_j^2+\lambda}{\bf u}_j^T{\bf y}$$ .&#160;
<p>
It is indeed the economy-sized SVD, note the summation runs up tp $$p$$ only and not $$n$$.&#160;
<p>
Here we have that $${\bf X} = {\bf U}{\bf \Sigma}{\bf V}^T$$, with $$\Sigma$$ being an $$ n\times p$$ matrix and $${\bf V}$$ being a $$ p\times p$$ matrix. We also have assumed here that $$ n > p$$.&#160;
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/html/week35.html
+29 -127
View File
@@ -264,21 +264,15 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'ridge-and-lasso-regression'),
('More on Ridge Regression', 2, None, 'more-on-ridge-regression'),
('Deriving the Ridge Regression Equations',
2,
None,
'deriving-the-ridge-regression-equations'),
('Interpreting the Ridge results',
2,
None,
'interpreting-the-ridge-results'),
('More interpretations', 2, None, 'more-interpretations'),
('A better understanding of regularization',
2,
None,
'a-better-understanding-of-regularization'),
('Decomposing the OLS and Ridge expressions',
2,
None,
'decomposing-the-ols-and-ridge-expressions'),
('Mathematical Properties', 2, None, 'mathematical-properties'),
('Exercises for week 36, September 6-10',
2,
None,
@@ -2800,10 +2794,15 @@ $\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \ti
$$
<p>
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and measure how much correlations are contained in the rows of \( \m{X} \).
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by
the non-zero singular values plus now a series of zeros. The column
vectors of \( \boldsymbol{U} \) are the eigenvectors of \( \boldsymbol{X}\boldsymbol{X}^T \) and
measure how much correlations are contained in the rows of \( \boldsymbol{X} \).
<p>
Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of \( \boldsymbol{V} \).
Since we will mainly be interested in the correlations among the features
of our data (the columns of \( \boldsymbol{X} \), the quantity of interest for us are the non-zero singular
values and the column vectors of \( \boldsymbol{V} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2866,7 +2865,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-ridge-regression">More on Ridge Regression </h2>
<h2 id="deriving-the-ridge-regression-equations">Deriving the Ridge Regression Equations </h2>
<p>
Using the matrix-vector expression for Ridge regression,
@@ -2879,9 +2878,9 @@ $$
by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
$$
\boldsymbol{\beta}^{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \left(\boldsymbol{X}^T\boldsymbol{X}+\lambda\boldsymbol{I}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
<p>
@@ -2895,28 +2894,32 @@ $$
with \( t \) a finite positive number.
<p>
We see that Ridge regression is nothing but the standard
OLS with a modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The
consequences, in particular for our discussion of the bias-variance tradeoff
are rather interesting.
<p>
Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had
When we compare this with the ordinary least squares result we have
$$
(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D}.
\hat{\boldsymbol{\beta}}_{\mathrm{OLS}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
$$
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \).
<p>
We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
We see that Ridge regression is nothing but the standard OLS with a
modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
<p>
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
$$
\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}
\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
<p>
For Ridge regression this becomes
$$
\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
<p>
@@ -2983,107 +2986,6 @@ We will come back to more interpreations after we have gone through some of the
For more discussions of Ridge and Lasso regression, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.
Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et al's article</a> is also recommended.
<p>
<!-- !split -->
<h2 id="a-better-understanding-of-regularization">A better understanding of regularization </h2>
<p>
The parameter \( \lambda \) that we have introduced in the Ridge (and
Lasso as well) regression is often called a regularization parameter
or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?
<p>
Here we will first look at how to analyze the difference between the
standard OLS equations and the Ridge expressions in terms of a linear
algebra analysis using the SVD algorithm. Thereafter, we will link
(see the material on the bias-variance tradeoff below) these
observation to the statisical analysis of the results. In particular
we consider how the variance of the parameters \( \boldsymbol{\beta} \) is
affected by changing the parameter \( \lambda \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="decomposing-the-ols-and-ridge-expressions">Decomposing the OLS and Ridge expressions </h2>
<p>
We have our design matrix
\( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). With the SVD we decompose it as
$$
\boldsymbol{X} = \boldsymbol{U\Sigma V^T},
$$
<p>
with \( \boldsymbol{U}\in {\mathbb{R}}^{n\times n} \), \( \boldsymbol{\Sigma}\in {\mathbb{R}}^{n\times p} \)
and \( \boldsymbol{V}\in {\mathbb{R}}^{p\times p} \).
<p>
The matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \) are unitary/orthonormal matrices, that is in case the matrices are real we have \( \boldsymbol{U}^T\boldsymbol{U}=\boldsymbol{U}\boldsymbol{U}^T=\boldsymbol{I} \) and \( \boldsymbol{V}^T\boldsymbol{V}=\boldsymbol{V}\boldsymbol{V}^T=\boldsymbol{I} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="mathematical-properties">Mathematical Properties </h2>
<p>
There are several interesting mathematical properties which will be
relevant when we are going to discuss the differences between say
ordinary least squares (OLS) and <b>Ridge</b> regression.
<p>
We have from OLS that the parameters of the linear approximation are given by
$$
\boldsymbol{\tilde{y}} = \boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>
The matrix to invert can be rewritten in terms of our SVD decomposition as
$$
\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
Using the orthogonality properties of \( \boldsymbol{U} \) we have
$$
\boldsymbol{X}^T\boldsymbol{X} = \boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T = \boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T,
$$
with \( \boldsymbol{D} \) being a diagonal matrix with values along the diagonal given by the singular values squared.
<p>
This means that
$$
(\boldsymbol{X}^T\boldsymbol{X})\boldsymbol{V} = \boldsymbol{V}\boldsymbol{D},
$$
that is the eigenvectors of \( (\boldsymbol{X}^T\boldsymbol{X}) \) are given by the columns of the right singular matrix of \( \boldsymbol{X} \) and the eigenvalues are the squared singular values. It is easy to show (show this) that
$$
(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U} = \boldsymbol{U}\boldsymbol{D},
$$
that is, the eigenvectors of \( (\boldsymbol{X}\boldsymbol{X})^T \) are the columns of the left singular matrix and the eigenvalues are the same.
<p>
Going back to our OLS equation we have
$$
\boldsymbol{X}\boldsymbol{\beta} = \boldsymbol{X}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}\boldsymbol{X}^T\boldsymbol{y}=\boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{D}\boldsymbol{V}^T \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
We will come back to this expression when we discuss Ridge regression.
$$ \tilde{y}^{OLS}={\bf X}\hat{\beta}^{OLS}=\sum_{j=1}^p {\bf u}_j{\bf u}_j^T{\bf y}$$ and for Ridge we have&#160;
$$ \tilde{y}^{Ridge}={\bf X}\hat{\beta}^{Ridge}=\sum_{j=1}^p {\bf u}_j\frac{\sigma_j^2}{\sigma_j^2+\lambda}{\bf u}_j^T{\bf y}$$ .&#160;
<p>
It is indeed the economy-sized SVD, note the summation runs up tp $$p$$ only and not $$n$$.&#160;
<p>
Here we have that $${\bf X} = {\bf U}{\bf \Sigma}{\bf V}^T$$, with $$\Sigma$$ being an $$ n\times p$$ matrix and $${\bf V}$$ being a $$ p\times p$$ matrix. We also have assumed here that $$ n > p$$.&#160;
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+26 -175
View File
@@ -3653,9 +3653,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of $\\boldsymbol{U}$ are the eigenvectors of $\\boldsymbol{X}\\boldsymbol{X}^T$ and measure how much correlations are contained in the rows of $\\m{X}$.\n",
"It means that the eigenvalues of $\\boldsymbol{X}\\boldsymbol{X}^T$ are again given by\n",
"the non-zero singular values plus now a series of zeros. The column\n",
"vectors of $\\boldsymbol{U}$ are the eigenvectors of $\\boldsymbol{X}\\boldsymbol{X}^T$ and\n",
"measure how much correlations are contained in the rows of $\\boldsymbol{X}$.\n",
"\n",
"Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of $\\boldsymbol{V}$.\n",
"Since we will mainly be interested in the correlations among the features\n",
"of our data (the columns of $\\boldsymbol{X}$, the quantity of interest for us are the non-zero singular\n",
"values and the column vectors of $\\boldsymbol{V}$.\n",
"\n",
"\n",
"## Ridge and LASSO Regression\n",
@@ -3783,7 +3788,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## More on Ridge Regression\n",
"## Deriving the Ridge Regression Equations\n",
"\n",
"Using the matrix-vector expression for Ridge regression,"
]
@@ -3803,7 +3808,8 @@
"source": [
"by taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n",
"a slightly modified matrix inversion problem which for finite values\n",
"of $\\lambda$ does not suffer from singularity problems. We obtain"
"of $\\lambda$ does not suffer from singularity problems. We obtain\n",
"the optimal parameters"
]
},
{
@@ -3811,7 +3817,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{Ridge}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}+\\lambda\\boldsymbol{I}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
"$$"
]
},
@@ -3837,12 +3843,7 @@
"source": [
"with $t$ a finite positive number. \n",
"\n",
"We see that Ridge regression is nothing but the standard\n",
"OLS with a modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The\n",
"consequences, in particular for our discussion of the bias-variance tradeoff \n",
"are rather interesting.\n",
"\n",
"Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had"
"When we compare this with the ordinary least squares result we have"
]
},
{
@@ -3850,7 +3851,7 @@
"metadata": {},
"source": [
"$$\n",
"(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U} = \\boldsymbol{U}\\boldsymbol{D}.\n",
"\\hat{\\boldsymbol{\\beta}}_{\\mathrm{OLS}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
"$$"
]
},
@@ -3858,7 +3859,17 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as"
"which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$.\n",
"\n",
"\n",
"We see that Ridge regression is nothing but the standard OLS with a\n",
"modified diagonal term added to $\\boldsymbol{X}^T\\boldsymbol{X}$. The consequences, in\n",
"particular for our discussion of the bias-variance tradeoff are rather\n",
"interesting. We will see that for specific values of $\\lambda$, we may\n",
"even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n",
"\n",
"Using our insights about the SVD of the design matrix $\\boldsymbol{X}$ \n",
"We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as"
]
},
{
@@ -3866,7 +3877,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}\n",
"\\boldsymbol{X}\\boldsymbol{\\beta} = =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
"$$"
]
},
@@ -3882,7 +3893,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
"\\boldsymbol{X}\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
"$$"
]
},
@@ -3981,166 +3992,6 @@
"Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n",
"\n",
"\n",
"<!-- !split -->\n",
"## A better understanding of regularization\n",
"\n",
"The parameter $\\lambda$ that we have introduced in the Ridge (and\n",
"Lasso as well) regression is often called a regularization parameter\n",
"or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?\n",
"\n",
"Here we will first look at how to analyze the difference between the\n",
"standard OLS equations and the Ridge expressions in terms of a linear\n",
"algebra analysis using the SVD algorithm. Thereafter, we will link\n",
"(see the material on the bias-variance tradeoff below) these\n",
"observation to the statisical analysis of the results. In particular\n",
"we consider how the variance of the parameters $\\boldsymbol{\\beta}$ is\n",
"affected by changing the parameter $\\lambda$.\n",
"\n",
"## Decomposing the OLS and Ridge expressions\n",
"\n",
"We have our design matrix\n",
" $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. With the SVD we decompose it as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X} = \\boldsymbol{U\\Sigma V^T},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"with $\\boldsymbol{U}\\in {\\mathbb{R}}^{n\\times n}$, $\\boldsymbol{\\Sigma}\\in {\\mathbb{R}}^{n\\times p}$\n",
"and $\\boldsymbol{V}\\in {\\mathbb{R}}^{p\\times p}$.\n",
"\n",
"The matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\\boldsymbol{U}^T\\boldsymbol{U}=\\boldsymbol{U}\\boldsymbol{U}^T=\\boldsymbol{I}$ and $\\boldsymbol{V}^T\\boldsymbol{V}=\\boldsymbol{V}\\boldsymbol{V}^T=\\boldsymbol{I}$.\n",
"\n",
"\n",
"\n",
"\n",
"## Mathematical Properties\n",
"\n",
"There are several interesting mathematical properties which will be\n",
"relevant when we are going to discuss the differences between say\n",
"ordinary least squares (OLS) and **Ridge** regression.\n",
"\n",
"We have from OLS that the parameters of the linear approximation are given by"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\tilde{y}} = \\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The matrix to invert can be rewritten in terms of our SVD decomposition as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the orthogonality properties of $\\boldsymbol{U}$ we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X} = \\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T = \\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"with $\\boldsymbol{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared. \n",
"\n",
"This means that"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"(\\boldsymbol{X}^T\\boldsymbol{X})\\boldsymbol{V} = \\boldsymbol{V}\\boldsymbol{D},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"that is the eigenvectors of $(\\boldsymbol{X}^T\\boldsymbol{X})$ are given by the columns of the right singular matrix of $\\boldsymbol{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U} = \\boldsymbol{U}\\boldsymbol{D},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"that is, the eigenvectors of $(\\boldsymbol{X}\\boldsymbol{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same. \n",
"\n",
"Going back to our OLS equation we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}\\boldsymbol{\\beta} = \\boldsymbol{X}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}=\\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{D}\\boldsymbol{V}^T \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will come back to this expression when we discuss Ridge regression. \n",
"\n",
"\n",
"$$ \\tilde{y}^{OLS}={\\bf X}\\hat{\\beta}^{OLS}=\\sum_{j=1}^p {\\bf u}_j{\\bf u}_j^T{\\bf y}$$ and for Ridge we have \n",
"\n",
"$$ \\tilde{y}^{Ridge}={\\bf X}\\hat{\\beta}^{Ridge}=\\sum_{j=1}^p {\\bf u}_j\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}{\\bf u}_j^T{\\bf y}$$ . \n",
"\n",
"It is indeed the economy-sized SVD, note the summation runs up tp $$p$$ only and not $$n$$. \n",
"\n",
"Here we have that $${\\bf X} = {\\bf U}{\\bf \\Sigma}{\\bf V}^T$$, with $$\\Sigma$$ being an $$ n\\times p$$ matrix and $${\\bf V}$$ being a $$ p\\times p$$ matrix. We also have assumed here that $$ n > p$$. \n",
"\n",
"\n",
"\n",
"\n",
"## Exercises for week 36, September 6-10\n",
doc/src/week35/week35.do.txt
+24 -112
View File
@@ -2281,9 +2281,14 @@ $\bm{X}\bm{X}^T$\bm{U}=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm
\]
!et
It means that the eigenvalues of $\bm{X}\bm{X}^T$ are again given by the non-zero singular values plus now a series of zeros. However, when performing the matrix-matrix multiplications. The column vectors of $\bm{U}$ are the eigenvectors of $\bm{X}\bm{X}^T$ and measure how much correlations are contained in the rows of $\m{X}$.
It means that the eigenvalues of $\bm{X}\bm{X}^T$ are again given by
the non-zero singular values plus now a series of zeros. The column
vectors of $\bm{U}$ are the eigenvectors of $\bm{X}\bm{X}^T$ and
measure how much correlations are contained in the rows of $\bm{X}$.
Since we will mainly be interested in the correlations among features of our data, the quantity of interest for us are the non-zero singular values and the column vectors of $\bm{V}$.
Since we will mainly be interested in the correlations among the features
of our data (the columns of $\bm{X}$, the quantity of interest for us are the non-zero singular
values and the column vectors of $\bm{V}$.
!split
@@ -2350,7 +2355,7 @@ Here we have defined the norm-1 as
!split
===== More on Ridge Regression =====
===== Deriving the Ridge Regression Equations =====
Using the matrix-vector expression for Ridge regression,
@@ -2363,10 +2368,10 @@ C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}
by taking the derivatives with respect to $\bm{\beta}$ we obtain then
a slightly modified matrix inversion problem which for finite values
of $\lambda$ does not suffer from singularity problems. We obtain
the optimal parameters
!bt
\[
\bm{\beta}^{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
\hat{\bm{\beta}}_{\mathrm{Ridge}} = \left(\bm{X}^T\bm{X}+\lambda\bm{I}\right)^{-1}\bm{X}^T\bm{y},
\]
!et
@@ -2380,22 +2385,26 @@ with $\bm{I}$ being a $p\times p$ identity matrix with the constraint that
with $t$ a finite positive number.
We see that Ridge regression is nothing but the standard
OLS with a modified diagonal term added to $\bm{X}^T\bm{X}$. The
consequences, in particular for our discussion of the bias-variance tradeoff
are rather interesting.
Furthermore, if we use the result above in terms of the SVD decomposition (our analysis was done for the OLS method), we had
When we compare this with the ordinary least squares result we have
!bt
\[
(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D}.
\hat{\bm{\beta}}_{\mathrm{OLS}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
\]
!et
which can lead to singular matrices. However, with the SVD, we can always compute the inverse of the matrix $\bm{X}^T\bm{X}$.
We can analyse the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
We see that Ridge regression is nothing but the standard OLS with a
modified diagonal term added to $\bm{X}^T\bm{X}$. The consequences, in
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of $\lambda$, we may
even reduce the variance of the optimal parameters $\bm{\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
Using our insights about the SVD of the design matrix $\bm{X}$
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
!bt
\[
\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}
\bm{X}\bm{\beta} = =\bm{U}\bm{U}^T\bm{y}.
\]
!et
@@ -2404,7 +2413,7 @@ For Ridge regression this becomes
!bt
\[
\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
\]
!et
@@ -2468,103 +2477,6 @@ For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"ht
Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended.
!split
===== A better understanding of regularization =====
The parameter $\lambda$ that we have introduced in the Ridge (and
Lasso as well) regression is often called a regularization parameter
or shrinkage parameter. It is common to call it a hyperparameter. What does it mean mathemtically?
Here we will first look at how to analyze the difference between the
standard OLS equations and the Ridge expressions in terms of a linear
algebra analysis using the SVD algorithm. Thereafter, we will link
(see the material on the bias-variance tradeoff below) these
observation to the statisical analysis of the results. In particular
we consider how the variance of the parameters $\bm{\beta}$ is
affected by changing the parameter $\lambda$.
!split
===== Decomposing the OLS and Ridge expressions =====
We have our design matrix
$\bm{X}\in {\mathbb{R}}^{n\times p}$. With the SVD we decompose it as
!bt
\[
\bm{X} = \bm{U\Sigma V^T},
\]
!et
with $\bm{U}\in {\mathbb{R}}^{n\times n}$, $\bm{\Sigma}\in {\mathbb{R}}^{n\times p}$
and $\bm{V}\in {\mathbb{R}}^{p\times p}$.
The matrices $\bm{U}$ and $\bm{V}$ are unitary/orthonormal matrices, that is in case the matrices are real we have $\bm{U}^T\bm{U}=\bm{U}\bm{U}^T=\bm{I}$ and $\bm{V}^T\bm{V}=\bm{V}\bm{V}^T=\bm{I}$.
!split
===== Mathematical Properties =====
There are several interesting mathematical properties which will be
relevant when we are going to discuss the differences between say
ordinary least squares (OLS) and _Ridge_ regression.
We have from OLS that the parameters of the linear approximation are given by
!bt
\[
\bm{\tilde{y}} = \bm{X}\bm{\beta} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
The matrix to invert can be rewritten in terms of our SVD decomposition as
!bt
\[
\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T.
\]
!et
Using the orthogonality properties of $\bm{U}$ we have
!bt
\[
\bm{X}^T\bm{X} = \bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T = \bm{V}\bm{D}\bm{V}^T,
\]
!et
with $\bm{D}$ being a diagonal matrix with values along the diagonal given by the singular values squared.
This means that
!bt
\[
(\bm{X}^T\bm{X})\bm{V} = \bm{V}\bm{D},
\]
!et
that is the eigenvectors of $(\bm{X}^T\bm{X})$ are given by the columns of the right singular matrix of $\bm{X}$ and the eigenvalues are the squared singular values. It is easy to show (show this) that
!bt
\[
(\bm{X}\bm{X}^T)\bm{U} = \bm{U}\bm{D},
\]
!et
that is, the eigenvectors of $(\bm{X}\bm{X})^T$ are the columns of the left singular matrix and the eigenvalues are the same.
Going back to our OLS equation we have
!bt
\[
\bm{X}\bm{\beta} = \bm{X}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}\bm{X}^T\bm{y}=\bm{U\Sigma V^T}\left(\bm{V}\bm{D}\bm{V}^T \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\bm{U}\bm{U}^T\bm{y}.
\]
!et
We will come back to this expression when we discuss Ridge regression.
$$ \tilde{y}^{OLS}={\bf X}\hat{\beta}^{OLS}=\sum_{j=1}^p {\bf u}_j{\bf u}_j^T{\bf y}$$ and for Ridge we have 
$$ \tilde{y}^{Ridge}={\bf X}\hat{\beta}^{Ridge}=\sum_{j=1}^p {\bf u}_j\frac{\sigma_j^2}{\sigma_j^2+\lambda}{\bf u}_j^T{\bf y}$$ . 
It is indeed the economy-sized SVD, note the summation runs up tp $$p$$ only and not $$n$$. 
Here we have that $${\bf X} = {\bf U}{\bf \Sigma}{\bf V}^T$$, with $$\Sigma$$ being an $$ n\times p$$ matrix and $${\bf V}$$ being a $$ p\times p$$ matrix. We also have assumed here that $$ n > p$$. 
!split