lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-09 06:19:09 +02:00
parent e44e4093a3
commit 79156f2150
7 changed files with 481 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week36/html/week36-bs.html
+43 -36
View File
@@ -88,6 +88,11 @@ Automatically generated HTML file from DocOnce source
('Yet another Example', 2, None, 'yet-another-example'),
('The OLS case', 2, None, 'the-ols-case'),
('The Ridge case', 2, None, 'the-ridge-case'),
('Writing the Cost Function',
2,
None,
'writing-the-cost-function'),
('Lasso case', 2, None, 'lasso-case'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -255,40 +260,42 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week36-bs017.html#yet-another-example" style="font-size: 80%;">Yet another Example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs018.html#the-ols-case" style="font-size: 80%;">The OLS case</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs019.html#the-ridge-case" style="font-size: 80%;">The Ridge case</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs020.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs021.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#independent-and-identically-distrubuted-iid" style="font-size: 80%;">Independent and Identically Distrubuted (iid)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs053.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs020.html#writing-the-cost-function" style="font-size: 80%;">Writing the Cost Function</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs021.html#lasso-case" style="font-size: 80%;">Lasso case</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#deriving-ols-from-a-probability-distribution" style="font-size: 80%;">Deriving OLS from a probability distribution</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#independent-and-identically-distrubuted-iid" style="font-size: 80%;">Independent and Identically Distrubuted (iid)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs048.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs049.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs050.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs051.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs052.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs053.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs054.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs055.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
</ul>
</li>
@@ -323,7 +330,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 8, 2021</h4></center> <!-- date -->
<center><h4>Sep 9, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -347,7 +354,7 @@ MathJax.Hub.Config({
<li><a href="._week36-bs008.html">9</a></li>
<li><a href="._week36-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week36-bs053.html">54</a></li>
<li><a href="._week36-bs055.html">56</a></li>
<li><a href="._week36-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week36/html/week36-reveal.html
+84 -8
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 8, 2021</h4></center> <!-- date -->
<center><h4>Sep 9, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -654,7 +654,7 @@ $$
<p>&nbsp;<br>
<p>
Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \).
Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).
<p>
We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
@@ -677,7 +677,7 @@ $$
and our inputs as a \( 3\times 2 \) design matrix
<p>&nbsp;<br>
$$
\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix},
$$
<p>&nbsp;<br>
@@ -701,13 +701,10 @@ Inserting the above values we obtain that
<p>&nbsp;<br>
$$
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}\frac{11}{5} \\ 2\end{bmatrix},
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
$$
<p>&nbsp;<br>
<p>
Computing the mean squared error we obtian a value of \( 0.27 \).
<p>
The code which implements this simpler case is presented after the discussion of Ridge and Lasso.
</section>
@@ -729,7 +726,7 @@ Inserting the above values we obtain that
<p>&nbsp;<br>
$$
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
$$
<p>&nbsp;<br>
@@ -742,6 +739,85 @@ To see this, let us write the cost function for Ridge regression.
</section>
<section>
<h2 id="writing-the-cost-function">Writing the Cost Function </h2>
<p>
We define the MSE without the \( 1/n \) factor and have then, using that
<p>&nbsp;<br>
$$
\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix},
$$
<p>&nbsp;<br>
<p>&nbsp;<br>
$$
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2),
$$
<p>&nbsp;<br>
and taking the derivative with respect to \( \beta_0 \) we get
<p>&nbsp;<br>
$$
\beta_0=\frac{8}{4+\lambda},
$$
<p>&nbsp;<br>
and for \( \beta_1 \) we have
<p>&nbsp;<br>
$$
\beta_1=\frac{2}{1+\lambda},
$$
<p>&nbsp;<br>
<p>
Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving
<p>&nbsp;<br>
$$
\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1,
$$
<p>&nbsp;<br>
which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
</section>
<section>
<h2 id="lasso-case">Lasso case </h2>
<p>
For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \)
and \( beta_1 \). This gives us the following derivatives of the cost function
<p>&nbsp;<br>
$$
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
$$
<p>&nbsp;<br>
<p>&nbsp;<br>
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
$$
<p>&nbsp;<br>
and
<p>&nbsp;<br>
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
$$
<p>&nbsp;<br>
We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely
<ol>
<p><li> \( \beta_0 > 0 \) and \( \beta_1 > 0 \),</li>
<p><li> \( \beta_0 > 0 \) and \( \beta_1 < 0 \),</li>
<p><li> \( \beta_0 < 0 \) and \( \beta_1 > 0 \),</li>
<p><li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \),</li>
</ol>
</section>
<section>
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
doc/pub/week36/html/week36-solarized.html
+74 -8
View File
@@ -108,6 +108,11 @@ div { text-align: justify; text-justify: inter-word; }
('Yet another Example', 2, None, 'yet-another-example'),
('The OLS case', 2, None, 'the-ols-case'),
('The Ridge case', 2, None, 'the-ridge-case'),
('Writing the Cost Function',
2,
None,
'writing-the-cost-function'),
('Lasso case', 2, None, 'lasso-case'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -260,7 +265,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 8, 2021</h4></center> <!-- date -->
<center><h4>Sep 9, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -681,7 +686,7 @@ $$
$$
<p>
Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \).
Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).
<p>
We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
@@ -701,7 +706,7 @@ $$
and our inputs as a \( 3\times 2 \) design matrix
$$
\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix},
$$
meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.
@@ -721,12 +726,9 @@ $$
Inserting the above values we obtain that
$$
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}\frac{11}{5} \\ 2\end{bmatrix},
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
$$
<p>
Computing the mean squared error we obtian a value of \( 0.27 \).
<p>
The code which implements this simpler case is presented after the discussion of Ridge and Lasso.
@@ -745,7 +747,7 @@ $$
Inserting the above values we obtain that
$$
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
$$
<p>
@@ -756,6 +758,70 @@ Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint.
To see this, let us write the cost function for Ridge regression.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="writing-the-cost-function">Writing the Cost Function </h2>
<p>
We define the MSE without the \( 1/n \) factor and have then, using that
$$
\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix},
$$
$$
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2),
$$
and taking the derivative with respect to \( \beta_0 \) we get
$$
\beta_0=\frac{8}{4+\lambda},
$$
and for \( \beta_1 \) we have
$$
\beta_1=\frac{2}{1+\lambda},
$$
<p>
Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving
$$
\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1,
$$
which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="lasso-case">Lasso case </h2>
<p>
For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \)
and \( beta_1 \). This gives us the following derivatives of the cost function
$$
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
$$
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
$$
and
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
$$
We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely
<ol>
<li> \( \beta_0 > 0 \) and \( \beta_1 > 0 \),</li>
<li> \( \beta_0 > 0 \) and \( \beta_1 < 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 > 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \),</li>
</ol>
<!-- !split -->
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
doc/pub/week36/html/week36.html
+74 -8
View File
@@ -113,6 +113,11 @@ div { text-align: justify; text-justify: inter-word; }
('Yet another Example', 2, None, 'yet-another-example'),
('The OLS case', 2, None, 'the-ols-case'),
('The Ridge case', 2, None, 'the-ridge-case'),
('Writing the Cost Function',
2,
None,
'writing-the-cost-function'),
('Lasso case', 2, None, 'lasso-case'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -265,7 +270,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 8, 2021</h4></center> <!-- date -->
<center><h4>Sep 9, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -686,7 +691,7 @@ $$
$$
<p>
Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the hand the values of \( \beta_i \) as function of \( \lambda \).
Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of \( \beta_i \) for specific values of \( \lambda \). Ridge regression reduces on the other hand the values of \( \beta_i \) as function of \( \lambda \).
<p>
We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
@@ -706,7 +711,7 @@ $$
and our inputs as a \( 3\times 2 \) design matrix
$$
\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
\boldsymbol{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix},
$$
meaning that we have two features and two unknown parameters \( \beta_0 \) and \( \beta_1 \) to be determined either by ordinary least squares, Ridge or Lasso regression.
@@ -726,12 +731,9 @@ $$
Inserting the above values we obtain that
$$
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}\frac{11}{5} \\ 2\end{bmatrix},
\hat{\boldsymbol{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
$$
<p>
Computing the mean squared error we obtian a value of \( 0.27 \).
<p>
The code which implements this simpler case is presented after the discussion of Ridge and Lasso.
@@ -750,7 +752,7 @@ $$
Inserting the above values we obtain that
$$
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
\hat{\boldsymbol{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
$$
<p>
@@ -761,6 +763,70 @@ Let us for simplicity assume now that \( \beta_0^2+\beta_1^2=1 \) as constraint.
To see this, let us write the cost function for Ridge regression.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="writing-the-cost-function">Writing the Cost Function </h2>
<p>
We define the MSE without the \( 1/n \) factor and have then, using that
$$
\boldsymbol{X}\boldsymbol{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix},
$$
$$
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2),
$$
and taking the derivative with respect to \( \beta_0 \) we get
$$
\beta_0=\frac{8}{4+\lambda},
$$
and for \( \beta_1 \) we have
$$
\beta_1=\frac{2}{1+\lambda},
$$
<p>
Using the constraint for \( \beta_0^2+\beta_1^2=1 \) we can constrain \( \lambda \) by solving
$$
\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1,
$$
which gives \( \lambda=4.571 \) and \( \beta_0=0.933 \) and \( \beta_1=0.359 \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="lasso-case">Lasso case </h2>
<p>
For Lasso we need now, keeping the same constraint on \( \beta_0^2+\beta_1^2=1 \), to take the derivative of the absolute values of \( \beta_0 \)
and \( beta_1 \). This gives us the following derivatives of the cost function
$$
C(\boldsymbol{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
$$
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
$$
and
$$
\frac{\partialC(\boldsymbol{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
$$
We have now four cases to solve besides the trivial cases \( \beta_0 \) and/or \( \beta_1 \) are zero, namely
<ol>
<li> \( \beta_0 > 0 \) and \( \beta_1 > 0 \),</li>
<li> \( \beta_0 > 0 \) and \( \beta_1 < 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 > 0 \),</li>
<li> \( \beta_0 < 0 \) and \( \beta_1 < 0 \),</li>
</ol>
<!-- !split -->
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week36/ipynb/week36.ipynb
+135 -7
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 8, 2021**\n",
"Date: **Sep 9, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -772,7 +772,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the hand the values of $\\beta_i$ as function of $\\lambda$.\n",
"Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of $\\beta_i$ for specific values of $\\lambda$. Ridge regression reduces on the other hand the values of $\\beta_i$ as function of $\\lambda$.\n",
"\n",
"We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.\n",
"We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\\beta$.\n",
@@ -804,7 +804,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 1 & 0\\end{bmatrix},\n",
"\\boldsymbol{X}=\\begin{bmatrix}2 & 0\\\\ 0 & 1 \\\\ 0 & 0\\end{bmatrix},\n",
"$$"
]
},
@@ -840,7 +840,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}\\frac{11}{5} \\\\ 2\\end{bmatrix},\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\begin{bmatrix}2 \\\\ 2\\end{bmatrix},\n",
"$$"
]
},
@@ -848,8 +848,6 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Computing the mean squared error we obtian a value of $0.27$. \n",
"\n",
"The code which implements this simpler case is presented after the discussion of Ridge and Lasso.\n",
"\n",
"## The Ridge case\n",
@@ -878,7 +876,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{11}{5+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n",
"\\hat{\\boldsymbol{\\beta}}^{\\mathrm{Ridge}}=\\begin{bmatrix}\\frac{8}{4+\\lambda} \\\\ \\frac{2}{1+\\lambda}\\end{bmatrix},\n",
"$$"
]
},
@@ -892,6 +890,136 @@
"To see this, let us write the cost function for Ridge regression. \n",
"\n",
"\n",
"## Writing the Cost Function\n",
"\n",
"We define the MSE without the $1/n$ factor and have then, using that"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}\\boldsymbol{\\beta}=\\begin{bmatrix} 2\\beta_0 \\\\ \\beta_1 \\\\0 \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\beta_0^2+\\beta_1^2),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and taking the derivative with respect to $\\beta_0$ we get"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_0=\\frac{8}{4+\\lambda},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and for $\\beta_1$ we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\beta_1=\\frac{2}{1+\\lambda},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the constraint for $\\beta_0^2+\\beta_1^2=1$ we can constrain $\\lambda$ by solving"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\left(\\frac{8}{4+\\lambda}\\right)^2+\\left(\\frac{2}{1+\\lambda}\\right)^2=1,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which gives $\\lambda=4.571$ and $\\beta_0=0.933$ and $\\beta_1=0.359$.\n",
"\n",
"## Lasso case\n",
"\n",
"For Lasso we need now, keeping the same constraint on $\\beta_0^2+\\beta_1^2=1$, to take the derivative of the absolute values of $\\beta_0$\n",
"and $beta_1$. This gives us the following derivatives of the cost function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=(4-2\\beta_0)^2+(2-\\beta_1)^2+\\lambda(\\vert\\beta_0\\vert+\\vert\\beta_1\\vert),\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partialC(\\boldsymbol{\\beta})}{\\partial \\beta_0}=-2(4-2\\beta_0)+\\lambda\\mathrm{sgn}\\beta_0=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partialC(\\boldsymbol{\\beta})}{\\partial \\beta_1}=-2(2-\\beta_1)+\\lambda\\mathrm{sgn}\\beta_1=0.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We have now four cases to solve besides the trivial cases $\\beta_0$ and/or $\\beta_1$ are zero, namely\n",
"1. $\\beta_0 > 0$ and $\\beta_1 > 0$,\n",
"\n",
"2. $\\beta_0 > 0$ and $\\beta_1 < 0$,\n",
"\n",
"3. $\\beta_0 < 0$ and $\\beta_1 > 0$,\n",
"\n",
"4. $\\beta_0 < 0$ and $\\beta_1 < 0$,\n",
"\n",
"<!-- !split -->\n",
"## Linking the regression analysis with a statistical interpretation\n",
"\n",
doc/src/week36/week36.do.txt
+71 -6
View File
@@ -423,7 +423,7 @@ which leads to
\]
!et
Plotting these results (figure to come) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the hand the values of $\beta_i$ as function of $\lambda$.
Plotting these results (figure in handwritten notes for week 36) shows clearly that Lasso regression suppresses (sets to zero) values of $\beta_i$ for specific values of $\lambda$. Ridge regression reduces on the other hand the values of $\beta_i$ as function of $\lambda$.
We will now couple the discussions of ordinary least squares, Ridge and Lasso regression with a statistical interpretation, that is we move from a linear algebra analysis to a statistical analysis. In particular, we will focus on what the regularization terms can result in.
We will amongst other things show that the regularization parameter can reduce considerably the variance of the parameters $\beta$.
@@ -442,7 +442,7 @@ Let us assume we have a data set with outputs/targets given by the vector
and our inputs as a $3\times 2$ design matrix
!bt
\[
\bm{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 1 & 0\end{bmatrix},
\bm{X}=\begin{bmatrix}2 & 0\\ 0 & 1 \\ 0 & 0\end{bmatrix},
\]
!et
meaning that we have two features and two unknown parameters $\beta_0$ and $\beta_1$ to be determined either by ordinary least squares, Ridge or Lasso regression.
@@ -461,12 +461,10 @@ Inserting the above values we obtain that
!bt
\[
\hat{\bm{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}\frac{11}{5} \\ 2\end{bmatrix},
\hat{\bm{\beta}}^{\mathrm{OLS}}=\begin{bmatrix}2 \\ 2\end{bmatrix},
\]
!et
Computing the mean squared error we obtian a value of $0.27$.
The code which implements this simpler case is presented after the discussion of Ridge and Lasso.
!split
@@ -483,7 +481,7 @@ Inserting the above values we obtain that
!bt
\[
\hat{\bm{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{11}{5+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
\hat{\bm{\beta}}^{\mathrm{Ridge}}=\begin{bmatrix}\frac{8}{4+\lambda} \\ \frac{2}{1+\lambda}\end{bmatrix},
\]
!et
@@ -493,6 +491,73 @@ Let us for simplicity assume now that $\beta_0^2+\beta_1^2=1$ as constraint. Thi
To see this, let us write the cost function for Ridge regression.
!split
===== Writing the Cost Function =====
We define the MSE without the $1/n$ factor and have then, using that
!bt
\[
\bm{X}\bm{\beta}=\begin{bmatrix} 2\beta_0 \\ \beta_1 \\0 \end{bmatrix},
\]
!et
!bt
\[
C(\bm{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\beta_0^2+\beta_1^2),
\]
!et
and taking the derivative with respect to $\beta_0$ we get
!bt
\[
\beta_0=\frac{8}{4+\lambda},
\]
!et
and for $\beta_1$ we have
!bt
\[
\beta_1=\frac{2}{1+\lambda},
\]
!et
Using the constraint for $\beta_0^2+\beta_1^2=1$ we can constrain $\lambda$ by solving
!bt
\[
\left(\frac{8}{4+\lambda}\right)^2+\left(\frac{2}{1+\lambda}\right)^2=1,
\]
!et
which gives $\lambda=4.571$ and $\beta_0=0.933$ and $\beta_1=0.359$.
!split
===== Lasso case =====
For Lasso we need now, keeping the same constraint on $\beta_0^2+\beta_1^2=1$, to take the derivative of the absolute values of $\beta_0$
and $beta_1$. This gives us the following derivatives of the cost function
!bt
\[
C(\bm{\beta})=(4-2\beta_0)^2+(2-\beta_1)^2+\lambda(\vert\beta_0\vert+\vert\beta_1\vert),
\]
!et
!bt
\[
\frac{\partialC(\bm{\beta})}{\partial \beta_0}=-2(4-2\beta_0)+\lambda\mathrm{sgn}\beta_0=0,
\]
!et
and
!bt
\[
\frac{\partialC(\bm{\beta})}{\partial \beta_1}=-2(2-\beta_1)+\lambda\mathrm{sgn}\beta_1=0.
\]
!et
We have now four cases to solve besides the trivial cases $\beta_0$ and/or $\beta_1$ are zero, namely
o $\beta_0 > 0$ and $\beta_1 > 0$,
o $\beta_0 > 0$ and $\beta_1 < 0$,
o $\beta_0 < 0$ and $\beta_1 > 0$,
o $\beta_0 < 0$ and $\beta_1 < 0$,
!split
===== Linking the regression analysis with a statistical interpretation =====