lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week 36

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-08 05:53:54 +02:00
parent 24d9042ec7
commit 5d36d12c1d
7 changed files with 186 additions and 68 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week36/html/week36-bs.html
+52 -42
View File
@@ -47,7 +47,10 @@ Automatically generated HTML file from DocOnce source
2,
None,
'summary-from-last-week-and-examples'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('Linear Regression and the SVD',
2,
None,
'linear-regression-and-the-svd'),
('What does it mean?', 2, None, 'what-does-it-mean'),
('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$',
2,
@@ -57,6 +60,10 @@ Automatically generated HTML file from DocOnce source
2,
None,
'ridge-and-lasso-regression'),
('From OLS to Ridge and Lasso',
2,
None,
'from-ols-to-ridge-and-lasso'),
('Deriving the Ridge Regression Equations',
2,
None,
@@ -76,6 +83,7 @@ Automatically generated HTML file from DocOnce source
None,
'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
('Ridge Regression', 2, None, 'ridge-regression'),
('Lasso Rgeression', 2, None, 'lasso-rgeression'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -219,48 +227,50 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week36-bs001.html#plans-for-week-36" style="font-size: 80%;">Plans for week 36</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs002.html#thursday-september-9" style="font-size: 80%;">Thursday September 9</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs003.html#summary-from-last-week-and-examples" style="font-size: 80%;">Summary from last Week and Examples</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs004.html#linking-with-the-svd" style="font-size: 80%;">Linking with the SVD</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs004.html#linear-regression-and-the-svd" style="font-size: 80%;">Linear Regression and the SVD</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs005.html#what-does-it-mean" style="font-size: 80%;">What does it mean?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs006.html#and-finally-boldsymbol-x-boldsymbol-x-t" style="font-size: 80%;">And finally \( \boldsymbol{X}\boldsymbol{X}^T \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs007.html#ridge-and-lasso-regression" style="font-size: 80%;">Ridge and LASSO Regression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs008.html#deriving-the-ridge-regression-equations" style="font-size: 80%;">Deriving the Ridge Regression Equations</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs009.html#interpreting-the-ridge-results" style="font-size: 80%;">Interpreting the Ridge results</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs010.html#more-interpretations" style="font-size: 80%;">More interpretations</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs011.html#deriving-the-lasso-regression-equations" style="font-size: 80%;">Deriving the Lasso Regression Equations</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs012.html#simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression" style="font-size: 80%;">Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs013.html#ridge-regression" style="font-size: 80%;">Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs014.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs015.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs016.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs017.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs018.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs021.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs008.html#from-ols-to-ridge-and-lasso" style="font-size: 80%;">From OLS to Ridge and Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs009.html#deriving-the-ridge-regression-equations" style="font-size: 80%;">Deriving the Ridge Regression Equations</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs010.html#interpreting-the-ridge-results" style="font-size: 80%;">Interpreting the Ridge results</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs011.html#more-interpretations" style="font-size: 80%;">More interpretations</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs012.html#deriving-the-lasso-regression-equations" style="font-size: 80%;">Deriving the Lasso Regression Equations</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs013.html#simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression" style="font-size: 80%;">Simple example to illustrate Ordinary Least Squares, Ridge and Lasso Regression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs014.html#ridge-regression" style="font-size: 80%;">Ridge Regression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs015.html#lasso-rgeression" style="font-size: 80%;">Lasso Rgeression</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs016.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs017.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs018.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs019.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs020.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs035.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs036.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs037.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs038.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs039.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs040.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs041.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs042.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs043.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs044.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs045.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs046.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs047.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
</ul>
</li>
@@ -295,7 +305,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 7, 2021</h4></center> <!-- date -->
<center><h4>Sep 8, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -319,7 +329,7 @@ MathJax.Hub.Config({
<li><a href="._week36-bs008.html">9</a></li>
<li><a href="._week36-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week36-bs045.html">46</a></li>
<li><a href="._week36-bs047.html">48</a></li>
<li><a href="._week36-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week36/html/week36-reveal.html
+25 -5
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 7, 2021</h4></center> <!-- date -->
<center><h4>Sep 8, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -179,10 +179,10 @@ MathJax.Hub.Config({
<section>
<h2 id="linking-with-the-svd">Linking with the SVD </h2>
<h2 id="linear-regression-and-the-svd">Linear Regression and the SVD </h2>
<p>
We saw earlier that
We used the SVD to analyse the matrix to invert in ordinary lineat regression
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
@@ -345,6 +345,11 @@ $$
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="from-ols-to-ridge-and-lasso">From OLS to Ridge and Lasso </h2>
<p>
By minimizing the above equation with respect to the parameters
@@ -537,7 +542,7 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
<h2 id="deriving-the-lasso-regression-equations">Deriving the Lasso Regression Equations </h2>
<p>
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following <b>cost</b> function
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following <b>cost</b> function
<p>&nbsp;<br>
$$
@@ -568,7 +573,7 @@ $$
$$
<p>&nbsp;<br>
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
</section>
@@ -616,6 +621,21 @@ $$
</section>
<section>
<h2 id="lasso-rgeression">Lasso Rgeression </h2>
<p>
For Ridge regression our cost function is
<p>&nbsp;<br>
$$
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
$$
<p>&nbsp;<br>
and minimizing we have that
</section>
<section>
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
doc/pub/week36/html/week36-solarized.html
+32 -6
View File
@@ -67,7 +67,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'summary-from-last-week-and-examples'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('Linear Regression and the SVD',
2,
None,
'linear-regression-and-the-svd'),
('What does it mean?', 2, None, 'what-does-it-mean'),
('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$',
2,
@@ -77,6 +80,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'ridge-and-lasso-regression'),
('From OLS to Ridge and Lasso',
2,
None,
'from-ols-to-ridge-and-lasso'),
('Deriving the Ridge Regression Equations',
2,
None,
@@ -96,6 +103,7 @@ div { text-align: justify; text-justify: inter-word; }
None,
'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
('Ridge Regression', 2, None, 'ridge-regression'),
('Lasso Rgeression', 2, None, 'lasso-rgeression'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -240,7 +248,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 7, 2021</h4></center> <!-- date -->
<center><h4>Sep 8, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -264,10 +272,10 @@ MathJax.Hub.Config({
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="linking-with-the-svd">Linking with the SVD </h2>
<h2 id="linear-regression-and-the-svd">Linear Regression and the SVD </h2>
<p>
We saw earlier that
We used the SVD to analyse the matrix to invert in ordinary lineat regression
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
@@ -403,6 +411,11 @@ $$
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="from-ols-to-ridge-and-lasso">From OLS to Ridge and Lasso </h2>
<p>
By minimizing the above equation with respect to the parameters
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
@@ -566,7 +579,7 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
<h2 id="deriving-the-lasso-regression-equations">Deriving the Lasso Regression Equations </h2>
<p>
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following <b>cost</b> function
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following <b>cost</b> function
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
@@ -589,7 +602,7 @@ $$
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -627,6 +640,19 @@ $$
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \frac{\boldsymbol{y}}{1+\lambda}.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="lasso-rgeression">Lasso Rgeression </h2>
<p>
For Ridge regression our cost function is
$$
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
$$
and minimizing we have that
<p>
<!-- !split -->
doc/pub/week36/html/week36.html
+32 -6
View File
@@ -72,7 +72,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'summary-from-last-week-and-examples'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('Linear Regression and the SVD',
2,
None,
'linear-regression-and-the-svd'),
('What does it mean?', 2, None, 'what-does-it-mean'),
('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$',
2,
@@ -82,6 +85,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'ridge-and-lasso-regression'),
('From OLS to Ridge and Lasso',
2,
None,
'from-ols-to-ridge-and-lasso'),
('Deriving the Ridge Regression Equations',
2,
None,
@@ -101,6 +108,7 @@ div { text-align: justify; text-justify: inter-word; }
None,
'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
('Ridge Regression', 2, None, 'ridge-regression'),
('Lasso Rgeression', 2, None, 'lasso-rgeression'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -245,7 +253,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 7, 2021</h4></center> <!-- date -->
<center><h4>Sep 8, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -269,10 +277,10 @@ MathJax.Hub.Config({
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="linking-with-the-svd">Linking with the SVD </h2>
<h2 id="linear-regression-and-the-svd">Linear Regression and the SVD </h2>
<p>
We saw earlier that
We used the SVD to analyse the matrix to invert in ordinary lineat regression
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
@@ -408,6 +416,11 @@ $$
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="from-ols-to-ridge-and-lasso">From OLS to Ridge and Lasso </h2>
<p>
By minimizing the above equation with respect to the parameters
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
@@ -571,7 +584,7 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
<h2 id="deriving-the-lasso-regression-equations">Deriving the Lasso Regression Equations </h2>
<p>
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following <b>cost</b> function
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following <b>cost</b> function
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
@@ -594,7 +607,7 @@ $$
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -632,6 +645,19 @@ $$
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \frac{\boldsymbol{y}}{1+\lambda}.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="lasso-rgeression">Lasso Rgeression </h2>
<p>
For Ridge regression our cost function is
$$
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
$$
and minimizing we have that
<p>
<!-- !split -->
doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week36/ipynb/week36.ipynb
+28 -5
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 7, 2021**\n",
"Date: **Sep 8, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -28,9 +28,9 @@
"\n",
"## Summary from last Week and Examples\n",
"\n",
"## Linking with the SVD\n",
"## Linear Regression and the SVD\n",
"\n",
"We saw earlier that"
"We used the SVD to analyse the matrix to invert in ordinary lineat regression"
]
},
{
@@ -299,6 +299,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## From OLS to Ridge and Lasso\n",
"\n",
"By minimizing the above equation with respect to the parameters\n",
"$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n",
"parameters $\\boldsymbol{\\beta}$. We can add a regularization parameter $\\lambda$ by\n",
@@ -576,7 +578,7 @@
"\n",
"## Deriving the Lasso Regression Equations\n",
"\n",
"Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following **cost** function"
"Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard mean squared error equation, we have the following **cost** function"
]
},
{
@@ -640,7 +642,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later. \n",
"This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later. \n",
"\n",
"\n",
"\n",
@@ -716,6 +718,27 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Lasso Rgeression\n",
"\n",
"For Ridge regression our cost function is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and minimizing we have that\n",
"\n",
"\n",
"<!-- !split -->\n",
"## Linking the regression analysis with a statistical interpretation\n",
"\n",
doc/src/week36/week36.do.txt
+17 -4
View File
@@ -18,9 +18,9 @@ DATE: today
===== Summary from last Week and Examples =====
!split
===== Linking with the SVD =====
===== Linear Regression and the SVD =====
We saw earlier that
We used the SVD to analyse the matrix to invert in ordinary lineat regression
!bt
\[
\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T.
@@ -163,6 +163,9 @@ where we have used the definition of a norm-2 vector, that is
\]
!et
!split
===== From OLS to Ridge and Lasso =====
By minimizing the above equation with respect to the parameters
$\bm{\beta}$ we could then obtain an analytical expression for the
parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by
@@ -325,7 +328,7 @@ Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also re
!split
===== Deriving the Lasso Regression Equations =====
Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following _cost_ function
Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard mean squared error equation, we have the following _cost_ function
!bt
\[
@@ -352,7 +355,7 @@ and reordering we have
\bm{X}^T\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=2\bm{X}^T(\bm{y}.
\]
!et
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later.
This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later.
@@ -393,6 +396,16 @@ and minimizing we have that
!et
!split
===== Lasso Rgeression =====
For Ridge regression our cost function is
!bt
\[
C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
\]
!et
and minimizing we have that
!split