-
@@ -319,7 +329,7 @@ MathJax.Hub.Config({
-
@@ -179,10 +179,10 @@ MathJax.Hub.Config({
-We saw earlier that
+We used the SVD to analyse the matrix to invert in ordinary lineat regression
By minimizing the above equation with respect to the parameters
@@ -537,7 +542,7 @@ Similarly, Mehta et a
-Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function
+Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function
+For Ridge regression our cost function is
+
-
-We saw earlier that
+We used the SVD to analyse the matrix to invert in ordinary lineat regression
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
@@ -403,6 +411,11 @@ $$
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
$$
+
+
By minimizing the above equation with respect to the parameters
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
@@ -566,7 +579,7 @@ Similarly, Mehta et a
-Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function
+Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
@@ -589,7 +602,7 @@ $$
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
$$
-This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+
+For Ridge regression our cost function is
+$$
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+$$
+
+and minimizing we have that
+
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html
index 358a9abf0..dde24d44d 100644
--- a/doc/pub/week36/html/week36.html
+++ b/doc/pub/week36/html/week36.html
@@ -72,7 +72,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'summary-from-last-week-and-examples'),
- ('Linking with the SVD', 2, None, 'linking-with-the-svd'),
+ ('Linear Regression and the SVD',
+ 2,
+ None,
+ 'linear-regression-and-the-svd'),
('What does it mean?', 2, None, 'what-does-it-mean'),
('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$',
2,
@@ -82,6 +85,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'ridge-and-lasso-regression'),
+ ('From OLS to Ridge and Lasso',
+ 2,
+ None,
+ 'from-ols-to-ridge-and-lasso'),
('Deriving the Ridge Regression Equations',
2,
None,
@@ -101,6 +108,7 @@ div { text-align: justify; text-justify: inter-word; }
None,
'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
('Ridge Regression', 2, None, 'ridge-regression'),
+ ('Lasso Rgeression', 2, None, 'lasso-rgeression'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -245,7 +253,7 @@ MathJax.Hub.Config({
-
-We saw earlier that
+We used the SVD to analyse the matrix to invert in ordinary lineat regression
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
@@ -408,6 +416,11 @@ $$
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
$$
+
+
By minimizing the above equation with respect to the parameters
\( \boldsymbol{\beta} \) we could then obtain an analytical expression for the
@@ -571,7 +584,7 @@ Similarly, Mehta et a
-Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following cost function
+Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard mean squared error equation, we have the following cost function
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
@@ -594,7 +607,7 @@ $$
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
$$
-This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+
+For Ridge regression our cost function is
+$$
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+$$
+
+and minimizing we have that
+
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
index 8d24c23d6..1a865aee4 100644
Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ
diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb
index a2a7ec0b0..96657f4bc 100644
--- a/doc/pub/week36/ipynb/week36.ipynb
+++ b/doc/pub/week36/ipynb/week36.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Sep 7, 2021**\n",
+ "Date: **Sep 8, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -28,9 +28,9 @@
"\n",
"## Summary from last Week and Examples\n",
"\n",
- "## Linking with the SVD\n",
+ "## Linear Regression and the SVD\n",
"\n",
- "We saw earlier that"
+ "We used the SVD to analyse the matrix to invert in ordinary lineat regression"
]
},
{
@@ -299,6 +299,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "## From OLS to Ridge and Lasso\n",
+ "\n",
"By minimizing the above equation with respect to the parameters\n",
"$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n",
"parameters $\\boldsymbol{\\beta}$. We can add a regularization parameter $\\lambda$ by\n",
@@ -576,7 +578,7 @@
"\n",
"## Deriving the Lasso Regression Equations\n",
"\n",
- "Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following **cost** function"
+ "Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard mean squared error equation, we have the following **cost** function"
]
},
{
@@ -640,7 +642,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later. \n",
+ "This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later. \n",
"\n",
"\n",
"\n",
@@ -716,6 +718,27 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "## Lasso Rgeression\n",
+ "\n",
+ "For Ridge regression our cost function is"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "C(\\boldsymbol{\\beta})=\\sum_{i=0}^{p-1}(y_i-\\beta_i)^2+\\lambda\\sum_{i=0}^{p-1}\\beta_i^2,,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and minimizing we have that\n",
+ "\n",
+ "\n",
"\n",
"## Linking the regression analysis with a statistical interpretation\n",
"\n",
diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt
index 02d6bbf19..fd9b5cbd3 100644
--- a/doc/src/week36/week36.do.txt
+++ b/doc/src/week36/week36.do.txt
@@ -18,9 +18,9 @@ DATE: today
===== Summary from last Week and Examples =====
!split
-===== Linking with the SVD =====
+===== Linear Regression and the SVD =====
-We saw earlier that
+We used the SVD to analyse the matrix to invert in ordinary lineat regression
!bt
\[
\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T.
@@ -163,6 +163,9 @@ where we have used the definition of a norm-2 vector, that is
\]
!et
+!split
+===== From OLS to Ridge and Lasso =====
+
By minimizing the above equation with respect to the parameters
$\bm{\beta}$ we could then obtain an analytical expression for the
parameters $\bm{\beta}$. We can add a regularization parameter $\lambda$ by
@@ -325,7 +328,7 @@ Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also re
!split
===== Deriving the Lasso Regression Equations =====
-Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following _cost_ function
+Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard mean squared error equation, we have the following _cost_ function
!bt
\[
@@ -352,7 +355,7 @@ and reordering we have
\bm{X}^T\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=2\bm{X}^T(\bm{y}.
\]
!et
-This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later.
+This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later.
@@ -393,6 +396,16 @@ and minimizing we have that
!et
+!split
+===== Lasso Rgeression =====
+
+For Ridge regression our cost function is
+!bt
+\[
+C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+\]
+!et
+and minimizing we have that
!split
Linking with the SVD
+Linear Regression and the SVD
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
@@ -345,6 +345,11 @@ $$
\vert\vert \boldsymbol{x}\vert\vert_2 = \sqrt{\sum_i x_i^2}.
$$
+From OLS to Ridge and Lasso
Deriving the Lasso Regression Equations
$$
@@ -568,7 +573,7 @@ $$
$$
-This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
+This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
Lasso Rgeression
+
+
+$$
+C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2+\lambda\sum_{i=0}^{p-1}\beta_i^2,,
+$$
+
+
+and minimizing we have that
+Linking the regression analysis with a statistical interpretation
diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html
index 836b7a150..6deff0044 100644
--- a/doc/pub/week36/html/week36-solarized.html
+++ b/doc/pub/week36/html/week36-solarized.html
@@ -67,7 +67,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'summary-from-last-week-and-examples'),
- ('Linking with the SVD', 2, None, 'linking-with-the-svd'),
+ ('Linear Regression and the SVD',
+ 2,
+ None,
+ 'linear-regression-and-the-svd'),
('What does it mean?', 2, None, 'what-does-it-mean'),
('And finally $\\boldsymbol{X}\\boldsymbol{X}^T$',
2,
@@ -77,6 +80,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'ridge-and-lasso-regression'),
+ ('From OLS to Ridge and Lasso',
+ 2,
+ None,
+ 'from-ols-to-ridge-and-lasso'),
('Deriving the Ridge Regression Equations',
2,
None,
@@ -96,6 +103,7 @@ div { text-align: justify; text-justify: inter-word; }
None,
'simple-example-to-illustrate-ordinary-least-squares-ridge-and-lasso-regression'),
('Ridge Regression', 2, None, 'ridge-regression'),
+ ('Lasso Rgeression', 2, None, 'lasso-rgeression'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
@@ -240,7 +248,7 @@ MathJax.Hub.Config({
Sep 7, 2021
Sep 8, 2021
@@ -264,10 +272,10 @@ MathJax.Hub.Config({
-Linking with the SVD
+Linear Regression and the SVD
+
+From OLS to Ridge and Lasso
+
Deriving the Lasso Regression Equations
@@ -627,6 +640,19 @@ $$
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \frac{\boldsymbol{y}}{1+\lambda}.
$$
+
+
+Lasso Rgeression
+
+
Sep 7, 2021
Sep 8, 2021
@@ -269,10 +277,10 @@ MathJax.Hub.Config({
-Linking with the SVD
+Linear Regression and the SVD
+
+From OLS to Ridge and Lasso
+
Deriving the Lasso Regression Equations
@@ -632,6 +645,19 @@ $$
\hat{\boldsymbol{\beta}}_{\mathrm{Ridge}} = \frac{\boldsymbol{y}}{1+\lambda}.
$$
+
+
+Lasso Rgeression
+
+