lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

final update of week 35

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-06 21:58:48 +02:00
parent 6dfcb94649
commit 8ba5a0e89b
7 changed files with 233 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+9 -4
View File
@@ -248,6 +248,10 @@ Automatically generated HTML file from DocOnce source
None,
'interpreting-the-ridge-results'),
('More interpretations', 2, None, 'more-interpretations'),
('Deriving the Lasso Regression Equations',
2,
None,
'deriving-the-lasso-regression-equations'),
('Exercises for week 36, September 6-10',
2,
None,
@@ -364,9 +368,10 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs065.html#deriving-the-ridge-regression-equations" style="font-size: 80%;"><b>Deriving the Ridge Regression Equations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#deriving-the-lasso-regression-equations" style="font-size: 80%;"><b>Deriving the Lasso Regression Equations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -425,7 +430,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs068.html">69</a></li>
<li><a href="._week35-bs069.html">70</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+43 -4
View File
@@ -2899,16 +2899,16 @@ $$
<h2 id="deriving-the-ridge-regression-equations">Deriving the Ridge Regression Equations </h2>
<p>
Using the matrix-vector expression for Ridge regression,
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
<p>&nbsp;<br>
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
$$
<p>&nbsp;<br>
<p>
by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
and
taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
@@ -3039,6 +3039,45 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
</section>
<section>
<h2 id="deriving-the-lasso-regression-equations">Deriving the Lasso Regression Equations </h2>
<p>
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following <b>cost</b> function
<p>&nbsp;<br>
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
$$
<p>&nbsp;<br>
<p>
Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
<p>&nbsp;<br>
$$
\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right.
$$
<p>&nbsp;<br>
we have that the derivative of the cost function is
<p>&nbsp;<br>
$$
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
$$
<p>&nbsp;<br>
and reordering we have
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
$$
<p>&nbsp;<br>
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
</section>
<section>
<h2 id="exercises-for-week-36-september-6-10">Exercises for week 36, September 6-10 </h2>
doc/pub/week35/html/week35-solarized.html
+39 -4
View File
@@ -268,6 +268,10 @@ div { text-align: justify; text-justify: inter-word; }
None,
'interpreting-the-ridge-results'),
('More interpretations', 2, None, 'more-interpretations'),
('Deriving the Lasso Regression Equations',
2,
None,
'deriving-the-lasso-regression-equations'),
('Exercises for week 36, September 6-10',
2,
None,
@@ -2863,14 +2867,14 @@ $$
<h2 id="deriving-the-ridge-regression-equations">Deriving the Ridge Regression Equations </h2>
<p>
Using the matrix-vector expression for Ridge regression,
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
$$
<p>
by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
and
taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
@@ -2984,6 +2988,37 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="deriving-the-lasso-regression-equations">Deriving the Lasso Regression Equations </h2>
<p>
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following <b>cost</b> function
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
$$
<p>
Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
$$
\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right.
$$
we have that the derivative of the cost function is
$$
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
$$
and reordering we have
$$
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="exercises-for-week-36-september-6-10">Exercises for week 36, September 6-10 </h2>
<p>
doc/pub/week35/html/week35.html
+39 -4
View File
@@ -273,6 +273,10 @@ div { text-align: justify; text-justify: inter-word; }
None,
'interpreting-the-ridge-results'),
('More interpretations', 2, None, 'more-interpretations'),
('Deriving the Lasso Regression Equations',
2,
None,
'deriving-the-lasso-regression-equations'),
('Exercises for week 36, September 6-10',
2,
None,
@@ -2868,14 +2872,14 @@ $$
<h2 id="deriving-the-ridge-regression-equations">Deriving the Ridge Regression Equations </h2>
<p>
Using the matrix-vector expression for Ridge regression,
Using the matrix-vector expression for Ridge regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\frac{1}{n}\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\boldsymbol{\beta}^T\boldsymbol{\beta},
$$
<p>
by taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
and
taking the derivatives with respect to \( \boldsymbol{\beta} \) we obtain then
a slightly modified matrix inversion problem which for finite values
of \( \lambda \) does not suffer from singularity problems. We obtain
the optimal parameters
@@ -2989,6 +2993,37 @@ Similarly, <a href="https://arxiv.org/abs/1803.08823" target="_blank">Mehta et a
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="deriving-the-lasso-regression-equations">Deriving the Lasso Regression Equations </h2>
<p>
Using the matrix-vector expression for Lasso regression and dropping the parameter \( 1/n \) in front of the standard means squared error equation, we have the following <b>cost</b> function
$$
C(\boldsymbol{X},\boldsymbol{\beta})=\left\{(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})\right\}+\lambda\vert\vert\boldsymbol{\beta}\vert\vert_1,
$$
<p>
Taking the derivative with respect to \( \boldsymbol{\beta} \) and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
$$
\frac{d \vert \beta\vert}{d \boldsymbol{\beta}}=\mathrm{sgn}(\boldsymbol{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right.
$$
we have that the derivative of the cost function is
$$
\frac{\partial C(\boldsymbol{X},\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-2\boldsymbol{X}^T(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=0,
$$
and reordering we have
$$
\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package <a href="https://cvxopt.org/" target="_blank">CVXOPT</a>. We will discuss this later.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="exercises-for-week-36-september-6-10">Exercises for week 36, September 6-10 </h2>
<p>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+70 -4
View File
@@ -3790,7 +3790,7 @@
"source": [
"## Deriving the Ridge Regression Equations\n",
"\n",
"Using the matrix-vector expression for Ridge regression,"
"Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have"
]
},
{
@@ -3798,7 +3798,7 @@
"metadata": {},
"source": [
"$$\n",
"C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n",
"C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\boldsymbol{\\beta}^T\\boldsymbol{\\beta},\n",
"$$"
]
},
@@ -3806,7 +3806,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"by taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n",
"and \n",
"taking the derivatives with respect to $\\boldsymbol{\\beta}$ we obtain then\n",
"a slightly modified matrix inversion problem which for finite values\n",
"of $\\lambda$ does not suffer from singularity problems. We obtain\n",
"the optimal parameters"
@@ -3991,8 +3992,73 @@
"For more discussions of Ridge and Lasso regression, [Wessel van Wieringen's](https://arxiv.org/abs/1509.09169) article is highly recommended.\n",
"Similarly, [Mehta et al's article](https://arxiv.org/abs/1803.08823) is also recommended.\n",
"\n",
"## Deriving the Lasso Regression Equations\n",
"\n",
"\n",
"Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following **cost** function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\left\\{(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\right\\}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Taking the derivative with respect to $\\boldsymbol{\\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{d \\vert \\beta\\vert}{d \\boldsymbol{\\beta}}=\\mathrm{sgn}(\\boldsymbol{\\beta})=\\left\\{\\begin{array}{cc} 1 & \\beta > 0 \\\\ 0 & \\beta =0\\\\-1 & \\beta < 0, \\end{array}\\right.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"we have that the derivative of the cost function is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial C(\\boldsymbol{X},\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}=-2\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=0,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and reordering we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T(\\boldsymbol{y}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package [CVXOPT](https://cvxopt.org/). We will discuss this later. \n",
"\n",
"## Exercises for week 36, September 6-10\n",
"\n",
doc/src/week35/week35.do.txt
+33 -4
View File
@@ -2357,15 +2357,15 @@ Here we have defined the norm-1 as
!split
===== Deriving the Ridge Regression Equations =====
Using the matrix-vector expression for Ridge regression,
Using the matrix-vector expression for Ridge regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have
!bt
\[
C(\bm{X},\bm{\beta})=\frac{1}{n}\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\bm{\beta}^T\bm{\beta},
\]
!et
by taking the derivatives with respect to $\bm{\beta}$ we obtain then
and
taking the derivatives with respect to $\bm{\beta}$ we obtain then
a slightly modified matrix inversion problem which for finite values
of $\lambda$ does not suffer from singularity problems. We obtain
the optimal parameters
@@ -2476,8 +2476,37 @@ We will come back to more interpreations after we have gone through some of the
For more discussions of Ridge and Lasso regression, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
Similarly, "Mehta et al's article":"https://arxiv.org/abs/1803.08823" is also recommended.
!split
===== Deriving the Lasso Regression Equations =====
Using the matrix-vector expression for Lasso regression and dropping the parameter $1/n$ in front of the standard means squared error equation, we have the following _cost_ function
!bt
\[
C(\bm{X},\bm{\beta})=\left\{(\bm{y}-\bm{X}\bm{\beta})^T(\bm{y}-\bm{X}\bm{\beta})\right\}+\lambda\vert\vert\bm{\beta}\vert\vert_1,
\]
!et
Taking the derivative with respect to $\bm{\beta}$ and recalling that the derivative of the absolute value is (we drop the boldfaced vector symbol for simplicty)
!bt
\[
\frac{d \vert \beta\vert}{d \bm{\beta}}=\mathrm{sgn}(\bm{\beta})=\left\{\begin{array}{cc} 1 & \beta > 0 \\ 0 & \beta =0\\-1 & \beta < 0, \end{array}\right.
\]
!et
we have that the derivative of the cost function is
!bt
\[
\frac{\partial C(\bm{X},\bm{\beta})}{\partial \bm{\beta}}=-2\bm{X}^T(\bm{y}-\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=0,
\]
!et
and reordering we have
!bt
\[
\bm{X}^T\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=2\bm{X}^T(\bm{y}.
\]
!et
This equation does not lead to a nice analytical equation as in either Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later.
!split
===== Exercises for week 36, September 6-10 =====