lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on week 35 lectures

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-06 09:18:05 +02:00
parent 81e322f510
commit 30ebf09753
7 changed files with 359 additions and 95 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+27 -22
View File
@@ -192,10 +192,14 @@ Automatically generated HTML file from DocOnce source
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Further properties (important gems for our analysis)',
('Further properties (important for our analyses later)',
2,
None,
'further-properties-important-gems-for-our-analysis'),
'further-properties-important-for-our-analyses-later'),
('Meet the Covariance Matrix',
2,
None,
'meet-the-covariance-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -339,24 +343,25 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs048.html#matheamtics-of-the-svd-and-implications" style="font-size: 80%;"><b>Matheamtics of the SVD and implications</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs049.html#example-matrix" style="font-size: 80%;"><b>Example Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs050.html#setting-up-the-matrix-to-be-inverted" style="font-size: 80%;"><b>Setting up the Matrix to be inverted</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#further-properties-important-gems-for-our-analysis" style="font-size: 80%;"><b>Further properties (important gems for our analysis)</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#further-properties-important-for-our-analyses-later" style="font-size: 80%;"><b>Further properties (important for our analyses later)</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#meet-the-covariance-matrix" style="font-size: 80%;"><b>Meet the Covariance Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -391,7 +396,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 5, 2021</h4></center> <!-- date -->
<center><h4>Sep 6, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -415,7 +420,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs066.html">67</a></li>
<li><a href="._week35-bs067.html">68</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+63 -14
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 5, 2021</h4></center> <!-- date -->
<center><h4>Sep 6, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -768,12 +768,15 @@ $$
<h2 id="meet-the-hessian-matrix">Meet the Hessian Matrix </h2>
<p>
A very important matrix we will meet again and again in Machine Learning is the Hessian.
It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
A very important matrix we will meet again and again in Machine
Learning is the Hessian. It is given by the second derivative of the
cost function with respect to the parameter \( \beta \). Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the cost function is,
<p>&nbsp;<br>
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>&nbsp;<br>
@@ -787,8 +790,13 @@ $$
<p>&nbsp;<br>
<p>
For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is
(beside \( 1/n \)) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning
problem.
<p>
<b>Linear algebra question:</b> Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
@@ -2001,11 +2009,11 @@ Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\t
$$
\boldsymbol{X}=
\begin{bmatrix}
x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
\dots& \dots &\dots & \dots & \dots &\dots\\
x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
\end{bmatrix}
$$
<p>&nbsp;<br>
@@ -2083,7 +2091,7 @@ contains only the singular values. Note also (and we will use this below) that
<p>&nbsp;<br>
$$
\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
@@ -2094,7 +2102,7 @@ $$
which is a \( 2\times 2 \) matrix while
<p>&nbsp;<br>
$$
\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
@@ -2182,7 +2190,7 @@ It means that the ordinary least square model (with the optimal parameters) \( \
<section>
<h2 id="further-properties-important-gems-for-our-analysis">Further properties (important gems for our analysis) </h2>
<h2 id="further-properties-important-for-our-analyses-later">Further properties (important for our analyses later) </h2>
<p>
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
@@ -2231,6 +2239,47 @@ $$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2.
$$
<p>&nbsp;<br>
<p>
<b>Important note</b>: we have defined our design matrix \( \boldsymbol{X} \) to be an
\( n\times p \) matrix. In most supervised learning cases we have that \( n
\ge p \), and quite often we have \( n >> p \). The number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix.
</section>
<section>
<h2 id="meet-the-covariance-matrix">Meet the Covariance Matrix </h2>
<p>
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
<p>
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have
<p>&nbsp;<br>
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>&nbsp;<br>
<p>
The Hessian matrix plays an important role and is defined here as
<p>&nbsp;<br>
$$
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>&nbsp;<br>
<p>
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values.
</section>
doc/pub/week35/html/week35-solarized.html
+65 -16
View File
@@ -212,10 +212,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Further properties (important gems for our analysis)',
('Further properties (important for our analyses later)',
2,
None,
'further-properties-important-gems-for-our-analysis'),
'further-properties-important-for-our-analyses-later'),
('Meet the Covariance Matrix',
2,
None,
'meet-the-covariance-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -313,7 +317,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 5, 2021</h4></center> <!-- date -->
<center><h4>Sep 6, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -884,11 +888,14 @@ $$
<h2 id="meet-the-hessian-matrix">Meet the Hessian Matrix </h2>
<p>
A very important matrix we will meet again and again in Machine Learning is the Hessian.
It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
A very important matrix we will meet again and again in Machine
Learning is the Hessian. It is given by the second derivative of the
cost function with respect to the parameter \( \beta \). Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the cost function is,
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
@@ -899,8 +906,13 @@ $$
$$
<p>
For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is
(beside \( 1/n \)) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning
problem.
<p>
<b>Linear algebra question:</b> Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
@@ -2073,11 +2085,11 @@ Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\t
$$
\boldsymbol{X}=
\begin{bmatrix}
x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
\dots& \dots &\dots & \dots & \dots &\dots\\
x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
\end{bmatrix}
$$
@@ -2143,7 +2155,7 @@ $$
contains only the singular values. Note also (and we will use this below) that
$$
\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
@@ -2152,7 +2164,7 @@ $$
which is a \( 2\times 2 \) matrix while
$$
\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
@@ -2225,7 +2237,7 @@ It means that the ordinary least square model (with the optimal parameters) \( \
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="further-properties-important-gems-for-our-analysis">Further properties (important gems for our analysis) </h2>
<h2 id="further-properties-important-for-our-analyses-later">Further properties (important for our analyses later) </h2>
<p>
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
@@ -2263,6 +2275,43 @@ $$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2.
$$
<p>
<b>Important note</b>: we have defined our design matrix \( \boldsymbol{X} \) to be an
\( n\times p \) matrix. In most supervised learning cases we have that \( n
\ge p \), and quite often we have \( n >> p \). The number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="meet-the-covariance-matrix">Meet the Covariance Matrix </h2>
<p>
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
<p>
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
The Hessian matrix plays an important role and is defined here as
$$
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/html/week35.html
+65 -16
View File
@@ -217,10 +217,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'setting-up-the-matrix-to-be-inverted'),
('Further properties (important gems for our analysis)',
('Further properties (important for our analyses later)',
2,
None,
'further-properties-important-gems-for-our-analysis'),
'further-properties-important-for-our-analyses-later'),
('Meet the Covariance Matrix',
2,
None,
'meet-the-covariance-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -318,7 +322,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 5, 2021</h4></center> <!-- date -->
<center><h4>Sep 6, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -889,11 +893,14 @@ $$
<h2 id="meet-the-hessian-matrix">Meet the Hessian Matrix </h2>
<p>
A very important matrix we will meet again and again in Machine Learning is the Hessian.
It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
A very important matrix we will meet again and again in Machine
Learning is the Hessian. It is given by the second derivative of the
cost function with respect to the parameter \( \beta \). Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the cost function is,
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
@@ -904,8 +911,13 @@ $$
$$
<p>
For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is
(beside \( 1/n \)) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning
problem.
<p>
<b>Linear algebra question:</b> Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
@@ -2078,11 +2090,11 @@ Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\t
$$
\boldsymbol{X}=
\begin{bmatrix}
x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
\dots& \dots &\dots & \dots & \dots &\dots\\
x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
\end{bmatrix}
$$
@@ -2148,7 +2160,7 @@ $$
contains only the singular values. Note also (and we will use this below) that
$$
\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
@@ -2157,7 +2169,7 @@ $$
which is a \( 2\times 2 \) matrix while
$$
\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
@@ -2230,7 +2242,7 @@ It means that the ordinary least square model (with the optimal parameters) \( \
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="further-properties-important-gems-for-our-analysis">Further properties (important gems for our analysis) </h2>
<h2 id="further-properties-important-for-our-analyses-later">Further properties (important for our analyses later) </h2>
<p>
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
@@ -2268,6 +2280,43 @@ $$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2.
$$
<p>
<b>Important note</b>: we have defined our design matrix \( \boldsymbol{X} \) to be an
\( n\times p \) matrix. In most supervised learning cases we have that \( n
\ge p \), and quite often we have \( n >> p \). The number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="meet-the-covariance-matrix">Meet the Covariance Matrix </h2>
<p>
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
<p>
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
The Hessian matrix plays an important role and is defined here as
$$
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+78 -14
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 5, 2021**\n",
"Date: **Sep 6, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -847,8 +847,11 @@
"source": [
"## Meet the Hessian Matrix\n",
"\n",
"A very important matrix we will meet again and again in Machine Learning is the Hessian.\n",
"It is given by the second derivative of the cost function with respect to the parameter $\\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,"
"A very important matrix we will meet again and again in Machine\n",
"Learning is the Hessian. It is given by the second derivative of the\n",
"cost function with respect to the parameter $\\beta$. Using the above\n",
"expression for derivatives of vectors and matrices, we find that the\n",
"second derivative of the cost function is,"
]
},
{
@@ -856,7 +859,7 @@
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
@@ -880,8 +883,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters\n",
"$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.\n",
"For ordinary least squares, it is inversely proportional (derivation\n",
"next week) with the variance of the optimal parameters\n",
"$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that is\n",
"(beside $1/n$) equal to the covariance matrix. It plays also a very\n",
"important role in optmization algorithms and Principal Component\n",
"Analysis as a way to reduce the dimensionality of a machine learning\n",
"problem.\n",
"\n",
"**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). \n",
"\n",
@@ -2463,11 +2471,11 @@
"$$\n",
"\\boldsymbol{X}=\n",
"\\begin{bmatrix}\n",
"x_{0,0}& x_{0,1} &x_{0,2}& \\dots & \\dots &x_{0,p-1}\\\\\n",
"x_{1,0}& x_{1,1} &x_{1,2& \\dots & \\dots &x_{1,p-1}\\\\\n",
"x_{2,0}& x_{2,1} &x_{2,2}& \\dots & \\dots &x_{2,p-1}\\\\\n",
"\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
"x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \\dots & \\dots &x_{n-1,p-1}\\\\\n",
"x_{0,0} & x_{0,1} & x_{0,2}& \\dots & \\dots & x_{0,p-1}\\\\\n",
"x_{1,0} & x_{1,1} & x_{1,2& \\dots & \\dots & x_{1,p-1}\\\\\n",
"x_{2,0} & x_{2,1} & x_{2,2}& \\dots & \\dots & x_{2,p-1}\\\\\n",
"\\dots& \\dots &\\dots & \\dots & \\dots &\\dots\\\\\n",
"x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \\dots & \\dots & x_{n-1,p-1}\\\\\n",
"\\end{bmatrix}\n",
"$$"
]
@@ -2585,7 +2593,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}^T\\boldsymbol{\\sigma}=\n",
"\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}=\n",
"\\begin{bmatrix}\n",
"4& 0 \\\\\n",
"0 & 1 \\\\\n",
@@ -2605,7 +2613,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}\\boldsymbol{\\sigma}^T=\n",
"\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T=\n",
"\\begin{bmatrix}\n",
"4& 0 & 0\\\\\n",
"0 & 1 & 0\\\\\n",
@@ -2743,7 +2751,7 @@
"\n",
"It means that the ordinary least square model (with the optimal parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\\boldsymbol{y}$ by the vectors of the matrix $\\boldsymbol{U}$.\n",
"\n",
"## Further properties (important gems for our analysis)\n",
"## Further properties (important for our analyses later)\n",
"\n",
"Let us study again $\\boldsymbol{X}^T\\boldsymbol{X}$ in terms of our SVD,"
]
@@ -2843,6 +2851,62 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**Important note**: we have defined our design matrix $\\boldsymbol{X}$ to be an\n",
"$n\\times p$ matrix. In most supervised learning cases we have that $n\n",
"\\ge p$, and quite often we have $n >> p$. The number of columns will\n",
"always refer to the number of features in our data set, while the\n",
"number of rows represents the number of data inputs. Note that in\n",
"other texts you may find the opposite notation. This has consequences\n",
"for the definition of for example the covariance matrix.\n",
"\n",
"\n",
"## Meet the Covariance Matrix\n",
"\n",
"\n",
"Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.\n",
"\n",
"We have already noted that the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The Hessian matrix plays an important role and is defined here as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The Hessian matrix for ordinary least squares is also proportional to\n",
"the covariance matrix. This means also that we can use the SVD to find\n",
"the eigenvalues of the covariance matrix and the Hessian matrix in\n",
"terms of the singular values.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Ridge and LASSO Regression\n",
"\n",
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
doc/src/week35/week35.do.txt
+61 -13
View File
@@ -512,12 +512,15 @@ matrices as upper case boldfaced letters.
!split
===== Meet the Hessian Matrix =====
A very important matrix we will meet again and again in Machine Learning is the Hessian.
It is given by the second derivative of the cost function with respect to the parameter $\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
A very important matrix we will meet again and again in Machine
Learning is the Hessian. It is given by the second derivative of the
cost function with respect to the parameter $\beta$. Using the above
expression for derivatives of vectors and matrices, we find that the
second derivative of the cost function is,
!bt
\[
\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{\partial}{\partial \bm{\beta}^T}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{\partial}{\partial \bm{\beta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
\]
!et
@@ -531,8 +534,13 @@ The Hessian matrix plays an important role and is defined here as
!et
For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
$\hat{\bm{\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
For ordinary least squares, it is inversely proportional (derivation
next week) with the variance of the optimal parameters
$\hat{\bm{\beta}}$. Furthermore, we will see later this week that is
(beside $1/n$) equal to the covariance matrix. It plays also a very
important role in optmization algorithms and Principal Component
Analysis as a way to reduce the dimensionality of a machine learning
problem.
_Linear algebra question:_ Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
@@ -1575,11 +1583,11 @@ Our starting point is our design matrix $\bm{X}$ of dimension $n\times p$
\[
\bm{X}=
\begin{bmatrix}
x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
\dots& \dots &\dots& \dots & \dots &\dots\\
x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
\dots& \dots &\dots & \dots & \dots &\dots\\
x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
\end{bmatrix}
\]
!et
@@ -1646,7 +1654,7 @@ contains only the singular values. Note also (and we will use this below) that
!bt
\[
\bm{\Sigma}^T\bm{\sigma}=
\bm{\Sigma}^T\bm{\Sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
@@ -1656,7 +1664,7 @@ contains only the singular values. Note also (and we will use this below) that
which is a $2\times 2 $ matrix while
!bt
\[
\bm{\Sigma}\bm{\sigma}^T=
\bm{\Sigma}\bm{\Sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
@@ -1730,7 +1738,7 @@ that belong to $i>p-1$, give all zeros when we perform the multiplications. This
It means that the ordinary least square model (with the optimal parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\bm{y}$ by the vectors of the matrix $\bm{U}$.
!split
===== Further properties (important gems for our analysis) =====
===== Further properties (important for our analyses later) =====
Let us study again $\bm{X}^T\bm{X}$ in terms of our SVD,
!bt
@@ -1774,6 +1782,46 @@ with eigenvalues given by the singular values squared, that is
\]
!et
_Important note_: we have defined our design matrix $\bm{X}$ to be an
$n\times p$ matrix. In most supervised learning cases we have that $n
\ge p$, and quite often we have $n >> p$. The number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix.
!split
===== Meet the Covariance Matrix =====
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have
!bt
\[
\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}.
\]
!et
The Hessian matrix plays an important role and is defined here as
!bt
\[
\bm{H}=\bm{X}^T\bm{X}.
\]
!et
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values.