lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 35

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-02 11:57:56 +02:00
parent 23178af02a
commit 270b0beff9
7 changed files with 310 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+57 -50
View File
@@ -102,6 +102,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'some-useful-matrix-and-vector-expressions'),
('Meet the Hessian Matrix', 2, None, 'meet-the-hessian-matrix'),
('Interpretations and optimizing our parameters',
2,
None,
@@ -177,6 +178,10 @@ Automatically generated HTML file from DocOnce source
'the-svd-a-fantastic-algorithm'),
('Economy-size SVD', 2, None, 'economy-size-svd'),
('Codes for the SVD', 2, None, 'codes-for-the-svd'),
('Note about SVD Calculations',
2,
None,
'note-about-svd-calculations'),
('Mathematical Properties', 2, None, 'mathematical-properties'),
('Friday September 3', 2, None, 'friday-september-3'),
('Ridge and LASSO Regression',
@@ -277,7 +282,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs003.html#why-linear-regression-aka-ordinary-least-squares-and-family-repeat-from-last-week" style="font-size: 80%;"><b>Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs004.html#regression-analysis-overarching-aims" style="font-size: 80%;"><b>Regression analysis, overarching aims</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs005.html#regression-analysis-overarching-aims-ii" style="font-size: 80%;"><b>Regression analysis, overarching aims II</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs022.html#examples" style="font-size: 80%;"><b>Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs023.html#examples" style="font-size: 80%;"><b>Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs007.html#general-linear-models" style="font-size: 80%;"><b>General linear models</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs008.html#rewriting-the-fitting-procedure-as-a-linear-algebra-problem" style="font-size: 80%;"><b>Rewriting the fitting procedure as a linear algebra problem</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs009.html#rewriting-the-fitting-procedure-as-a-linear-algebra-problem-more-details" style="font-size: 80%;"><b>Rewriting the fitting procedure as a linear algebra problem, more details</b></a></li>
@@ -286,55 +291,57 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs012.html#optimizing-our-parameters" style="font-size: 80%;"><b>Optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs013.html#our-model-for-the-nuclear-binding-energies" style="font-size: 80%;"><b>Our model for the nuclear binding energies</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs014.html#optimizing-our-parameters-more-details" style="font-size: 80%;"><b>Optimizing our parameters, more details</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs018.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs018.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs019.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs019.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs017.html#some-useful-matrix-and-vector-expressions" style="font-size: 80%;"><b>Some useful matrix and vector expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs018.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs019.html#own-code-for-ordinary-least-squares" style="font-size: 80%;"><b>Own code for Ordinary Least Squares</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs020.html#adding-error-analysis-and-training-set-up" style="font-size: 80%;"><b>Adding error analysis and training set up</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs021.html#splitting-our-data-in-training-and-test-data" style="font-size: 80%;"><b>Splitting our Data in Training and Test data</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs022.html#examples" style="font-size: 80%;"><b>Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs023.html#making-your-own-test-train-splitting" style="font-size: 80%;"><b>Making your own test-train splitting</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs024.html#the-boston-housing-data-example" style="font-size: 80%;"><b>The Boston housing data example</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs025.html#housing-data-the-code" style="font-size: 80%;"><b>Housing data, the code</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs026.html#reducing-the-number-of-degrees-of-freedom-overarching-view" style="font-size: 80%;"><b>Reducing the number of degrees of freedom, overarching view</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs027.html#preprocessing-our-data" style="font-size: 80%;"><b>Preprocessing our data</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs028.html#functionality-in-scikit-learn" style="font-size: 80%;"><b>Functionality in Scikit-Learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs029.html#more-preprocessing" style="font-size: 80%;"><b>More preprocessing</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs030.html#frequently-used-scaling-functions" style="font-size: 80%;"><b>Frequently used scaling functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs031.html#example-of-own-standard-scaling" style="font-size: 80%;"><b>Example of own Standard scaling</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs032.html#min-max-scaling" style="font-size: 80%;"><b>Min-Max Scaling</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs033.html#testing-the-means-squared-error-as-function-of-complexity" style="font-size: 80%;"><b>Testing the Means Squared Error as function of Complexity</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs034.html#more-preprocessing-examples-franke-function-and-regression" style="font-size: 80%;"><b>More preprocessing examples, Franke function and regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs035.html#mathematical-interpretation-of-ordinary-least-squares" style="font-size: 80%;"><b>Mathematical Interpretation of Ordinary Least Squares</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs036.html#residual-error" style="font-size: 80%;"><b>Residual Error</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs037.html#simple-case" style="font-size: 80%;"><b>Simple case</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs038.html#the-singular-value-decomposition" style="font-size: 80%;"><b>The singular value decomposition</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs039.html#linear-regression-problems" style="font-size: 80%;"><b>Linear Regression Problems</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs040.html#fixing-the-singularity" style="font-size: 80%;"><b>Fixing the singularity</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs041.html#basic-math-of-the-svd" style="font-size: 80%;"><b>Basic math of the SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs042.html#the-svd-a-fantastic-algorithm" style="font-size: 80%;"><b>The SVD, a Fantastic Algorithm</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs043.html#economy-size-svd" style="font-size: 80%;"><b>Economy-size SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs044.html#codes-for-the-svd" style="font-size: 80%;"><b>Codes for the SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs045.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs046.html#friday-september-3" style="font-size: 80%;"><b>Friday September 3</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs047.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs048.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs049.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs050.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#linking-with-svd" style="font-size: 80%;"><b>Linking with SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#exercises-for-week-37-september-6-10" style="font-size: 80%;"><b>Exercises for week 37, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs018.html#meet-the-hessian-matrix" style="font-size: 80%;"><b>Meet the Hessian Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs019.html#interpretations-and-optimizing-our-parameters" style="font-size: 80%;"><b>Interpretations and optimizing our parameters</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs020.html#own-code-for-ordinary-least-squares" style="font-size: 80%;"><b>Own code for Ordinary Least Squares</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs021.html#adding-error-analysis-and-training-set-up" style="font-size: 80%;"><b>Adding error analysis and training set up</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs022.html#splitting-our-data-in-training-and-test-data" style="font-size: 80%;"><b>Splitting our Data in Training and Test data</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs023.html#examples" style="font-size: 80%;"><b>Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs024.html#making-your-own-test-train-splitting" style="font-size: 80%;"><b>Making your own test-train splitting</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs025.html#the-boston-housing-data-example" style="font-size: 80%;"><b>The Boston housing data example</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs026.html#housing-data-the-code" style="font-size: 80%;"><b>Housing data, the code</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs027.html#reducing-the-number-of-degrees-of-freedom-overarching-view" style="font-size: 80%;"><b>Reducing the number of degrees of freedom, overarching view</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs028.html#preprocessing-our-data" style="font-size: 80%;"><b>Preprocessing our data</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs029.html#functionality-in-scikit-learn" style="font-size: 80%;"><b>Functionality in Scikit-Learn</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs030.html#more-preprocessing" style="font-size: 80%;"><b>More preprocessing</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs031.html#frequently-used-scaling-functions" style="font-size: 80%;"><b>Frequently used scaling functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs032.html#example-of-own-standard-scaling" style="font-size: 80%;"><b>Example of own Standard scaling</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs033.html#min-max-scaling" style="font-size: 80%;"><b>Min-Max Scaling</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs034.html#testing-the-means-squared-error-as-function-of-complexity" style="font-size: 80%;"><b>Testing the Means Squared Error as function of Complexity</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs035.html#more-preprocessing-examples-franke-function-and-regression" style="font-size: 80%;"><b>More preprocessing examples, Franke function and regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs036.html#mathematical-interpretation-of-ordinary-least-squares" style="font-size: 80%;"><b>Mathematical Interpretation of Ordinary Least Squares</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs037.html#residual-error" style="font-size: 80%;"><b>Residual Error</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs038.html#simple-case" style="font-size: 80%;"><b>Simple case</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs039.html#the-singular-value-decomposition" style="font-size: 80%;"><b>The singular value decomposition</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs040.html#linear-regression-problems" style="font-size: 80%;"><b>Linear Regression Problems</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs041.html#fixing-the-singularity" style="font-size: 80%;"><b>Fixing the singularity</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs042.html#basic-math-of-the-svd" style="font-size: 80%;"><b>Basic math of the SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs043.html#the-svd-a-fantastic-algorithm" style="font-size: 80%;"><b>The SVD, a Fantastic Algorithm</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs044.html#economy-size-svd" style="font-size: 80%;"><b>Economy-size SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs045.html#codes-for-the-svd" style="font-size: 80%;"><b>Codes for the SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs046.html#note-about-svd-calculations" style="font-size: 80%;"><b>Note about SVD Calculations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs047.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs048.html#friday-september-3" style="font-size: 80%;"><b>Friday September 3</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs049.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs050.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs051.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#linking-with-svd" style="font-size: 80%;"><b>Linking with SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#exercises-for-week-37-september-6-10" style="font-size: 80%;"><b>Exercises for week 37, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -393,7 +400,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs061.html">62</a></li>
<li><a href="._week35-bs063.html">64</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+51
View File
@@ -762,6 +762,37 @@ $$
</section>
<section>
<h2 id="meet-the-hessian-matrix">Meet the Hessian Matrix </h2>
<p>
A very important matrix we will meet again and again in Machine Learning is the Hessian.
It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
<p>&nbsp;<br>
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>&nbsp;<br>
<p>
The Hessian matrix plays an important role and is defined here as
<p>&nbsp;<br>
$$
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>&nbsp;<br>
<p>
For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
<p>
<b>Linear algebra question:</b> Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
</section>
<section>
<h2 id="interpretations-and-optimizing-our-parameters">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
@@ -1906,6 +1937,26 @@ in the program terminating due to a singular matrix.
</section>
<section>
<h2 id="note-about-svd-calculations">Note about SVD Calculations </h2>
<p>
The \( U \), \( S \), and \( V \) matrices returned from the <b>svd()</b> function
cannot be multiplied directly.
<p>
As you can see from the code, the \( S \)
vector must be converted into a diagonal matrix. This may cause a
as
the size of the matrices do not fit the rules of matrix
multiplication, where the number of columns in a matrix must match the
number of rows in the subsequent matrix.
<p>
If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
</section>
<section>
<h2 id="mathematical-properties">Mathematical Properties </h2>
doc/pub/week35/html/week35-solarized.html
+52
View File
@@ -122,6 +122,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'some-useful-matrix-and-vector-expressions'),
('Meet the Hessian Matrix', 2, None, 'meet-the-hessian-matrix'),
('Interpretations and optimizing our parameters',
2,
None,
@@ -197,6 +198,10 @@ div { text-align: justify; text-justify: inter-word; }
'the-svd-a-fantastic-algorithm'),
('Economy-size SVD', 2, None, 'economy-size-svd'),
('Codes for the SVD', 2, None, 'codes-for-the-svd'),
('Note about SVD Calculations',
2,
None,
'note-about-svd-calculations'),
('Mathematical Properties', 2, None, 'mathematical-properties'),
('Friday September 3', 2, None, 'friday-september-3'),
('Ridge and LASSO Regression',
@@ -862,6 +867,33 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="meet-the-hessian-matrix">Meet the Hessian Matrix </h2>
<p>
A very important matrix we will meet again and again in Machine Learning is the Hessian.
It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
The Hessian matrix plays an important role and is defined here as
$$
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
<p>
<b>Linear algebra question:</b> Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="interpretations-and-optimizing-our-parameters">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
@@ -1967,6 +1999,26 @@ in the program terminating due to a singular matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="note-about-svd-calculations">Note about SVD Calculations </h2>
<p>
The \( U \), \( S \), and \( V \) matrices returned from the <b>svd()</b> function
cannot be multiplied directly.
<p>
As you can see from the code, the \( S \)
vector must be converted into a diagonal matrix. This may cause a
as
the size of the matrices do not fit the rules of matrix
multiplication, where the number of columns in a matrix must match the
number of rows in the subsequent matrix.
<p>
If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="mathematical-properties">Mathematical Properties </h2>
<p>
doc/pub/week35/html/week35.html
+52
View File
@@ -127,6 +127,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'some-useful-matrix-and-vector-expressions'),
('Meet the Hessian Matrix', 2, None, 'meet-the-hessian-matrix'),
('Interpretations and optimizing our parameters',
2,
None,
@@ -202,6 +203,10 @@ div { text-align: justify; text-justify: inter-word; }
'the-svd-a-fantastic-algorithm'),
('Economy-size SVD', 2, None, 'economy-size-svd'),
('Codes for the SVD', 2, None, 'codes-for-the-svd'),
('Note about SVD Calculations',
2,
None,
'note-about-svd-calculations'),
('Mathematical Properties', 2, None, 'mathematical-properties'),
('Friday September 3', 2, None, 'friday-september-3'),
('Ridge and LASSO Regression',
@@ -867,6 +872,33 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="meet-the-hessian-matrix">Meet the Hessian Matrix </h2>
<p>
A very important matrix we will meet again and again in Machine Learning is the Hessian.
It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
The Hessian matrix plays an important role and is defined here as
$$
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
$$
<p>
For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
<p>
<b>Linear algebra question:</b> Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="interpretations-and-optimizing-our-parameters">Interpretations and optimizing our parameters </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
@@ -1972,6 +2004,26 @@ in the program terminating due to a singular matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="note-about-svd-calculations">Note about SVD Calculations </h2>
<p>
The \( U \), \( S \), and \( V \) matrices returned from the <b>svd()</b> function
cannot be multiplied directly.
<p>
As you can see from the code, the \( S \)
vector must be converted into a diagonal matrix. This may cause a
as
the size of the matrices do not fit the rules of matrix
multiplication, where the number of columns in a matrix must match the
number of rows in the subsequent matrix.
<p>
If you wish to include the zero singular values, you will need to resize the matrices. More about this later.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="mathematical-properties">Mathematical Properties </h2>
<p>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+55
View File
@@ -841,6 +841,46 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Meet the Hessian Matrix\n",
"\n",
"A very important matrix we will meet again and again in Machine Learning is the Hessian.\n",
"It is given by the second derivative of the cost function with respect to the parameter $\\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The Hessian matrix plays an important role and is defined here as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters\n",
"$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.\n",
"\n",
"**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). \n",
"\n",
"## Interpretations and optimizing our parameters\n",
"The residuals $\\boldsymbol{\\epsilon}$ are in turn given by"
]
@@ -2360,6 +2400,21 @@
"in the program terminating due to a singular matrix.\n",
"\n",
"\n",
"## Note about SVD Calculations\n",
"\n",
"The $U$, $S$, and $V$ matrices returned from the **svd()** function\n",
"cannot be multiplied directly.\n",
"\n",
"As you can see from the code, the $S$\n",
"vector must be converted into a diagonal matrix. This may cause a \n",
"as\n",
"the size of the matrices do not fit the rules of matrix\n",
"multiplication, where the number of columns in a matrix must match the\n",
"number of rows in the subsequent matrix.\n",
"\n",
"If you wish to include the zero singular values, you will need to resize the matrices. More about this later.\n",
"\n",
"\n",
"\n",
"## Mathematical Properties\n",
"\n",