-
@@ -415,7 +420,7 @@ MathJax.Hub.Config({
-
@@ -768,12 +768,15 @@ $$
-A very important matrix we will meet again and again in Machine Learning is the Hessian. -It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, +A very important matrix we will meet again and again in Machine +Learning is the Hessian. It is given by the second derivative of the +cost function with respect to the parameter \( \beta \). Using the above +expression for derivatives of vectors and matrices, we find that the +second derivative of the cost function is,
$$
-\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
@@ -787,8 +790,13 @@ $$
-For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters -\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. +For ordinary least squares, it is inversely proportional (derivation +next week) with the variance of the optimal parameters +\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is +(beside \( 1/n \)) equal to the covariance matrix. It plays also a very +important role in optmization algorithms and Principal Component +Analysis as a way to reduce the dimensionality of a machine learning +problem.
Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). @@ -2001,11 +2009,11 @@ Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\t $$ \boldsymbol{X}= \begin{bmatrix} -x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\ -x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\ -x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\ -\dots& \dots &\dots& \dots & \dots &\dots\\ -x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\ +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\ +\dots& \dots &\dots & \dots & \dots &\dots\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\ \end{bmatrix} $$
@@ -2083,7 +2091,7 @@ contains only the singular values. Note also (and we will use this below) that
$$
-\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=
+\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
@@ -2094,7 +2102,7 @@ $$
which is a \( 2\times 2 \) matrix while
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD,
@@ -2231,6 +2239,47 @@ $$
\left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2.
$$
+Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an
+\( n\times p \) matrix. In most supervised learning cases we have that \( n
+\ge p \), and quite often we have \( n >> p \). The number of columns will
+always refer to the number of features in our data set, while the
+number of rows represents the number of data inputs. Note that in
+other texts you may find the opposite notation. This has consequences
+for the definition of for example the covariance matrix.
+
+Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
+
+
+We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have
+
+
+The Hessian matrix plays an important role and is defined here as
+
+
+The Hessian matrix for ordinary least squares is also proportional to
+the covariance matrix. This means also that we can use the SVD to find
+the eigenvalues of the covariance matrix and the Hessian matrix in
+terms of the singular values.
$$
-\boldsymbol{\Sigma}\boldsymbol{\sigma}^T=
+\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
@@ -2182,7 +2190,7 @@ It means that the ordinary least square model (with the optimal parameters) \( \
Further properties (important gems for our analysis)
+Further properties (important for our analyses later)
+
+Meet the Covariance Matrix
+
+
+$$
+\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+$$
+
+
+
+$$
+\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
+$$
+
+
+
-
@@ -884,11 +888,14 @@ $$
-A very important matrix we will meet again and again in Machine Learning is the Hessian. -It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, +A very important matrix we will meet again and again in Machine +Learning is the Hessian. It is given by the second derivative of the +cost function with respect to the parameter \( \beta \). Using the above +expression for derivatives of vectors and matrices, we find that the +second derivative of the cost function is, $$ -\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. +\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$
@@ -899,8 +906,13 @@ $$ $$
-For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters -\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. +For ordinary least squares, it is inversely proportional (derivation +next week) with the variance of the optimal parameters +\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is +(beside \( 1/n \)) equal to the covariance matrix. It plays also a very +important role in optmization algorithms and Principal Component +Analysis as a way to reduce the dimensionality of a machine learning +problem.
Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). @@ -2073,11 +2085,11 @@ Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\t $$ \boldsymbol{X}= \begin{bmatrix} -x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\ -x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\ -x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\ -\dots& \dots &\dots& \dots & \dots &\dots\\ -x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\ +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\ +\dots& \dots &\dots & \dots & \dots &\dots\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\ \end{bmatrix} $$ @@ -2143,7 +2155,7 @@ $$ contains only the singular values. Note also (and we will use this below) that $$ -\boldsymbol{\Sigma}^T\boldsymbol{\sigma}= +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}= \begin{bmatrix} 4& 0 \\ 0 & 1 \\ @@ -2152,7 +2164,7 @@ $$ which is a \( 2\times 2 \) matrix while $$ -\boldsymbol{\Sigma}\boldsymbol{\sigma}^T= +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T= \begin{bmatrix} 4& 0 & 0\\ 0 & 1 & 0\\ @@ -2225,7 +2237,7 @@ It means that the ordinary least square model (with the optimal parameters) \( \
-
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD, @@ -2263,6 +2275,43 @@ $$ \left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. $$ +
+Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an +\( n\times p \) matrix. In most supervised learning cases we have that \( n +\ge p \), and quite often we have \( n >> p \). The number of columns will +always refer to the number of features in our data set, while the +number of rows represents the number of data inputs. Note that in +other texts you may find the opposite notation. This has consequences +for the definition of for example the covariance matrix. + +
+
+
+
+Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above. + +
+We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have + +$$ +\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. +$$ + +
+The Hessian matrix plays an important role and is defined here as + +$$ +\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. +$$ + +
+The Hessian matrix for ordinary least squares is also proportional to +the covariance matrix. This means also that we can use the SVD to find +the eigenvalues of the covariance matrix and the Hessian matrix in +terms of the singular values. +
diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html
index cc8a3a179..25d8e5fa9 100644
--- a/doc/pub/week35/html/week35.html
+++ b/doc/pub/week35/html/week35.html
@@ -217,10 +217,14 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'setting-up-the-matrix-to-be-inverted'),
- ('Further properties (important gems for our analysis)',
+ ('Further properties (important for our analyses later)',
2,
None,
- 'further-properties-important-gems-for-our-analysis'),
+ 'further-properties-important-for-our-analyses-later'),
+ ('Meet the Covariance Matrix',
+ 2,
+ None,
+ 'meet-the-covariance-matrix'),
('Ridge and LASSO Regression',
2,
None,
@@ -318,7 +322,7 @@ MathJax.Hub.Config({
-
@@ -889,11 +893,14 @@ $$
-A very important matrix we will meet again and again in Machine Learning is the Hessian. -It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, +A very important matrix we will meet again and again in Machine +Learning is the Hessian. It is given by the second derivative of the +cost function with respect to the parameter \( \beta \). Using the above +expression for derivatives of vectors and matrices, we find that the +second derivative of the cost function is, $$ -\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. +\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$
@@ -904,8 +911,13 @@ $$ $$
-For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters -\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. +For ordinary least squares, it is inversely proportional (derivation +next week) with the variance of the optimal parameters +\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is +(beside \( 1/n \)) equal to the covariance matrix. It plays also a very +important role in optmization algorithms and Principal Component +Analysis as a way to reduce the dimensionality of a machine learning +problem.
Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). @@ -2078,11 +2090,11 @@ Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\t $$ \boldsymbol{X}= \begin{bmatrix} -x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\ -x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\ -x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\ -\dots& \dots &\dots& \dots & \dots &\dots\\ -x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\ +x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\ +x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\ +x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\ +\dots& \dots &\dots & \dots & \dots &\dots\\ +x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\ \end{bmatrix} $$ @@ -2148,7 +2160,7 @@ $$ contains only the singular values. Note also (and we will use this below) that $$ -\boldsymbol{\Sigma}^T\boldsymbol{\sigma}= +\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}= \begin{bmatrix} 4& 0 \\ 0 & 1 \\ @@ -2157,7 +2169,7 @@ $$ which is a \( 2\times 2 \) matrix while $$ -\boldsymbol{\Sigma}\boldsymbol{\sigma}^T= +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T= \begin{bmatrix} 4& 0 & 0\\ 0 & 1 & 0\\ @@ -2230,7 +2242,7 @@ It means that the ordinary least square model (with the optimal parameters) \( \
-
Let us study again \( \boldsymbol{X}^T\boldsymbol{X} \) in terms of our SVD, @@ -2268,6 +2280,43 @@ $$ \left(\boldsymbol{X}\boldsymbol{X}^T\right)\boldsymbol{u}_i=\boldsymbol{u}_i\sigma_i^2. $$ +
+Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an +\( n\times p \) matrix. In most supervised learning cases we have that \( n +\ge p \), and quite often we have \( n >> p \). The number of columns will +always refer to the number of features in our data set, while the +number of rows represents the number of data inputs. Note that in +other texts you may find the opposite notation. This has consequences +for the definition of for example the covariance matrix. + +
+
+
+
+Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above. + +
+We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have + +$$ +\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. +$$ + +
+The Hessian matrix plays an important role and is defined here as + +$$ +\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. +$$ + +
+The Hessian matrix for ordinary least squares is also proportional to +the covariance matrix. This means also that we can use the SVD to find +the eigenvalues of the covariance matrix and the Hessian matrix in +terms of the singular values. +
diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
index 9141f751b..9e24a8184 100644
Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ
diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb
index 42790a125..7e23aacc6 100644
--- a/doc/pub/week35/ipynb/week35.ipynb
+++ b/doc/pub/week35/ipynb/week35.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Sep 5, 2021**\n",
+ "Date: **Sep 6, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -847,8 +847,11 @@
"source": [
"## Meet the Hessian Matrix\n",
"\n",
- "A very important matrix we will meet again and again in Machine Learning is the Hessian.\n",
- "It is given by the second derivative of the cost function with respect to the parameter $\\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,"
+ "A very important matrix we will meet again and again in Machine\n",
+ "Learning is the Hessian. It is given by the second derivative of the\n",
+ "cost function with respect to the parameter $\\beta$. Using the above\n",
+ "expression for derivatives of vectors and matrices, we find that the\n",
+ "second derivative of the cost function is,"
]
},
{
@@ -856,7 +859,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+ "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
"$$"
]
},
@@ -880,8 +883,13 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters\n",
- "$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.\n",
+ "For ordinary least squares, it is inversely proportional (derivation\n",
+ "next week) with the variance of the optimal parameters\n",
+ "$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that is\n",
+ "(beside $1/n$) equal to the covariance matrix. It plays also a very\n",
+ "important role in optmization algorithms and Principal Component\n",
+ "Analysis as a way to reduce the dimensionality of a machine learning\n",
+ "problem.\n",
"\n",
"**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). \n",
"\n",
@@ -2463,11 +2471,11 @@
"$$\n",
"\\boldsymbol{X}=\n",
"\\begin{bmatrix}\n",
- "x_{0,0}& x_{0,1} &x_{0,2}& \\dots & \\dots &x_{0,p-1}\\\\\n",
- "x_{1,0}& x_{1,1} &x_{1,2& \\dots & \\dots &x_{1,p-1}\\\\\n",
- "x_{2,0}& x_{2,1} &x_{2,2}& \\dots & \\dots &x_{2,p-1}\\\\\n",
- "\\dots& \\dots &\\dots& \\dots & \\dots &\\dots\\\\\n",
- "x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \\dots & \\dots &x_{n-1,p-1}\\\\\n",
+ "x_{0,0} & x_{0,1} & x_{0,2}& \\dots & \\dots & x_{0,p-1}\\\\\n",
+ "x_{1,0} & x_{1,1} & x_{1,2& \\dots & \\dots & x_{1,p-1}\\\\\n",
+ "x_{2,0} & x_{2,1} & x_{2,2}& \\dots & \\dots & x_{2,p-1}\\\\\n",
+ "\\dots& \\dots &\\dots & \\dots & \\dots &\\dots\\\\\n",
+ "x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \\dots & \\dots & x_{n-1,p-1}\\\\\n",
"\\end{bmatrix}\n",
"$$"
]
@@ -2585,7 +2593,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\sigma}=\n",
+ "\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}=\n",
"\\begin{bmatrix}\n",
"4& 0 \\\\\n",
"0 & 1 \\\\\n",
@@ -2605,7 +2613,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\boldsymbol{\\Sigma}\\boldsymbol{\\sigma}^T=\n",
+ "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T=\n",
"\\begin{bmatrix}\n",
"4& 0 & 0\\\\\n",
"0 & 1 & 0\\\\\n",
@@ -2743,7 +2751,7 @@
"\n",
"It means that the ordinary least square model (with the optimal parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\\boldsymbol{y}$ by the vectors of the matrix $\\boldsymbol{U}$.\n",
"\n",
- "## Further properties (important gems for our analysis)\n",
+ "## Further properties (important for our analyses later)\n",
"\n",
"Let us study again $\\boldsymbol{X}^T\\boldsymbol{X}$ in terms of our SVD,"
]
@@ -2843,6 +2851,62 @@
"cell_type": "markdown",
"metadata": {},
"source": [
+ "**Important note**: we have defined our design matrix $\\boldsymbol{X}$ to be an\n",
+ "$n\\times p$ matrix. In most supervised learning cases we have that $n\n",
+ "\\ge p$, and quite often we have $n >> p$. The number of columns will\n",
+ "always refer to the number of features in our data set, while the\n",
+ "number of rows represents the number of data inputs. Note that in\n",
+ "other texts you may find the opposite notation. This has consequences\n",
+ "for the definition of for example the covariance matrix.\n",
+ "\n",
+ "\n",
+ "## Meet the Covariance Matrix\n",
+ "\n",
+ "\n",
+ "Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.\n",
+ "\n",
+ "We have already noted that the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The Hessian matrix plays an important role and is defined here as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The Hessian matrix for ordinary least squares is also proportional to\n",
+ "the covariance matrix. This means also that we can use the SVD to find\n",
+ "the eigenvalues of the covariance matrix and the Hessian matrix in\n",
+ "terms of the singular values.\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
"## Ridge and LASSO Regression\n",
"\n",
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt
index 9cee6b018..8882cf666 100644
--- a/doc/src/week35/week35.do.txt
+++ b/doc/src/week35/week35.do.txt
@@ -512,12 +512,15 @@ matrices as upper case boldfaced letters.
!split
===== Meet the Hessian Matrix =====
-A very important matrix we will meet again and again in Machine Learning is the Hessian.
-It is given by the second derivative of the cost function with respect to the parameter $\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is,
+A very important matrix we will meet again and again in Machine
+Learning is the Hessian. It is given by the second derivative of the
+cost function with respect to the parameter $\beta$. Using the above
+expression for derivatives of vectors and matrices, we find that the
+second derivative of the cost function is,
!bt
\[
-\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{\partial}{\partial \bm{\beta}^T}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
+\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{\partial}{\partial \bm{\beta}}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}.
\]
!et
@@ -531,8 +534,13 @@ The Hessian matrix plays an important role and is defined here as
!et
-For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters
-$\hat{\bm{\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.
+For ordinary least squares, it is inversely proportional (derivation
+next week) with the variance of the optimal parameters
+$\hat{\bm{\beta}}$. Furthermore, we will see later this week that is
+(beside $1/n$) equal to the covariance matrix. It plays also a very
+important role in optmization algorithms and Principal Component
+Analysis as a way to reduce the dimensionality of a machine learning
+problem.
_Linear algebra question:_ Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!).
@@ -1575,11 +1583,11 @@ Our starting point is our design matrix $\bm{X}$ of dimension $n\times p$
\[
\bm{X}=
\begin{bmatrix}
-x_{0,0}& x_{0,1} &x_{0,2}& \dots & \dots &x_{0,p-1}\\
-x_{1,0}& x_{1,1} &x_{1,2& \dots & \dots &x_{1,p-1}\\
-x_{2,0}& x_{2,1} &x_{2,2}& \dots & \dots &x_{2,p-1}\\
-\dots& \dots &\dots& \dots & \dots &\dots\\
-x_{n-1,0}& x_{n-1,1} &x_{n-1,2}& \dots & \dots &x_{n-1,p-1}\\
+x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
+x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
+x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
+\dots& \dots &\dots & \dots & \dots &\dots\\
+x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
\end{bmatrix}
\]
!et
@@ -1646,7 +1654,7 @@ contains only the singular values. Note also (and we will use this below) that
!bt
\[
-\bm{\Sigma}^T\bm{\sigma}=
+\bm{\Sigma}^T\bm{\Sigma}=
\begin{bmatrix}
4& 0 \\
0 & 1 \\
@@ -1656,7 +1664,7 @@ contains only the singular values. Note also (and we will use this below) that
which is a $2\times 2 $ matrix while
!bt
\[
-\bm{\Sigma}\bm{\sigma}^T=
+\bm{\Sigma}\bm{\Sigma}^T=
\begin{bmatrix}
4& 0 & 0\\
0 & 1 & 0\\
@@ -1730,7 +1738,7 @@ that belong to $i>p-1$, give all zeros when we perform the multiplications. This
It means that the ordinary least square model (with the optimal parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\bm{y}$ by the vectors of the matrix $\bm{U}$.
!split
-===== Further properties (important gems for our analysis) =====
+===== Further properties (important for our analyses later) =====
Let us study again $\bm{X}^T\bm{X}$ in terms of our SVD,
!bt
@@ -1774,6 +1782,46 @@ with eigenvalues given by the singular values squared, that is
\]
!et
+_Important note_: we have defined our design matrix $\bm{X}$ to be an
+$n\times p$ matrix. In most supervised learning cases we have that $n
+\ge p$, and quite often we have $n >> p$. The number of columns will
+always refer to the number of features in our data set, while the
+number of rows represents the number of data inputs. Note that in
+other texts you may find the opposite notation. This has consequences
+for the definition of for example the covariance matrix.
+
+
+!split
+===== Meet the Covariance Matrix =====
+
+
+Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
+
+We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have
+
+!bt
+\[
+\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}.
+\]
+!et
+
+
+The Hessian matrix plays an important role and is defined here as
+
+!bt
+\[
+\bm{H}=\bm{X}^T\bm{X}.
+\]
+!et
+
+
+The Hessian matrix for ordinary least squares is also proportional to
+the covariance matrix. This means also that we can use the SVD to find
+the eigenvalues of the covariance matrix and the Hessian matrix in
+terms of the singular values.
+
+
+