+A very important matrix we will meet again and again in Machine Learning is the Hessian. +It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, + +
+$$
+\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
+$$
+
+
+
+The Hessian matrix plays an important role and is defined here as + +
+$$
+\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
+$$
+
+
+
+For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters +\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. + +
+Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). +
+The \( U \), \( S \), and \( V \) matrices returned from the svd() function +cannot be multiplied directly. + +
+As you can see from the code, the \( S \) +vector must be converted into a diagonal matrix. This may cause a +as +the size of the matrices do not fit the rules of matrix +multiplication, where the number of columns in a matrix must match the +number of rows in the subsequent matrix. + +
+If you wish to include the zero singular values, you will need to resize the matrices. More about this later. +
+
+A very important matrix we will meet again and again in Machine Learning is the Hessian. +It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, + +$$ +\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. +$$ + +
+The Hessian matrix plays an important role and is defined here as + +$$ +\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. +$$ + +
+For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters +\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. + +
+Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). + +
+
+
+
+The \( U \), \( S \), and \( V \) matrices returned from the svd() function +cannot be multiplied directly. + +
+As you can see from the code, the \( S \) +vector must be converted into a diagonal matrix. This may cause a +as +the size of the matrices do not fit the rules of matrix +multiplication, where the number of columns in a matrix must match the +number of rows in the subsequent matrix. + +
+If you wish to include the zero singular values, you will need to resize the matrices. More about this later. + +
+
+
diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 86ba19579..258737935 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -127,6 +127,7 @@ div { text-align: justify; text-justify: inter-word; } 2, None, 'some-useful-matrix-and-vector-expressions'), + ('Meet the Hessian Matrix', 2, None, 'meet-the-hessian-matrix'), ('Interpretations and optimizing our parameters', 2, None, @@ -202,6 +203,10 @@ div { text-align: justify; text-justify: inter-word; } 'the-svd-a-fantastic-algorithm'), ('Economy-size SVD', 2, None, 'economy-size-svd'), ('Codes for the SVD', 2, None, 'codes-for-the-svd'), + ('Note about SVD Calculations', + 2, + None, + 'note-about-svd-calculations'), ('Mathematical Properties', 2, None, 'mathematical-properties'), ('Friday September 3', 2, None, 'friday-september-3'), ('Ridge and LASSO Regression', @@ -867,6 +872,33 @@ $$
+
+A very important matrix we will meet again and again in Machine Learning is the Hessian. +It is given by the second derivative of the cost function with respect to the parameter \( \beta \). Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, + +$$ +\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{\partial}{\partial \boldsymbol{\beta}^T}\left[-\frac{2}{n}\boldsymbol{X}^T\left( \boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]=\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. +$$ + +
+The Hessian matrix plays an important role and is defined here as + +$$ +\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. +$$ + +
+For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters +\( \hat{\boldsymbol{\beta}} \). Furthermore, we will see later this week that is (beside \( 1/n \)) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. + +
+Linear algebra question: Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). + +
+
+
+
+The \( U \), \( S \), and \( V \) matrices returned from the svd() function +cannot be multiplied directly. + +
+As you can see from the code, the \( S \) +vector must be converted into a diagonal matrix. This may cause a +as +the size of the matrices do not fit the rules of matrix +multiplication, where the number of columns in a matrix must match the +number of rows in the subsequent matrix. + +
+If you wish to include the zero singular values, you will need to resize the matrices. More about this later. + +
+
+
diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 91a436d1e..079540bef 100644 Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 73ef692d9..a1102bae4 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -841,6 +841,46 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "## Meet the Hessian Matrix\n", + "\n", + "A very important matrix we will meet again and again in Machine Learning is the Hessian.\n", + "It is given by the second derivative of the cost function with respect to the parameter $\\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is," + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\left[-\\frac{2}{n}\\boldsymbol{X}^T\\left( \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right]=\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The Hessian matrix plays an important role and is defined here as" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{H}=\\boldsymbol{X}^T\\boldsymbol{X}.\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters\n", + "$\\hat{\\boldsymbol{\\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem.\n", + "\n", + "**Linear algebra question:** Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). \n", + "\n", "## Interpretations and optimizing our parameters\n", "The residuals $\\boldsymbol{\\epsilon}$ are in turn given by" ] @@ -2360,6 +2400,21 @@ "in the program terminating due to a singular matrix.\n", "\n", "\n", + "## Note about SVD Calculations\n", + "\n", + "The $U$, $S$, and $V$ matrices returned from the **svd()** function\n", + "cannot be multiplied directly.\n", + "\n", + "As you can see from the code, the $S$\n", + "vector must be converted into a diagonal matrix. This may cause a \n", + "as\n", + "the size of the matrices do not fit the rules of matrix\n", + "multiplication, where the number of columns in a matrix must match the\n", + "number of rows in the subsequent matrix.\n", + "\n", + "If you wish to include the zero singular values, you will need to resize the matrices. More about this later.\n", + "\n", + "\n", "\n", "## Mathematical Properties\n", "\n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index ba241260d..bca18e8b5 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -503,6 +503,33 @@ matrices as upper case boldfaced letters. \] !et +!split +===== Meet the Hessian Matrix ===== + +A very important matrix we will meet again and again in Machine Learning is the Hessian. +It is given by the second derivative of the cost function with respect to the parameter $\beta$. Using the above expression for derivatives of vectors and matrices, we find that the second derivative of the cost function is, + +!bt +\[ +\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{\partial}{\partial \bm{\beta}^T}\left[-\frac{2}{n}\bm{X}^T\left( \bm{y}-\bm{X}\bm{\beta}\right)\right]=\frac{2}{n}\bm{X}^T\bm{X}. +\] +!et + + +The Hessian matrix plays an important role and is defined here as + +!bt +\[ +\bm{H}=\bm{X}^T\bm{X}. +\] +!et + + +For ordinary least squares, it is inversely proportional (derivation next week) with the variance of the optimal parameters +$\hat{\bm{\beta}}$. Furthermore, we will see later this week that is (beside $1/n$) equal to the covariance matrix. It plays also a very important role in optmization algorithms and Principal Component Analysis as a way to reduce the dimensionality of a machine learning problem. + +_Linear algebra question:_ Can we use the Hessian matrix to say something about properties of the cost function (our optmization problem)? (hint: think about convex or concave problems and how to relate these to a matrix!). + !split ===== Interpretations and optimizing our parameters ===== !bblock @@ -1483,6 +1510,22 @@ inversion algorithm for matrix inversion with $\bm{X}^T\bm{X}$ results in the program terminating due to a singular matrix. +!split +===== Note about SVD Calculations ===== + +The $U$, $S$, and $V$ matrices returned from the _svd()_ function +cannot be multiplied directly. + +As you can see from the code, the $S$ +vector must be converted into a diagonal matrix. This may cause a +as +the size of the matrices do not fit the rules of matrix +multiplication, where the number of columns in a matrix must match the +number of rows in the subsequent matrix. + +If you wish to include the zero singular values, you will need to resize the matrices. More about this later. + + !split ===== Mathematical Properties =====