Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an \( n\times p \) matrix. In most supervised learning cases we have that \( n -\ge p \), and quite often we have \( n >> p \). The number of columns will +\ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). + +
+In our lectures, the number of columns will always refer to the number of features in our data set, while the number of rows represents the number of data inputs. Note that in other texts you may find the opposite notation. This has consequences -for the definition of for example the covariance matrix. +for the definition of for example the covariance matrix and its relation to the SVD. @@ -2258,7 +2261,9 @@ for the definition of for example the covariance matrix. Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
-We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have +We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary +least squares is proportional to the second derivative of the cost +function, that is we have
$$
@@ -2266,8 +2271,10 @@ $$
$$
+This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
+
-The Hessian matrix plays an important role and is defined here as +The Hessian matrix plays an important role and is defined in this course as
$$
@@ -2279,7 +2286,7 @@ $$
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
-terms of the singular values.
+terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.
@@ -2324,6 +2331,16 @@ $$
$$
+
+Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. +What you will find in essentially all statistics texts are equations +with a factor \( 1/(n-1) \). This is called Bessel's correction. This +method corrects the bias in the estimation of the population variance +and covariance. It also partially corrects the bias in the estimation +of the population standard deviation. If you use a library like +Scikit-Learn or nunmpy's function calculate the covariance, this +quantity will be computed with a factor \( 1/(n-1) \). +
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index ce90fe276..4c3ca31e1 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -2278,11 +2278,14 @@ $$
Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an \( n\times p \) matrix. In most supervised learning cases we have that \( n -\ge p \), and quite often we have \( n >> p \). The number of columns will +\ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). + +
+In our lectures, the number of columns will always refer to the number of features in our data set, while the number of rows represents the number of data inputs. Note that in other texts you may find the opposite notation. This has consequences -for the definition of for example the covariance matrix. +for the definition of for example the covariance matrix and its relation to the SVD.
@@ -2293,14 +2296,18 @@ for the definition of for example the covariance matrix.
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
-We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have +We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary +least squares is proportional to the second derivative of the cost +function, that is we have $$ \frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ +This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize). +
-The Hessian matrix plays an important role and is defined here as +The Hessian matrix plays an important role and is defined in this course as $$ \boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. @@ -2310,7 +2317,7 @@ $$ The Hessian matrix for ordinary least squares is also proportional to the covariance matrix. This means also that we can use the SVD to find the eigenvalues of the covariance matrix and the Hessian matrix in -terms of the singular values. +terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.
@@ -2347,6 +2354,16 @@ $$
\end{bmatrix}.
$$
+
+Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. +What you will find in essentially all statistics texts are equations +with a factor \( 1/(n-1) \). This is called Bessel's correction. This +method corrects the bias in the estimation of the population variance +and covariance. It also partially corrects the bias in the estimation +of the population standard deviation. If you use a library like +Scikit-Learn or nunmpy's function calculate the covariance, this +quantity will be computed with a factor \( 1/(n-1) \). +
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html index 6e4ea5930..4c69e39eb 100644 --- a/doc/pub/week35/html/week35.html +++ b/doc/pub/week35/html/week35.html @@ -2283,11 +2283,14 @@ $$
Important note: we have defined our design matrix \( \boldsymbol{X} \) to be an \( n\times p \) matrix. In most supervised learning cases we have that \( n -\ge p \), and quite often we have \( n >> p \). The number of columns will +\ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). + +
+In our lectures, the number of columns will always refer to the number of features in our data set, while the number of rows represents the number of data inputs. Note that in other texts you may find the opposite notation. This has consequences -for the definition of for example the covariance matrix. +for the definition of for example the covariance matrix and its relation to the SVD.
@@ -2298,14 +2301,18 @@ for the definition of for example the covariance matrix.
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
-We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have +We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary +least squares is proportional to the second derivative of the cost +function, that is we have $$ \frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}. $$ +This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize). +
-The Hessian matrix plays an important role and is defined here as +The Hessian matrix plays an important role and is defined in this course as $$ \boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}. @@ -2315,7 +2322,7 @@ $$ The Hessian matrix for ordinary least squares is also proportional to the covariance matrix. This means also that we can use the SVD to find the eigenvalues of the covariance matrix and the Hessian matrix in -terms of the singular values. +terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.
@@ -2352,6 +2359,16 @@ $$
\end{bmatrix}.
$$
+
+Note: we have used \( 1/n \) in the above definitions of the sample variance and covariance. We assume then that we can calculate the exact mean value. +What you will find in essentially all statistics texts are equations +with a factor \( 1/(n-1) \). This is called Bessel's correction. This +method corrects the bias in the estimation of the population variance +and covariance. It also partially corrects the bias in the estimation +of the population standard deviation. If you use a library like +Scikit-Learn or nunmpy's function calculate the covariance, this +quantity will be computed with a factor \( 1/(n-1) \). +
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz index 37755550c..21138d176 100644 Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb index 76c24500b..a931e3f06 100644 --- a/doc/pub/week35/ipynb/week35.ipynb +++ b/doc/pub/week35/ipynb/week35.ipynb @@ -2853,11 +2853,13 @@ "source": [ "**Important note**: we have defined our design matrix $\\boldsymbol{X}$ to be an\n", "$n\\times p$ matrix. In most supervised learning cases we have that $n\n", - "\\ge p$, and quite often we have $n >> p$. The number of columns will\n", + "\\ge p$, and quite often we have $n >> p$. For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations).\n", + "\n", + "In our lectures, the number of columns will\n", "always refer to the number of features in our data set, while the\n", "number of rows represents the number of data inputs. Note that in\n", "other texts you may find the opposite notation. This has consequences\n", - "for the definition of for example the covariance matrix.\n", + "for the definition of for example the covariance matrix and its relation to the SVD.\n", "\n", "\n", "## Meet the Covariance Matrix\n", @@ -2865,7 +2867,9 @@ "\n", "Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.\n", "\n", - "We have already noted that the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have" + "We have already noted that the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ in ordinary\n", + "least squares is proportional to the second derivative of the cost\n", + "function, that is we have" ] }, { @@ -2881,7 +2885,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "The Hessian matrix plays an important role and is defined here as" + "This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).\n", + "\n", + "The Hessian matrix plays an important role and is defined in this course as" ] }, { @@ -2900,7 +2906,7 @@ "The Hessian matrix for ordinary least squares is also proportional to\n", "the covariance matrix. This means also that we can use the SVD to find\n", "the eigenvalues of the covariance matrix and the Hessian matrix in\n", - "terms of the singular values.\n", + "terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.\n", "\n", "\n", "## Introducing the Covariance and Correlation functions\n", @@ -2977,6 +2983,17 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "**Note:** we have used $1/n$ in the above definitions of the *sample* variance and covariance. We assume then that we can calculate the exact mean value. \n", + "What you will find in essentially all statistics texts are equations\n", + "with a factor $1/(n-1)$. This is called [Bessel's correction](https://mathworld.wolfram.com/BesselsCorrection.html). This\n", + "method corrects the bias in the estimation of the population variance\n", + "and covariance. It also partially corrects the bias in the estimation\n", + "of the population standard deviation. If you use a library like\n", + "**Scikit-Learn** or **nunmpy's** function calculate the covariance, this\n", + "quantity will be computed with a factor $1/(n-1)$.\n", + "\n", + "\n", + "\n", "The covariance takes values between zero and infinity and may thus\n", "lead to problems with loss of numerical precision for particularly\n", "large values. It is common to scale the covariance matrix by\n", diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt index a7de5d036..6331b028a 100644 --- a/doc/src/week35/week35.do.txt +++ b/doc/src/week35/week35.do.txt @@ -1784,11 +1784,13 @@ with eigenvalues given by the singular values squared, that is _Important note_: we have defined our design matrix $\bm{X}$ to be an $n\times p$ matrix. In most supervised learning cases we have that $n -\ge p$, and quite often we have $n >> p$. The number of columns will +\ge p$, and quite often we have $n >> p$. For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix $\bm{X}^T\bm{X}$ which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations). + +In our lectures, the number of columns will always refer to the number of features in our data set, while the number of rows represents the number of data inputs. Note that in other texts you may find the opposite notation. This has consequences -for the definition of for example the covariance matrix. +for the definition of for example the covariance matrix and its relation to the SVD. !split @@ -1797,16 +1799,18 @@ for the definition of for example the covariance matrix. Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above. -We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have +We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary +least squares is proportional to the second derivative of the cost +function, that is we have !bt \[ \frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}. \] !et +This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize). - -The Hessian matrix plays an important role and is defined here as +The Hessian matrix plays an important role and is defined in this course as !bt \[ @@ -1818,7 +1822,7 @@ The Hessian matrix plays an important role and is defined here as The Hessian matrix for ordinary least squares is also proportional to the covariance matrix. This means also that we can use the SVD to find the eigenvalues of the covariance matrix and the Hessian matrix in -terms of the singular values. +terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies. !split @@ -1857,6 +1861,17 @@ we can rewrite the covariance matrix as \] !et +_Note:_ we have used $1/n$ in the above definitions of the *sample* variance and covariance. We assume then that we can calculate the exact mean value. +What you will find in essentially all statistics texts are equations +with a factor $1/(n-1)$. This is called "Bessel's correction":"https://mathworld.wolfram.com/BesselsCorrection.html". This +method corrects the bias in the estimation of the population variance +and covariance. It also partially corrects the bias in the estimation +of the population standard deviation. If you use a library like +_Scikit-Learn_ or _nunmpy's_ function calculate the covariance, this +quantity will be computed with a factor $1/(n-1)$. + + + The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by