lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update week 35

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-06 15:41:21 +02:00
parent 0ceb1d5a5a
commit 2951056a34
6 changed files with 109 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-reveal.html
+22 -5
View File
@@ -2243,11 +2243,14 @@ $$
<p>
<b>Important note</b>: we have defined our design matrix \( \boldsymbol{X} \) to be an
\( n\times p \) matrix. In most supervised learning cases we have that \( n
\ge p \), and quite often we have \( n >> p \). The number of columns will
\ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations).
<p>
In our lectures, the number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix.
for the definition of for example the covariance matrix and its relation to the SVD.
</section>
@@ -2258,7 +2261,9 @@ for the definition of for example the covariance matrix.
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
<p>
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary
least squares is proportional to the second derivative of the cost
function, that is we have
<p>&nbsp;<br>
$$
@@ -2266,8 +2271,10 @@ $$
$$
<p>&nbsp;<br>
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
<p>
The Hessian matrix plays an important role and is defined here as
The Hessian matrix plays an important role and is defined in this course as
<p>&nbsp;<br>
$$
@@ -2279,7 +2286,7 @@ $$
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values.
terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.
</section>
@@ -2324,6 +2331,16 @@ $$
$$
<p>&nbsp;<br>
<p>
<b>Note:</b> we have used \( 1/n \) in the above definitions of the <em>sample</em> variance and covariance. We assume then that we can calculate the exact mean value.
What you will find in essentially all statistics texts are equations
with a factor \( 1/(n-1) \). This is called <a href="https://mathworld.wolfram.com/BesselsCorrection.html" target="_blank">Bessel's correction</a>. This
method corrects the bias in the estimation of the population variance
and covariance. It also partially corrects the bias in the estimation
of the population standard deviation. If you use a library like
<b>Scikit-Learn</b> or <b>nunmpy's</b> function calculate the covariance, this
quantity will be computed with a factor \( 1/(n-1) \).
<p>
The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
doc/pub/week35/html/week35-solarized.html
+22 -5
View File
@@ -2278,11 +2278,14 @@ $$
<p>
<b>Important note</b>: we have defined our design matrix \( \boldsymbol{X} \) to be an
\( n\times p \) matrix. In most supervised learning cases we have that \( n
\ge p \), and quite often we have \( n >> p \). The number of columns will
\ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations).
<p>
In our lectures, the number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix.
for the definition of for example the covariance matrix and its relation to the SVD.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2293,14 +2296,18 @@ for the definition of for example the covariance matrix.
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
<p>
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary
least squares is proportional to the second derivative of the cost
function, that is we have
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
<p>
The Hessian matrix plays an important role and is defined here as
The Hessian matrix plays an important role and is defined in this course as
$$
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
@@ -2310,7 +2317,7 @@ $$
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values.
terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2347,6 +2354,16 @@ $$
\end{bmatrix}.
$$
<p>
<b>Note:</b> we have used \( 1/n \) in the above definitions of the <em>sample</em> variance and covariance. We assume then that we can calculate the exact mean value.
What you will find in essentially all statistics texts are equations
with a factor \( 1/(n-1) \). This is called <a href="https://mathworld.wolfram.com/BesselsCorrection.html" target="_blank">Bessel's correction</a>. This
method corrects the bias in the estimation of the population variance
and covariance. It also partially corrects the bias in the estimation
of the population standard deviation. If you use a library like
<b>Scikit-Learn</b> or <b>nunmpy's</b> function calculate the covariance, this
quantity will be computed with a factor \( 1/(n-1) \).
<p>
The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
doc/pub/week35/html/week35.html
+22 -5
View File
@@ -2283,11 +2283,14 @@ $$
<p>
<b>Important note</b>: we have defined our design matrix \( \boldsymbol{X} \) to be an
\( n\times p \) matrix. In most supervised learning cases we have that \( n
\ge p \), and quite often we have \( n >> p \). The number of columns will
\ge p \), and quite often we have \( n >> p \). For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix \( \boldsymbol{X}^T\boldsymbol{X} \) which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations).
<p>
In our lectures, the number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix.
for the definition of for example the covariance matrix and its relation to the SVD.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2298,14 +2301,18 @@ for the definition of for example the covariance matrix.
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
<p>
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary least squares is proportional to the second derivative of the cost function, that is we have
We have already noted that the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) in ordinary
least squares is proportional to the second derivative of the cost
function, that is we have
$$
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
$$
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
<p>
The Hessian matrix plays an important role and is defined here as
The Hessian matrix plays an important role and is defined in this course as
$$
\boldsymbol{H}=\boldsymbol{X}^T\boldsymbol{X}.
@@ -2315,7 +2322,7 @@ $$
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values.
terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -2352,6 +2359,16 @@ $$
\end{bmatrix}.
$$
<p>
<b>Note:</b> we have used \( 1/n \) in the above definitions of the <em>sample</em> variance and covariance. We assume then that we can calculate the exact mean value.
What you will find in essentially all statistics texts are equations
with a factor \( 1/(n-1) \). This is called <a href="https://mathworld.wolfram.com/BesselsCorrection.html" target="_blank">Bessel's correction</a>. This
method corrects the bias in the estimation of the population variance
and covariance. It also partially corrects the bias in the estimation
of the population standard deviation. If you use a library like
<b>Scikit-Learn</b> or <b>nunmpy's</b> function calculate the covariance, this
quantity will be computed with a factor \( 1/(n-1) \).
<p>
The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+22 -5
View File
@@ -2853,11 +2853,13 @@
"source": [
"**Important note**: we have defined our design matrix $\\boldsymbol{X}$ to be an\n",
"$n\\times p$ matrix. In most supervised learning cases we have that $n\n",
"\\ge p$, and quite often we have $n >> p$. The number of columns will\n",
"\\ge p$, and quite often we have $n >> p$. For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations).\n",
"\n",
"In our lectures, the number of columns will\n",
"always refer to the number of features in our data set, while the\n",
"number of rows represents the number of data inputs. Note that in\n",
"other texts you may find the opposite notation. This has consequences\n",
"for the definition of for example the covariance matrix.\n",
"for the definition of for example the covariance matrix and its relation to the SVD.\n",
"\n",
"\n",
"## Meet the Covariance Matrix\n",
@@ -2865,7 +2867,9 @@
"\n",
"Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.\n",
"\n",
"We have already noted that the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have"
"We have already noted that the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ in ordinary\n",
"least squares is proportional to the second derivative of the cost\n",
"function, that is we have"
]
},
{
@@ -2881,7 +2885,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The Hessian matrix plays an important role and is defined here as"
"This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).\n",
"\n",
"The Hessian matrix plays an important role and is defined in this course as"
]
},
{
@@ -2900,7 +2906,7 @@
"The Hessian matrix for ordinary least squares is also proportional to\n",
"the covariance matrix. This means also that we can use the SVD to find\n",
"the eigenvalues of the covariance matrix and the Hessian matrix in\n",
"terms of the singular values.\n",
"terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.\n",
"\n",
"\n",
"## Introducing the Covariance and Correlation functions\n",
@@ -2977,6 +2983,17 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"**Note:** we have used $1/n$ in the above definitions of the *sample* variance and covariance. We assume then that we can calculate the exact mean value. \n",
"What you will find in essentially all statistics texts are equations\n",
"with a factor $1/(n-1)$. This is called [Bessel's correction](https://mathworld.wolfram.com/BesselsCorrection.html). This\n",
"method corrects the bias in the estimation of the population variance\n",
"and covariance. It also partially corrects the bias in the estimation\n",
"of the population standard deviation. If you use a library like\n",
"**Scikit-Learn** or **nunmpy's** function calculate the covariance, this\n",
"quantity will be computed with a factor $1/(n-1)$.\n",
"\n",
"\n",
"\n",
"The covariance takes values between zero and infinity and may thus\n",
"lead to problems with loss of numerical precision for particularly\n",
"large values. It is common to scale the covariance matrix by\n",
doc/src/week35/week35.do.txt
+21 -6
View File
@@ -1784,11 +1784,13 @@ with eigenvalues given by the singular values squared, that is
_Important note_: we have defined our design matrix $\bm{X}$ to be an
$n\times p$ matrix. In most supervised learning cases we have that $n
\ge p$, and quite often we have $n >> p$. The number of columns will
\ge p$, and quite often we have $n >> p$. For linear algebra based methods like ordinary least squares or Ridge regression, this leads to a matrix $\bm{X}^T\bm{X}$ which is small and thereby easier to handle from a computational point of view (in terms of number of floating point operations).
In our lectures, the number of columns will
always refer to the number of features in our data set, while the
number of rows represents the number of data inputs. Note that in
other texts you may find the opposite notation. This has consequences
for the definition of for example the covariance matrix.
for the definition of for example the covariance matrix and its relation to the SVD.
!split
@@ -1797,16 +1799,18 @@ for the definition of for example the covariance matrix.
Before we move on to a discussion of Ridge and Lasso regression, we want to show an important example of the above.
We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary least squares is proportional to the second derivative of the cost function, that is we have
We have already noted that the matrix $\bm{X}^T\bm{X}$ in ordinary
least squares is proportional to the second derivative of the cost
function, that is we have
!bt
\[
\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}.
\]
!et
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
The Hessian matrix plays an important role and is defined here as
The Hessian matrix plays an important role and is defined in this course as
!bt
\[
@@ -1818,7 +1822,7 @@ The Hessian matrix plays an important role and is defined here as
The Hessian matrix for ordinary least squares is also proportional to
the covariance matrix. This means also that we can use the SVD to find
the eigenvalues of the covariance matrix and the Hessian matrix in
terms of the singular values.
terms of the singular values. Let us develop these arguments, as they will play an important role in our machine learning studies.
!split
@@ -1857,6 +1861,17 @@ we can rewrite the covariance matrix as
\]
!et
_Note:_ we have used $1/n$ in the above definitions of the *sample* variance and covariance. We assume then that we can calculate the exact mean value.
What you will find in essentially all statistics texts are equations
with a factor $1/(n-1)$. This is called "Bessel's correction":"https://mathworld.wolfram.com/BesselsCorrection.html". This
method corrects the bias in the estimation of the population variance
and covariance. It also partially corrects the bias in the estimation
of the population standard deviation. If you use a library like
_Scikit-Learn_ or _nunmpy's_ function calculate the covariance, this
quantity will be computed with a factor $1/(n-1)$.
The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
large values. It is common to scale the covariance matrix by