update of week 35
This commit is contained in:
Morten Hjorth-Jensen
@@ -183,10 +183,10 @@ Automatically generated HTML file from DocOnce source
|
||||
None,
|
||||
'note-about-svd-calculations'),
|
||||
('Friday September 3', 2, None, 'friday-september-3'),
|
||||
('Matheamtics of the SVD and implications',
|
||||
('Mathematics of the SVD and implications',
|
||||
2,
|
||||
None,
|
||||
'matheamtics-of-the-svd-and-implications'),
|
||||
'mathematics-of-the-svd-and-implications'),
|
||||
('Example Matrix', 2, None, 'example-matrix'),
|
||||
('Setting up the Matrix to be inverted',
|
||||
2,
|
||||
@@ -350,7 +350,7 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._week35-bs045.html#codes-for-the-svd" style="font-size: 80%;"><b>Codes for the SVD</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week35-bs046.html#note-about-svd-calculations" style="font-size: 80%;"><b>Note about SVD Calculations</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week35-bs047.html#friday-september-3" style="font-size: 80%;"><b>Friday September 3</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week35-bs048.html#matheamtics-of-the-svd-and-implications" style="font-size: 80%;"><b>Matheamtics of the SVD and implications</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week35-bs048.html#mathematics-of-the-svd-and-implications" style="font-size: 80%;"><b>Mathematics of the SVD and implications</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week35-bs049.html#example-matrix" style="font-size: 80%;"><b>Example Matrix</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week35-bs050.html#setting-up-the-matrix-to-be-inverted" style="font-size: 80%;"><b>Setting up the Matrix to be inverted</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._week35-bs051.html#further-properties-important-for-our-analyses-later" style="font-size: 80%;"><b>Further properties (important for our analyses later)</b></a></li>
|
||||
|
||||
@@ -1615,7 +1615,7 @@ We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being re
|
||||
We have defined the residual error as
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=(\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)\boldsymbol{y}.
|
||||
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -1998,7 +1998,7 @@ More material will be added here, see handwritten notes also. Note that this m
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="matheamtics-of-the-svd-and-implications">Matheamtics of the SVD and implications </h2>
|
||||
<h2 id="mathematics-of-the-svd-and-implications">Mathematics of the SVD and implications </h2>
|
||||
|
||||
<p>
|
||||
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
|
||||
@@ -2007,14 +2007,14 @@ Let us take a closer look at the mathematics of the SVD and the various implicat
|
||||
Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{X}=
|
||||
\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
|
||||
\dots& \dots &\dots & \dots & \dots &\dots\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
|
||||
\end{bmatrix}
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
|
||||
\dots & \dots & \dots & \dots \dots & \dots \\
|
||||
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
|
||||
\end{bmatrix}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2138,7 +2138,7 @@ $$
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \).
|
||||
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \).
|
||||
|
||||
<p>
|
||||
This means, using the orthogonality of \( \boldsymbol{V} \), that we get
|
||||
@@ -2267,7 +2267,7 @@ function, that is we have
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -2706,7 +2706,7 @@ where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimensio
|
||||
$$
|
||||
\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
|
||||
0 & \sigma_1 & 0 & \dots & 0 & 0 \\
|
||||
0 & 0 & \sigma_2 & \dots & 0 & 0 \
|
||||
0 & 0 & \sigma_2 & \dots & 0 & 0 \\
|
||||
0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
|
||||
0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
|
||||
\end{bmatrix},
|
||||
|
||||
@@ -203,10 +203,10 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
None,
|
||||
'note-about-svd-calculations'),
|
||||
('Friday September 3', 2, None, 'friday-september-3'),
|
||||
('Matheamtics of the SVD and implications',
|
||||
('Mathematics of the SVD and implications',
|
||||
2,
|
||||
None,
|
||||
'matheamtics-of-the-svd-and-implications'),
|
||||
'mathematics-of-the-svd-and-implications'),
|
||||
('Example Matrix', 2, None, 'example-matrix'),
|
||||
('Setting up the Matrix to be inverted',
|
||||
2,
|
||||
@@ -1726,7 +1726,7 @@ We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being re
|
||||
<p>
|
||||
We have defined the residual error as
|
||||
$$
|
||||
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=(\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)\boldsymbol{y}.
|
||||
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -2085,7 +2085,7 @@ More material will be added here, see handwritten notes also. Note that this m
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="matheamtics-of-the-svd-and-implications">Matheamtics of the SVD and implications </h2>
|
||||
<h2 id="mathematics-of-the-svd-and-implications">Mathematics of the SVD and implications </h2>
|
||||
|
||||
<p>
|
||||
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
|
||||
@@ -2093,14 +2093,14 @@ Let us take a closer look at the mathematics of the SVD and the various implicat
|
||||
<p>
|
||||
Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
|
||||
$$
|
||||
\boldsymbol{X}=
|
||||
\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
|
||||
\dots& \dots &\dots & \dots & \dots &\dots\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
|
||||
\end{bmatrix}
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
|
||||
\dots & \dots & \dots & \dots \dots & \dots \\
|
||||
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
|
||||
\end{bmatrix}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -2205,7 +2205,7 @@ $$
|
||||
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
|
||||
$$
|
||||
|
||||
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \).
|
||||
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \).
|
||||
|
||||
<p>
|
||||
This means, using the orthogonality of \( \boldsymbol{V} \), that we get
|
||||
@@ -2311,7 +2311,7 @@ least squares is proportional to the second derivative of the cost
|
||||
function, that is we have
|
||||
|
||||
$$
|
||||
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
|
||||
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
|
||||
@@ -2707,7 +2707,7 @@ where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimensio
|
||||
$$
|
||||
\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
|
||||
0 & \sigma_1 & 0 & \dots & 0 & 0 \\
|
||||
0 & 0 & \sigma_2 & \dots & 0 & 0 \
|
||||
0 & 0 & \sigma_2 & \dots & 0 & 0 \\
|
||||
0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
|
||||
0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
|
||||
\end{bmatrix},
|
||||
|
||||
@@ -208,10 +208,10 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
None,
|
||||
'note-about-svd-calculations'),
|
||||
('Friday September 3', 2, None, 'friday-september-3'),
|
||||
('Matheamtics of the SVD and implications',
|
||||
('Mathematics of the SVD and implications',
|
||||
2,
|
||||
None,
|
||||
'matheamtics-of-the-svd-and-implications'),
|
||||
'mathematics-of-the-svd-and-implications'),
|
||||
('Example Matrix', 2, None, 'example-matrix'),
|
||||
('Setting up the Matrix to be inverted',
|
||||
2,
|
||||
@@ -1731,7 +1731,7 @@ We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being re
|
||||
<p>
|
||||
We have defined the residual error as
|
||||
$$
|
||||
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=(\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)\boldsymbol{y}.
|
||||
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=\left[\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\right]\boldsymbol{y}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -2090,7 +2090,7 @@ More material will be added here, see handwritten notes also. Note that this m
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="matheamtics-of-the-svd-and-implications">Matheamtics of the SVD and implications </h2>
|
||||
<h2 id="mathematics-of-the-svd-and-implications">Mathematics of the SVD and implications </h2>
|
||||
|
||||
<p>
|
||||
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
|
||||
@@ -2098,14 +2098,14 @@ Let us take a closer look at the mathematics of the SVD and the various implicat
|
||||
<p>
|
||||
Our starting point is our design matrix \( \boldsymbol{X} \) of dimension \( n\times p \)
|
||||
$$
|
||||
\boldsymbol{X}=
|
||||
\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
|
||||
\dots& \dots &\dots & \dots & \dots &\dots\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
|
||||
\end{bmatrix}
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
|
||||
\dots & \dots & \dots & \dots \dots & \dots \\
|
||||
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
|
||||
\end{bmatrix}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
@@ -2210,7 +2210,7 @@ $$
|
||||
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
|
||||
$$
|
||||
|
||||
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \).
|
||||
We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( p \times p \).
|
||||
|
||||
<p>
|
||||
This means, using the orthogonality of \( \boldsymbol{V} \), that we get
|
||||
@@ -2316,7 +2316,7 @@ least squares is proportional to the second derivative of the cost
|
||||
function, that is we have
|
||||
|
||||
$$
|
||||
\frac{\partial}{\partial \boldsymbol{\beta}^T}\frac{\partial C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
\frac{\partial^2 C(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}^T\partial \boldsymbol{\beta}} =\frac{2}{n}\boldsymbol{X}^T\boldsymbol{X}.
|
||||
$$
|
||||
|
||||
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
|
||||
@@ -2712,7 +2712,7 @@ where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimensio
|
||||
$$
|
||||
\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
|
||||
0 & \sigma_1 & 0 & \dots & 0 & 0 \\
|
||||
0 & 0 & \sigma_2 & \dots & 0 & 0 \
|
||||
0 & 0 & \sigma_2 & \dots & 0 & 0 \\
|
||||
0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
|
||||
0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
|
||||
\end{bmatrix},
|
||||
|
||||
Binary file not shown.
@@ -1992,7 +1992,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=(\\boldsymbol{I}-\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T)\\boldsymbol{y}.\n",
|
||||
"\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=\\left[\\boldsymbol{I}-\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\right]\\boldsymbol{y}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -2457,7 +2457,7 @@
|
||||
"More material will be added here, see handwritten notes also. Note that this material will be cleaned up after the lecture of Friday September 3. See the handwritten notes from Friday's lecture at <https://github.com/CompPhysics/MachineLearning/tree/master/doc/HandWrittenNotes/2021>.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Matheamtics of the SVD and implications\n",
|
||||
"## Mathematics of the SVD and implications\n",
|
||||
"\n",
|
||||
"Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.\n",
|
||||
"\n",
|
||||
@@ -2469,14 +2469,14 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{X}=\n",
|
||||
"\\begin{bmatrix}\n",
|
||||
"x_{0,0} & x_{0,1} & x_{0,2}& \\dots & \\dots & x_{0,p-1}\\\\\n",
|
||||
"x_{1,0} & x_{1,1} & x_{1,2& \\dots & \\dots & x_{1,p-1}\\\\\n",
|
||||
"x_{2,0} & x_{2,1} & x_{2,2}& \\dots & \\dots & x_{2,p-1}\\\\\n",
|
||||
"\\dots& \\dots &\\dots & \\dots & \\dots &\\dots\\\\\n",
|
||||
"x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \\dots & \\dots & x_{n-1,p-1}\\\\\n",
|
||||
"\\end{bmatrix}\n",
|
||||
"\\boldsymbol{X}=\\begin{bmatrix}\n",
|
||||
"x_{0,0} & x_{0,1} & x_{0,2}& \\dots & \\dots x_{0,p-1}\\\\\n",
|
||||
"x_{1,0} & x_{1,1} & x_{1,2}& \\dots & \\dots x_{1,p-1}\\\\\n",
|
||||
"x_{2,0} & x_{2,1} & x_{2,2}& \\dots & \\dots x_{2,p-1}\\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots \\dots & \\dots \\\\\n",
|
||||
"x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \\dots & \\dots x_{n-2,p-1}\\\\\n",
|
||||
"x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \\dots & \\dots x_{n-1,p-1}\\\\\n",
|
||||
"\\end{bmatrix}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -2665,7 +2665,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We define $\\boldsymbol{\\Sigma}^T\\boldsymbol{\\sigma}=\\boldsymbol{D}^2, where $\\boldsymbol{D}$ is a diagonal matrix containing only the singular values squared. It has dimensionality $\\p \\times p$.\n",
|
||||
"We define $\\boldsymbol{\\Sigma}^T\\boldsymbol{\\sigma}=\\boldsymbol{D}^2, where $\\boldsymbol{D}$ is a diagonal matrix containing only the singular values squared. It has dimensionality $p \\times p$.\n",
|
||||
"\n",
|
||||
"This means, using the orthogonality of $\\boldsymbol{V}$, that we get"
|
||||
]
|
||||
@@ -2877,7 +2877,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\frac{\\partial}{\\partial \\boldsymbol{\\beta}^T}\\frac{\\partial C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
|
||||
"\\frac{\\partial^2 C(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}^T\\partial \\boldsymbol{\\beta}} =\\frac{2}{n}\\boldsymbol{X}^T\\boldsymbol{X}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -3488,7 +3488,7 @@
|
||||
"$$\n",
|
||||
"\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n",
|
||||
" 0 & \\sigma_1 & 0 & \\dots & 0 & 0 \\\\\n",
|
||||
"\t\t\t\t 0 & 0 & \\sigma_2 & \\dots & 0 & 0 \\\n",
|
||||
"\t\t\t\t 0 & 0 & \\sigma_2 & \\dots & 0 & 0 \\\\\n",
|
||||
"\t\t\t\t 0 & 0 & 0 & \\dots & \\sigma_{p-2} & 0 \\\\\n",
|
||||
"\t\t\t\t 0 & 0 & 0 & \\dots & 0 & \\sigma_{p-1} \\\\\n",
|
||||
"\\end{bmatrix},\n",
|
||||
|
||||
@@ -1242,7 +1242,7 @@ We can then interpret our optimal model $\tilde{\bm{y}}$ as being represented b
|
||||
We have defined the residual error as
|
||||
!bt
|
||||
\[
|
||||
\bm{\epsilon}=\bm{y}-\tilde{\bm{y}}=(\bm{I}-\bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T)\bm{y}.
|
||||
\bm{\epsilon}=\bm{y}-\tilde{\bm{y}}=\left[\bm{I}-\bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\right]\bm{y}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -1574,21 +1574,21 @@ More material will be added here, see handwritten notes also. Note that this m
|
||||
|
||||
|
||||
!split
|
||||
===== Matheamtics of the SVD and implications =====
|
||||
===== Mathematics of the SVD and implications =====
|
||||
|
||||
Let us take a closer look at the mathematics of the SVD and the various implications for machine learning studies.
|
||||
|
||||
Our starting point is our design matrix $\bm{X}$ of dimension $n\times p$
|
||||
!bt
|
||||
\[
|
||||
\bm{X}=
|
||||
\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots & x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2& \dots & \dots & x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots & x_{2,p-1}\\
|
||||
\dots& \dots &\dots & \dots & \dots &\dots\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2} & \dots & \dots & x_{n-1,p-1}\\
|
||||
\end{bmatrix}
|
||||
\bm{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
|
||||
\dots & \dots & \dots & \dots \dots & \dots \\
|
||||
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
|
||||
\end{bmatrix}.
|
||||
\]
|
||||
!et
|
||||
|
||||
@@ -1695,7 +1695,7 @@ and using the orthogonality of the matrix $\bm{U}$ we have
|
||||
\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T.
|
||||
\]
|
||||
!et
|
||||
We define $\bm{\Sigma}^T\bm{\sigma}=\bm{D}^2, where $\bm{D}$ is a diagonal matrix containing only the singular values squared. It has dimensionality $\p \times p$.
|
||||
We define $\bm{\Sigma}^T\bm{\sigma}=\bm{D}^2, where $\bm{D}$ is a diagonal matrix containing only the singular values squared. It has dimensionality $p \times p$.
|
||||
|
||||
This means, using the orthogonality of $\bm{V}$, that we get
|
||||
|
||||
@@ -1805,7 +1805,7 @@ function, that is we have
|
||||
|
||||
!bt
|
||||
\[
|
||||
\frac{\partial}{\partial \bm{\beta}^T}\frac{\partial C(\bm{\beta})}{\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}.
|
||||
\frac{\partial^2 C(\bm{\beta})}{\partial \bm{\beta}^T\partial \bm{\beta}} =\frac{2}{n}\bm{X}^T\bm{X}.
|
||||
\]
|
||||
!et
|
||||
This quantity defines was what is called the Hessian matrix (the second derivative of a function we want to optimize).
|
||||
@@ -2190,7 +2190,7 @@ where the tilde-matrix $\tilde{\bm{\Sigma}}$ is a matrix of dimension $p\times p
|
||||
\[
|
||||
\tilde{\bm{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
|
||||
0 & \sigma_1 & 0 & \dots & 0 & 0 \\
|
||||
0 & 0 & \sigma_2 & \dots & 0 & 0 \
|
||||
0 & 0 & \sigma_2 & \dots & 0 & 0 \\
|
||||
0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
|
||||
0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
|
||||
\end{bmatrix},
|
||||
|
||||
Reference in New Issue
Block a user