lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week 35

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-06 16:08:28 +02:00
parent 2951056a34
commit 28a9f6ee81
7 changed files with 375 additions and 58 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-bs.html
+24 -17
View File
@@ -204,6 +204,10 @@ Automatically generated HTML file from DocOnce source
2,
None,
'introducing-the-covariance-and-correlation-functions'),
('Covariance and Correlation Matrix',
2,
None,
'covariance-and-correlation-matrix'),
('Correlation Function and Design/Feature Matrix',
2,
None,
@@ -225,6 +229,7 @@ Automatically generated HTML file from DocOnce source
2,
None,
'rewriting-the-covariance-and-or-correlation-matrix'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('Ridge and LASSO Regression',
2,
None,
@@ -346,22 +351,24 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week35-bs051.html#further-properties-important-for-our-analyses-later" style="font-size: 80%;"><b>Further properties (important for our analyses later)</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs052.html#meet-the-covariance-matrix" style="font-size: 80%;"><b>Meet the Covariance Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs053.html#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
<!-- navigation toc: --> <li><a href="._week35-bs054.html#covariance-and-correlation-matrix" style="font-size: 80%;"><b>Covariance and Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs055.html#correlation-function-and-design-feature-matrix" style="font-size: 80%;"><b>Correlation Function and Design/Feature Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs056.html#covariance-matrix-examples" style="font-size: 80%;"><b>Covariance Matrix Examples</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs057.html#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs058.html#correlation-matrix-with-pandas" style="font-size: 80%;"><b>Correlation Matrix with Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs059.html#correlation-matrix-with-pandas-and-the-franke-function" style="font-size: 80%;"><b>Correlation Matrix with Pandas and the Franke function</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs060.html#rewriting-the-covariance-and-or-correlation-matrix" style="font-size: 80%;"><b>Rewriting the Covariance and/or Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs061.html#linking-with-the-svd" style="font-size: 80%;"><b>Linking with the SVD</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs062.html#ridge-and-lasso-regression" style="font-size: 80%;"><b>Ridge and LASSO Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs063.html#more-on-ridge-regression" style="font-size: 80%;"><b>More on Ridge Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs064.html#interpreting-the-ridge-results" style="font-size: 80%;"><b>Interpreting the Ridge results</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs065.html#more-interpretations" style="font-size: 80%;"><b>More interpretations</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs066.html#a-better-understanding-of-regularization" style="font-size: 80%;"><b>A better understanding of regularization</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs067.html#decomposing-the-ols-and-ridge-expressions" style="font-size: 80%;"><b>Decomposing the OLS and Ridge expressions</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs068.html#mathematical-properties" style="font-size: 80%;"><b>Mathematical Properties</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercises-for-week-36-september-6-10" style="font-size: 80%;"><b>Exercises for week 36, September 6-10</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercise-1-adding-ridge-and-lasso-regression" style="font-size: 80%;"><b>Exercise 1: Adding Ridge and Lasso Regression</b></a></li>
<!-- navigation toc: --> <li><a href="._week35-bs069.html#exercise-linear-regression-for-a-two-dimensional-function" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Exercise: Linear Regression for a two-dimensional function</a></li>
</ul>
</li>
@@ -420,7 +427,7 @@ MathJax.Hub.Config({
<li><a href="._week35-bs008.html">9</a></li>
<li><a href="._week35-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._week35-bs067.html">68</a></li>
<li><a href="._week35-bs069.html">70</a></li>
<li><a href="._week35-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/week35/html/week35-reveal.html
+66 -7
View File
@@ -2340,6 +2340,11 @@ and covariance. It also partially corrects the bias in the estimation
of the population standard deviation. If you use a library like
<b>Scikit-Learn</b> or <b>nunmpy's</b> function calculate the covariance, this
quantity will be computed with a factor \( 1/(n-1) \).
</section>
<section>
<h2 id="covariance-and-correlation-matrix">Covariance and Correlation Matrix </h2>
<p>
The covariance takes values between zero and infinity and may thus
@@ -2411,7 +2416,7 @@ $$
<p>
With these definitions, we can now rewrite our \( 2\times 2 \)
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
@@ -2454,14 +2459,12 @@ the exact mean values. The following simple function uses the
<b>np.vstack</b> function which takes each vector of dimension \( 1\times n \)
and produces a \( 2\times n \) matrix \( \boldsymbol{W} \)
<p>
Note that this assumes you have the features as the rows, and the inputs as columns, that is
<p>&nbsp;<br>
$$
\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\
x_1 & y_1 \\
x_2 & y_2\\
\dots & \dots \\
x_{n-2} & y_{n-2}\\
x_{n-1} & y_{n-1} &
\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\
y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\
\end{bmatrix},
$$
<p>&nbsp;<br>
@@ -2550,6 +2553,7 @@ x = np.random.normal(size=n)
x = x - np.mean(x)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.normal(size=n)
y = y - np.mean(y)
<span style="color: #228B22"># Note that we transpose the matrix in order to stay with our ordering n x p</span>
X = (np.vstack((x, y))).T
<span style="color: #658b00">print</span>(X)
Xpd = pd.DataFrame(X)
@@ -2678,6 +2682,61 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
</section>
<section>
<h2 id="linking-with-the-svd">Linking with the SVD </h2>
<p>
We saw earlier that
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
<p>&nbsp;<br>
Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
<p>&nbsp;<br>
$$
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
$$
<p>&nbsp;<br>
where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is
<p>&nbsp;<br>
$$
\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
0 & \sigma_1 & 0 & \dots & 0 & 0 \\
0 & 0 & \sigma_2 & \dots & 0 & 0 \
0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
\end{bmatrix},
$$
<p>&nbsp;<br>
meaning we can write
<p>&nbsp;<br>
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T.
$$
<p>&nbsp;<br>
Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
<p>&nbsp;<br>
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2.
$$
<p>&nbsp;<br>
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
<p>&nbsp;<br>
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
doc/pub/week35/html/week35-solarized.html
+59 -7
View File
@@ -224,6 +224,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'introducing-the-covariance-and-correlation-functions'),
('Covariance and Correlation Matrix',
2,
None,
'covariance-and-correlation-matrix'),
('Correlation Function and Design/Feature Matrix',
2,
None,
@@ -245,6 +249,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'rewriting-the-covariance-and-or-correlation-matrix'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('Ridge and LASSO Regression',
2,
None,
@@ -2364,6 +2369,11 @@ of the population standard deviation. If you use a library like
<b>Scikit-Learn</b> or <b>nunmpy's</b> function calculate the covariance, this
quantity will be computed with a factor \( 1/(n-1) \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="covariance-and-correlation-matrix">Covariance and Correlation Matrix </h2>
<p>
The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
@@ -2424,7 +2434,7 @@ $$
<p>
With these definitions, we can now rewrite our \( 2\times 2 \)
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
@@ -2463,13 +2473,11 @@ the exact mean values. The following simple function uses the
<b>np.vstack</b> function which takes each vector of dimension \( 1\times n \)
and produces a \( 2\times n \) matrix \( \boldsymbol{W} \)
<p>
Note that this assumes you have the features as the rows, and the inputs as columns, that is
$$
\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\
x_1 & y_1 \\
x_2 & y_2\\
\dots & \dots \\
x_{n-2} & y_{n-2}\\
x_{n-1} & y_{n-1} &
\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\
y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\
\end{bmatrix},
$$
@@ -2556,6 +2564,7 @@ x = np.random.normal(size=n)
x = x - np.mean(x)
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.normal(size=n)
y = y - np.mean(y)
<span style="color: #228B22"># Note that we transpose the matrix in order to stay with our ordering n x p</span>
X = (np.vstack((x, y))).T
<span style="color: #658b00">print</span>(X)
Xpd = pd.DataFrame(X)
@@ -2675,6 +2684,49 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="linking-with-the-svd">Linking with the SVD </h2>
<p>
We saw earlier that
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
$$
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
$$
where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is
$$
\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
0 & \sigma_1 & 0 & \dots & 0 & 0 \\
0 & 0 & \sigma_2 & \dots & 0 & 0 \
0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
\end{bmatrix},
$$
meaning we can write
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T.
$$
Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2.
$$
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
<p>
doc/pub/week35/html/week35.html
+59 -7
View File
@@ -229,6 +229,10 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'introducing-the-covariance-and-correlation-functions'),
('Covariance and Correlation Matrix',
2,
None,
'covariance-and-correlation-matrix'),
('Correlation Function and Design/Feature Matrix',
2,
None,
@@ -250,6 +254,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'rewriting-the-covariance-and-or-correlation-matrix'),
('Linking with the SVD', 2, None, 'linking-with-the-svd'),
('Ridge and LASSO Regression',
2,
None,
@@ -2369,6 +2374,11 @@ of the population standard deviation. If you use a library like
<b>Scikit-Learn</b> or <b>nunmpy's</b> function calculate the covariance, this
quantity will be computed with a factor \( 1/(n-1) \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="covariance-and-correlation-matrix">Covariance and Correlation Matrix </h2>
<p>
The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
@@ -2429,7 +2439,7 @@ $$
<p>
With these definitions, we can now rewrite our \( 2\times 2 \)
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
@@ -2468,13 +2478,11 @@ the exact mean values. The following simple function uses the
<b>np.vstack</b> function which takes each vector of dimension \( 1\times n \)
and produces a \( 2\times n \) matrix \( \boldsymbol{W} \)
<p>
Note that this assumes you have the features as the rows, and the inputs as columns, that is
$$
\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\
x_1 & y_1 \\
x_2 & y_2\\
\dots & \dots \\
x_{n-2} & y_{n-2}\\
x_{n-1} & y_{n-1} &
\boldsymbol{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\
y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\
\end{bmatrix},
$$
@@ -2561,6 +2569,7 @@ x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>r
x <span style="color: #666666">=</span> x <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(x)
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
y <span style="color: #666666">=</span> y <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y)
<span style="color: #408080; font-style: italic"># Note that we transpose the matrix in order to stay with our ordering n x p</span>
X <span style="color: #666666">=</span> (np<span style="color: #666666">.</span>vstack((x, y)))<span style="color: #666666">.</span>T
<span style="color: #008000">print</span>(X)
Xpd <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(X)
@@ -2680,6 +2689,49 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="linking-with-the-svd">Linking with the SVD </h2>
<p>
We saw earlier that
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
$$
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
$$
where the tilde-matrix \( \tilde{\boldsymbol{\Sigma}} \) is a matrix of dimension \( p\times p \) containing only the singular values \( \sigma_i \), that is
$$
\tilde{\boldsymbol{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
0 & \sigma_1 & 0 & \dots & 0 & 0 \\
0 & 0 & \sigma_2 & \dots & 0 & 0 \
0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
\end{bmatrix},
$$
meaning we can write
$$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2\boldsymbol{V}^T.
$$
Multiplying from the right with \( \boldsymbol{V} \) (using the orthogonality of \( \boldsymbol{V} \)) we get
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{V}=\boldsymbol{V}\tilde{\boldsymbol{\Sigma}}^2.
$$
This means the vectors \( \boldsymbol{v}_i \) of the orthogonal matrix \( \boldsymbol{V} \) are the eigenvectors of the matrix \( \boldsymbol{X}^T\boldsymbol{X} \)
with eigenvalues given by the singular values squared, that is
$$
\left(\boldsymbol{X}^T\boldsymbol{X}\right)\boldsymbol{v}_i=\boldsymbol{v}_i\sigma_i^2.
$$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="ridge-and-lasso-regression">Ridge and LASSO Regression </h2>
<p>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+111 -11
View File
@@ -2992,7 +2992,7 @@
"**Scikit-Learn** or **nunmpy's** function calculate the covariance, this\n",
"quantity will be computed with a factor $1/(n-1)$.\n",
"\n",
"\n",
"## Covariance and Correlation Matrix\n",
"\n",
"The covariance takes values between zero and infinity and may thus\n",
"lead to problems with loss of numerical precision for particularly\n",
@@ -3098,7 +3098,7 @@
"metadata": {},
"source": [
"With these definitions, we can now rewrite our $2\\times 2$\n",
"correaltion/covariance matrix in terms of a moe general design/feature\n",
"correlation/covariance matrix in terms of a moe general design/feature\n",
"matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. This leads to a $p\\times p$\n",
"covariance matrix for the vectors $\\boldsymbol{x}_i$ with $i=0,1,\\dots,p-1$"
]
@@ -3153,7 +3153,9 @@
"the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have\n",
"the exact mean values. The following simple function uses the\n",
"**np.vstack** function which takes each vector of dimension $1\\times n$\n",
"and produces a $2\\times n$ matrix $\\boldsymbol{W}$"
"and produces a $2\\times n$ matrix $\\boldsymbol{W}$\n",
"\n",
"Note that this assumes you have the features as the rows, and the inputs as columns, that is"
]
},
{
@@ -3161,12 +3163,8 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{W} = \\begin{bmatrix} x_0 & y_0 \\\\\n",
" x_1 & y_1 \\\\\n",
" x_2 & y_2\\\\\n",
" \\dots & \\dots \\\\\n",
" x_{n-2} & y_{n-2}\\\\\n",
" x_{n-1} & y_{n-1} & \n",
"\\boldsymbol{W} = \\begin{bmatrix} x_0 & x_1 & x_2 & \\dots & x_{n-2} & x_{n-1} \\\\\n",
" y_0 & y_1 & y_2 & \\dots & y_{n-2} & y_{n-1} \\\\\n",
" \\end{bmatrix},\n",
"$$"
]
@@ -3279,6 +3277,7 @@
"x = x - np.mean(x)\n",
"y = 4+3*x+np.random.normal(size=n)\n",
"y = y - np.mean(y)\n",
"# Note that we transpose the matrix in order to stay with our ordering n x p\n",
"X = (np.vstack((x, y))).T\n",
"print(X)\n",
"Xpd = pd.DataFrame(X)\n",
@@ -3445,9 +3444,110 @@
"It is easy to generalize this to a matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$.\n",
"\n",
"\n",
"## Linking with the SVD\n",
"\n",
"\n",
"\n",
"We saw earlier that"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma} = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where the tilde-matrix $\\tilde{\\boldsymbol{\\Sigma}}$ is a matrix of dimension $p\\times p$ containing only the singular values $\\sigma_i$, that is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\tilde{\\boldsymbol{\\Sigma}}=\\begin{bmatrix} \\sigma_0 & 0 & 0 & \\dots & 0 & 0 \\\\\n",
" 0 & \\sigma_1 & 0 & \\dots & 0 & 0 \\\\\n",
"\t\t\t\t 0 & 0 & \\sigma_2 & \\dots & 0 & 0 \\\n",
"\t\t\t\t 0 & 0 & 0 & \\dots & \\sigma_{p-2} & 0 \\\\\n",
"\t\t\t\t 0 & 0 & 0 & \\dots & 0 & \\sigma_{p-1} \\\\\n",
"\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"meaning we can write"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2\\boldsymbol{V}^T.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Multiplying from the right with $\\boldsymbol{V}$ (using the orthogonality of $\\boldsymbol{V}$) we get"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{V}=\\boldsymbol{V}\\tilde{\\boldsymbol{\\Sigma}}^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This means the vectors $\\boldsymbol{v}_i$ of the orthogonal matrix $\\boldsymbol{V}$ are the eigenvectors of the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$\n",
"with eigenvalues given by the singular values squared, that is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)\\boldsymbol{v}_i=\\boldsymbol{v}_i\\sigma_i^2.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Ridge and LASSO Regression\n",
"\n",
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
doc/src/week35/week35.do.txt
+56 -9
View File
@@ -1870,7 +1870,8 @@ of the population standard deviation. If you use a library like
_Scikit-Learn_ or _nunmpy's_ function calculate the covariance, this
quantity will be computed with a factor $1/(n-1)$.
!split
===== Covariance and Correlation Matrix =====
The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
@@ -1933,7 +1934,7 @@ with a given vector
!et
With these definitions, we can now rewrite our $2\times 2$
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$
covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$
@@ -1974,15 +1975,11 @@ the exact mean values. The following simple function uses the
_np.vstack_ function which takes each vector of dimension $1\times n$
and produces a $2\times n$ matrix $\bm{W}$
Note that this assumes you have the features as the rows, and the inputs as columns, that is
!bt
\[
\bm{W} = \begin{bmatrix} x_0 & y_0 \\
x_1 & y_1 \\
x_2 & y_2\\
\dots & \dots \\
x_{n-2} & y_{n-2}\\
x_{n-1} & y_{n-1} &
\bm{W} = \begin{bmatrix} x_0 & x_1 & x_2 & \dots & x_{n-2} & x_{n-1} \\
y_0 & y_1 & y_2 & \dots & y_{n-2} & y_{n-1} \\
\end{bmatrix},
\]
!et
@@ -2057,6 +2054,7 @@ x = np.random.normal(size=n)
x = x - np.mean(x)
y = 4+3*x+np.random.normal(size=n)
y = y - np.mean(y)
# Note that we transpose the matrix in order to stay with our ordering n x p
X = (np.vstack((x, y))).T
print(X)
Xpd = pd.DataFrame(X)
@@ -2171,6 +2169,55 @@ where we wrote $$\bm{C}[\bm{x}_0,\bm{x}_1] = \bm{C}[\bm{x}]$$ to indicate that t
It is easy to generalize this to a matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$.
!split
===== Linking with the SVD =====
We saw earlier that
!bt
\[
\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T.
\]
!et
Since the matrices here have dimension $p\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix
!bt
\[
\bm{\Sigma}^T\bm{\Sigma} = \begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix},
\]
!et
where the tilde-matrix $\tilde{\bm{\Sigma}}$ is a matrix of dimension $p\times p$ containing only the singular values $\sigma_i$, that is
!bt
\[
\tilde{\bm{\Sigma}}=\begin{bmatrix} \sigma_0 & 0 & 0 & \dots & 0 & 0 \\
0 & \sigma_1 & 0 & \dots & 0 & 0 \\
0 & 0 & \sigma_2 & \dots & 0 & 0 \
0 & 0 & 0 & \dots & \sigma_{p-2} & 0 \\
0 & 0 & 0 & \dots & 0 & \sigma_{p-1} \\
\end{bmatrix},
\]
!et
meaning we can write
!bt
\[
\bm{X}^T\bm{X}=\bm{V}\tilde{\bm{\Sigma}}^2\bm{V}^T.
\]
!et
Multiplying from the right with $\bm{V}$ (using the orthogonality of $\bm{V}$) we get
!bt
\[
\left(\bm{X}^T\bm{X}\right)\bm{V}=\bm{V}\tilde{\bm{\Sigma}}^2.
\]
!et
This means the vectors $\bm{v}_i$ of the orthogonal matrix $\bm{V}$ are the eigenvectors of the matrix $\bm{X}^T\bm{X}$
with eigenvalues given by the singular values squared, that is
!bt
\[
\left(\bm{X}^T\bm{X}\right)\bm{v}_i=\bm{v}_i\sigma_i^2.
\]
!et