-meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$, -the number of samples, are the eigenvalues of the covariance +meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is -self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the +self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \). @@ -2782,31 +2782,33 @@ absolute value of the eigenvalues of \( \boldsymbol{X} \).
For \( \boldsymbol{X}\boldsymbol{X}^T \) we found -$$ -$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T.
$$
+\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T.
+$$
+
Since the matrices here have dimension \( n\times n \), we have
-$$
-
-\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix},
$$
+\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix},
+$$
+
leading to
-$$
-
-$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T.
$$
+\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T.
+$$
+
Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem +
+$$
+(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}.
$$
-$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}.
-$$
It means that the eigenvalues of \( \boldsymbol{X}\boldsymbol{X}^T \) are again given by @@ -2950,7 +2952,7 @@ Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
$$
-\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
+\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}.
$$
@@ -2959,12 +2961,12 @@ For Ridge regression this becomes
$$
-\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
+\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y},
$$
-with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). +with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \). diff --git a/doc/pub/week35/html/week35-solarized.html b/doc/pub/week35/html/week35-solarized.html index 62835ff47..f4ef35128 100644 --- a/doc/pub/week35/html/week35-solarized.html +++ b/doc/pub/week35/html/week35-solarized.html @@ -2753,11 +2753,11 @@ $$ $$
-meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$, -the number of samples, are the eigenvalues of the covariance +meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is -self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the +self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \).
@@ -2769,23 +2769,23 @@ absolute value of the eigenvalues of \( \boldsymbol{X} \). For \( \boldsymbol{X}\boldsymbol{X}^T \) we found $$ -$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ Since the matrices here have dimension \( n\times n \), we have $$ -\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ leading to $$ -$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$
Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem $$ -$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. $$
@@ -2907,18 +2907,18 @@ even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). The Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as $$ -\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$
For Ridge regression this becomes $$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$
-with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). +with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).
diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html
index 6cc72bd9a..fc18d1a1d 100644
--- a/doc/pub/week35/html/week35.html
+++ b/doc/pub/week35/html/week35.html
@@ -2758,11 +2758,11 @@ $$
$$
-meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by$n$, -the number of samples, are the eigenvalues of the covariance +meaning that every squared non-singular value of \( \boldsymbol{X} \) divided by \( n \) ( +the number of samples) are the eigenvalues of the covariance matrix. Every singular value of \( \boldsymbol{X} \) is thus a positive square root of an eigenvalue of \( \boldsymbol{X}^T\boldsymbol{X} \). If the matrix \( \boldsymbol{X} \) is -self-adjoint, the the sinular values of \( \boldsymbol{X} \) are equal to the +self-adjoint, the singular values of \( \boldsymbol{X} \) are equal to the absolute value of the eigenvalues of \( \boldsymbol{X} \).
@@ -2774,23 +2774,23 @@ absolute value of the eigenvalues of \( \boldsymbol{X} \). For \( \boldsymbol{X}\boldsymbol{X}^T \) we found $$ -$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T=\boldsymbol{U}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{U}^T. $$ Since the matrices here have dimension \( n\times n \), we have $$ -\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} 0 \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, +\boldsymbol{\Sigma}\boldsymbol{\Sigma}^T = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \boldsymbol{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}, $$ leading to $$ -$\boldsymbol{X}\boldsymbol{X}^T$=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. +\boldsymbol{X}\boldsymbol{X}^T=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}\boldsymbol{U}^T. $$
Multiplying with \( \boldsymbol{U} \) from the right gives us the eigenvalue problem $$ -$\boldsymbol{X}\boldsymbol{X}^T$\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. +(\boldsymbol{X}\boldsymbol{X}^T)\boldsymbol{U}=\boldsymbol{U}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0} \\ \boldsymbol{0} & \boldsymbol{0}\\ \end{bmatrix}. $$
@@ -2912,18 +2912,18 @@ even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). The Using our insights about the SVD of the design matrix \( \boldsymbol{X} \) We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as $$ -\boldsymbol{X}\boldsymbol{\beta} = =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. +\tilde{\boldsymbol{y}}_{\mathrm{OLS}}=\boldsymbol{X}\boldsymbol{\beta} =\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}. $$
For Ridge regression this becomes $$ -\boldsymbol{X}\boldsymbol{\beta}^{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, +\tilde{\boldsymbol{y}}_{\mathrm{Ridge}}=\boldsymbol{X}\boldsymbol{\beta}_{\mathrm{Ridge}} = \boldsymbol{U\Sigma V^T}\left(\boldsymbol{V}\boldsymbol{\Sigma}^2\boldsymbol{V}^T+\lambda\boldsymbol{I} \right)^{-1}(\boldsymbol{U\Sigma V^T})^T\boldsymbol{y}=\sum_{j=0}^{p-1}\boldsymbol{u}_j\boldsymbol{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\boldsymbol{y}, $$
-with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \). +with the vectors \( \boldsymbol{u}_j \) being the columns of \( \boldsymbol{U} \) from the SVD of the matrix \( \boldsymbol{X} \).
diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
index 1f3502f6d..4e7f65506 100644
Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ
diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb
index fb01baf4e..a1a54d36b 100644
--- a/doc/pub/week35/ipynb/week35.ipynb
+++ b/doc/pub/week35/ipynb/week35.ipynb
@@ -3580,11 +3580,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by$n$,\n",
- "the number of samples, are the eigenvalues of the covariance\n",
+ "meaning that every squared non-singular value of $\\boldsymbol{X}$ divided by $n$ (\n",
+ "the number of samples) are the eigenvalues of the covariance\n",
"matrix. Every singular value of $\\boldsymbol{X}$ is thus a positive square\n",
"root of an eigenvalue of $\\boldsymbol{X}^T\\boldsymbol{X}$. If the matrix $\\boldsymbol{X}$ is\n",
- "self-adjoint, the the sinular values of $\\boldsymbol{X}$ are equal to the\n",
+ "self-adjoint, the singular values of $\\boldsymbol{X}$ are equal to the\n",
"absolute value of the eigenvalues of $\\boldsymbol{X}$.\n",
"\n",
"## And finally $\\boldsymbol{X}\\boldsymbol{X}^T$\n",
@@ -3597,7 +3597,7 @@
"metadata": {},
"source": [
"$$\n",
- "$\\boldsymbol{X}\\boldsymbol{X}^T$=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n",
+ "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T=\\boldsymbol{U}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{U}^T.\n",
"$$"
]
},
@@ -3613,7 +3613,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} 0 \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n",
+ "\\boldsymbol{\\Sigma}\\boldsymbol{\\Sigma}^T = \\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\\\ \\boldsymbol{0}\\\\ \\end{bmatrix}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} \\boldsymbol{0}\\\\ \\end{bmatrix}=\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix},\n",
"$$"
]
},
@@ -3629,7 +3629,7 @@
"metadata": {},
"source": [
"$$\n",
- "$\\boldsymbol{X}\\boldsymbol{X}^T$=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n",
+ "\\boldsymbol{X}\\boldsymbol{X}^T=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}\\boldsymbol{U}^T.\n",
"$$"
]
},
@@ -3645,7 +3645,7 @@
"metadata": {},
"source": [
"$$\n",
- "$\\boldsymbol{X}\\boldsymbol{X}^T$\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n",
+ "(\\boldsymbol{X}\\boldsymbol{X}^T)\\boldsymbol{U}=\\boldsymbol{U}\\begin{bmatrix} \\tilde{\\boldsymbol{\\Sigma}} & \\boldsymbol{0} \\\\ \\boldsymbol{0} & \\boldsymbol{0}\\\\ \\end{bmatrix}.\n",
"$$"
]
},
@@ -3877,7 +3877,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\boldsymbol{X}\\boldsymbol{\\beta} = =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
+ "\\tilde{\\boldsymbol{y}}_{\\mathrm{OLS}}=\\boldsymbol{X}\\boldsymbol{\\beta} =\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}.\n",
"$$"
]
},
@@ -3893,7 +3893,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\boldsymbol{X}\\boldsymbol{\\beta}^{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
+ "\\tilde{\\boldsymbol{y}}_{\\mathrm{Ridge}}=\\boldsymbol{X}\\boldsymbol{\\beta}_{\\mathrm{Ridge}} = \\boldsymbol{U\\Sigma V^T}\\left(\\boldsymbol{V}\\boldsymbol{\\Sigma}^2\\boldsymbol{V}^T+\\lambda\\boldsymbol{I} \\right)^{-1}(\\boldsymbol{U\\Sigma V^T})^T\\boldsymbol{y}=\\sum_{j=0}^{p-1}\\boldsymbol{u}_j\\boldsymbol{u}_j^T\\frac{\\sigma_j^2}{\\sigma_j^2+\\lambda}\\boldsymbol{y},\n",
"$$"
]
},
@@ -3901,7 +3901,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$. \n",
+ "with the vectors $\\boldsymbol{u}_j$ being the columns of $\\boldsymbol{U}$ from the SVD of the matrix $\\boldsymbol{X}$. \n",
"\n",
"## Interpreting the Ridge results\n",
"\n",
diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt
index 27a8ce41b..8f168817a 100644
--- a/doc/src/week35/week35.do.txt
+++ b/doc/src/week35/week35.do.txt
@@ -2244,11 +2244,11 @@ Bessel's correction) we have
\]
!et
-meaning that every squared non-singular value of $\bm{X}$ divided by$n$,
-the number of samples, are the eigenvalues of the covariance
+meaning that every squared non-singular value of $\bm{X}$ divided by $n$ (
+the number of samples) are the eigenvalues of the covariance
matrix. Every singular value of $\bm{X}$ is thus a positive square
root of an eigenvalue of $\bm{X}^T\bm{X}$. If the matrix $\bm{X}$ is
-self-adjoint, the the sinular values of $\bm{X}$ are equal to the
+self-adjoint, the singular values of $\bm{X}$ are equal to the
absolute value of the eigenvalues of $\bm{X}$.
!split
@@ -2258,26 +2258,26 @@ For $\bm{X}\bm{X}^T$ we found
!bt
\[
-$\bm{X}\bm{X}^T$=\bm{U}\bm{\Sigma}\bm{V}^T\bm{V}\bm{\Sigma}^T\bm{U}^T=\bm{U}\bm{\Sigma}^T\bm{\Sigma}\bm{U}^T.
+\bm{X}\bm{X}^T=\bm{U}\bm{\Sigma}\bm{V}^T\bm{V}\bm{\Sigma}^T\bm{U}^T=\bm{U}\bm{\Sigma}^T\bm{\Sigma}\bm{U}^T.
\]
!et
Since the matrices here have dimension $n\times n$, we have
!bt
\[
-\bm{\Sigma}\bm{\Sigma}^T = \begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} 0 \bm{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix},
+\bm{\Sigma}\bm{\Sigma}^T = \begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} \bm{0}\\ \end{bmatrix}=\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix},
\]
!et
leading to
!bt
\[
-$\bm{X}\bm{X}^T$=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}\bm{U}^T.
+\bm{X}\bm{X}^T=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}\bm{U}^T.
\]
!et
Multiplying with $\bm{U}$ from the right gives us the eigenvalue problem
!bt
\[
-$\bm{X}\bm{X}^T$\bm{U}=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}.
+(\bm{X}\bm{X}^T)\bm{U}=\bm{U}\begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0} \\ \bm{0} & \bm{0}\\ \end{bmatrix}.
\]
!et
@@ -2404,7 +2404,7 @@ Using our insights about the SVD of the design matrix $\bm{X}$
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
!bt
\[
-\bm{X}\bm{\beta} = =\bm{U}\bm{U}^T\bm{y}.
+\tilde{\bm{y}}_{\mathrm{OLS}}=\bm{X}\bm{\beta} =\bm{U}\bm{U}^T\bm{y}.
\]
!et
@@ -2413,11 +2413,11 @@ For Ridge regression this becomes
!bt
\[
-\bm{X}\bm{\beta}^{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
+\tilde{\bm{y}}_{\mathrm{Ridge}}=\bm{X}\bm{\beta}_{\mathrm{Ridge}} = \bm{U\Sigma V^T}\left(\bm{V}\bm{\Sigma}^2\bm{V}^T+\lambda\bm{I} \right)^{-1}(\bm{U\Sigma V^T})^T\bm{y}=\sum_{j=0}^{p-1}\bm{u}_j\bm{u}_j^T\frac{\sigma_j^2}{\sigma_j^2+\lambda}\bm{y},
\]
!et
-with the vectors $\bm{u}_j$ being the columns of $\bm{U}$.
+with the vectors $\bm{u}_j$ being the columns of $\bm{U}$ from the SVD of the matrix $\bm{X}$.
!split
===== Interpreting the Ridge results =====