-Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
+Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
@@ -569,7 +574,7 @@ $$
and reordering we have
-Our model approximation is just \( \tild{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))
+Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))
+
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
@@ -599,7 +605,7 @@ $$
and reordering we have
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
@@ -614,7 +620,7 @@ Let us assume that our design matrix is given by unit (identity) matrix, that is
diagonal. In this case we have an equal number of rows and columns \( n=p \).
-Our model approximation is just \( \tild{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))
+Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))
$$
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
$$
@@ -660,7 +666,7 @@ which leads to
$$
\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\
- 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\
+ 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
$$
diff --git a/doc/pub/week36/html/week36.html b/doc/pub/week36/html/week36.html
index 5d9cdcfc9..b129c6975 100644
--- a/doc/pub/week36/html/week36.html
+++ b/doc/pub/week36/html/week36.html
@@ -93,6 +93,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'deriving-the-ridge-regression-equations'),
+ ('SVD analysis', 2, None, 'svd-analysis'),
('Interpreting the Ridge results',
2,
None,
@@ -285,7 +286,7 @@ $$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
-Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
+Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix
$$
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
$$
@@ -502,6 +503,11 @@ particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+
+
Using our insights about the SVD of the design matrix \( \boldsymbol{X} \)
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix \( \boldsymbol{U} \) as
@@ -604,7 +610,7 @@ $$
and reordering we have
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package CVXOPT. We will discuss this later.
@@ -619,7 +625,7 @@ Let us assume that our design matrix is given by unit (identity) matrix, that is
diagonal. In this case we have an equal number of rows and columns \( n=p \).
-Our model approximation is just \( \tild{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))
+Our model approximation is just \( \tilde{\boldsymbol{y}}=\boldsymbol{\beta} \) and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term \( 1/n \))
$$
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
$$
@@ -665,7 +671,7 @@ which leads to
$$
\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\
- 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\
+ 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
$$
diff --git a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
index 141346627..ed6bbb6e8 100644
Binary files a/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz and b/doc/pub/week36/ipynb/ipynb-week36-src.tar.gz differ
diff --git a/doc/pub/week36/ipynb/week36.ipynb b/doc/pub/week36/ipynb/week36.ipynb
index f2bf4eaaa..6008a4b74 100644
--- a/doc/pub/week36/ipynb/week36.ipynb
+++ b/doc/pub/week36/ipynb/week36.ipynb
@@ -46,7 +46,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix"
+ "Since the matrices here have dimension $p\\times p$, with $p$ corresponding to the singular values, we defined last week the matrix"
]
},
{
@@ -455,6 +455,8 @@
"interesting. We will see that for specific values of $\\lambda$, we may\n",
"even reduce the variance of the optimal parameters $\\boldsymbol{\\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.\n",
"\n",
+ "## SVD analysis\n",
+ "\n",
"Using our insights about the SVD of the design matrix $\\boldsymbol{X}$ \n",
"We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\\boldsymbol{U}$ as"
]
@@ -634,7 +636,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta})+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T(\\boldsymbol{y}.\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}\\boldsymbol{\\beta}+\\lambda sgn(\\boldsymbol{\\beta})=2\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
@@ -652,7 +654,7 @@
"Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the\n",
"diagonal. In this case we have an equal number of rows and columns $n=p$.\n",
"\n",
- "Our model approximation is just $\\tild{\\boldsymbol{y}}=\\boldsymbol{\\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$)"
+ "Our model approximation is just $\\tilde{\\boldsymbol{y}}=\\boldsymbol{\\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$)"
]
},
{
@@ -762,7 +764,7 @@
"$$\n",
"\\hat{\\boldsymbol{\\beta}}_i^{\\mathrm{Lasso}} = \\left\\{\\begin{array}{ccc}y_i-\\frac{\\lambda}{2} &\\mathrm{if} & y_i> \\frac{\\lambda}{2}\\\\\n",
" y_i+\\frac{\\lambda}{2} &\\mathrm{if} & y_i< -\\frac{\\lambda}{2}\\\\\n",
- "\t\t\t\t\t\t\t 0 &\\mathrm{if} & \\verty_i\\vert\\le \\frac{\\lambda}{2}\\\\\n",
+ "\t\t\t\t\t\t\t 0 &\\mathrm{if} & \\verty_i\\vert\\le \\frac{\\lambda}{2}\\end{array}\\right.\\\\.\n",
"$$"
]
},
diff --git a/doc/src/week36/week36.do.txt b/doc/src/week36/week36.do.txt
index 4299321aa..826c4575f 100644
--- a/doc/src/week36/week36.do.txt
+++ b/doc/src/week36/week36.do.txt
@@ -26,7 +26,7 @@ We used the SVD to analyse the matrix to invert in ordinary lineat regression
\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T.
\]
!et
-Since the matrices here have dimension $p\times p$, with $p$ corresponding to the singular values, we defined earlier the matrix
+Since the matrices here have dimension $p\times p$, with $p$ corresponding to the singular values, we defined last week the matrix
!bt
\[
\bm{\Sigma}^T\bm{\Sigma} = \begin{bmatrix} \tilde{\bm{\Sigma}} & \bm{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\bm{\Sigma}} \\ \bm{0}\\ \end{bmatrix},
@@ -251,6 +251,9 @@ particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of $\lambda$, we may
even reduce the variance of the optimal parameters $\bm{\beta}$. These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+!split
+===== SVD analysis =====
+
Using our insights about the SVD of the design matrix $\bm{X}$
We have already analyzed the OLS solutions in terms of the eigenvectors (the columns) of the right singular value matrix $\bm{U}$ as
!bt
@@ -352,7 +355,7 @@ we have that the derivative of the cost function is
and reordering we have
!bt
\[
-\bm{X}^T\bm{X}\bm{\beta})+\lambda sgn(\bm{\beta})=2\bm{X}^T(\bm{y}.
+\bm{X}^T\bm{X}\bm{\beta}+\lambda sgn(\bm{\beta})=2\bm{X}^T\bm{y}.
\]
!et
This equation does not lead to a nice analytical equation as in Ridge regression or ordinary least squares. This equation can however be solved by using standard convex optimization algorithms using for example the Python package "CVXOPT":"https://cvxopt.org/". We will discuss this later.
@@ -366,7 +369,7 @@ This equation does not lead to a nice analytical equation as in Ridge regression
Let us assume that our design matrix is given by unit (identity) matrix, that is a square diagonal matrix with ones only along the
diagonal. In this case we have an equal number of rows and columns $n=p$.
-Our model approximation is just $\tild{\bm{y}}=\bm{\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$)
+Our model approximation is just $\tilde{\bm{y}}=\bm{\beta}$ and the mean squared error and thereby the cost function for ordinary least sqquares (OLS) is then (we drop the term $1/n$)
!bt
\[
C(\bm{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
@@ -416,7 +419,7 @@ which leads to
\[
\hat{\bm{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
y_i+\frac{\lambda}{2} &\mathrm{if} & y_i< -\frac{\lambda}{2}\\
- 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\
+ 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
\]
!et
$$
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
@@ -447,6 +447,11 @@ modified diagonal term added to \( \boldsymbol{X}^T\boldsymbol{X} \). The conseq
particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+
+
+
+SVD analysis
$$
-\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta})+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T(\boldsymbol{y}.
+\boldsymbol{X}^T\boldsymbol{X}\boldsymbol{\beta}+\lambda sgn(\boldsymbol{\beta})=2\boldsymbol{X}^T\boldsymbol{y}.
$$
@@ -585,7 +590,7 @@ Let us assume that our design matrix is given by unit (identity) matrix, that is
diagonal. In this case we have an equal number of rows and columns \( n=p \).
$$
C(\boldsymbol{\beta})=\sum_{i=0}^{p-1}(y_i-\beta_i)^2,
@@ -644,7 +649,7 @@ which leads to
$$
\hat{\boldsymbol{\beta}}_i^{\mathrm{Lasso}} = \left\{\begin{array}{ccc}y_i-\frac{\lambda}{2} &\mathrm{if} & y_i> \frac{\lambda}{2}\\
y_i+\frac{\lambda}{2} &\mathrm{if} & y_i < -\frac{\lambda}{2}\\
- 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\\
+ 0 &\mathrm{if} & \verty_i\vert\le \frac{\lambda}{2}\end{array}\right.\\.
$$
diff --git a/doc/pub/week36/html/week36-solarized.html b/doc/pub/week36/html/week36-solarized.html
index 1e86b8369..f60448f4d 100644
--- a/doc/pub/week36/html/week36-solarized.html
+++ b/doc/pub/week36/html/week36-solarized.html
@@ -88,6 +88,7 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'deriving-the-ridge-regression-equations'),
+ ('SVD analysis', 2, None, 'svd-analysis'),
('Interpreting the Ridge results',
2,
None,
@@ -280,7 +281,7 @@ $$
\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
$$
-Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined earlier the matrix
+Since the matrices here have dimension \( p\times p \), with \( p \) corresponding to the singular values, we defined last week the matrix
$$
\boldsymbol{\Sigma}^T\boldsymbol{\Sigma} = \begin{bmatrix} \tilde{\boldsymbol{\Sigma}} & \boldsymbol{0}\\ \end{bmatrix}\begin{bmatrix} \tilde{\boldsymbol{\Sigma}} \\ \boldsymbol{0}\\ \end{bmatrix},
$$
@@ -497,6 +498,11 @@ particular for our discussion of the bias-variance tradeoff are rather
interesting. We will see that for specific values of \( \lambda \), we may
even reduce the variance of the optimal parameters \( \boldsymbol{\beta} \). These topics and other related ones, will be discussed after the more linear algebra oriented analysis here.
+
+
+SVD analysis
+
+
+SVD analysis
+