-is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
+
+is a \( 3\times 3 \) matrix. The last row and column of this last matrix
+contain only zeros. This will have important consequences for our SVD
+decomposition of the design matrix.
+
+
+
+
+The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as
+
+
+This means, using the orthogonality of \( \boldsymbol{V} \), that we get
+
+
+We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where
+
+
+Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \)
+
+It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).
Setting up the Matrix to be inverted
+
+
+$$
+\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T,
+$$
+
+
+and using the orthogonality of the matrix \( \boldsymbol{U} \) we have
+
+
+$$
+\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T.
+$$
+
+
+We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \).
+
+
+$$
+\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{D}^2.
+$$
+
+
+
+$$
+\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y},
+$$
+
+
+and using our SVD decomposition of \( \boldsymbol{X} \) we have
+
+
+$$
+\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{D}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y},
+$$
+
+
+which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \),
+
+
+$$
+\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y},
+$$
+
+
+
+$$
+\boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}],
+$$
+
+
+that belong to \( i>p-1 \), give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero).
+
+
+is a \( 3\times 3 \) matrix. The last row and column of this last matrix +contain only zeros. This will have important consequences for our SVD +decomposition of the design matrix. + +
+
+
+
+The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as + +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +and using the orthogonality of the matrix \( \boldsymbol{U} \) we have + +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \). + +
+This means, using the orthogonality of \( \boldsymbol{V} \), that we get + +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{D}^2. +$$ + +
+We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where + +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +and using our SVD decomposition of \( \boldsymbol{X} \) we have + +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{D}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, +$$ + +which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), + +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y}, +$$ + +
+Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \) +$$ +\boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}], +$$ + +that belong to \( i>p-1 \), give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero). + +
+It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).
diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html
index 3d05a9b45..fd3b8ab8f 100644
--- a/doc/pub/week35/html/week35.html
+++ b/doc/pub/week35/html/week35.html
@@ -213,6 +213,10 @@ div { text-align: justify; text-justify: inter-word; }
None,
'matheamtics-of-the-svd-and-implications'),
('Example Matrix', 2, None, 'example-matrix'),
+ ('Setting up the Matrix to be inverted',
+ 2,
+ None,
+ 'setting-up-the-matrix-to-be-inverted'),
('Ridge and LASSO Regression',
2,
None,
@@ -2157,7 +2161,67 @@ $$
\end{bmatrix},
$$
-is a \( 3\times 3 \) matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
+
+is a \( 3\times 3 \) matrix. The last row and column of this last matrix +contain only zeros. This will have important consequences for our SVD +decomposition of the design matrix. + +
+
+
+
+The matrix that may cause problems for us is \( \boldsymbol{X}^T\boldsymbol{X} \). Using the SVD we can rewrite this matrix as + +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T, +$$ + +and using the orthogonality of the matrix \( \boldsymbol{U} \) we have + +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{\Sigma}\boldsymbol{V}^T. +$$ + +We define $\boldsymbol{\Sigma}^T\boldsymbol{\sigma}=\boldsymbol{D}^2, where \( \boldsymbol{D} \) is a diagonal matrix containing only the singular values squared. It has dimensionality \( \p \times p \). + +
+This means, using the orthogonality of \( \boldsymbol{V} \), that we get + +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{D}^2. +$$ + +
+We can now insert the result for the matrix \( \boldsymbol{X}^T\boldsymbol{X} \) into our equation for ordinary least squares where + +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}, +$$ + +and using our SVD decomposition of \( \boldsymbol{X} \) we have + +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{\Sigma}\boldsymbol{V}^T\boldsymbol{D}^{-2}\boldsymbol{V}\boldsymbol{\Sigma}^T\boldsymbol{U}^T\boldsymbol{y}, +$$ + +which gives us, using the orthogonality of the matrices \( \boldsymbol{U} \) and \( \boldsymbol{V} \), + +$$ +\tilde{y}_{\mathrm{OLS}}=\boldsymbol{U}\boldsymbol{U}^T\boldsymbol{y}=\sum_{i=0}^{p-1}\boldsymbol{u}_i\boldsymbol{u}^T_j\boldsymbol{y}, +$$ + +
+Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix \( \boldsymbol{U} \) +$$ +\boldsymbol{U}=[\boldsymbol{u}_0,\boldsymbol{u}_1,\dots,\boldsymbol{u}_{n-1}], +$$ + +that belong to \( i>p-1 \), give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to \( i=p-1 \). This corresponds also to the number of singular values (these are all non-zero). + +
+It means that the ordinary least square model (with the optimal parameters) \( \boldsymbol{\tilde{y}} \), corresponds to an orthogonal transformation of the output (or target) vector \( \boldsymbol{y} \) by the vectors of the matrix \( \boldsymbol{U} \).
diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
index 8e0cda615..246c38ddc 100644
Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ
diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb
index 31946a305..65762ee72 100644
--- a/doc/pub/week35/ipynb/week35.ipynb
+++ b/doc/pub/week35/ipynb/week35.ipynb
@@ -2618,9 +2618,131 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "is a $3\\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.\n",
+ "is a $3\\times 3 $ matrix. The last row and column of this last matrix\n",
+ "contain only zeros. This will have important consequences for our SVD\n",
+ "decomposition of the design matrix.\n",
"\n",
"\n",
+ "## Setting up the Matrix to be inverted\n",
+ "\n",
+ "The matrix that may cause problems for us is $\\boldsymbol{X}^T\\boldsymbol{X}$. Using the SVD we can rewrite this matrix as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and using the orthogonality of the matrix $\\boldsymbol{U}$ we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{\\Sigma}\\boldsymbol{V}^T.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We define $\\boldsymbol{\\Sigma}^T\\boldsymbol{\\sigma}=\\boldsymbol{D}^2, where $\\boldsymbol{D}$ is a diagonal matrix containing only the singular values squared. It has dimensionality $\\p \\times p$.\n",
+ "\n",
+ "This means, using the orthogonality of $\\boldsymbol{V}$, that we get"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{D}^2.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can now insert the result for the matrix $\\boldsymbol{X}^T\\boldsymbol{X}$ into our equation for ordinary least squares where"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and using our SVD decomposition of $\\boldsymbol{X}$ we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{\\Sigma}\\boldsymbol{V}^T\\boldsymbol{D}^{-2}\\boldsymbol{V}\\boldsymbol{\\Sigma}^T\\boldsymbol{U}^T\\boldsymbol{y},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which gives us, using the orthogonality of the matrices $\\boldsymbol{U}$ and $\\boldsymbol{V}$,"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\tilde{y}_{\\mathrm{OLS}}=\\boldsymbol{U}\\boldsymbol{U}^T\\boldsymbol{y}=\\sum_{i=0}^{p-1}\\boldsymbol{u}_i\\boldsymbol{u}^T_j\\boldsymbol{y},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix $\\boldsymbol{U}$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{U}=[\\boldsymbol{u}_0,\\boldsymbol{u}_1,\\dots,\\boldsymbol{u}_{n-1}],\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "that belong to $i>p-1$, give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to $i=p-1$. This corresponds also to the number of singular values (these are all non-zero).\n",
+ "\n",
+ "It means that the ordinary least square model (with the optimal parameters) $\\boldsymbol{\\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\\boldsymbol{y}$ by the vectors of the matrix $\\boldsymbol{U}$.\n",
+ "\n",
"## Ridge and LASSO Regression\n",
"\n",
"Let us remind ourselves about the expression for the standard Mean Squared Error (MSE) which we used to define our cost function and the equations for the ordinary least squares (OLS) method, that is \n",
diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt
index 2fd0133d0..c38b44933 100644
--- a/doc/src/week35/week35.do.txt
+++ b/doc/src/week35/week35.do.txt
@@ -1664,8 +1664,70 @@ which is a $2\times 2 $ matrix while
\end{bmatrix},
\]
!et
-is a $3\times 3 $ matrix. The last row and column of this last matrix contain only zeros. This will have important consequences for our SVD decomposition of the design matrix.
+is a $3\times 3 $ matrix. The last row and column of this last matrix
+contain only zeros. This will have important consequences for our SVD
+decomposition of the design matrix.
+
+
+!split
+===== Setting up the Matrix to be inverted =====
+
+The matrix that may cause problems for us is $\bm{X}^T\bm{X}$. Using the SVD we can rewrite this matrix as
+
+!bt
+\[
+\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{U}\bm{\Sigma}\bm{V}^T,
+\]
+!et
+and using the orthogonality of the matrix $\bm{U}$ we have
+
+!bt
+\[
+\bm{X}^T\bm{X}=\bm{V}\bm{\Sigma}^T\bm{\Sigma}\bm{V}^T.
+\]
+!et
+We define $\bm{\Sigma}^T\bm{\sigma}=\bm{D}^2, where $\bm{D}$ is a diagonal matrix containing only the singular values squared. It has dimensionality $\p \times p$.
+
+This means, using the orthogonality of $\bm{V}$, that we get
+
+!bt
+\[
+\bm{X}^T\bm{X}=\bm{D}^2.
+\]
+!et
+
+We can now insert the result for the matrix $\bm{X}^T\bm{X}$ into our equation for ordinary least squares where
+
+!bt
+\[
+\tilde{y}_{\mathrm{OLS}}=\bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y},
+\]
+!et
+and using our SVD decomposition of $\bm{X}$ we have
+
+!bt
+\[
+\tilde{y}_{\mathrm{OLS}}=\bm{U}\bm{\Sigma}\bm{V}^T\bm{D}^{-2}\bm{V}\bm{\Sigma}^T\bm{U}^T\bm{y},
+\]
+!et
+which gives us, using the orthogonality of the matrices $\bm{U}$ and $\bm{V}$,
+
+!bt
+\[
+\tilde{y}_{\mathrm{OLS}}=\bm{U}\bm{U}^T\bm{y}=\sum_{i=0}^{p-1}\bm{u}_i\bm{u}^T_j\bm{y},
+\]
+!et
+
+Note here that when we perform the multiplication of the various matrices, the orthogonal vectors of the matrix $\bm{U}$
+!bt
+\[
+\bm{U}=[\bm{u}_0,\bm{u}_1,\dots,\bm{u}_{n-1}],
+\]
+!et
+that belong to $i>p-1$, give all zeros when we perform the multiplications. This means the sum above has non-zero elements only up to $i=p-1$. This corresponds also to the number of singular values (these are all non-zero).
+
+It means that the ordinary least square model (with the optimal parameters) $\bm{\tilde{y}}$, corresponds to an orthogonal transformation of the output (or target) vector $\bm{y}$ by the vectors of the matrix $\bm{U}$.
!split
===== Ridge and LASSO Regression =====