What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). + +
+We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by + +
+$$
+\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+$$
+
+
+
+This means that our best model is defined as + +
+$$
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}.
+$$
+
+
+
+We define now a matrix +
+$$
+\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T.
+$$
+
+
+
+It means that we can rewrite +
+$$
+\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
+$$
+
+
+
+The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
+We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \).
+
+
+
+
+We have defined the residual error as
+
+The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \).
+
+If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have
+
+
+In this case the matrix \( \boldsymbol{A} \) becomes
+ Residual Error
+
+
+$$
+\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=(\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)\boldsymbol{y}.
+$$
+
+
+Simple case
+
+
+$$
+\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{X}\boldsymbol{X}^T = \boldsymbol{I}.
+$$
+
+
+
+$$
+\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I},
+$$
+
+
+and we have the obvious case
+
+$$
+\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0.
+$$
+
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). +
+We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by + +$$ +\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +
+This means that our best model is defined as + +$$ +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +
+We define now a matrix +$$ +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T. +$$ + +
+It means that we can rewrite +$$ +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. +$$ + +
+The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. +We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). + +
+
+
+
+We have defined the residual error as +$$ +\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=(\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)\boldsymbol{y}. +$$ + +
+The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \). + +
+
+
+
+If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have + +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{X}\boldsymbol{X}^T = \boldsymbol{I}. +$$ + +
+In this case the matrix \( \boldsymbol{A} \) becomes +$$ +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, +$$ + +and we have the obvious case +$$ +\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. +$$ +
diff --git a/doc/pub/week35/html/week35.html b/doc/pub/week35/html/week35.html
index 10114f03e..86ba19579 100644
--- a/doc/pub/week35/html/week35.html
+++ b/doc/pub/week35/html/week35.html
@@ -184,6 +184,8 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'mathematical-interpretation-of-ordinary-least-squares'),
+ ('Residual Error', 2, None, 'residual-error'),
+ ('Simple case', 2, None, 'simple-case'),
('The singular value decomposition',
2,
None,
@@ -322,7 +324,7 @@ The main topics on Thursday are:
+We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by + +$$ +\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +
+This means that our best model is defined as + +$$ +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}. +$$ + +
+We define now a matrix +$$ +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T. +$$ + +
+It means that we can rewrite +$$ +\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}. +$$ + +
+The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix. +We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). + +
+
+
+
+We have defined the residual error as +$$ +\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=(\boldsymbol{I}-\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)\boldsymbol{y}. +$$ + +
+The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \). + +
+
+
+
+If the matrix \( \boldsymbol{X} \) is an orthogonal (or unitary in case of complex values) matrix, we have + +$$ +\boldsymbol{X}^T\boldsymbol{X}=\boldsymbol{X}\boldsymbol{X}^T = \boldsymbol{I}. +$$ + +
+In this case the matrix \( \boldsymbol{A} \) becomes +$$ +\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I}, +$$ + +and we have the obvious case +$$ +\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0. +$$ +
diff --git a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
index 8af419c5e..91a436d1e 100644
Binary files a/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz and b/doc/pub/week35/ipynb/ipynb-week35-src.tar.gz differ
diff --git a/doc/pub/week35/ipynb/week35.ipynb b/doc/pub/week35/ipynb/week35.ipynb
index 8de0757f5..73ef692d9 100644
--- a/doc/pub/week35/ipynb/week35.ipynb
+++ b/doc/pub/week35/ipynb/week35.ipynb
@@ -32,7 +32,7 @@
"\n",
"3. Mathematical interpretations of Linear Regression\n",
"\n",
- "4. Start discussing Ridge regression and Singular Value Decomposition\n",
+ "4. Start discussing Ridge and Lasso regression and Singular Value Decomposition\n",
"\n",
"## Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week\n",
"\n",
@@ -1858,6 +1858,143 @@
"What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n",
"\n",
"\n",
+ "We have shown that in ordinary least squares that the optimal parameter $\\beta$ are given by"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This means that our best model is defined as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We define now a matrix"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "It means that we can rewrite"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{A}\\boldsymbol{y}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The matrix $\\boldsymbol{A}$ has the important property that $\\boldsymbol{A}^2=\\boldsymbol{A}$. This is the definition of a projection matrix.\n",
+ "We can then interpret that our optimal model $\\tilde{\\boldsymbol{y}}$ ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$.\n",
+ "\n",
+ "## Residual Error\n",
+ "\n",
+ "We have defined the residual error as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=(\\boldsymbol{I}-\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T)\\boldsymbol{y}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal components of the space defined by the column vectors of $\\boldsymbol{X}$.\n",
+ "\n",
+ "## Simple case\n",
+ "\n",
+ "If the matrix $\\boldsymbol{X}$ is an orthogonal (or unitary in case of complex values) matrix, we have"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{X}^T\\boldsymbol{X}=\\boldsymbol{X}\\boldsymbol{X}^T = \\boldsymbol{I}.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In this case the matrix $\\boldsymbol{A}$ becomes"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T)=\\boldsymbol{I},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "and we have the obvious case"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{\\epsilon}=\\boldsymbol{y}-\\tilde{\\boldsymbol{y}}=0.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
"## The singular value decomposition\n",
"\n",
"\n",
diff --git a/doc/src/week35/week35.do.txt b/doc/src/week35/week35.do.txt
index bc398b0eb..ba241260d 100644
--- a/doc/src/week35/week35.do.txt
+++ b/doc/src/week35/week35.do.txt
@@ -17,7 +17,7 @@ The main topics on Thursday are:
o Repetition from last week on linear regression
o Discussion of how to prepare data and examples of applications of linear regression
o Mathematical interpretations of Linear Regression
-o Start discussing Ridge regression and Singular Value Decomposition
+o Start discussing Ridge and Lasso regression and Singular Value Decomposition
!split
===== Why Linear Regression (aka Ordinary Least Squares and family), repeat from last week =====
@@ -1157,6 +1157,77 @@ print("R2 score for scaled data: {:.2f}".format(clf.score(X_test_scaled,y_test)
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
+We have shown that in ordinary least squares that the optimal parameter $\beta$ are given by
+
+!bt
+\[
+\hat{\bm{\beta}} = \left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T\bm{y}.
+\]
+!et
+
+This means that our best model is defined as
+
+!bt
+\[
+\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T\bm{y}.
+\]
+!et
+
+We define now a matrix
+!bt
+\[
+\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T.
+\]
+!et
+
+It means that we can rewrite
+!bt
+\[
+\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{A}\bm{y}.
+\]
+!et
+
+The matrix $\bm{A}$ has the important property that $\bm{A}^2=\bm{A}$. This is the definition of a projection matrix.
+We can then interpret that our optimal model $\tilde{\bm{y}}$ ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of $\bm{y}$ onto a space defined by the column vectors of $\bm{X}$.
+
+!split
+===== Residual Error =====
+
+We have defined the residual error as
+!bt
+\[
+\bm{\epsilon}=\bm{y}-\tilde{\bm{y}}=(\bm{I}-\bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T)\bm{y}.
+\]
+!et
+
+The residual errors are then the projections of $\bm{y}$ onto the orthogonal components of the space defined by the column vectors of $\bm{X}$.
+
+!split
+===== Simple case =====
+
+If the matrix $\bm{X}$ is an orthogonal (or unitary in case of complex values) matrix, we have
+
+!bt
+\[
+\bm{X}^T\bm{X}=\bm{X}\bm{X}^T = \bm{I}.
+\]
+!et
+
+In this case the matrix $\bm{A}$ becomes
+!bt
+\[
+\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T)=\bm{I},
+\]
+!et
+and we have the obvious case
+!bt
+\[
+\bm{\epsilon}=\bm{y}-\tilde{\bm{y}}=0.
+\]
+!et
+
+
+
!split
===== The singular value decomposition =====