lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added more material to week 35

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-02 16:24:20 +02:00
parent c35b7a2fd3
commit b605ae4461
6 changed files with 84 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week35/html/week35-reveal.html
+17 -10
View File
@@ -1352,13 +1352,14 @@ display(XPandas)
<span style="color: #658b00">print</span>(XPandas.std())
XPandas = (XPandas -XPandas.mean())
display(XPandas)
<span style="color: #228B22"># This option does not include the standard deviation</span>
scaler = StandardScaler(with_std=<span style="color: #8B008B; font-weight: bold">False</span>)
scaler.fit(X)
Xscaled = scaler.transform(X)
display(XPandas-Xscaled)
</pre></div>
<p>
Small exercise: perform the standard scaling by including the standard deviation.
Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.
</section>
@@ -1556,33 +1557,36 @@ clf = skl.LinearRegression().fit(X_train_scaled, y_train)
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
<p>
We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by
We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by
<p>&nbsp;<br>
$$
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>&nbsp;<br>
<p>
The <b>hat</b> over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.
<p>
This means that our best model is defined as
<p>&nbsp;<br>
$$
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>&nbsp;<br>
<p>
We define now a matrix
We now define a matrix
<p>&nbsp;<br>
$$
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T.
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T.
$$
<p>&nbsp;<br>
<p>
It means that we can rewrite
We can rewrite
<p>&nbsp;<br>
$$
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
@@ -1591,7 +1595,7 @@ $$
<p>
The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \).
We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.
</section>
@@ -1607,7 +1611,7 @@ $$
<p>&nbsp;<br>
<p>
The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \).
The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).
</section>
@@ -1627,7 +1631,7 @@ $$
In this case the matrix \( \boldsymbol{A} \) becomes
<p>&nbsp;<br>
$$
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I},
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I},
$$
<p>&nbsp;<br>
@@ -1637,6 +1641,9 @@ $$
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0.
$$
<p>&nbsp;<br>
<p>
This serves also as a useful test of our codes.
</section>
doc/pub/week35/html/week35-solarized.html
+17 -10
View File
@@ -1450,13 +1450,14 @@ display(XPandas)
<span style="color: #658b00">print</span>(XPandas.std())
XPandas = (XPandas -XPandas.mean())
display(XPandas)
<span style="color: #228B22"># This option does not include the standard deviation</span>
scaler = StandardScaler(with_std=<span style="color: #8B008B; font-weight: bold">False</span>)
scaler.fit(X)
Xscaled = scaler.transform(X)
display(XPandas-Xscaled)
</pre></div>
<p>
Small exercise: perform the standard scaling by including the standard deviation.
Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1650,34 +1651,37 @@ clf = skl.LinearRegression().fit(X_train_scaled, y_train)
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
<p>
We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by
We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by
$$
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>
The <b>hat</b> over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.
<p>
This means that our best model is defined as
$$
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>
We define now a matrix
We now define a matrix
$$
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T.
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T.
$$
<p>
It means that we can rewrite
We can rewrite
$$
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
$$
<p>
The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \).
We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1691,7 +1695,7 @@ $$
$$
<p>
The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \).
The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1708,7 +1712,7 @@ $$
<p>
In this case the matrix \( \boldsymbol{A} \) becomes
$$
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I},
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I},
$$
and we have the obvious case
@@ -1716,6 +1720,9 @@ $$
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0.
$$
<p>
This serves also as a useful test of our codes.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/html/week35.html
+17 -10
View File
@@ -1455,13 +1455,14 @@ display(XPandas)
<span style="color: #008000">print</span>(XPandas<span style="color: #666666">.</span>std())
XPandas <span style="color: #666666">=</span> (XPandas <span style="color: #666666">-</span>XPandas<span style="color: #666666">.</span>mean())
display(XPandas)
<span style="color: #408080; font-style: italic"># This option does not include the standard deviation</span>
scaler <span style="color: #666666">=</span> StandardScaler(with_std<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>)
scaler<span style="color: #666666">.</span>fit(X)
Xscaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X)
display(XPandas<span style="color: #666666">-</span>Xscaled)
</pre></div>
<p>
Small exercise: perform the standard scaling by including the standard deviation.
Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1655,34 +1656,37 @@ clf <span style="color: #666666">=</span> skl<span style="color: #666666">.</spa
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
<p>
We have shown that in ordinary least squares that the optimal parameter \( \beta \) are given by
We have shown that in ordinary least squares the optimal parameters \( \beta \) are given by
$$
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\hat{\boldsymbol{\beta}} = \left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>
The <b>hat</b> over \( \boldsymbol{\beta} \) means we have the optimal parameters after minimization of the cost function.
<p>
This means that our best model is defined as
$$
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T\boldsymbol{y}.
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T\boldsymbol{y}.
$$
<p>
We define now a matrix
We now define a matrix
$$
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T.
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T.
$$
<p>
It means that we can rewrite
We can rewrite
$$
\tilde{\boldsymbol{y}}=\boldsymbol{X}\hat{\boldsymbol{\beta}} = \boldsymbol{A}\boldsymbol{y}.
$$
<p>
The matrix \( \boldsymbol{A} \) has the important property that \( \boldsymbol{A}^2=\boldsymbol{A} \). This is the definition of a projection matrix.
We can then interpret that our optimal model \( \tilde{\boldsymbol{y}} \) ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \).
We can then interpret our optimal model \( \tilde{\boldsymbol{y}} \) as being represented by an orthogonal projection of \( \boldsymbol{y} \) onto a space defined by the column vectors of \( \boldsymbol{X} \). In our case here the matrix \( \boldsymbol{A} \) is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1696,7 +1700,7 @@ $$
$$
<p>
The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal components of the space defined by the column vectors of \( \boldsymbol{X} \).
The residual errors are then the projections of \( \boldsymbol{y} \) onto the orthogonal component of the space defined by the column vectors of \( \boldsymbol{X} \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1713,7 +1717,7 @@ $$
<p>
In this case the matrix \( \boldsymbol{A} \) becomes
$$
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right^{-1}\boldsymbol{X}^T)=\boldsymbol{I},
\boldsymbol{A}=\boldsymbol{X}\left(\boldsymbol{X}^T\boldsymbol{X}\right)^{-1}\boldsymbol{X}^T)=\boldsymbol{I},
$$
and we have the obvious case
@@ -1721,6 +1725,9 @@ $$
\boldsymbol{\epsilon}=\boldsymbol{y}-\tilde{\boldsymbol{y}}=0.
$$
<p>
This serves also as a useful test of our codes.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week35/ipynb/ipynb-week35-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week35/ipynb/week35.ipynb
+17 -10
View File
@@ -1663,6 +1663,7 @@
"print(XPandas.std())\n",
"XPandas = (XPandas -XPandas.mean())\n",
"display(XPandas)\n",
"# This option does not include the standard deviation\n",
"scaler = StandardScaler(with_std=False)\n",
"scaler.fit(X)\n",
"Xscaled = scaler.transform(X)\n",
@@ -1673,7 +1674,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Small exercise: perform the standard scaling by including the standard deviation.\n",
"Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.\n",
"\n",
"## Min-Max Scaling\n",
"\n",
@@ -1900,7 +1901,7 @@
"What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD). \n",
"\n",
"\n",
"We have shown that in ordinary least squares that the optimal parameter $\\beta$ are given by"
"We have shown that in ordinary least squares the optimal parameters $\\beta$ are given by"
]
},
{
@@ -1908,7 +1909,7 @@
"metadata": {},
"source": [
"$$\n",
"\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"\\hat{\\boldsymbol{\\beta}} = \\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
@@ -1916,6 +1917,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The **hat** over $\\boldsymbol{\\beta}$ means we have the optimal parameters after minimization of the cost function.\n",
"\n",
"This means that our best model is defined as"
]
},
@@ -1924,7 +1927,7 @@
"metadata": {},
"source": [
"$$\n",
"\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"\\tilde{\\boldsymbol{y}}=\\boldsymbol{X}\\hat{\\boldsymbol{\\beta}} = \\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}.\n",
"$$"
]
},
@@ -1932,7 +1935,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We define now a matrix"
"We now define a matrix"
]
},
{
@@ -1940,7 +1943,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T.\n",
"\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T.\n",
"$$"
]
},
@@ -1948,7 +1951,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"It means that we can rewrite"
"We can rewrite"
]
},
{
@@ -1965,7 +1968,9 @@
"metadata": {},
"source": [
"The matrix $\\boldsymbol{A}$ has the important property that $\\boldsymbol{A}^2=\\boldsymbol{A}$. This is the definition of a projection matrix.\n",
"We can then interpret that our optimal model $\\tilde{\\boldsymbol{y}}$ ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$.\n",
"We can then interpret our optimal model $\\tilde{\\boldsymbol{y}}$ as being represented by an orthogonal projection of $\\boldsymbol{y}$ onto a space defined by the column vectors of $\\boldsymbol{X}$. In our case here the matrix $\\boldsymbol{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.\n",
"\n",
"\n",
"\n",
"## Residual Error\n",
"\n",
@@ -1985,7 +1990,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal components of the space defined by the column vectors of $\\boldsymbol{X}$.\n",
"The residual errors are then the projections of $\\boldsymbol{y}$ onto the orthogonal component of the space defined by the column vectors of $\\boldsymbol{X}$.\n",
"\n",
"## Simple case\n",
"\n",
@@ -2013,7 +2018,7 @@
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right^{-1}\\boldsymbol{X}^T)=\\boldsymbol{I},\n",
"\\boldsymbol{A}=\\boldsymbol{X}\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T)=\\boldsymbol{I},\n",
"$$"
]
},
@@ -2037,6 +2042,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"This serves also as a useful test of our codes. \n",
"\n",
"## The singular value decomposition\n",
"\n",
"\n",
doc/src/week35/week35.do.txt
+16 -11
View File
@@ -999,13 +999,14 @@ print(XPandas.mean())
print(XPandas.std())
XPandas = (XPandas -XPandas.mean())
display(XPandas)
# This option does not include the standard deviation
scaler = StandardScaler(with_std=False)
scaler.fit(X)
Xscaled = scaler.transform(X)
display(XPandas-Xscaled)
!ec
Small exercise: perform the standard scaling by including the standard deviation.
Small exercise: perform the standard scaling by including the standard deviation and compare with what Scikit-Learn gives.
!split
===== Min-Max Scaling =====
@@ -1189,30 +1190,32 @@ print("R2 score for scaled data: {:.2f}".format(clf.score(X_test_scaled,y_test)
What is presented here is a mathematical analysis of various regression algorithms (ordinary least squares, Ridge and Lasso Regression). The analysis is based on an important algorithm in linear algebra, the so-called Singular Value Decomposition (SVD).
We have shown that in ordinary least squares that the optimal parameter $\beta$ are given by
We have shown that in ordinary least squares the optimal parameters $\beta$ are given by
!bt
\[
\hat{\bm{\beta}} = \left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T\bm{y}.
\hat{\bm{\beta}} = \left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
The _hat_ over $\bm{\beta}$ means we have the optimal parameters after minimization of the cost function.
This means that our best model is defined as
!bt
\[
\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T\bm{y}.
\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T\bm{y}.
\]
!et
We define now a matrix
We now define a matrix
!bt
\[
\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T.
\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T.
\]
!et
It means that we can rewrite
We can rewrite
!bt
\[
\tilde{\bm{y}}=\bm{X}\hat{\bm{\beta}} = \bm{A}\bm{y}.
@@ -1220,7 +1223,9 @@ It means that we can rewrite
!et
The matrix $\bm{A}$ has the important property that $\bm{A}^2=\bm{A}$. This is the definition of a projection matrix.
We can then interpret that our optimal model $\tilde{\bm{y}}$ ir represented by an orthogonal (it has to be a square matrix like ours, if not we have an oblique projection matrix ) projection of $\bm{y}$ onto a space defined by the column vectors of $\bm{X}$.
We can then interpret our optimal model $\tilde{\bm{y}}$ as being represented by an orthogonal projection of $\bm{y}$ onto a space defined by the column vectors of $\bm{X}$. In our case here the matrix $\bm{A}$ is a square matrix. If it is a general rectangular matrix we have an oblique projection matrix.
!split
===== Residual Error =====
@@ -1232,7 +1237,7 @@ We have defined the residual error as
\]
!et
The residual errors are then the projections of $\bm{y}$ onto the orthogonal components of the space defined by the column vectors of $\bm{X}$.
The residual errors are then the projections of $\bm{y}$ onto the orthogonal component of the space defined by the column vectors of $\bm{X}$.
!split
===== Simple case =====
@@ -1248,7 +1253,7 @@ If the matrix $\bm{X}$ is an orthogonal (or unitary in case of complex values) m
In this case the matrix $\bm{A}$ becomes
!bt
\[
\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right^{-1}\bm{X}^T)=\bm{I},
\bm{A}=\bm{X}\left(\bm{X}^T\bm{X}\right)^{-1}\bm{X}^T)=\bm{I},
\]
!et
and we have the obvious case
@@ -1258,7 +1263,7 @@ and we have the obvious case
\]
!et
This serves also as a useful test of our codes.
!split
===== The singular value decomposition =====