lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

typos in reg slides

Browse Source
This commit is contained in:
mhjensen
2020-09-11 07:00:56 +02:00
parent 1d3bcc1c2c
commit e0e961e384
11 changed files with 26 additions and 260 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/Regression/html/._Regression-bs000.html
+1 -1
View File
@@ -465,7 +465,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 10, 2020</h4></center> <!-- date -->
<center><h4>Sep 11, 2020</h4></center> <!-- date -->
<br>
<p>
doc/pub/Regression/html/._Regression-bs030.html
+2 -4
View File
@@ -461,11 +461,9 @@ often possible to reduce the number of features considerably, turning
an intractable problem into a tractable one.
<p>
Here we will discuss some of the most popular dimensionality reduction
Later we will discuss some of the most popular dimensionality reduction
techniques: the principal component analysis (PCA), Kernel PCA, and
Locally Linear Embedding (LLE). Furthermore, we will start by looking
at some simple preprocessing of the data which allow us to rescale the
data.
Locally Linear Embedding (LLE).
<p>
Principal component analysis and its various variants deal with the
doc/pub/Regression/html/._Regression-bs055.html
+1 -1
View File
@@ -451,7 +451,7 @@ MathJax.Hub.Config({
<p>
We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T].
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}].
$$
<p>
doc/pub/Regression/html/._Regression-bs056.html
+1 -44
View File
@@ -449,50 +449,7 @@ MathJax.Hub.Config({
<h2 id="___sec55" class="anchor">Linking with SVD </h2>
<p>
We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T].
$$
Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \).
These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \).
<p>
Assume also that there is a transformation \( \boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \).
<p>
That is we have
$$
\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}\boldsymbol{X}\boldsymbol{X}^T\boldsymbol{S}^T]=\boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T,
$$
since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S}^T \) from the left we have
$$
\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T,
$$
and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that
$$
\boldsymbol{S}^T_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T_i.
$$
<p>
In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
<p>
The eigenvalues tell us then how much we need to stretch the
corresponding eigenvectors. Dimensions with large eigenvalues have
thus large variations (large variance) and define therefore useful
dimensions. The data points are more spread out in the direction of
these eigenvectors. Smaller eigenvalues mean on the other hand that
the corresponding eigenvectors are shrunk accordingly and the data
points are tightly bunched together and there is not much variation in
these specific directions. Hopefully then we could leave it out
dimensions where the eigenvalues are very small. If \( p \) is very large,
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
features/predictors.
See lecture september 11. More text to be added here soon.
<p>
<p>
doc/pub/Regression/html/Regression-bs.html
+1 -1
View File
@@ -465,7 +465,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 10, 2020</h4></center> <!-- date -->
<center><h4>Sep 11, 2020</h4></center> <!-- date -->
<br>
<p>
doc/pub/Regression/html/Regression-reveal.html
+5 -58
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 10, 2020</h4></center> <!-- date -->
<center><h4>Sep 11, 2020</h4></center> <!-- date -->
<br>
<p>
@@ -1493,11 +1493,9 @@ often possible to reduce the number of features considerably, turning
an intractable problem into a tractable one.
<p>
Here we will discuss some of the most popular dimensionality reduction
Later we will discuss some of the most popular dimensionality reduction
techniques: the principal component analysis (PCA), Kernel PCA, and
Locally Linear Embedding (LLE). Furthermore, we will start by looking
at some simple preprocessing of the data which allow us to rescale the
data.
Locally Linear Embedding (LLE).
<p>
Principal component analysis and its various variants deal with the
@@ -2632,7 +2630,7 @@ matrix without these elements.
We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as
<p>&nbsp;<br>
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T].
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}].
$$
<p>&nbsp;<br>
@@ -2682,58 +2680,7 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
<h2 id="___sec55">Linking with SVD </h2>
<p>
We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as
<p>&nbsp;<br>
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T].
$$
<p>&nbsp;<br>
Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \).
These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \).
<p>
Assume also that there is a transformation \( \boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \).
<p>
That is we have
<p>&nbsp;<br>
$$
\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}\boldsymbol{X}\boldsymbol{X}^T\boldsymbol{S}^T]=\boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T,
$$
<p>&nbsp;<br>
since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S}^T \) from the left we have
<p>&nbsp;<br>
$$
\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T,
$$
<p>&nbsp;<br>
and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that
<p>&nbsp;<br>
$$
\boldsymbol{S}^T_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T_i.
$$
<p>&nbsp;<br>
<p>
In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
<p>
The eigenvalues tell us then how much we need to stretch the
corresponding eigenvectors. Dimensions with large eigenvalues have
thus large variations (large variance) and define therefore useful
dimensions. The data points are more spread out in the direction of
these eigenvectors. Smaller eigenvalues mean on the other hand that
the corresponding eigenvectors are shrunk accordingly and the data
points are tightly bunched together and there is not much variation in
these specific directions. Hopefully then we could leave it out
dimensions where the eigenvalues are very small. If \( p \) is very large,
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
features/predictors.
See lecture september 11. More text to be added here soon.
</section>
doc/pub/Regression/html/Regression-solarized.html
+5 -50
View File
@@ -334,7 +334,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 10, 2020</h4></center> <!-- date -->
<center><h4>Sep 11, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1602,11 +1602,9 @@ often possible to reduce the number of features considerably, turning
an intractable problem into a tractable one.
<p>
Here we will discuss some of the most popular dimensionality reduction
Later we will discuss some of the most popular dimensionality reduction
techniques: the principal component analysis (PCA), Kernel PCA, and
Locally Linear Embedding (LLE). Furthermore, we will start by looking
at some simple preprocessing of the data which allow us to rescale the
data.
Locally Linear Embedding (LLE).
<p>
Principal component analysis and its various variants deal with the
@@ -2651,7 +2649,7 @@ matrix without these elements.
<p>
We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T].
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}].
$$
<p>
@@ -2692,50 +2690,7 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
<h2 id="___sec55">Linking with SVD </h2>
<p>
We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T].
$$
Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \).
These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \).
<p>
Assume also that there is a transformation \( \boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \).
<p>
That is we have
$$
\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}\boldsymbol{X}\boldsymbol{X}^T\boldsymbol{S}^T]=\boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T,
$$
since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S}^T \) from the left we have
$$
\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T,
$$
and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that
$$
\boldsymbol{S}^T_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T_i.
$$
<p>
In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
<p>
The eigenvalues tell us then how much we need to stretch the
corresponding eigenvectors. Dimensions with large eigenvalues have
thus large variations (large variance) and define therefore useful
dimensions. The data points are more spread out in the direction of
these eigenvectors. Smaller eigenvalues mean on the other hand that
the corresponding eigenvectors are shrunk accordingly and the data
points are tightly bunched together and there is not much variation in
these specific directions. Hopefully then we could leave it out
dimensions where the eigenvalues are very small. If \( p \) is very large,
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
features/predictors.
See lecture september 11. More text to be added here soon.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/Regression/html/Regression.html
+5 -50
View File
@@ -339,7 +339,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 10, 2020</h4></center> <!-- date -->
<center><h4>Sep 11, 2020</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1607,11 +1607,9 @@ often possible to reduce the number of features considerably, turning
an intractable problem into a tractable one.
<p>
Here we will discuss some of the most popular dimensionality reduction
Later we will discuss some of the most popular dimensionality reduction
techniques: the principal component analysis (PCA), Kernel PCA, and
Locally Linear Embedding (LLE). Furthermore, we will start by looking
at some simple preprocessing of the data which allow us to rescale the
data.
Locally Linear Embedding (LLE).
<p>
Principal component analysis and its various variants deal with the
@@ -2656,7 +2654,7 @@ matrix without these elements.
<p>
We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T].
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}].
$$
<p>
@@ -2697,50 +2695,7 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
<h2 id="___sec55">Linking with SVD </h2>
<p>
We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T].
$$
Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \).
These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \).
<p>
Assume also that there is a transformation \( \boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \).
<p>
That is we have
$$
\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}\boldsymbol{X}\boldsymbol{X}^T\boldsymbol{S}^T]=\boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T,
$$
since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S}^T \) from the left we have
$$
\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T,
$$
and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that
$$
\boldsymbol{S}^T_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T_i.
$$
<p>
In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
<p>
The eigenvalues tell us then how much we need to stretch the
corresponding eigenvectors. Dimensions with large eigenvalues have
thus large variations (large variance) and define therefore useful
dimensions. The data points are more spread out in the direction of
these eigenvectors. Smaller eigenvalues mean on the other hand that
the corresponding eigenvectors are shrunk accordingly and the data
points are tightly bunched together and there is not much variation in
these specific directions. Hopefully then we could leave it out
dimensions where the eigenvalues are very small. If \( p \) is very large,
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
features/predictors.
See lecture september 11. More text to be added here soon.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/Regression/pdf/Regression-minted.pdf
BIN
View File
Binary file not shown.
doc/src/Regression/Regression.do.txt
+5 -51
View File
@@ -1131,11 +1131,10 @@ curse of dimensionality. Fortunately, in real-world problems, it is
often possible to reduce the number of features considerably, turning
an intractable problem into a tractable one.
Here we will discuss some of the most popular dimensionality reduction
Later we will discuss some of the most popular dimensionality reduction
techniques: the principal component analysis (PCA), Kernel PCA, and
Locally Linear Embedding (LLE). Furthermore, we will start by looking
at some simple preprocessing of the data which allow us to rescale the
data.
Locally Linear Embedding (LLE).
Principal component analysis and its various variants deal with the
problem of fitting a low-dimensional "affine
@@ -2134,7 +2133,7 @@ matrix without these elements.
We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix $\bm{X}$ as
!bt
\[
\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}\bm{X}^T= \mathbb{E}[\bm{X}\bm{X}^T].
\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}^T\bm{X}= \mathbb{E}[\bm{X}^T\bm{X}].
\]
!et
@@ -2175,52 +2174,7 @@ It is easy to generalize this to a matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$.
!split
===== Linking with SVD =====
We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as
!bt
\[
\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}\bm{X}^T= \mathbb{E}[\bm{X}\bm{X}^T].
\]
!et
Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices $\bm{S}$.
These matrices are defined as $\bm{S}\in {\mathbb{R}}^{p\times p}$ and obey the orthogonality requirements $\bm{S}\bm{S}^T=\bm{S}^T\bm{S}=\bm{I}$. The matrix can be written out in terms of the column vectors $\bm{s}_i$ as $\bm{S}=[\bm{s}_0,\bm{s}_1,\dots,\bm{s}_{p-1}]$ and $\bm{s}_i \in {\mathbb{R}}^{p}$.
Assume also that there is a transformation $\bm{S}\bm{C}[\bm{x}]\bm{S}^T=\bm{C}[\bm{y}]$ such that the new matrix $\bm{C}[\bm{y}]$ is diagonal with elements $[\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}]$.
That is we have
!bt
\[
\bm{C}[\bm{y}] = \mathbb{E}[\bm{S}\bm{X}\bm{X}^T\bm{S}^T]=\bm{S}\bm{C}[\bm{x}]\bm{S}^T,
\]
!et
since the matrix $\bm{S}$ is not a data dependent matrix. Multiplying with $\bm{S}^T$ from the left we have
!bt
\[
\bm{S}^T\bm{C}[\bm{y}] = \bm{C}[\bm{x}]\bm{S}^T,
\]
!et
and since $\bm{C}[\bm{y}]$ is diagonal we have for a given eigenvalue $i$ of the covariance matrix that
!bt
\[
\bm{S}^T_i\lambda_i = \bm{C}[\bm{x}]\bm{S}^T_i.
\]
!et
In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
$\lambda_0 > \lambda_1 > \dots > \lambda_{p-1}$.
The eigenvalues tell us then how much we need to stretch the
corresponding eigenvectors. Dimensions with large eigenvalues have
thus large variations (large variance) and define therefore useful
dimensions. The data points are more spread out in the direction of
these eigenvectors. Smaller eigenvalues mean on the other hand that
the corresponding eigenvectors are shrunk accordingly and the data
points are tightly bunched together and there is not much variation in
these specific directions. Hopefully then we could leave it out
dimensions where the eigenvalues are very small. If $p$ is very large,
we could then aim at reducing $p$ to $l << p$ and handle only $l$
features/predictors.
See lecture september 11. More text to be added here soon.