diff --git a/doc/pub/Regression/html/._Regression-bs000.html b/doc/pub/Regression/html/._Regression-bs000.html index 026f578e0..3ac741c21 100644 --- a/doc/pub/Regression/html/._Regression-bs000.html +++ b/doc/pub/Regression/html/._Regression-bs000.html @@ -465,7 +465,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 10, 2020

+

Sep 11, 2020


diff --git a/doc/pub/Regression/html/._Regression-bs030.html b/doc/pub/Regression/html/._Regression-bs030.html index 6cb4cf52c..8c6f360a3 100644 --- a/doc/pub/Regression/html/._Regression-bs030.html +++ b/doc/pub/Regression/html/._Regression-bs030.html @@ -461,11 +461,9 @@ often possible to reduce the number of features considerably, turning an intractable problem into a tractable one.

-Here we will discuss some of the most popular dimensionality reduction +Later we will discuss some of the most popular dimensionality reduction techniques: the principal component analysis (PCA), Kernel PCA, and -Locally Linear Embedding (LLE). Furthermore, we will start by looking -at some simple preprocessing of the data which allow us to rescale the -data. +Locally Linear Embedding (LLE).

Principal component analysis and its various variants deal with the diff --git a/doc/pub/Regression/html/._Regression-bs055.html b/doc/pub/Regression/html/._Regression-bs055.html index 1dfd25f53..6f0822d91 100644 --- a/doc/pub/Regression/html/._Regression-bs055.html +++ b/doc/pub/Regression/html/._Regression-bs055.html @@ -451,7 +451,7 @@ MathJax.Hub.Config({

We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as $$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T]. +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. $$

diff --git a/doc/pub/Regression/html/._Regression-bs056.html b/doc/pub/Regression/html/._Regression-bs056.html index 3c78814da..abe730d4c 100644 --- a/doc/pub/Regression/html/._Regression-bs056.html +++ b/doc/pub/Regression/html/._Regression-bs056.html @@ -449,50 +449,7 @@ MathJax.Hub.Config({

Linking with SVD

-We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as -$$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T]. -$$ - -Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \). -These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \). - -

-Assume also that there is a transformation \( \boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \). - -

-That is we have -$$ -\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}\boldsymbol{X}\boldsymbol{X}^T\boldsymbol{S}^T]=\boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T, -$$ - -since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S}^T \) from the left we have -$$ -\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T, -$$ - -and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that - -$$ -\boldsymbol{S}^T_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T_i. -$$ - -

-In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is -\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). - -

-The eigenvalues tell us then how much we need to stretch the -corresponding eigenvectors. Dimensions with large eigenvalues have -thus large variations (large variance) and define therefore useful -dimensions. The data points are more spread out in the direction of -these eigenvectors. Smaller eigenvalues mean on the other hand that -the corresponding eigenvectors are shrunk accordingly and the data -points are tightly bunched together and there is not much variation in -these specific directions. Hopefully then we could leave it out -dimensions where the eigenvalues are very small. If \( p \) is very large, -we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) -features/predictors. +See lecture september 11. More text to be added here soon.

diff --git a/doc/pub/Regression/html/Regression-bs.html b/doc/pub/Regression/html/Regression-bs.html index 026f578e0..3ac741c21 100644 --- a/doc/pub/Regression/html/Regression-bs.html +++ b/doc/pub/Regression/html/Regression-bs.html @@ -465,7 +465,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 10, 2020

+

Sep 11, 2020


diff --git a/doc/pub/Regression/html/Regression-reveal.html b/doc/pub/Regression/html/Regression-reveal.html index bb171a8b3..94984c17f 100644 --- a/doc/pub/Regression/html/Regression-reveal.html +++ b/doc/pub/Regression/html/Regression-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

 
-

Sep 10, 2020

+

Sep 11, 2020


@@ -1493,11 +1493,9 @@ often possible to reduce the number of features considerably, turning an intractable problem into a tractable one.

-Here we will discuss some of the most popular dimensionality reduction +Later we will discuss some of the most popular dimensionality reduction techniques: the principal component analysis (PCA), Kernel PCA, and -Locally Linear Embedding (LLE). Furthermore, we will start by looking -at some simple preprocessing of the data which allow us to rescale the -data. +Locally Linear Embedding (LLE).

Principal component analysis and its various variants deal with the @@ -2632,7 +2630,7 @@ matrix without these elements. We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as

 
$$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T]. +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. $$

 
@@ -2682,58 +2680,7 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t

Linking with SVD

-We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as -

 
-$$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T]. -$$ -

 
- -Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \). -These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \). - -

-Assume also that there is a transformation \( \boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \). - -

-That is we have -

 
-$$ -\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}\boldsymbol{X}\boldsymbol{X}^T\boldsymbol{S}^T]=\boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T, -$$ -

 
- -since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S}^T \) from the left we have -

 
-$$ -\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T, -$$ -

 
- -and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that - -

 
-$$ -\boldsymbol{S}^T_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T_i. -$$ -

 
- -

-In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is -\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). - -

-The eigenvalues tell us then how much we need to stretch the -corresponding eigenvectors. Dimensions with large eigenvalues have -thus large variations (large variance) and define therefore useful -dimensions. The data points are more spread out in the direction of -these eigenvectors. Smaller eigenvalues mean on the other hand that -the corresponding eigenvectors are shrunk accordingly and the data -points are tightly bunched together and there is not much variation in -these specific directions. Hopefully then we could leave it out -dimensions where the eigenvalues are very small. If \( p \) is very large, -we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) -features/predictors. +See lecture september 11. More text to be added here soon. diff --git a/doc/pub/Regression/html/Regression-solarized.html b/doc/pub/Regression/html/Regression-solarized.html index 8e4ba9023..4cb9edffe 100644 --- a/doc/pub/Regression/html/Regression-solarized.html +++ b/doc/pub/Regression/html/Regression-solarized.html @@ -334,7 +334,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 10, 2020

+

Sep 11, 2020












@@ -1602,11 +1602,9 @@ often possible to reduce the number of features considerably, turning an intractable problem into a tractable one.

-Here we will discuss some of the most popular dimensionality reduction +Later we will discuss some of the most popular dimensionality reduction techniques: the principal component analysis (PCA), Kernel PCA, and -Locally Linear Embedding (LLE). Furthermore, we will start by looking -at some simple preprocessing of the data which allow us to rescale the -data. +Locally Linear Embedding (LLE).

Principal component analysis and its various variants deal with the @@ -2651,7 +2649,7 @@ matrix without these elements.

We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as $$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T]. +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. $$

@@ -2692,50 +2690,7 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t

Linking with SVD

-We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as -$$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T]. -$$ - -Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \). -These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \). - -

-Assume also that there is a transformation \( \boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \). - -

-That is we have -$$ -\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}\boldsymbol{X}\boldsymbol{X}^T\boldsymbol{S}^T]=\boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T, -$$ - -since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S}^T \) from the left we have -$$ -\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T, -$$ - -and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that - -$$ -\boldsymbol{S}^T_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T_i. -$$ - -

-In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is -\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). - -

-The eigenvalues tell us then how much we need to stretch the -corresponding eigenvectors. Dimensions with large eigenvalues have -thus large variations (large variance) and define therefore useful -dimensions. The data points are more spread out in the direction of -these eigenvectors. Smaller eigenvalues mean on the other hand that -the corresponding eigenvectors are shrunk accordingly and the data -points are tightly bunched together and there is not much variation in -these specific directions. Hopefully then we could leave it out -dimensions where the eigenvalues are very small. If \( p \) is very large, -we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) -features/predictors. +See lecture september 11. More text to be added here soon.











diff --git a/doc/pub/Regression/html/Regression.html b/doc/pub/Regression/html/Regression.html index 849520e87..b585f5a3c 100644 --- a/doc/pub/Regression/html/Regression.html +++ b/doc/pub/Regression/html/Regression.html @@ -339,7 +339,7 @@ MathJax.Hub.Config({

[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 10, 2020

+

Sep 11, 2020












@@ -1607,11 +1607,9 @@ often possible to reduce the number of features considerably, turning an intractable problem into a tractable one.

-Here we will discuss some of the most popular dimensionality reduction +Later we will discuss some of the most popular dimensionality reduction techniques: the principal component analysis (PCA), Kernel PCA, and -Locally Linear Embedding (LLE). Furthermore, we will start by looking -at some simple preprocessing of the data which allow us to rescale the -data. +Locally Linear Embedding (LLE).

Principal component analysis and its various variants deal with the @@ -2656,7 +2654,7 @@ matrix without these elements.

We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as $$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T]. +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}]. $$

@@ -2697,50 +2695,7 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t

Linking with SVD

-We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as -$$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T]. -$$ - -Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices \( \boldsymbol{S} \). -These matrices are defined as \( \boldsymbol{S}\in {\mathbb{R}}^{p\times p} \) and obey the orthogonality requirements \( \boldsymbol{S}\boldsymbol{S}^T=\boldsymbol{S}^T\boldsymbol{S}=\boldsymbol{I} \). The matrix can be written out in terms of the column vectors \( \boldsymbol{s}_i \) as \( \boldsymbol{S}=[\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \) and \( \boldsymbol{s}_i \in {\mathbb{R}}^{p} \). - -

-Assume also that there is a transformation \( \boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T=\boldsymbol{C}[\boldsymbol{y}] \) such that the new matrix \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal with elements \( [\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}] \). - -

-That is we have -$$ -\boldsymbol{C}[\boldsymbol{y}] = \mathbb{E}[\boldsymbol{S}\boldsymbol{X}\boldsymbol{X}^T\boldsymbol{S}^T]=\boldsymbol{S}\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T, -$$ - -since the matrix \( \boldsymbol{S} \) is not a data dependent matrix. Multiplying with \( \boldsymbol{S}^T \) from the left we have -$$ -\boldsymbol{S}^T\boldsymbol{C}[\boldsymbol{y}] = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T, -$$ - -and since \( \boldsymbol{C}[\boldsymbol{y}] \) is diagonal we have for a given eigenvalue \( i \) of the covariance matrix that - -$$ -\boldsymbol{S}^T_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}^T_i. -$$ - -

-In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is -\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \). - -

-The eigenvalues tell us then how much we need to stretch the -corresponding eigenvectors. Dimensions with large eigenvalues have -thus large variations (large variance) and define therefore useful -dimensions. The data points are more spread out in the direction of -these eigenvectors. Smaller eigenvalues mean on the other hand that -the corresponding eigenvectors are shrunk accordingly and the data -points are tightly bunched together and there is not much variation in -these specific directions. Hopefully then we could leave it out -dimensions where the eigenvalues are very small. If \( p \) is very large, -we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) -features/predictors. +See lecture september 11. More text to be added here soon.











diff --git a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz index b7d91d5a9..667d72509 100644 Binary files a/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz and b/doc/pub/Regression/ipynb/ipynb-Regression-src.tar.gz differ diff --git a/doc/pub/Regression/pdf/Regression-minted.pdf b/doc/pub/Regression/pdf/Regression-minted.pdf index 08fa6ccfe..1ad75de53 100644 Binary files a/doc/pub/Regression/pdf/Regression-minted.pdf and b/doc/pub/Regression/pdf/Regression-minted.pdf differ diff --git a/doc/src/Regression/Regression.do.txt b/doc/src/Regression/Regression.do.txt index b3b547891..33b034e21 100644 --- a/doc/src/Regression/Regression.do.txt +++ b/doc/src/Regression/Regression.do.txt @@ -1131,11 +1131,10 @@ curse of dimensionality. Fortunately, in real-world problems, it is often possible to reduce the number of features considerably, turning an intractable problem into a tractable one. -Here we will discuss some of the most popular dimensionality reduction +Later we will discuss some of the most popular dimensionality reduction techniques: the principal component analysis (PCA), Kernel PCA, and -Locally Linear Embedding (LLE). Furthermore, we will start by looking -at some simple preprocessing of the data which allow us to rescale the -data. +Locally Linear Embedding (LLE). + Principal component analysis and its various variants deal with the problem of fitting a low-dimensional "affine @@ -2134,7 +2133,7 @@ matrix without these elements. We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix $\bm{X}$ as !bt \[ -\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}\bm{X}^T= \mathbb{E}[\bm{X}\bm{X}^T]. +\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}^T\bm{X}= \mathbb{E}[\bm{X}^T\bm{X}]. \] !et @@ -2175,52 +2174,7 @@ It is easy to generalize this to a matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. !split ===== Linking with SVD ===== -We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as -!bt -\[ -\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}\bm{X}^T= \mathbb{E}[\bm{X}\bm{X}^T]. -\] -!et -Let us now assume that we can perform a series of orthogonal transformations where we employ some orthogonal matrices $\bm{S}$. -These matrices are defined as $\bm{S}\in {\mathbb{R}}^{p\times p}$ and obey the orthogonality requirements $\bm{S}\bm{S}^T=\bm{S}^T\bm{S}=\bm{I}$. The matrix can be written out in terms of the column vectors $\bm{s}_i$ as $\bm{S}=[\bm{s}_0,\bm{s}_1,\dots,\bm{s}_{p-1}]$ and $\bm{s}_i \in {\mathbb{R}}^{p}$. - -Assume also that there is a transformation $\bm{S}\bm{C}[\bm{x}]\bm{S}^T=\bm{C}[\bm{y}]$ such that the new matrix $\bm{C}[\bm{y}]$ is diagonal with elements $[\lambda_0,\lambda_1,\lambda_2,\dots,\lambda_{p-1}]$. - -That is we have -!bt -\[ -\bm{C}[\bm{y}] = \mathbb{E}[\bm{S}\bm{X}\bm{X}^T\bm{S}^T]=\bm{S}\bm{C}[\bm{x}]\bm{S}^T, -\] -!et -since the matrix $\bm{S}$ is not a data dependent matrix. Multiplying with $\bm{S}^T$ from the left we have -!bt -\[ -\bm{S}^T\bm{C}[\bm{y}] = \bm{C}[\bm{x}]\bm{S}^T, -\] -!et -and since $\bm{C}[\bm{y}]$ is diagonal we have for a given eigenvalue $i$ of the covariance matrix that - -!bt -\[ -\bm{S}^T_i\lambda_i = \bm{C}[\bm{x}]\bm{S}^T_i. -\] -!et - -In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is -$\lambda_0 > \lambda_1 > \dots > \lambda_{p-1}$. - - -The eigenvalues tell us then how much we need to stretch the -corresponding eigenvectors. Dimensions with large eigenvalues have -thus large variations (large variance) and define therefore useful -dimensions. The data points are more spread out in the direction of -these eigenvectors. Smaller eigenvalues mean on the other hand that -the corresponding eigenvectors are shrunk accordingly and the data -points are tightly bunched together and there is not much variation in -these specific directions. Hopefully then we could leave it out -dimensions where the eigenvalues are very small. If $p$ is very large, -we could then aim at reducing $p$ to $l << p$ and handle only $l$ -features/predictors. +See lecture september 11. More text to be added here soon.