diff --git a/doc/pub/DimRed/html/._DimRed-bs018.html b/doc/pub/DimRed/html/._DimRed-bs018.html index c172b2cd4..dcd6f46d7 100644 --- a/doc/pub/DimRed/html/._DimRed-bs018.html +++ b/doc/pub/DimRed/html/._DimRed-bs018.html @@ -193,14 +193,14 @@ each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).

We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as $$ -J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, +J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, $$ with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix -\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality. +\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.

-The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix). +The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).

diff --git a/doc/pub/DimRed/html/._DimRed-bs019.html b/doc/pub/DimRed/html/._DimRed-bs019.html index e4746e5e3..363c14d79 100644 --- a/doc/pub/DimRed/html/._DimRed-bs019.html +++ b/doc/pub/DimRed/html/._DimRed-bs019.html @@ -189,12 +189,12 @@ MathJax.Hub.Config({

To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{n}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), $$ which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). $$ Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that diff --git a/doc/pub/DimRed/html/._DimRed-bs020.html b/doc/pub/DimRed/html/._DimRed-bs020.html index ec98cd767..1b841aa38 100644 --- a/doc/pub/DimRed/html/._DimRed-bs020.html +++ b/doc/pub/DimRed/html/._DimRed-bs020.html @@ -187,15 +187,15 @@ MathJax.Hub.Config({

PCA Proof continued

-We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write +We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write $$ -J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2. +J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2. $$

-We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by +We can show that the variance of the projected coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by $$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2, +\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2, $$ since the expectation value of @@ -208,12 +208,12 @@ where we have used the fact that our data are centered.

Recalling our definition of the covariance as $$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], $$ we have thus that $$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. +\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. $$

diff --git a/doc/pub/DimRed/html/DimRed-reveal.html b/doc/pub/DimRed/html/DimRed-reveal.html index fb83652a5..74eb2f961 100644 --- a/doc/pub/DimRed/html/DimRed-reveal.html +++ b/doc/pub/DimRed/html/DimRed-reveal.html @@ -999,15 +999,15 @@ each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \). We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as

 
$$ -J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, +J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, $$

 
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix -\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality. +\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.

-The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix). +The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix). @@ -1018,14 +1018,14 @@ The PCA theorem states that minimizing the above reconstruction error correspond To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as

 
$$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{n}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), $$

 
which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as

 
$$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). $$

 
@@ -1044,18 +1044,18 @@ where the vectors on the rhs are known.

PCA Proof continued

-We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write +We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write

 
$$ -J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2. +J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2. $$

 

-We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by +We can show that the variance of the projected coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by

 
$$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2, +\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2, $$

 
@@ -1072,14 +1072,14 @@ where we have used the fact that our data are centered. Recalling our definition of the covariance as

 
$$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], $$

 
we have thus that

 
$$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. +\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. $$

 
diff --git a/doc/pub/DimRed/html/DimRed-solarized.html b/doc/pub/DimRed/html/DimRed-solarized.html index d48c7a463..249cb656a 100644 --- a/doc/pub/DimRed/html/DimRed-solarized.html +++ b/doc/pub/DimRed/html/DimRed-solarized.html @@ -964,14 +964,14 @@ each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).

We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as $$ -J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, +J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, $$ with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix -\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality. +\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.

-The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix). +The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).











@@ -981,12 +981,12 @@ The PCA theorem states that minimizing the above reconstruction error correspond

To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{n}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), $$ which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). $$ Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that @@ -1002,15 +1002,15 @@ where the vectors on the rhs are known.

PCA Proof continued

-We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write +We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write $$ -J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2. +J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2. $$

-We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by +We can show that the variance of the projected coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by $$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2, +\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2, $$ since the expectation value of @@ -1023,12 +1023,12 @@ where we have used the fact that our data are centered.

Recalling our definition of the covariance as $$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], $$ we have thus that $$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. +\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. $$

diff --git a/doc/pub/DimRed/html/DimRed.html b/doc/pub/DimRed/html/DimRed.html index effe9e5c9..b13959662 100644 --- a/doc/pub/DimRed/html/DimRed.html +++ b/doc/pub/DimRed/html/DimRed.html @@ -969,14 +969,14 @@ each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).

We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as $$ -J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, +J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{n}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, $$ with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix -\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality. +\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.

-The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix). +The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).











@@ -986,12 +986,12 @@ The PCA theorem states that minimizing the above reconstruction error correspond

To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{n}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), $$ which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{n}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). $$ Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that @@ -1007,15 +1007,15 @@ where the vectors on the rhs are known.

PCA Proof continued

-We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write +We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write $$ -J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2. +J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2. $$

-We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by +We can show that the variance of the projected coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by $$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2, +\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2, $$ since the expectation value of @@ -1028,12 +1028,12 @@ where we have used the fact that our data are centered.

Recalling our definition of the covariance as $$ -\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], +\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T=\mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T], $$ we have thus that $$ -\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. +\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0. $$

diff --git a/doc/pub/DimRed/ipynb/DimRed.ipynb b/doc/pub/DimRed/ipynb/DimRed.ipynb index 24a0aba51..c9ff60e5b 100644 --- a/doc/pub/DimRed/ipynb/DimRed.ipynb +++ b/doc/pub/DimRed/ipynb/DimRed.ipynb @@ -1064,7 +1064,7 @@ "metadata": {}, "source": [ "$$\n", - "J(\\boldsymbol{W},\\boldsymbol{Z}) = \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i - \\overline{\\boldsymbol{x}_i})^2,\n", + "J(\\boldsymbol{W},\\boldsymbol{Z}) = \\frac{1}{n}\\sum_i (\\boldsymbol{x}_i - \\overline{\\boldsymbol{x}_i})^2,\n", "$$" ] }, @@ -1073,9 +1073,9 @@ "metadata": {}, "source": [ "with $\\overline{\\boldsymbol{x}_i} = \\boldsymbol{W}\\boldsymbol{z}_i$, where $\\boldsymbol{z}_i$ is a row vector with dimension ${\\mathbb{R}}^{n}$ of the matrix\n", - "$\\boldsymbol{Z}\\in {\\mathbb{R}}^{p\\times n}$. When doing PCA we want to reduce this dimensionality. \n", + "$\\boldsymbol{Z}\\in{\\mathbb{R}}^{p\\times n}$. When doing PCA we want to reduce this dimensionality. \n", "\n", - "The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\\boldsymbol{W}=\\boldsymbol{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\\boldsymbol{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).\n", + "The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\\boldsymbol{W}=\\boldsymbol{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $\\boldsymbol{z}_i$ with at most $l$ vectors, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).\n", "\n", "\n", "\n", @@ -1089,7 +1089,7 @@ "metadata": {}, "source": [ "$$\n", - "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i - z_{i0}\\boldsymbol{w}_0)^2=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2\\boldsymbol{w}_0^T\\boldsymbol{w}_0),\n", + "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)= \\frac{1}{n}\\sum_i (\\boldsymbol{x}_i - z_{i0}\\boldsymbol{w}_0)^2=\\frac{1}{n}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2\\boldsymbol{w}_0^T\\boldsymbol{w}_0),\n", "$$" ] }, @@ -1105,7 +1105,7 @@ "metadata": {}, "source": [ "$$\n", - "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2).\n", + "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)=\\frac{1}{n}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2).\n", "$$" ] }, @@ -1134,7 +1134,7 @@ "\n", "## PCA Proof continued\n", "\n", - "We have now found the unknown parameters $z_{i0}$. These correspond to the project coordinates and we can write" + "We have now found the unknown parameters $z_{i0}$. These correspond to the projected coordinates and we can write" ] }, { @@ -1142,7 +1142,7 @@ "metadata": {}, "source": [ "$$\n", - "J(\\boldsymbol{w}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - z_{i0}^2)=\\mathrm{const}-\\frac{1}{p}\\sum_i z_{i0}^2.\n", + "J(\\boldsymbol{w}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - z_{i0}^2)=\\mathrm{const}-\\frac{1}{n}\\sum_i z_{i0}^2.\n", "$$" ] }, @@ -1150,7 +1150,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "We can show that the variance of the project coordinates defined by $\\boldsymbol{w}_0^T\\boldsymbol{x}_i$ are given by" + "We can show that the variance of the projected coordinates defined by $\\boldsymbol{w}_0^T\\boldsymbol{x}_i$ are given by" ] }, { @@ -1158,7 +1158,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathrm{var}[\\boldsymbol{w}_0^T\\boldsymbol{x}_i] = \\frac{1}{p}\\sum_i z_{i0}^2,\n", + "\\mathrm{var}[\\boldsymbol{w}_0^T\\boldsymbol{x}_i] = \\frac{1}{n}\\sum_i z_{i0}^2,\n", "$$" ] }, @@ -1192,7 +1192,7 @@ "metadata": {}, "source": [ "$$\n", - "\\boldsymbol{C}[\\boldsymbol{x}] = \\frac{1}{n}\\boldsymbol{X}\\boldsymbol{X}^T= \\mathbb{E}[\\boldsymbol{X}\\boldsymbol{X}^T],\n", + "\\boldsymbol{C}[\\boldsymbol{x}] = \\frac{1}{n}\\boldsymbol{X}\\boldsymbol{X}^T=\\mathbb{E}[\\boldsymbol{X}\\boldsymbol{X}^T],\n", "$$" ] }, @@ -1208,7 +1208,7 @@ "metadata": {}, "source": [ "$$\n", - "\\mathrm{var}[\\boldsymbol{w}_0^T\\boldsymbol{x}_i] = \\frac{1}{p}\\sum_i z_{i0}^2=\\boldsymbol{w}_0^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0.\n", + "\\mathrm{var}[\\boldsymbol{w}_0^T\\boldsymbol{x}_i] = \\frac{1}{n}\\sum_i z_{i0}^2=\\boldsymbol{w}_0^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0.\n", "$$" ] }, diff --git a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz index e8237b747..9a7f9668c 100644 Binary files a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz and b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz differ diff --git a/doc/pub/DimRed/pdf/DimRed-minted.pdf b/doc/pub/DimRed/pdf/DimRed-minted.pdf index 3812702b5..cc02e531d 100644 Binary files a/doc/pub/DimRed/pdf/DimRed-minted.pdf and b/doc/pub/DimRed/pdf/DimRed-minted.pdf differ diff --git a/doc/src/DimRed/DimRed.do.txt b/doc/src/DimRed/DimRed.do.txt index b11e82109..d7b7864d8 100644 --- a/doc/src/DimRed/DimRed.do.txt +++ b/doc/src/DimRed/DimRed.do.txt @@ -750,13 +750,13 @@ each with dimension $\bm{x}\in {\mathbb{R}}^{n}$. We assume also that we have an orthogonal transformation $\bm{W}\in {\mathbb{R}}^{p\times p}$. We define the reconstruction error (which is similar to the mean squared error we have seen before) as !bt \[ -J(\bm{W},\bm{Z}) = \frac{1}{p}\sum_i (\bm{x}_i - \overline{\bm{x}_i})^2, +J(\bm{W},\bm{Z}) = \frac{1}{n}\sum_i (\bm{x}_i - \overline{\bm{x}_i})^2, \] !et with $\overline{\bm{x}_i} = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix -$\bm{Z}\in {\mathbb{R}}^{p\times n}$. When doing PCA we want to reduce this dimensionality. +$\bm{Z}\in{\mathbb{R}}^{p\times n}$. When doing PCA we want to reduce this dimensionality. -The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\bm{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix). +The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $\bm{z}_i$ with at most $l$ vectors, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix). @@ -766,13 +766,13 @@ The PCA theorem states that minimizing the above reconstruction error correspond To show the PCA theorem let us start with the assumption that there is one vector $\bm{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as !bt \[ -J(\bm{w}_0,\bm{z}_0)= \frac{1}{p}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0), +J(\bm{w}_0,\bm{z}_0)= \frac{1}{n}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{n}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0), \] !et which we can rewrite due to the orthogonality of $\bm{w}_i$ as !bt \[ -J(\bm{w}_0,\bm{z}_0)=\frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2). +J(\bm{w}_0,\bm{z}_0)=\frac{1}{n}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2). \] !et Minimizing $J$ with respect to the unknown parameters $z_{0i}$ we obtain that @@ -787,17 +787,17 @@ where the vectors on the rhs are known. !split ===== PCA Proof continued ===== -We have now found the unknown parameters $z_{i0}$. These correspond to the project coordinates and we can write +We have now found the unknown parameters $z_{i0}$. These correspond to the projected coordinates and we can write !bt \[ -J(\bm{w}_0)= \frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2. +J(\bm{w}_0)= \frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{n}\sum_i z_{i0}^2. \] !et -We can show that the variance of the project coordinates defined by $\bm{w}_0^T\bm{x}_i$ are given by +We can show that the variance of the projected coordinates defined by $\bm{w}_0^T\bm{x}_i$ are given by !bt \[ -\mathrm{var}[\bm{w}_0^T\bm{x}_i] = \frac{1}{p}\sum_i z_{i0}^2, +\mathrm{var}[\bm{w}_0^T\bm{x}_i] = \frac{1}{n}\sum_i z_{i0}^2, \] !et since the expectation value of @@ -811,13 +811,13 @@ where we have used the fact that our data are centered. Recalling our definition of the covariance as !bt \[ -\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}\bm{X}^T= \mathbb{E}[\bm{X}\bm{X}^T], +\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}\bm{X}^T=\mathbb{E}[\bm{X}\bm{X}^T], \] !et we have thus that !bt \[ -\mathrm{var}[\bm{w}_0^T\bm{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\bm{w}_0^T\bm{C}[\bm{x}]\bm{w}_0. +\mathrm{var}[\bm{w}_0^T\bm{x}_i] = \frac{1}{n}\sum_i z_{i0}^2=\bm{w}_0^T\bm{C}[\bm{x}]\bm{w}_0. \] !et