+We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) +each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \). + +
+We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as +$$ +J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, +$$ + +with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix +\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). + +
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix). +
diff --git a/doc/pub/DimRed/html/._DimRed-bs019.html b/doc/pub/DimRed/html/._DimRed-bs019.html index 57a96ad5b..313bb5b43 100644 --- a/doc/pub/DimRed/html/._DimRed-bs019.html +++ b/doc/pub/DimRed/html/._DimRed-bs019.html @@ -90,7 +90,7 @@ Automatically generated HTML file from DocOnce source ('Towards the PCA theorem', 2, None, '___sec15'), ('The Algorithm before the Theorem', 2, None, '___sec16'), ('Classical PCA Theorem', 2, None, '___sec17'), - ('Prof of the PCA Theorem', 2, None, '___sec18'), + ('Proof of the PCA Theorem', 2, None, '___sec18'), ('Getting started with PCA', 2, None, '___sec19'), ('Principal Component Analysis', 2, None, '___sec20'), ('PCA and scikit-learn', 2, None, '___sec21'), @@ -155,7 +155,7 @@ MathJax.Hub.Config({
+To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as +$$ +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +$$ + +which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as +$$ +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +$$ + +Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that +$$ +z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i, +$$ + +where the vectors on the rhs are known.
diff --git a/doc/pub/DimRed/html/._DimRed-bs020.html b/doc/pub/DimRed/html/._DimRed-bs020.html index 5c47f1973..11b293e38 100644 --- a/doc/pub/DimRed/html/._DimRed-bs020.html +++ b/doc/pub/DimRed/html/._DimRed-bs020.html @@ -90,7 +90,7 @@ Automatically generated HTML file from DocOnce source ('Towards the PCA theorem', 2, None, '___sec15'), ('The Algorithm before the Theorem', 2, None, '___sec16'), ('Classical PCA Theorem', 2, None, '___sec17'), - ('Prof of the PCA Theorem', 2, None, '___sec18'), + ('Proof of the PCA Theorem', 2, None, '___sec18'), ('Getting started with PCA', 2, None, '___sec19'), ('Principal Component Analysis', 2, None, '___sec20'), ('PCA and scikit-learn', 2, None, '___sec21'), @@ -155,7 +155,7 @@ MathJax.Hub.Config({
+We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) +each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \). + +
+We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as +
+$$
+J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2,
+$$
+
+
+with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
+\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \).
+
+
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
+To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as +
+$$
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
+$$
+
+
+which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
+
+$$
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
+$$
+
+
+Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that
+
+$$
+z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i,
+$$
+
+
+where the vectors on the rhs are known.
+We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) +each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \). + +
+We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as +$$ +J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, +$$ + +with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix +\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). + +
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix). +
-
+To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as +$$ +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +$$ + +which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as +$$ +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +$$ + +Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that +$$ +z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i, +$$ + +where the vectors on the rhs are known.
diff --git a/doc/pub/DimRed/html/DimRed.html b/doc/pub/DimRed/html/DimRed.html
index f6e2efdb3..d9d23455b 100644
--- a/doc/pub/DimRed/html/DimRed.html
+++ b/doc/pub/DimRed/html/DimRed.html
@@ -115,7 +115,7 @@ div { text-align: justify; text-justify: inter-word; }
('Towards the PCA theorem', 2, None, '___sec15'),
('The Algorithm before the Theorem', 2, None, '___sec16'),
('Classical PCA Theorem', 2, None, '___sec17'),
- ('Prof of the PCA Theorem', 2, None, '___sec18'),
+ ('Proof of the PCA Theorem', 2, None, '___sec18'),
('Getting started with PCA', 2, None, '___sec19'),
('Principal Component Analysis', 2, None, '___sec20'),
('PCA and scikit-learn', 2, None, '___sec21'),
@@ -947,7 +947,7 @@ $$
+We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) +each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \). + +
+We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as +$$ +J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overline{\boldsymbol{x}_i})^2, +$$ + +with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix +\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). + +
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix). +
-
+To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as +$$ +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +$$ + +which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as +$$ +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +$$ + +Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that +$$ +z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i, +$$ + +where the vectors on the rhs are known.
diff --git a/doc/pub/DimRed/ipynb/DimRed.ipynb b/doc/pub/DimRed/ipynb/DimRed.ipynb
index 538fd7b6f..dfc5ebe65 100644
--- a/doc/pub/DimRed/ipynb/DimRed.ipynb
+++ b/doc/pub/DimRed/ipynb/DimRed.ipynb
@@ -1041,7 +1041,7 @@
"source": [
"* Center the data by subtracting the mean value for each column. This leads to a new matrix $\\boldsymbol{X}\\rightarrow \\overline{\\boldsymbol{X}}$.\n",
"\n",
- "* Compute then the covariance/correlation matrix $\\mathbb{E}[\\overline{\\boldsymbol{X}}\\overline{\\boldsymbol{X}}^T].\n",
+ "* Compute then the covariance/correlation matrix $\\mathbb{E}[\\overline{\\boldsymbol{X}}\\overline{\\boldsymbol{X}}^T]$.\n",
"\n",
"* Find the eigenpairs of $\\boldsymbol{C}$ with eigenvalues $[\\lambda_0,\\lambda_1,\\dots,\\lambda_{p-1}]$ and eigenvectors $[\\boldsymbol{s}_0,\\boldsymbol{s}_1,\\dots,\\boldsymbol{s}_{p-1}]$.\n",
"\n",
@@ -1053,14 +1053,83 @@
"\n",
"## Classical PCA Theorem\n",
"\n",
+ "We assume now that we have a design matrix $\\boldsymbol{X}$ which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors $[\\boldsymbol{x}_0,\\boldsymbol{x}_1,\\dots, \\boldsymbol{x}_{p-1}]$\n",
+ "each with dimension $\\boldsymbol{x}\\in {\\mathbb{R}}^{n}$.\n",
"\n",
+ "We assume also that we have an orthogonal transformation $\\boldsymbol{W}\\in {\\mathbb{R}}^{p\\times p}$. We define the reconstruction error (which is similar to the mean squared error we have seen before) as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "J(\\boldsymbol{W},\\boldsymbol{Z}) = \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i - \\overline{\\boldsymbol{x}_i})^2,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "with $\\overline{\\boldsymbol{x}_i} = \\boldsymbol{W}\\boldsymbol{z}_i$, where $\\boldsymbol{z}_i$ is a row vector with dimension ${\\mathbb{R}}^{n}$ of the matrix\n",
+ "$\\boldsymbol{Z}\\in {\\mathbb{R}}^{p\\times n}$. \n",
"\n",
- "## Prof of the PCA Theorem\n",
- "\n",
+ "The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\\boldsymbol{W}=\\boldsymbol{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\\boldsymbol{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).\n",
"\n",
"\n",
"\n",
+ "## Proof of the PCA Theorem\n",
"\n",
+ "To show the PCA theorem let us start with the assumption that there is a vector $\\boldsymbol{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\\boldsymbol{w}_0$ and $\\boldsymbol{z}_0$ as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i - z_{i0}\\boldsymbol{w}_0)^2=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2\\boldsymbol{w}_0^T\\boldsymbol{w}_0),\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "which we can rewrite due to the orthogonality of $\\boldsymbol{w}_i$ as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2).\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Minimizing $J$ with respect to the unknown parameters $z_{0i}$ we obtain that"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "z_{i0}=\\boldsymbol{w}_0^T\\boldsymbol{x}_i,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "where the vectors on the rhs are known. \n",
"\n",
"## Getting started with PCA"
]
diff --git a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz
index 554bc3eba..d669e7d78 100644
Binary files a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz and b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz differ
diff --git a/doc/pub/DimRed/pdf/DimRed-minted.pdf b/doc/pub/DimRed/pdf/DimRed-minted.pdf
index 6ae259989..0db06109c 100644
Binary files a/doc/pub/DimRed/pdf/DimRed-minted.pdf and b/doc/pub/DimRed/pdf/DimRed-minted.pdf differ
diff --git a/doc/src/DimRed/DimRed.do.txt b/doc/src/DimRed/DimRed.do.txt
index a7124c982..301664605 100644
--- a/doc/src/DimRed/DimRed.do.txt
+++ b/doc/src/DimRed/DimRed.do.txt
@@ -734,7 +734,7 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
\]
!et
* Center the data by subtracting the mean value for each column. This leads to a new matrix $\bm{X}\rightarrow \overline{\bm{X}}$.
-* Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\bm{X}}\overline{\bm{X}}^T].
+* Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\bm{X}}\overline{\bm{X}}^T]$.
* Find the eigenpairs of $\bm{C}$ with eigenvalues $[\lambda_0,\lambda_1,\dots,\lambda_{p-1}]$ and eigenvectors $[\bm{s}_0,\bm{s}_1,\dots,\bm{s}_{p-1}]$.
* Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
* Keep only those $l$ eigenvalues larger than a selected threshold value, discarding thus $p-l$ features since we expect small variations in the data here.
@@ -744,15 +744,44 @@ After this we ask ourselves how do we prove the link between the maximum varianc
!split
===== Classical PCA Theorem =====
+We assume now that we have a design matrix $\bm{X}$ which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors $[\bm{x}_0,\bm{x}_1,\dots, \bm{x}_{p-1}]$
+each with dimension $\bm{x}\in {\mathbb{R}}^{n}$.
+
+We assume also that we have an orthogonal transformation $\bm{W}\in {\mathbb{R}}^{p\times p}$. We define the reconstruction error (which is similar to the mean squared error we have seen before) as
+!bt
+\[
+J(\bm{W},\bm{Z}) = \frac{1}{p}\sum_i (\bm{x}_i - \overline{\bm{x}_i})^2,
+\]
+!et
+with $\overline{\bm{x}_i} = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix
+$\bm{Z}\in {\mathbb{R}}^{p\times n}$.
+
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\bm{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
+
!split
-===== Prof of the PCA Theorem =====
-
-
-
-
+===== Proof of the PCA Theorem =====
+To show the PCA theorem let us start with the assumption that there is a vector $\bm{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
+!bt
+\[
+J(\bm{w}_0,\bm{z}_0)= \frac{1}{p}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0),
+\]
+!et
+which we can rewrite due to the orthogonality of $\bm{w}_i$ as
+!bt
+\[
+J(\bm{w}_0,\bm{z}_0)=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2).
+\]
+!et
+Minimizing $J$ with respect to the unknown parameters $z_{0i}$ we obtain that
+!bt
+\[
+z_{i0}=\bm{w}_0^T\bm{x}_i,
+\]
+!et
+where the vectors on the rhs are known.
!split
===== Getting started with PCA =====