To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), $$ which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). $$ Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that diff --git a/doc/pub/DimRed/html/._DimRed-bs021.html b/doc/pub/DimRed/html/._DimRed-bs021.html index 8d010f1b0..9fae4423f 100644 --- a/doc/pub/DimRed/html/._DimRed-bs021.html +++ b/doc/pub/DimRed/html/._DimRed-bs021.html @@ -191,7 +191,7 @@ We could trivially maximize the variance of the projection (and thereby minimize the error in the reconstruction function) by letting the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by -\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a +\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a Lagrange multiplier we can then in turn maximize $$ @@ -211,10 +211,26 @@ $$ The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is $$ -\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. +\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. $$ -If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see Murphy chapter 12.2. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein. +
+If we want to maximize the variance (minimize the construction error) +we simply pick the eigenvector of the covariance matrix with the +largest eigenvalue. This establishes the link between the minimization +of the reconstruction function \( J \) in terms of an orthogonal matrix +and the maximization of the variance and thereby the covariance of our +observations encoded in the design/feature matrix \( \boldsymbol{X} \). + +
+The proof +for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be +established by applying the above arguments and using the fact that +our basis of eigenvectors is orthogonal, see Murphy chapter +12.2. The +discussion in chapter 12.2 of Murphy's text has also a nice link with +the Singular Value Decomposition theorem. For categorical data, see +chapter 12.4 and discussion therein.
diff --git a/doc/pub/DimRed/html/DimRed-reveal.html b/doc/pub/DimRed/html/DimRed-reveal.html index 217576ba5..fb83652a5 100644 --- a/doc/pub/DimRed/html/DimRed-reveal.html +++ b/doc/pub/DimRed/html/DimRed-reveal.html @@ -1018,14 +1018,14 @@ The PCA theorem states that minimizing the above reconstruction error correspond To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
$$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
$$
which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
$$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
$$
@@ -1096,7 +1096,7 @@ We could trivially maximize the variance of the projection (and
thereby minimize the error in the reconstruction function) by letting
the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
-\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
Lagrange multiplier we can then in turn maximize
@@ -1123,11 +1123,27 @@ $$
The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
$$
-\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
$$
-If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see Murphy chapter 12.2. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+
+If we want to maximize the variance (minimize the construction error) +we simply pick the eigenvector of the covariance matrix with the +largest eigenvalue. This establishes the link between the minimization +of the reconstruction function \( J \) in terms of an orthogonal matrix +and the maximization of the variance and thereby the covariance of our +observations encoded in the design/feature matrix \( \boldsymbol{X} \). + +
+The proof +for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be +established by applying the above arguments and using the fact that +our basis of eigenvectors is orthogonal, see Murphy chapter +12.2. The +discussion in chapter 12.2 of Murphy's text has also a nice link with +the Singular Value Decomposition theorem. For categorical data, see +chapter 12.4 and discussion therein. diff --git a/doc/pub/DimRed/html/DimRed-solarized.html b/doc/pub/DimRed/html/DimRed-solarized.html index e9cc33dae..d48c7a463 100644 --- a/doc/pub/DimRed/html/DimRed-solarized.html +++ b/doc/pub/DimRed/html/DimRed-solarized.html @@ -981,12 +981,12 @@ The PCA theorem states that minimizing the above reconstruction error correspond
To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), $$ which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). $$ Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that @@ -1044,7 +1044,7 @@ We could trivially maximize the variance of the projection (and thereby minimize the error in the reconstruction function) by letting the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by -\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a +\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a Lagrange multiplier we can then in turn maximize $$ @@ -1064,10 +1064,26 @@ $$ The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is $$ -\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. +\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. $$ -If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see Murphy chapter 12.2. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein. +
+If we want to maximize the variance (minimize the construction error) +we simply pick the eigenvector of the covariance matrix with the +largest eigenvalue. This establishes the link between the minimization +of the reconstruction function \( J \) in terms of an orthogonal matrix +and the maximization of the variance and thereby the covariance of our +observations encoded in the design/feature matrix \( \boldsymbol{X} \). + +
+The proof +for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be +established by applying the above arguments and using the fact that +our basis of eigenvectors is orthogonal, see Murphy chapter +12.2. The +discussion in chapter 12.2 of Murphy's text has also a nice link with +the Singular Value Decomposition theorem. For categorical data, see +chapter 12.4 and discussion therein.
diff --git a/doc/pub/DimRed/html/DimRed.html b/doc/pub/DimRed/html/DimRed.html
index e3bc0b44a..effe9e5c9 100644
--- a/doc/pub/DimRed/html/DimRed.html
+++ b/doc/pub/DimRed/html/DimRed.html
@@ -986,12 +986,12 @@ The PCA theorem states that minimizing the above reconstruction error correspond
To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), +J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), $$ which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as $$ -J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). +J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2). $$ Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that @@ -1049,7 +1049,7 @@ We could trivially maximize the variance of the projection (and thereby minimize the error in the reconstruction function) by letting the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by -\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a +\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a Lagrange multiplier we can then in turn maximize $$ @@ -1069,10 +1069,26 @@ $$ The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is $$ -\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. +\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0. $$ -If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see Murphy chapter 12.2. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein. +
+If we want to maximize the variance (minimize the construction error) +we simply pick the eigenvector of the covariance matrix with the +largest eigenvalue. This establishes the link between the minimization +of the reconstruction function \( J \) in terms of an orthogonal matrix +and the maximization of the variance and thereby the covariance of our +observations encoded in the design/feature matrix \( \boldsymbol{X} \). + +
+The proof +for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be +established by applying the above arguments and using the fact that +our basis of eigenvectors is orthogonal, see Murphy chapter +12.2. The +discussion in chapter 12.2 of Murphy's text has also a nice link with +the Singular Value Decomposition theorem. For categorical data, see +chapter 12.4 and discussion therein.
diff --git a/doc/pub/DimRed/ipynb/DimRed.ipynb b/doc/pub/DimRed/ipynb/DimRed.ipynb
index 4ac228c70..24a0aba51 100644
--- a/doc/pub/DimRed/ipynb/DimRed.ipynb
+++ b/doc/pub/DimRed/ipynb/DimRed.ipynb
@@ -1089,7 +1089,7 @@
"metadata": {},
"source": [
"$$\n",
- "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i - z_{i0}\\boldsymbol{w}_0)^2=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2\\boldsymbol{w}_0^T\\boldsymbol{w}_0),\n",
+ "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i - z_{i0}\\boldsymbol{w}_0)^2=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2\\boldsymbol{w}_0^T\\boldsymbol{w}_0),\n",
"$$"
]
},
@@ -1105,7 +1105,7 @@
"metadata": {},
"source": [
"$$\n",
- "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2).\n",
+ "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2).\n",
"$$"
]
},
@@ -1224,7 +1224,7 @@
"thereby minimize the error in the reconstruction function) by letting\n",
"the norm-2 of $\\boldsymbol{w}_0$ go to infinity. However, this norm since we\n",
"want the matrix $\\boldsymbol{W}$ to be an orthogonal matrix, is constrained by\n",
- "$$\\vert\\vert \\boldsymbol{w}_0 \\vert\\vert_2^2=1$. Imposing this condition via a\n",
+ "$\\vert\\vert \\boldsymbol{w}_0 \\vert\\vert_2^2=1$. Imposing this condition via a\n",
"Lagrange multiplier we can then in turn maximize"
]
},
@@ -1281,7 +1281,7 @@
"metadata": {},
"source": [
"$$\n",
- "\\boldsymbol{w}_^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0=\\lambda_0.\n",
+ "\\boldsymbol{w}_0^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0=\\lambda_0.\n",
"$$"
]
},
@@ -1289,7 +1289,21 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function $J$ in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix $\\boldsymbol{X}$. The proof for the other eigenvectors $\\boldsymbol{w}_1,\\boldsymbol{w}_2,\\dots$ cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see [Murphy chapter 12.2](https://mitpress.mit.edu/books/machine-learning-1). The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein. \n",
+ "If we want to maximize the variance (minimize the construction error)\n",
+ "we simply pick the eigenvector of the covariance matrix with the\n",
+ "largest eigenvalue. This establishes the link between the minimization\n",
+ "of the reconstruction function $J$ in terms of an orthogonal matrix\n",
+ "and the maximization of the variance and thereby the covariance of our\n",
+ "observations encoded in the design/feature matrix $\\boldsymbol{X}$.\n",
+ "\n",
+ "The proof\n",
+ "for the other eigenvectors $\\boldsymbol{w}_1,\\boldsymbol{w}_2,\\dots$ can be\n",
+ "established by applying the above arguments and using the fact that\n",
+ "our basis of eigenvectors is orthogonal, see [Murphy chapter\n",
+ "12.2](https://mitpress.mit.edu/books/machine-learning-1). The\n",
+ "discussion in chapter 12.2 of Murphy's text has also a nice link with\n",
+ "the Singular Value Decomposition theorem. For categorical data, see\n",
+ "chapter 12.4 and discussion therein.\n",
"\n",
"\n",
"\n",
diff --git a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz
index 6437b95f4..e8237b747 100644
Binary files a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz and b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz differ
diff --git a/doc/pub/DimRed/pdf/DimRed-minted.pdf b/doc/pub/DimRed/pdf/DimRed-minted.pdf
index 8b6bcbffa..3812702b5 100644
Binary files a/doc/pub/DimRed/pdf/DimRed-minted.pdf and b/doc/pub/DimRed/pdf/DimRed-minted.pdf differ
diff --git a/doc/src/DimRed/DimRed.do.txt b/doc/src/DimRed/DimRed.do.txt
index 9d33bb058..b11e82109 100644
--- a/doc/src/DimRed/DimRed.do.txt
+++ b/doc/src/DimRed/DimRed.do.txt
@@ -766,13 +766,13 @@ The PCA theorem states that minimizing the above reconstruction error correspond
To show the PCA theorem let us start with the assumption that there is one vector $\bm{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
!bt
\[
-J(\bm{w}_0,\bm{z}_0)= \frac{1}{p}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0),
+J(\bm{w}_0,\bm{z}_0)= \frac{1}{p}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0),
\]
!et
which we can rewrite due to the orthogonality of $\bm{w}_i$ as
!bt
\[
-J(\bm{w}_0,\bm{z}_0)=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2).
+J(\bm{w}_0,\bm{z}_0)=\frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2).
\]
!et
Minimizing $J$ with respect to the unknown parameters $z_{0i}$ we obtain that
@@ -830,7 +830,7 @@ We could trivially maximize the variance of the projection (and
thereby minimize the error in the reconstruction function) by letting
the norm-2 of $\bm{w}_0$ go to infinity. However, this norm since we
want the matrix $\bm{W}$ to be an orthogonal matrix, is constrained by
-$$\vert\vert \bm{w}_0 \vert\vert_2^2=1$. Imposing this condition via a
+$\vert\vert \bm{w}_0 \vert\vert_2^2=1$. Imposing this condition via a
Lagrange multiplier we can then in turn maximize
!bt
@@ -854,10 +854,25 @@ meaning that
_The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix_! If we left multiply with $\bm{w}_0^T$ we have the variance of the projected data is
!bt
\[
-\bm{w}_^T\bm{C}[\bm{x}]\bm{w}_0=\lambda_0.
+\bm{w}_0^T\bm{C}[\bm{x}]\bm{w}_0=\lambda_0.
\]
!et
-If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function $J$ in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix $\bm{X}$. The proof for the other eigenvectors $\bm{w}_1,\bm{w}_2,\dots$ cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see "Murphy chapter 12.2":"https://mitpress.mit.edu/books/machine-learning-1". The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+
+If we want to maximize the variance (minimize the construction error)
+we simply pick the eigenvector of the covariance matrix with the
+largest eigenvalue. This establishes the link between the minimization
+of the reconstruction function $J$ in terms of an orthogonal matrix
+and the maximization of the variance and thereby the covariance of our
+observations encoded in the design/feature matrix $\bm{X}$.
+
+The proof
+for the other eigenvectors $\bm{w}_1,\bm{w}_2,\dots$ can be
+established by applying the above arguments and using the fact that
+our basis of eigenvectors is orthogonal, see "Murphy chapter
+12.2":"https://mitpress.mit.edu/books/machine-learning-1". The
+discussion in chapter 12.2 of Murphy's text has also a nice link with
+the Singular Value Decomposition theorem. For categorical data, see
+chapter 12.4 and discussion therein.