fixing minor typos

2019-10-24 06:12:37 +02:00
parent 6712184d36
commit 871338800b
9 changed files with 123 additions and 30 deletions
@@ -189,12 +189,12 @@ MathJax.Hub.Config({
 <p>
 To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
 $$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
 $$

 which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
 $$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
 $$

 Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that
@@ -191,7 +191,7 @@ We could trivially maximize the variance of the projection (and
 thereby minimize the error in the reconstruction function) by letting
 the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
 want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
-\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
 Lagrange multiplier we can then in turn maximize

 $$
@@ -211,10 +211,26 @@ $$

 <b>The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix</b>! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
 $$
-\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
 $$

-If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_self">Murphy chapter 12.2</a>.  The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+<p>
+If we want to maximize the variance (minimize the construction error)
+we simply pick the eigenvector of the covariance matrix with the
+largest eigenvalue. This establishes the link between the minimization
+of the reconstruction function \( J \) in terms of an orthogonal matrix
+and the maximization of the variance and thereby the covariance of our
+observations encoded in the design/feature matrix \( \boldsymbol{X} \).
+
+<p>
+The proof
+for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be
+established by applying the above arguments and using the fact that
+our basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_self">Murphy chapter
+12.2</a>.  The
+discussion in chapter 12.2 of Murphy's text has also a nice link with
+the Singular Value Decomposition theorem. For categorical data, see
+chapter 12.4 and discussion therein.

 <p>
 <p>
@@ -1018,14 +1018,14 @@ The PCA theorem states that minimizing the above reconstruction error correspond
 To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
 <p>&nbsp;<br>
 $$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
 $$
 <p>&nbsp;<br>

 which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
 <p>&nbsp;<br>
 $$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
 $$
 <p>&nbsp;<br>

@@ -1096,7 +1096,7 @@ We could trivially maximize the variance of the projection (and
 thereby minimize the error in the reconstruction function) by letting
 the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
 want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
-\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
 Lagrange multiplier we can then in turn maximize

 <p>&nbsp;<br>
@@ -1123,11 +1123,27 @@ $$
 <b>The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix</b>! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
 <p>&nbsp;<br>
 $$
-\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
 $$
 <p>&nbsp;<br>

-If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_blank">Murphy chapter 12.2</a>.  The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+<p>
+If we want to maximize the variance (minimize the construction error)
+we simply pick the eigenvector of the covariance matrix with the
+largest eigenvalue. This establishes the link between the minimization
+of the reconstruction function \( J \) in terms of an orthogonal matrix
+and the maximization of the variance and thereby the covariance of our
+observations encoded in the design/feature matrix \( \boldsymbol{X} \).
+
+<p>
+The proof
+for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be
+established by applying the above arguments and using the fact that
+our basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_blank">Murphy chapter
+12.2</a>.  The
+discussion in chapter 12.2 of Murphy's text has also a nice link with
+the Singular Value Decomposition theorem. For categorical data, see
+chapter 12.4 and discussion therein.
 </section>


@@ -981,12 +981,12 @@ The PCA theorem states that minimizing the above reconstruction error correspond
 <p>
 To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
 $$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
 $$

 which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
 $$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
 $$

 Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that
@@ -1044,7 +1044,7 @@ We could trivially maximize the variance of the projection (and
 thereby minimize the error in the reconstruction function) by letting
 the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
 want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
-\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
 Lagrange multiplier we can then in turn maximize

 $$
@@ -1064,10 +1064,26 @@ $$

 <b>The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix</b>! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
 $$
-\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
 $$

-If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_blank">Murphy chapter 12.2</a>.  The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+<p>
+If we want to maximize the variance (minimize the construction error)
+we simply pick the eigenvector of the covariance matrix with the
+largest eigenvalue. This establishes the link between the minimization
+of the reconstruction function \( J \) in terms of an orthogonal matrix
+and the maximization of the variance and thereby the covariance of our
+observations encoded in the design/feature matrix \( \boldsymbol{X} \).
+
+<p>
+The proof
+for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be
+established by applying the above arguments and using the fact that
+our basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_blank">Murphy chapter
+12.2</a>.  The
+discussion in chapter 12.2 of Murphy's text has also a nice link with
+the Singular Value Decomposition theorem. For categorical data, see
+chapter 12.4 and discussion therein.

 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -986,12 +986,12 @@ The PCA theorem states that minimizing the above reconstruction error correspond
 <p>
 To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
 $$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
 $$

 which we can rewrite due to the orthogonality of \( \boldsymbol{w}_i \) as
 $$
-J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
+J(\boldsymbol{w}_0,\boldsymbol{z}_0)=\frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2).
 $$

 Minimizing \( J \) with respect to the unknown parameters \( z_{0i} \) we obtain that
@@ -1049,7 +1049,7 @@ We could trivially maximize the variance of the projection (and
 thereby minimize the error in the reconstruction function) by letting
 the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
 want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
-\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+\( \vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
 Lagrange multiplier we can then in turn maximize

 $$
@@ -1069,10 +1069,26 @@ $$

 <b>The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix</b>! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
 $$
-\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
 $$

-If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_blank">Murphy chapter 12.2</a>.  The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+<p>
+If we want to maximize the variance (minimize the construction error)
+we simply pick the eigenvector of the covariance matrix with the
+largest eigenvalue. This establishes the link between the minimization
+of the reconstruction function \( J \) in terms of an orthogonal matrix
+and the maximization of the variance and thereby the covariance of our
+observations encoded in the design/feature matrix \( \boldsymbol{X} \).
+
+<p>
+The proof
+for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) can be
+established by applying the above arguments and using the fact that
+our basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_blank">Murphy chapter
+12.2</a>.  The
+discussion in chapter 12.2 of Murphy's text has also a nice link with
+the Singular Value Decomposition theorem. For categorical data, see
+chapter 12.4 and discussion therein.

 <p>
 <!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1089,7 +1089,7 @@
   "metadata": {},
   "source": [
    "$$\n",
-    "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i - z_{i0}\\boldsymbol{w}_0)^2=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2\\boldsymbol{w}_0^T\\boldsymbol{w}_0),\n",
+    "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i - z_{i0}\\boldsymbol{w}_0)^2=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2\\boldsymbol{w}_0^T\\boldsymbol{w}_0),\n",
    "$$"
   ]
  },
@@ -1105,7 +1105,7 @@
   "metadata": {},
   "source": [
    "$$\n",
-    "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2).\n",
+    "J(\\boldsymbol{w}_0,\\boldsymbol{z}_0)=\\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - 2z_{i0}\\boldsymbol{w}_0^T\\boldsymbol{x}_i+z_{i0}^2).\n",
    "$$"
   ]
  },
@@ -1224,7 +1224,7 @@
    "thereby minimize the error in the reconstruction function) by letting\n",
    "the norm-2 of $\\boldsymbol{w}_0$ go to infinity. However, this norm since we\n",
    "want the matrix $\\boldsymbol{W}$ to be an orthogonal matrix, is constrained by\n",
-    "$$\\vert\\vert \\boldsymbol{w}_0 \\vert\\vert_2^2=1$. Imposing this condition via a\n",
+    "$\\vert\\vert \\boldsymbol{w}_0 \\vert\\vert_2^2=1$. Imposing this condition via a\n",
    "Lagrange multiplier we can then in turn maximize"
   ]
  },
@@ -1281,7 +1281,7 @@
   "metadata": {},
   "source": [
    "$$\n",
-    "\\boldsymbol{w}_^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0=\\lambda_0.\n",
+    "\\boldsymbol{w}_0^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0=\\lambda_0.\n",
    "$$"
   ]
  },
@@ -1289,7 +1289,21 @@
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function $J$ in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix $\\boldsymbol{X}$. The proof for the other eigenvectors $\\boldsymbol{w}_1,\\boldsymbol{w}_2,\\dots$ cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see [Murphy chapter 12.2](https://mitpress.mit.edu/books/machine-learning-1).  The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein. \n",
+    "If we want to maximize the variance (minimize the construction error)\n",
+    "we simply pick the eigenvector of the covariance matrix with the\n",
+    "largest eigenvalue. This establishes the link between the minimization\n",
+    "of the reconstruction function $J$ in terms of an orthogonal matrix\n",
+    "and the maximization of the variance and thereby the covariance of our\n",
+    "observations encoded in the design/feature matrix $\\boldsymbol{X}$.\n",
+    "\n",
+    "The proof\n",
+    "for the other eigenvectors $\\boldsymbol{w}_1,\\boldsymbol{w}_2,\\dots$ can be\n",
+    "established by applying the above arguments and using the fact that\n",
+    "our basis of eigenvectors is orthogonal, see [Murphy chapter\n",
+    "12.2](https://mitpress.mit.edu/books/machine-learning-1).  The\n",
+    "discussion in chapter 12.2 of Murphy's text has also a nice link with\n",
+    "the Singular Value Decomposition theorem. For categorical data, see\n",
+    "chapter 12.4 and discussion therein.\n",
    "\n",
    "\n",
    "\n",
@@ -766,13 +766,13 @@ The PCA theorem states that minimizing the above reconstruction error correspond
 To show the PCA theorem let us start with the assumption that there is one vector $\bm{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
 !bt
 \[
-J(\bm{w}_0,\bm{z}_0)= \frac{1}{p}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0),
+J(\bm{w}_0,\bm{z}_0)= \frac{1}{p}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0),
 \]
 !et
 which we can rewrite due to the orthogonality of $\bm{w}_i$ as
 !bt
 \[
-J(\bm{w}_0,\bm{z}_0)=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2).
+J(\bm{w}_0,\bm{z}_0)=\frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2).
 \]
 !et
 Minimizing $J$ with respect to the unknown parameters $z_{0i}$ we obtain that
@@ -830,7 +830,7 @@ We could trivially maximize the variance of the projection (and
 thereby minimize the error in the reconstruction function) by letting
 the norm-2 of $\bm{w}_0$ go to infinity. However, this norm since we
 want the matrix $\bm{W}$ to be an orthogonal matrix, is constrained by
-$$\vert\vert \bm{w}_0 \vert\vert_2^2=1$. Imposing this condition via a
+$\vert\vert \bm{w}_0 \vert\vert_2^2=1$. Imposing this condition via a
 Lagrange multiplier we can then in turn maximize

 !bt
@@ -854,10 +854,25 @@ meaning that
 _The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix_! If we left multiply with $\bm{w}_0^T$ we have the variance of the projected data is
 !bt
 \[
-\bm{w}_^T\bm{C}[\bm{x}]\bm{w}_0=\lambda_0.
+\bm{w}_0^T\bm{C}[\bm{x}]\bm{w}_0=\lambda_0.
 \]
 !et
-If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function $J$ in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix $\bm{X}$. The proof for the other eigenvectors $\bm{w}_1,\bm{w}_2,\dots$ cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see "Murphy chapter 12.2":"https://mitpress.mit.edu/books/machine-learning-1".  The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein. 
+
+If we want to maximize the variance (minimize the construction error)
+we simply pick the eigenvector of the covariance matrix with the
+largest eigenvalue. This establishes the link between the minimization
+of the reconstruction function $J$ in terms of an orthogonal matrix
+and the maximization of the variance and thereby the covariance of our
+observations encoded in the design/feature matrix $\bm{X}$.
+
+The proof
+for the other eigenvectors $\bm{w}_1,\bm{w}_2,\dots$ can be
+established by applying the above arguments and using the fact that
+our basis of eigenvectors is orthogonal, see "Murphy chapter
+12.2":"https://mitpress.mit.edu/books/machine-learning-1".  The
+discussion in chapter 12.2 of Murphy's text has also a nice link with
+the Singular Value Decomposition theorem. For categorical data, see
+chapter 12.4 and discussion therein.