-
@@ -223,7 +227,7 @@ MathJax.Hub.Config({
-The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix). +The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
@@ -224,7 +228,7 @@ The PCA theorem states that minimizing the above reconstruction error correspond
-To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as +To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as $$ J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0), $$ @@ -225,6 +229,8 @@ where the vectors on the rhs are known.
+We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write +$$ +J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2. +$$ - -
# Now add PCA
-from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-pca.fit(X_train_scaled)
+
+We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by
+$$
+\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2,
+$$
+
+since the expectation value of
+$$
+\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0,
+$$
+
+where we have used the fact that our data are centered.
+
+
+Recalling our definition of the covariance as
+$$
+\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T],
+$$
+
+we have thus that
+$$
+\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0.
+$$
+
+
+We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
-X_pca = pca.transform(X_train_scaled)
-
@@ -216,6 +243,9 @@ X_pca = pca.27
-Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
-First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
+
-The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components
-
+We could trivially maximize the variance of the projection (and
+thereby minimize the error in the reconstruction function) by letting
+the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
+want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
+\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+Lagrange multiplier we can then in turn maximize
-
-
-PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
-the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
-forget to center the data first.
+$$
+J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0).
+$$
-
-Once you have identified all the principal components, you can reduce the dimensionality of the dataset
-down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
-Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
-
+Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain
+
+$$
+\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0,
+$$
+
+meaning that
+$$
+\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0.
+$$
+
+The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
+$$
+\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+$$
+
+If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see Murphy chapter 12.2. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
-
-
@@ -236,6 +239,8 @@ X2D = X_centered27
-Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
-following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
-that it automatically takes care of centering the data):
-
-After fitting the PCA transformer to the dataset, you can access the principal components using the
-components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
-principal component is equal to
-
+pca.fit(X_train_scaled)
-
-
-Another very useful piece of information is the explained variance ratio of each principal component,
-available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
-variance that lies along the axis of each principal component.
-More material to come here.
-
@@ -230,6 +218,8 @@ More material to come here.
+Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
+First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
-Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
-choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
-Unless, of course, you are reducing dimensionality for data visualization — in that case you will
-generally want to reduce the dimensionality down to 2 or 3.
-The following code computes PCA without reducing dimensionality, then computes the minimum number
-of dimensions required to preserve 95% of the training set’s variance:
+The following Python code uses NumPy’s svd() function to obtain all the principal components of the
+training set, then extracts the first two principal components. First we center the data
-
-You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
-of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
-a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
+PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
+the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
+forget to center the data first.
+
+
+Once you have identified all the principal components, you can reduce the dimensionality of the dataset
+down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
+Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
-
@@ -228,6 +238,8 @@ X_reduced = pca
-One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
-memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
-been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
-at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
-instances arrive).
+Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
+following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
+that it automatically takes care of centering the data):
+
+
+
+
+After fitting the PCA transformer to the dataset, you can access the principal components using the
+components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
+principal component is equal to
+
+
+
+
+Another very useful piece of information is the explained variance ratio of each principal component,
+available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
+variance that lies along the axis of each principal component.
+More material to come here.
@@ -209,6 +232,8 @@ instances arrive).
-Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
-algorithm that quickly finds an approximation of the first d principal components. Its computational
-complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
-previous algorithms when \( d \) is much smaller than \( n \).
-
+Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
+choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
+Unless, of course, you are reducing dimensionality for data visualization — in that case you will
+generally want to reduce the dimensionality down to 2 or 3.
+The following code computes PCA without reducing dimensionality, then computes the minimum number
+of dimensions required to preserve 95% of the training set’s variance:
-
+You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
+of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
+a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
+
+
+
@@ -212,6 +230,8 @@ previous algorithms when \( d \) is much smaller than \( n \).
+
-The kernel trick is a mathematical technique that implicitly maps instances into a
-very high-dimensional space (called the feature space), enabling nonlinear classification and regression
-with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
-space corresponds to a complex nonlinear decision boundary in the original space.
-It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
-projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
-preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
-twisted manifold.
-For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an
-
-
-
-
-
@@ -225,6 +211,8 @@ X_reduced = rbf_pca27
-Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
-(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
-algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
-closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where
-these local relationships are best preserved (more details shortly).
+Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
+algorithm that quickly finds an approximation of the first d principal components. Its computational
+complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
+previous algorithms when \( d \) is much smaller than \( n \).
+
+
+
@@ -206,6 +214,8 @@ these local relationships are best preserved (more details shortly).
-
@@ -223,7 +227,7 @@ MathJax.Hub.Config({
@@ -1004,10 +1004,10 @@ $$
-The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
@@ -1015,7 +1015,7 @@ The PCA theorem states that minimizing the above reconstruction error correspond
-To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
+To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
+We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write
+
+We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by
+
+Recalling our definition of the covariance as
+
+We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
+
+We could trivially maximize the variance of the projection (and
+thereby minimize the error in the reconstruction function) by letting
+the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
+want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
+\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+Lagrange multiplier we can then in turn maximize
+
+
@@ -1057,7 +1148,7 @@ X_pca = pca.transform(X_train_scaled)
@@ -1066,7 +1157,7 @@ First it identifies the hyperplane that lies closest to the data, and then it pr
The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components
+training set, then extracts the first two principal components. First we center the data
@@ -1094,7 +1185,7 @@ X2D = X_centered.dot(W2)
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1125,7 +1216,7 @@ More material to come here.
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1156,7 +1247,7 @@ X_reduced = pca.fit_transform(X)
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1168,7 +1259,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1182,7 +1273,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -1208,7 +1299,7 @@ X_reduced = rbf_pca.fit_transform(X)
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1220,7 +1311,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
@@ -1255,42 +1346,6 @@ display(df)
print(df.std())
display(df**2)
-Thereafter we can select specific columns only and plot final results
-
-
-
-
-We can produce a \( 4\times 4 \) matrix
-
-
-
-
-and many other operations.
-
-The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
-To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
+To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
$$
J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
$$
@@ -997,7 +999,80 @@ where the vectors on the rhs are known.
+We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write
+$$
+J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2.
+$$
+
+
+We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by
+$$
+\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2,
+$$
+
+since the expectation value of
+$$
+\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0,
+$$
+
+where we have used the fact that our data are centered.
+
+
+Recalling our definition of the covariance as
+$$
+\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T],
+$$
+
+we have thus that
+$$
+\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0.
+$$
+
+
+We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
+
+
+
+We could trivially maximize the variance of the projection (and
+thereby minimize the error in the reconstruction function) by letting
+the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
+want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
+\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+Lagrange multiplier we can then in turn maximize
+
+$$
+J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0).
+$$
+
+Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain
+
+$$
+\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0,
+$$
+
+meaning that
+$$
+\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0.
+$$
+
+The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
+$$
+\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+$$
+
+If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see Murphy chapter 12.2. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+
+
+
@@ -1012,7 +1087,7 @@ X_pca = pca.transform(X_train_scaled)
@@ -1021,7 +1096,7 @@ First it identifies the hyperplane that lies closest to the data, and then it pr
The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components
+training set, then extracts the first two principal components. First we center the data
@@ -1048,7 +1123,7 @@ X2D = X_centered.dot(W2)
-
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1079,7 +1154,7 @@ More material to come here.
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1109,7 +1184,7 @@ X_reduced = pca.fit_transform(X)
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1121,7 +1196,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1136,7 +1211,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -1165,7 +1240,7 @@ X_reduced = rbf_pca.fit_transform(X)
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1177,7 +1252,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
@@ -1212,41 +1287,6 @@ display(df)
display(df**2)
-Thereafter we can select specific columns only and plot final results
-
-
-
-
-We can produce a \( 4\times 4 \) matrix
-
-
-
-
-and many other operations.
diff --git a/doc/pub/DimRed/html/DimRed.html b/doc/pub/DimRed/html/DimRed.html
index d9d23455b..e3bc0b44a 100644
--- a/doc/pub/DimRed/html/DimRed.html
+++ b/doc/pub/DimRed/html/DimRed.html
@@ -116,15 +116,17 @@ div { text-align: justify; text-justify: inter-word; }
('The Algorithm before the Theorem', 2, None, '___sec16'),
('Classical PCA Theorem', 2, None, '___sec17'),
('Proof of the PCA Theorem', 2, None, '___sec18'),
- ('Getting started with PCA', 2, None, '___sec19'),
- ('Principal Component Analysis', 2, None, '___sec20'),
- ('PCA and scikit-learn', 2, None, '___sec21'),
- ('More on the PCA', 2, None, '___sec22'),
- ('Incremental PCA', 2, None, '___sec23'),
- ('Randomized PCA', 2, None, '___sec24'),
- ('Kernel PCA', 2, None, '___sec25'),
- ('LLE', 2, None, '___sec26'),
- ('Other techniques', 2, None, '___sec27')]}
+ ('PCA Proof continued', 2, None, '___sec19'),
+ ('The final step', 2, None, '___sec20'),
+ ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
+ ('Principal Component Analysis', 2, None, '___sec22'),
+ ('PCA and scikit-learn', 2, None, '___sec23'),
+ ('More on the PCA', 2, None, '___sec24'),
+ ('Incremental PCA', 2, None, '___sec25'),
+ ('Randomized PCA', 2, None, '___sec26'),
+ ('Kernel PCA', 2, None, '___sec27'),
+ ('LLE', 2, None, '___sec28'),
+ ('Other techniques', 2, None, '___sec29')]}
end of tocinfo -->
-
-The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
-To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
+To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
$$
J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
$$
@@ -1002,7 +1004,80 @@ where the vectors on the rhs are known.
+We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write
+$$
+J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2.
+$$
+
+
+We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by
+$$
+\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2,
+$$
+
+since the expectation value of
+$$
+\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0,
+$$
+
+where we have used the fact that our data are centered.
+
+
+Recalling our definition of the covariance as
+$$
+\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T],
+$$
+
+we have thus that
+$$
+\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0.
+$$
+
+
+We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
+
+
+
+We could trivially maximize the variance of the projection (and
+thereby minimize the error in the reconstruction function) by letting
+the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
+want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
+\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
+Lagrange multiplier we can then in turn maximize
+
+$$
+J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0).
+$$
+
+Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain
+
+$$
+\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0,
+$$
+
+meaning that
+$$
+\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0.
+$$
+
+The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
+$$
+\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+$$
+
+If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see Murphy chapter 12.2. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+
+
+
@@ -1017,7 +1092,7 @@ X_pca = pca.
@@ -1026,7 +1101,7 @@ First it identifies the hyperplane that lies closest to the data, and then it pr
The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components
+training set, then extracts the first two principal components. First we center the data
@@ -1053,7 +1128,7 @@ X2D = X_centered
-
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1084,7 +1159,7 @@ More material to come here.
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1114,7 +1189,7 @@ X_reduced = pca
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1126,7 +1201,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1141,7 +1216,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -1170,7 +1245,7 @@ X_reduced = rbf_pcaLLE
+
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1182,7 +1257,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
@@ -1217,41 +1292,6 @@ display(df)
display(df**2)
-Thereafter we can select specific columns only and plot final results
-
-
-
-
-We can produce a \( 4\times 4 \) matrix
-
-
-
-
-and many other operations.
diff --git a/doc/pub/DimRed/ipynb/DimRed.ipynb b/doc/pub/DimRed/ipynb/DimRed.ipynb
index dfc5ebe65..4ac228c70 100644
--- a/doc/pub/DimRed/ipynb/DimRed.ipynb
+++ b/doc/pub/DimRed/ipynb/DimRed.ipynb
@@ -10,7 +10,7 @@
" \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
- "Date: **Oct 22, 2019**\n",
+ "Date: **Oct 24, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -1073,15 +1073,15 @@
"metadata": {},
"source": [
"with $\\overline{\\boldsymbol{x}_i} = \\boldsymbol{W}\\boldsymbol{z}_i$, where $\\boldsymbol{z}_i$ is a row vector with dimension ${\\mathbb{R}}^{n}$ of the matrix\n",
- "$\\boldsymbol{Z}\\in {\\mathbb{R}}^{p\\times n}$. \n",
+ "$\\boldsymbol{Z}\\in {\\mathbb{R}}^{p\\times n}$. When doing PCA we want to reduce this dimensionality. \n",
"\n",
- "The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\\boldsymbol{W}=\\boldsymbol{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\\boldsymbol{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).\n",
+ "The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\\boldsymbol{W}=\\boldsymbol{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\\boldsymbol{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).\n",
"\n",
"\n",
"\n",
"## Proof of the PCA Theorem\n",
"\n",
- "To show the PCA theorem let us start with the assumption that there is a vector $\\boldsymbol{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\\boldsymbol{w}_0$ and $\\boldsymbol{z}_0$ as"
+ "To show the PCA theorem let us start with the assumption that there is one vector $\\boldsymbol{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\\boldsymbol{w}_0$ and $\\boldsymbol{z}_0$ as"
]
},
{
@@ -1131,7 +1131,170 @@
"source": [
"where the vectors on the rhs are known. \n",
"\n",
- "## Getting started with PCA"
+ "\n",
+ "## PCA Proof continued\n",
+ "\n",
+ "We have now found the unknown parameters $z_{i0}$. These correspond to the project coordinates and we can write"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "J(\\boldsymbol{w}_0)= \\frac{1}{p}\\sum_i (\\boldsymbol{x}_i^T\\boldsymbol{x}_i - z_{i0}^2)=\\mathrm{const}-\\frac{1}{p}\\sum_i z_{i0}^2.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can show that the variance of the project coordinates defined by $\\boldsymbol{w}_0^T\\boldsymbol{x}_i$ are given by"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\mathrm{var}[\\boldsymbol{w}_0^T\\boldsymbol{x}_i] = \\frac{1}{p}\\sum_i z_{i0}^2,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "since the expectation value of"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\mathbb{E}[\\boldsymbol{w}_0^T\\boldsymbol{x}_i] = \\mathbb{E}[z_{i0}]= \\boldsymbol{w}_0^T\\mathbb{E}[\\boldsymbol{x}_i]=0,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "where we have used the fact that our data are centered.\n",
+ "\n",
+ "Recalling our definition of the covariance as"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{C}[\\boldsymbol{x}] = \\frac{1}{n}\\boldsymbol{X}\\boldsymbol{X}^T= \\mathbb{E}[\\boldsymbol{X}\\boldsymbol{X}^T],\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "we have thus that"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\mathrm{var}[\\boldsymbol{w}_0^T\\boldsymbol{x}_i] = \\frac{1}{p}\\sum_i z_{i0}^2=\\boldsymbol{w}_0^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data. \n",
+ "\n",
+ "## The final step\n",
+ "\n",
+ "We could trivially maximize the variance of the projection (and\n",
+ "thereby minimize the error in the reconstruction function) by letting\n",
+ "the norm-2 of $\\boldsymbol{w}_0$ go to infinity. However, this norm since we\n",
+ "want the matrix $\\boldsymbol{W}$ to be an orthogonal matrix, is constrained by\n",
+ "$$\\vert\\vert \\boldsymbol{w}_0 \\vert\\vert_2^2=1$. Imposing this condition via a\n",
+ "Lagrange multiplier we can then in turn maximize"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "J(\\boldsymbol{w}_0)= \\boldsymbol{w}_0^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0+\\lambda_0(1-\\boldsymbol{w}_0^T\\boldsymbol{w}_0).\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Taking the derivative with respect to $\\boldsymbol{w}_0$ we obtain"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\frac{\\partial J(\\boldsymbol{w}_0)}{\\partial \\boldsymbol{w}_0}= 2\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0-2\\lambda_0\\boldsymbol{w}_0=0,\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "meaning that"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0=\\lambda_0\\boldsymbol{w}_0.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix**! If we left multiply with $\\boldsymbol{w}_0^T$ we have the variance of the projected data is"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{w}_^T\\boldsymbol{C}[\\boldsymbol{x}]\\boldsymbol{w}_0=\\lambda_0.\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function $J$ in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix $\\boldsymbol{X}$. The proof for the other eigenvectors $\\boldsymbol{w}_1,\\boldsymbol{w}_2,\\dots$ cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see [Murphy chapter 12.2](https://mitpress.mit.edu/books/machine-learning-1). The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein. \n",
+ "\n",
+ "\n",
+ "\n",
+ "\n",
+ "## PCA and Scikit-Learn Functionality"
]
},
{
@@ -1159,7 +1322,7 @@
"First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.\n",
"\n",
"The following Python code uses NumPy’s **svd()** function to obtain all the principal components of the\n",
- "training set, then extracts the first two principal components"
+ "training set, then extracts the first two principal components. First we center the data"
]
},
{
@@ -1402,69 +1565,6 @@
"print(df.std())\n",
"display(df**2)"
]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Thereafter we can select specific columns only and plot final results"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 20,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']\n",
- "df.index = np.arange(10)\n",
- "\n",
- "display(df)\n",
- "print(df['Second'].mean() )\n",
- "print(df.info())\n",
- "print(df.describe())\n",
- "\n",
- "from pylab import plt, mpl\n",
- "plt.style.use('seaborn')\n",
- "mpl.rcParams['font.family'] = 'serif'\n",
- "\n",
- "df.cumsum().plot(lw=2.0, figsize=(10,6))\n",
- "plt.show()\n",
- "\n",
- "\n",
- "df.plot.bar(figsize=(10,6), rot=15)\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can produce a $4\\times 4$ matrix"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 21,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "b = np.arange(16).reshape((4,4))\n",
- "print(b)\n",
- "df1 = pd.DataFrame(b)\n",
- "print(df1)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "and many other operations."
- ]
}
],
"metadata": {},
diff --git a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz
index d669e7d78..6437b95f4 100644
Binary files a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz and b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz differ
diff --git a/doc/pub/DimRed/pdf/DimRed-minted.pdf b/doc/pub/DimRed/pdf/DimRed-minted.pdf
index 0db06109c..8b6bcbffa 100644
Binary files a/doc/pub/DimRed/pdf/DimRed-minted.pdf and b/doc/pub/DimRed/pdf/DimRed-minted.pdf differ
diff --git a/doc/src/DimRed/DimRed.do.txt b/doc/src/DimRed/DimRed.do.txt
index 301664605..9d33bb058 100644
--- a/doc/src/DimRed/DimRed.do.txt
+++ b/doc/src/DimRed/DimRed.do.txt
@@ -754,16 +754,16 @@ J(\bm{W},\bm{Z}) = \frac{1}{p}\sum_i (\bm{x}_i - \overline{\bm{x}_i})^2,
\]
!et
with $\overline{\bm{x}_i} = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix
-$\bm{Z}\in {\mathbb{R}}^{p\times n}$.
+$\bm{Z}\in {\mathbb{R}}^{p\times n}$. When doing PCA we want to reduce this dimensionality.
-The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\bm{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
+The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $l$ $\bm{z}_i$, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
!split
===== Proof of the PCA Theorem =====
-To show the PCA theorem let us start with the assumption that there is a vector $\bm{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
+To show the PCA theorem let us start with the assumption that there is one vector $\bm{w}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
!bt
\[
J(\bm{w}_0,\bm{z}_0)= \frac{1}{p}\sum_i (\bm{x}_i - z_{i0}\bm{w}_0)^2=\frac{1}{p}\sum_i (\bm{x}_^T\bm{x}_i - 2z_{i0}\bm{w}_0^T\bm{x}_i+z_{i0}^2\bm{w}_0^T\bm{w}_0),
@@ -783,8 +783,87 @@ z_{i0}=\bm{w}_0^T\bm{x}_i,
!et
where the vectors on the rhs are known.
+
!split
-===== Getting started with PCA =====
+===== PCA Proof continued =====
+
+We have now found the unknown parameters $z_{i0}$. These correspond to the project coordinates and we can write
+!bt
+\[
+J(\bm{w}_0)= \frac{1}{p}\sum_i (\bm{x}_i^T\bm{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2.
+\]
+!et
+
+We can show that the variance of the project coordinates defined by $\bm{w}_0^T\bm{x}_i$ are given by
+!bt
+\[
+\mathrm{var}[\bm{w}_0^T\bm{x}_i] = \frac{1}{p}\sum_i z_{i0}^2,
+\]
+!et
+since the expectation value of
+!bt
+\[
+\mathbb{E}[\bm{w}_0^T\bm{x}_i] = \mathbb{E}[z_{i0}]= \bm{w}_0^T\mathbb{E}[\bm{x}_i]=0,
+\]
+!et
+where we have used the fact that our data are centered.
+
+Recalling our definition of the covariance as
+!bt
+\[
+\bm{C}[\bm{x}] = \frac{1}{n}\bm{X}\bm{X}^T= \mathbb{E}[\bm{X}\bm{X}^T],
+\]
+!et
+we have thus that
+!bt
+\[
+\mathrm{var}[\bm{w}_0^T\bm{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\bm{w}_0^T\bm{C}[\bm{x}]\bm{w}_0.
+\]
+!et
+
+We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
+
+!split
+===== The final step =====
+
+We could trivially maximize the variance of the projection (and
+thereby minimize the error in the reconstruction function) by letting
+the norm-2 of $\bm{w}_0$ go to infinity. However, this norm since we
+want the matrix $\bm{W}$ to be an orthogonal matrix, is constrained by
+$$\vert\vert \bm{w}_0 \vert\vert_2^2=1$. Imposing this condition via a
+Lagrange multiplier we can then in turn maximize
+
+!bt
+\[
+J(\bm{w}_0)= \bm{w}_0^T\bm{C}[\bm{x}]\bm{w}_0+\lambda_0(1-\bm{w}_0^T\bm{w}_0).
+\]
+!et
+Taking the derivative with respect to $\bm{w}_0$ we obtain
+
+!bt
+\[
+\frac{\partial J(\bm{w}_0)}{\partial \bm{w}_0}= 2\bm{C}[\bm{x}]\bm{w}_0-2\lambda_0\bm{w}_0=0,
+\]
+!et
+meaning that
+!bt
+\[
+\bm{C}[\bm{x}]\bm{w}_0=\lambda_0\bm{w}_0.
+\]
+!et
+_The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix_! If we left multiply with $\bm{w}_0^T$ we have the variance of the projected data is
+!bt
+\[
+\bm{w}_^T\bm{C}[\bm{x}]\bm{w}_0=\lambda_0.
+\]
+!et
+If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function $J$ in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix $\bm{X}$. The proof for the other eigenvectors $\bm{w}_1,\bm{w}_2,\dots$ cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see "Murphy chapter 12.2":"https://mitpress.mit.edu/books/machine-learning-1". The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+
+
+
+
+!split
+===== PCA and Scikit-Learn Functionality =====
!bc pycod
@@ -805,7 +884,7 @@ Principal Component Analysis (PCA) is by far the most popular dimensionality red
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
The following Python code uses NumPy’s _svd()_ function to obtain all the principal components of the
-training set, then extracts the first two principal components
+training set, then extracts the first two principal components. First we center the data
!bc pycod
X_centered = X - X.mean(axis=0)
U, s, V = np.linalg.svd(X_centered)
@@ -960,33 +1039,3 @@ print(df.std())
display(df**2)
!ec
-Thereafter we can select specific columns only and plot final results
-!bc pycod
-df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
-df.index = np.arange(10)
-
-display(df)
-print(df['Second'].mean() )
-print(df.info())
-print(df.describe())
-
-from pylab import plt, mpl
-plt.style.use('seaborn')
-mpl.rcParams['font.family'] = 'serif'
-
-df.cumsum().plot(lw=2.0, figsize=(10,6))
-plt.show()
-
-
-df.plot.bar(figsize=(10,6), rot=15)
-plt.show()
-!ec
-We can produce a $4\times 4$ matrix
-!bc pycod
-b = np.arange(16).reshape((4,4))
-print(b)
-df1 = pd.DataFrame(b)
-print(df1)
-!ec
-and many other operations.
-
Principal Component Analysis
-The final step
X_centered = X - X.mean(axis=0)
-U, s, V = np.linalg.svd(X_centered)
-c1 = V.T[:, 0]
-c2 = V.T[:, 1]
-
W2 = V.T[:, :2]
-X2D = X_centered.dot(W2)
-
PCA and scikit-learn
+PCA and Scikit-Learn Functionality
-from sklearn.decomposition import PCA
+
# Now add PCA
+from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
-X2D = pca.fit_transform(X)
-
pca.components_.T[:, 0]).
+X_pca = pca.transform(X_train_scaled)
More on the PCA
+Principal Component Analysis
+pca = PCA()
-pca.fit(X)
-cumsum = np.cumsum(pca.explained_variance_ratio_)
-d = np.argmax(cumsum >= 0.95) + 1
+
X_centered = X - X.mean(axis=0)
+U, s, V = np.linalg.svd(X_centered)
+c1 = V.T[:, 0]
+c2 = V.T[:, 1]
pca = PCA(n_components=0.95)
-X_reduced = pca.fit_transform(X)
+
W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
Incremental PCA
+PCA and scikit-learn
from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D = pca.fit_transform(X)
+
pca.components_.T[:, 0]).
+
Randomized PCA
+More on the PCA
pca = PCA()
+pca.fit(X)
+cumsum = np.cumsum(pca.explained_variance_ratio_)
+d = np.argmax(cumsum >= 0.95) + 1
+
pca = PCA(n_components=0.95)
+X_reduced = pca.fit_transform(X)
+
Kernel PCA
-Incremental PCA
from sklearn.decomposition import KernelPCA
-rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
-X_reduced = rbf_pca.fit_transform(X)
-
LLE
+Randomized PCA
Oct 22, 2019
Oct 24, 2019
-Oct 22, 2019
Oct 24, 2019
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
-\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \).
+\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
Proof of the PCA Theorem
$$
J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
@@ -1041,7 +1041,98 @@ where the vectors on the rhs are known.
Getting started with PCA
+PCA Proof continued
+
+
+$$
+J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2.
+$$
+
+
+
+$$
+\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2,
+$$
+
+
+since the expectation value of
+
+$$
+\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0,
+$$
+
+
+where we have used the fact that our data are centered.
+
+
+$$
+\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T],
+$$
+
+
+we have thus that
+
+$$
+\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0.
+$$
+
+
+The final step
+
+
+$$
+J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0).
+$$
+
+
+Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain
+
+
+$$
+\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0,
+$$
+
+
+meaning that
+
+$$
+\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0.
+$$
+
+
+The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
+
+$$
+\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
+$$
+
+
+If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see Murphy chapter 12.2. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
+PCA and Scikit-Learn Functionality
Principal Component Analysis
+Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
More on the PCA
+More on the PCA
Incremental PCA
+Incremental PCA
Randomized PCA
+Randomized PCA
Kernel PCA
+Kernel PCA
LLE
+LLE
Other techniques
+Other techniques
df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
-df.index = np.arange(10)
-
-display(df)
-print(df['Second'].mean() )
-print(df.info())
-print(df.describe())
-
-from pylab import plt, mpl
-plt.style.use('seaborn')
-mpl.rcParams['font.family'] = 'serif'
-
-df.cumsum().plot(lw=2.0, figsize=(10,6))
-plt.show()
-
-
-df.plot.bar(figsize=(10,6), rot=15)
-plt.show()
-
b = np.arange(16).reshape((4,4))
-print(b)
-df1 = pd.DataFrame(b)
-print(df1)
-
Oct 22, 2019
Oct 24, 2019
@@ -966,10 +968,10 @@ J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overli
$$
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
-\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \).
+\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
@@ -977,7 +979,7 @@ The PCA theorem states that minimizing the above reconstruction error correspond
Proof of the PCA Theorem
-Getting started with PCA
+PCA Proof continued
+
+
+
+The final step
+
+
+
+PCA and Scikit-Learn Functionality
-Principal Component Analysis
+Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
-More on the PCA
+More on the PCA
-Incremental PCA
+Incremental PCA
-Randomized PCA
+Randomized PCA
-Kernel PCA
+Kernel PCA
-LLE
+LLE
-Other techniques
+Other techniques
df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
-df.index = np.arange(10)
-
-display(df)
-print(df['Second'].mean() )
-print(df.info())
-print(df.describe())
-
-from pylab import plt, mpl
-plt.style.use('seaborn')
-mpl.rcParams['font.family'] = 'serif'
-
-df.cumsum().plot(lw=2.0, figsize=(10,6))
-plt.show()
-
-
-df.plot.bar(figsize=(10,6), rot=15)
-plt.show()
-
b = np.arange(16).reshape((4,4))
-print(b)
-df1 = pd.DataFrame(b)
-print(df1)
-
Oct 22, 2019
Oct 24, 2019
@@ -971,10 +973,10 @@ J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overli
$$
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
-\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \).
+\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
@@ -982,7 +984,7 @@ The PCA theorem states that minimizing the above reconstruction error correspond
Proof of the PCA Theorem
-Getting started with PCA
+PCA Proof continued
+
+
+
+The final step
+
+
+
+PCA and Scikit-Learn Functionality
-Principal Component Analysis
+Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
-More on the PCA
+More on the PCA
-Incremental PCA
+Incremental PCA
-Randomized PCA
+Randomized PCA
-Kernel PCA
+Kernel PCA
LLE
-Other techniques
+Other techniques
df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
-df.index = np.arange(10)
-
-display(df)
-print(df['Second'].mean() )
-print(df.info())
-print(df.describe())
-
-from pylab import plt, mpl
-plt.style.use('seaborn')
-mpl.rcParams['font.family'] = 'serif'
-
-df.cumsum().plot(lw=2.0, figsize=(10,6))
-plt.show()
-
-
-df.plot.bar(figsize=(10,6), rot=15)
-plt.show()
-
b = np.arange(16).reshape((4,4))
-print(b)
-df1 = pd.DataFrame(b)
-print(df1)
-