+Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm. +First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it. +
+The following Python code uses NumPy’s svd() function to obtain all the principal components of the +training set, then extracts the first two principal components. First we center the data using either pandas or our own code
-
# Now add PCA
-from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-pca.fit(X_train_scaled)
+import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
-X_pca = pca.transform(X_train_scaled)
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
+U, s, V = np.linalg.svd(X_centered)
+c1 = V.T[:, 0]
+c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
+
+
+PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
+the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
+forget to center the data first.
+
+
+Once you have identified all the principal components, you can reduce the dimensionality of the dataset
+down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
+Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
+
+
+
+
W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
@@ -219,7 +258,6 @@ X_pca = pca.28
29
30
- 31
»
diff --git a/doc/pub/DimRed/html/._DimRed-bs023.html b/doc/pub/DimRed/html/._DimRed-bs023.html
index cf2b2049f..51348ffde 100644
--- a/doc/pub/DimRed/html/._DimRed-bs023.html
+++ b/doc/pub/DimRed/html/._DimRed-bs023.html
@@ -93,15 +93,14 @@ Automatically generated HTML file from DocOnce source
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -160,15 +159,14 @@ MathJax.Hub.Config({
Proof of the PCA Theorem
PCA Proof continued
The final step
- PCA and Scikit-Learn Functionality
- Principal Component Analysis
- PCA and scikit-learn
- More on the PCA
- Incremental PCA
- Randomized PCA
- Kernel PCA
- LLE
- Other techniques
+ Principal Component Analysis
+ PCA and scikit-learn
+ More on the PCA
+ Incremental PCA
+ Randomized PCA
+ Kernel PCA
+ LLE
+ Other techniques
@@ -182,41 +180,38 @@ MathJax.Hub.Config({
-
+
-Principal Component Analysis
-
-
-
-Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
-First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
+
PCA and scikit-learn
-The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components. First we center the data
+Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
+following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
+that it automatically takes care of centering the data):
-
X_centered = X - X.mean(axis=0)
-U, s, V = np.linalg.svd(X_centered)
-c1 = V.T[:, 0]
-c2 = V.T[:, 1]
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D = pca.fit_transform(X)
+print(X2D)
-PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
-the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
-forget to center the data first.
-
-
-Once you have identified all the principal components, you can reduce the dimensionality of the dataset
-down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
-Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
+After fitting the PCA transformer to the dataset, you can access the principal components using the
+components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
+principal component is equal to
-
W2 = V.T[:, :2]
-X2D = X_centered.dot(W2)
+pca.components_.T[:, 0].
+
+Another very useful piece of information is the explained variance ratio of each principal component,
+available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
+variance that lies along the axis of each principal component.
+More material to come here.
+
@@ -239,7 +234,6 @@ X2D = X_centered28
29
30
- 31
»
diff --git a/doc/pub/DimRed/html/._DimRed-bs024.html b/doc/pub/DimRed/html/._DimRed-bs024.html
index 70af1ae21..c9ffe037b 100644
--- a/doc/pub/DimRed/html/._DimRed-bs024.html
+++ b/doc/pub/DimRed/html/._DimRed-bs024.html
@@ -93,15 +93,14 @@ Automatically generated HTML file from DocOnce source
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -160,15 +159,14 @@ MathJax.Hub.Config({
Proof of the PCA Theorem
PCA Proof continued
The final step
- PCA and Scikit-Learn Functionality
- Principal Component Analysis
- PCA and scikit-learn
- More on the PCA
- Incremental PCA
- Randomized PCA
- Kernel PCA
- LLE
- Other techniques
+ Principal Component Analysis
+ PCA and scikit-learn
+ More on the PCA
+ Incremental PCA
+ Randomized PCA
+ Kernel PCA
+ LLE
+ Other techniques
@@ -182,36 +180,35 @@ MathJax.Hub.Config({
-
+
-PCA and scikit-learn
+More on the PCA
-Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
-following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
-that it automatically takes care of centering the data):
+Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
+choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
+Unless, of course, you are reducing dimensionality for data visualization — in that case you will
+generally want to reduce the dimensionality down to 2 or 3.
+The following code computes PCA without reducing dimensionality, then computes the minimum number
+of dimensions required to preserve 95% of the training set’s variance:
-
from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-X2D = pca.fit_transform(X)
+pca = PCA()
+pca.fit(X)
+cumsum = np.cumsum(pca.explained_variance_ratio_)
+d = np.argmax(cumsum >= 0.95) + 1
-After fitting the PCA transformer to the dataset, you can access the principal components using the
-components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
-principal component is equal to
+You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
+of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
+a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
-
pca.components_.T[:, 0]).
+pca = PCA(n_components=0.95)
+X_reduced = pca.fit_transform(X)
-
-Another very useful piece of information is the explained variance ratio of each principal component,
-available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
-variance that lies along the axis of each principal component.
-More material to come here.
-
@@ -233,7 +230,6 @@ More material to come here.
28
29
30
- 31
»
diff --git a/doc/pub/DimRed/html/._DimRed-bs025.html b/doc/pub/DimRed/html/._DimRed-bs025.html
index 44a5f82f1..dc43d79f6 100644
--- a/doc/pub/DimRed/html/._DimRed-bs025.html
+++ b/doc/pub/DimRed/html/._DimRed-bs025.html
@@ -93,15 +93,14 @@ Automatically generated HTML file from DocOnce source
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -160,15 +159,14 @@ MathJax.Hub.Config({
Proof of the PCA Theorem
PCA Proof continued
The final step
- PCA and Scikit-Learn Functionality
- Principal Component Analysis
- PCA and scikit-learn
- More on the PCA
- Incremental PCA
- Randomized PCA
- Kernel PCA
- LLE
- Other techniques
+ Principal Component Analysis
+ PCA and scikit-learn
+ More on the PCA
+ Incremental PCA
+ Randomized PCA
+ Kernel PCA
+ LLE
+ Other techniques
@@ -184,33 +182,15 @@ MathJax.Hub.Config({
-More on the PCA
+Incremental PCA
-Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
-choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
-Unless, of course, you are reducing dimensionality for data visualization — in that case you will
-generally want to reduce the dimensionality down to 2 or 3.
-The following code computes PCA without reducing dimensionality, then computes the minimum number
-of dimensions required to preserve 95% of the training set’s variance:
-
+One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
+memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
+been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
+at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
+instances arrive).
-
-
pca = PCA()
-pca.fit(X)
-cumsum = np.cumsum(pca.explained_variance_ratio_)
-d = np.argmax(cumsum >= 0.95) + 1
-
-
-You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
-of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
-a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
-
-
-
-
pca = PCA(n_components=0.95)
-X_reduced = pca.fit_transform(X)
-
@@ -231,7 +211,6 @@ X_reduced = pca
28
29
30
- 31
»
diff --git a/doc/pub/DimRed/html/._DimRed-bs026.html b/doc/pub/DimRed/html/._DimRed-bs026.html
index 1dacaf321..a4e1b973d 100644
--- a/doc/pub/DimRed/html/._DimRed-bs026.html
+++ b/doc/pub/DimRed/html/._DimRed-bs026.html
@@ -93,15 +93,14 @@ Automatically generated HTML file from DocOnce source
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -160,15 +159,14 @@ MathJax.Hub.Config({
Proof of the PCA Theorem
PCA Proof continued
The final step
- PCA and Scikit-Learn Functionality
- Principal Component Analysis
- PCA and scikit-learn
- More on the PCA
- Incremental PCA
- Randomized PCA
- Kernel PCA
- LLE
- Other techniques
+ Principal Component Analysis
+ PCA and scikit-learn
+ More on the PCA
+ Incremental PCA
+ Randomized PCA
+ Kernel PCA
+ LLE
+ Other techniques
@@ -184,14 +182,18 @@ MathJax.Hub.Config({
-Incremental PCA
+Randomized PCA
-One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
-memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
-been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
-at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
-instances arrive).
+Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
+algorithm that quickly finds an approximation of the first d principal components. Its computational
+complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
+previous algorithms when \( d \) is much smaller than \( n \).
+
+
+
+
+
@@ -212,7 +214,6 @@ instances arrive).
28
29
30
- 31
»
diff --git a/doc/pub/DimRed/html/._DimRed-bs027.html b/doc/pub/DimRed/html/._DimRed-bs027.html
index 0aeb436e1..20413ce9d 100644
--- a/doc/pub/DimRed/html/._DimRed-bs027.html
+++ b/doc/pub/DimRed/html/._DimRed-bs027.html
@@ -93,15 +93,14 @@ Automatically generated HTML file from DocOnce source
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -160,15 +159,14 @@ MathJax.Hub.Config({
Proof of the PCA Theorem
PCA Proof continued
The final step
- PCA and Scikit-Learn Functionality
- Principal Component Analysis
- PCA and scikit-learn
- More on the PCA
- Incremental PCA
- Randomized PCA
- Kernel PCA
- LLE
- Other techniques
+ Principal Component Analysis
+ PCA and scikit-learn
+ More on the PCA
+ Incremental PCA
+ Randomized PCA
+ Kernel PCA
+ LLE
+ Other techniques
@@ -184,14 +182,28 @@ MathJax.Hub.Config({
-Randomized PCA
+Kernel PCA
+
+
+
-Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
-algorithm that quickly finds an approximation of the first d principal components. Its computational
-complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
-previous algorithms when \( d \) is much smaller than \( n \).
+The kernel trick is a mathematical technique that implicitly maps instances into a
+very high-dimensional space (called the feature space), enabling nonlinear classification and regression
+with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
+space corresponds to a complex nonlinear decision boundary in the original space.
+It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
+projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
+preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
+twisted manifold.
+For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an
+
+
+
from sklearn.decomposition import KernelPCA
+rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
+X_reduced = rbf_pca.fit_transform(X)
+
@@ -215,7 +227,6 @@ previous algorithms when \( d \) is much smaller than \( n \).
28
29
30
- 31
»
diff --git a/doc/pub/DimRed/html/._DimRed-bs028.html b/doc/pub/DimRed/html/._DimRed-bs028.html
index d3de994a0..a221d679b 100644
--- a/doc/pub/DimRed/html/._DimRed-bs028.html
+++ b/doc/pub/DimRed/html/._DimRed-bs028.html
@@ -93,15 +93,14 @@ Automatically generated HTML file from DocOnce source
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -160,15 +159,14 @@ MathJax.Hub.Config({
Proof of the PCA Theorem
PCA Proof continued
The final step
- PCA and Scikit-Learn Functionality
- Principal Component Analysis
- PCA and scikit-learn
- More on the PCA
- Incremental PCA
- Randomized PCA
- Kernel PCA
- LLE
- Other techniques
+ Principal Component Analysis
+ PCA and scikit-learn
+ More on the PCA
+ Incremental PCA
+ Randomized PCA
+ Kernel PCA
+ LLE
+ Other techniques
@@ -184,32 +182,14 @@ MathJax.Hub.Config({
-Kernel PCA
-
-
-
+
LLE
-The kernel trick is a mathematical technique that implicitly maps instances into a
-very high-dimensional space (called the feature space), enabling nonlinear classification and regression
-with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
-space corresponds to a complex nonlinear decision boundary in the original space.
-It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
-projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
-preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
-twisted manifold.
-For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an
-
-
-
-
from sklearn.decomposition import KernelPCA
-rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
-X_reduced = rbf_pca.fit_transform(X)
-
-
-
-
-
+Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
+(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
+algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
+closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where
+these local relationships are best preserved (more details shortly).
@@ -228,7 +208,6 @@ X_reduced = rbf_pca28
29
30
- 31
»
diff --git a/doc/pub/DimRed/html/._DimRed-bs029.html b/doc/pub/DimRed/html/._DimRed-bs029.html
index fd62370d4..b53f979ac 100644
--- a/doc/pub/DimRed/html/._DimRed-bs029.html
+++ b/doc/pub/DimRed/html/._DimRed-bs029.html
@@ -93,15 +93,14 @@ Automatically generated HTML file from DocOnce source
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -160,15 +159,14 @@ MathJax.Hub.Config({
Proof of the PCA Theorem
PCA Proof continued
The final step
- PCA and Scikit-Learn Functionality
- Principal Component Analysis
- PCA and scikit-learn
- More on the PCA
- Incremental PCA
- Randomized PCA
- Kernel PCA
- LLE
- Other techniques
+ Principal Component Analysis
+ PCA and scikit-learn
+ More on the PCA
+ Incremental PCA
+ Randomized PCA
+ Kernel PCA
+ LLE
+ Other techniques
@@ -184,16 +182,22 @@ MathJax.Hub.Config({
-LLE
+Other techniques
-Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
-(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
-algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
-closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where
-these local relationships are best preserved (more details shortly).
+There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
+Here are some of the most popular:
+
+
+- Multidimensional Scaling (MDS) reduces dimensionality while trying to preserve the distances between the instances.
+- Isomap creates a graph by connecting each instance to its nearest neighbors, then reduces dimensionality while trying to preserve the geodesic distances between the instances.
+- t-Distributed Stochastic Neighbor Embedding (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).
+- Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.
+
+
+
@@ -209,8 +213,6 @@ these local relationships are best preserved (more details shortly).
- 28
- 29
- 30
- - 31
- - »
diff --git a/doc/pub/DimRed/html/DimRed-bs.html b/doc/pub/DimRed/html/DimRed-bs.html
index c3c8a51c9..396bd951a 100644
--- a/doc/pub/DimRed/html/DimRed-bs.html
+++ b/doc/pub/DimRed/html/DimRed-bs.html
@@ -93,15 +93,14 @@ Automatically generated HTML file from DocOnce source
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -160,15 +159,14 @@ MathJax.Hub.Config({
Proof of the PCA Theorem
PCA Proof continued
The final step
- PCA and Scikit-Learn Functionality
- Principal Component Analysis
- PCA and scikit-learn
- More on the PCA
- Incremental PCA
- Randomized PCA
- Kernel PCA
- LLE
- Other techniques
+ Principal Component Analysis
+ PCA and scikit-learn
+ More on the PCA
+ Incremental PCA
+ Randomized PCA
+ Kernel PCA
+ LLE
+ Other techniques
@@ -227,7 +225,7 @@ MathJax.Hub.Config({
9
10
...
- 31
+ 30
»
diff --git a/doc/pub/DimRed/html/DimRed-reveal.html b/doc/pub/DimRed/html/DimRed-reveal.html
index bb2c0b941..be7cae712 100644
--- a/doc/pub/DimRed/html/DimRed-reveal.html
+++ b/doc/pub/DimRed/html/DimRed-reveal.html
@@ -1148,23 +1148,7 @@ chapter 12.4 and discussion therein.
-PCA and Scikit-Learn Functionality
-
-
-
-
-
# Now add PCA
-from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-pca.fit(X_train_scaled)
-
-X_pca = pca.transform(X_train_scaled)
-
-
-
-
-
-Principal Component Analysis
+Principal Component Analysis
@@ -1173,14 +1157,34 @@ First it identifies the hyperplane that lies closest to the data, and then it pr
The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components. First we center the data
+training set, then extracts the first two principal components. First we center the data using either pandas or our own code
-
X_centered = X - X.mean(axis=0)
+import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
+
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
U, s, V = np.linalg.svd(X_centered)
c1 = V.T[:, 0]
c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
@@ -1201,7 +1205,7 @@ X2D = X_centered.dot(W2)
-PCA and scikit-learn
+PCA and scikit-learn
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1210,9 +1214,11 @@ that it automatically takes care of centering the data):
-
from sklearn.decomposition import PCA
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
X2D = pca.fit_transform(X)
+print(X2D)
After fitting the PCA transformer to the dataset, you can access the principal components using the
@@ -1221,7 +1227,7 @@ principal component is equal to
-
pca.components_.T[:, 0]).
+pca.components_.T[:, 0].
Another very useful piece of information is the explained variance ratio of each principal component,
@@ -1232,7 +1238,7 @@ More material to come here.
-More on the PCA
+More on the PCA
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1263,7 +1269,7 @@ X_reduced = pca.fit_transform(X)
-Incremental PCA
+Incremental PCA
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1275,7 +1281,7 @@ instances arrive).
-Randomized PCA
+Randomized PCA
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1289,7 +1295,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
-Kernel PCA
+Kernel PCA
@@ -1315,7 +1321,7 @@ X_reduced = rbf_pca.fit_transform(X)
-LLE
+LLE
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1327,7 +1333,7 @@ these local relationships are best preserved (more details shortly).
-Other techniques
+Other techniques
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/DimRed/html/DimRed-solarized.html b/doc/pub/DimRed/html/DimRed-solarized.html
index 4ec81ffb3..0925346bb 100644
--- a/doc/pub/DimRed/html/DimRed-solarized.html
+++ b/doc/pub/DimRed/html/DimRed-solarized.html
@@ -113,15 +113,14 @@ div { text-align: justify; text-justify: inter-word; }
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -1088,22 +1087,7 @@ chapter 12.4 and discussion therein.
-
PCA and Scikit-Learn Functionality
-
-
-
-
-
# Now add PCA
-from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-pca.fit(X_train_scaled)
-
-X_pca = pca.transform(X_train_scaled)
-
-
-
-
-
Principal Component Analysis
+Principal Component Analysis
@@ -1112,14 +1096,34 @@ First it identifies the hyperplane that lies closest to the data, and then it pr
The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components. First we center the data
+training set, then extracts the first two principal components. First we center the data using either pandas or our own code
-
X_centered = X - X.mean(axis=0)
+import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
+
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
U, s, V = np.linalg.svd(X_centered)
c1 = V.T[:, 0]
c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
@@ -1139,7 +1143,7 @@ X2D = X_centered.dot(W2)
-
PCA and scikit-learn
+PCA and scikit-learn
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1148,9 +1152,11 @@ that it automatically takes care of centering the data):
-
from sklearn.decomposition import PCA
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
X2D = pca.fit_transform(X)
+print(X2D)
After fitting the PCA transformer to the dataset, you can access the principal components using the
@@ -1159,7 +1165,7 @@ principal component is equal to
-
pca.components_.T[:, 0]).
+pca.components_.T[:, 0].
Another very useful piece of information is the explained variance ratio of each principal component,
@@ -1170,7 +1176,7 @@ More material to come here.
-
More on the PCA
+More on the PCA
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1200,7 +1206,7 @@ X_reduced = pca.fit_transform(X)
-
Incremental PCA
+Incremental PCA
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1212,7 +1218,7 @@ instances arrive).
-
Randomized PCA
+Randomized PCA
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1227,7 +1233,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
-
Kernel PCA
+Kernel PCA
@@ -1256,7 +1262,7 @@ X_reduced = rbf_pca.fit_transform(X)
-
LLE
+LLE
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1268,7 +1274,7 @@ these local relationships are best preserved (more details shortly).
-
Other techniques
+Other techniques
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/DimRed/html/DimRed.html b/doc/pub/DimRed/html/DimRed.html
index 4d2c857e4..60f22749e 100644
--- a/doc/pub/DimRed/html/DimRed.html
+++ b/doc/pub/DimRed/html/DimRed.html
@@ -118,15 +118,14 @@ div { text-align: justify; text-justify: inter-word; }
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
- ('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
- ('Principal Component Analysis', 2, None, '___sec22'),
- ('PCA and scikit-learn', 2, None, '___sec23'),
- ('More on the PCA', 2, None, '___sec24'),
- ('Incremental PCA', 2, None, '___sec25'),
- ('Randomized PCA', 2, None, '___sec26'),
- ('Kernel PCA', 2, None, '___sec27'),
- ('LLE', 2, None, '___sec28'),
- ('Other techniques', 2, None, '___sec29')]}
+ ('Principal Component Analysis', 2, None, '___sec21'),
+ ('PCA and scikit-learn', 2, None, '___sec22'),
+ ('More on the PCA', 2, None, '___sec23'),
+ ('Incremental PCA', 2, None, '___sec24'),
+ ('Randomized PCA', 2, None, '___sec25'),
+ ('Kernel PCA', 2, None, '___sec26'),
+ ('LLE', 2, None, '___sec27'),
+ ('Other techniques', 2, None, '___sec28')]}
end of tocinfo -->
@@ -1093,22 +1092,7 @@ chapter 12.4 and discussion therein.
-
PCA and Scikit-Learn Functionality
-
-
-
-
-
# Now add PCA
-from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-pca.fit(X_train_scaled)
-
-X_pca = pca.transform(X_train_scaled)
-
-
-
-
-
Principal Component Analysis
+Principal Component Analysis
@@ -1117,14 +1101,34 @@ First it identifies the hyperplane that lies closest to the data, and then it pr
The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components. First we center the data
+training set, then extracts the first two principal components. First we center the data using either pandas or our own code
-
X_centered = X - X.mean(axis=0)
+import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
+
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
U, s, V = np.linalg.svd(X_centered)
c1 = V.T[:, 0]
c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
@@ -1144,7 +1148,7 @@ X2D = X_centered
-PCA and scikit-learn
+PCA and scikit-learn
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1153,9 +1157,11 @@ that it automatically takes care of centering the data):
-
from sklearn.decomposition import PCA
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
X2D = pca.fit_transform(X)
+print(X2D)
After fitting the PCA transformer to the dataset, you can access the principal components using the
@@ -1164,7 +1170,7 @@ principal component is equal to
-
pca.components_.T[:, 0]).
+pca.components_.T[:, 0].
Another very useful piece of information is the explained variance ratio of each principal component,
@@ -1175,7 +1181,7 @@ More material to come here.
-
More on the PCA
+More on the PCA
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1205,7 +1211,7 @@ X_reduced = pca
-
Incremental PCA
+Incremental PCA
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1217,7 +1223,7 @@ instances arrive).
-
Randomized PCA
+Randomized PCA
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1232,7 +1238,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
-
Kernel PCA
+Kernel PCA
@@ -1261,7 +1267,7 @@ X_reduced = rbf_pcaLLE
+LLE
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1273,7 +1279,7 @@ these local relationships are best preserved (more details shortly).
-
Other techniques
+Other techniques
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/DimRed/ipynb/DimRed.ipynb b/doc/pub/DimRed/ipynb/DimRed.ipynb
index 312f09d3c..c76867fcf 100644
--- a/doc/pub/DimRed/ipynb/DimRed.ipynb
+++ b/doc/pub/DimRed/ipynb/DimRed.ipynb
@@ -1308,7 +1308,14 @@
"\n",
"\n",
"\n",
- "## PCA and Scikit-Learn Functionality"
+ "\n",
+ "\n",
+ "## Principal Component Analysis\n",
+ "Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.\n",
+ "First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.\n",
+ "\n",
+ "The following Python code uses NumPy’s **svd()** function to obtain all the principal components of the\n",
+ "training set, then extracts the first two principal components. First we center the data using either **pandas** or our own code"
]
},
{
@@ -1319,38 +1326,30 @@
},
"outputs": [],
"source": [
- "# Now add PCA\n",
- "from sklearn.decomposition import PCA\n",
- "pca = PCA(n_components = 2)\n",
- "pca.fit(X_train_scaled)\n",
+ "import numpy as np\n",
+ "import pandas as pd\n",
+ "from IPython.display import display\n",
+ "np.random.seed(100)\n",
+ "# setting up a 10 x 5 vanilla matrix \n",
+ "rows = 10\n",
+ "cols = 5\n",
+ "X = np.random.randn(rows,cols)\n",
+ "df = pd.DataFrame(X)\n",
+ "# Pandas does the centering for us\n",
+ "df = df -df.mean()\n",
+ "display(df)\n",
"\n",
- "X_pca = pca.transform(X_train_scaled)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Principal Component Analysis\n",
- "Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.\n",
- "First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.\n",
- "\n",
- "The following Python code uses NumPy’s **svd()** function to obtain all the principal components of the\n",
- "training set, then extracts the first two principal components. First we center the data"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
+ "# we center it ourselves\n",
"X_centered = X - X.mean(axis=0)\n",
+ "# Then check the difference between pandas and our own set up\n",
+ "print(X_centered-df)\n",
+ "#Now we do an SVD\n",
"U, s, V = np.linalg.svd(X_centered)\n",
"c1 = V.T[:, 0]\n",
- "c2 = V.T[:, 1]"
+ "c2 = V.T[:, 1]\n",
+ "W2 = V.T[:, :2]\n",
+ "X2D = X_centered.dot(W2)\n",
+ "print(X2D)"
]
},
{
@@ -1368,7 +1367,7 @@
},
{
"cell_type": "code",
- "execution_count": 13,
+ "execution_count": 12,
"metadata": {
"collapsed": false
},
@@ -1392,15 +1391,17 @@
},
{
"cell_type": "code",
- "execution_count": 14,
+ "execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
+ "#thereafter we do a PCA with Scikit-learn\n",
"from sklearn.decomposition import PCA\n",
"pca = PCA(n_components = 2)\n",
- "X2D = pca.fit_transform(X)"
+ "X2D = pca.fit_transform(X)\n",
+ "print(X2D)"
]
},
{
@@ -1414,13 +1415,13 @@
},
{
"cell_type": "code",
- "execution_count": 15,
+ "execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
- "pca.components_.T[:, 0])."
+ "pca.components_.T[:, 0]."
]
},
{
@@ -1444,7 +1445,7 @@
},
{
"cell_type": "code",
- "execution_count": 16,
+ "execution_count": 15,
"metadata": {
"collapsed": false
},
@@ -1467,7 +1468,7 @@
},
{
"cell_type": "code",
- "execution_count": 17,
+ "execution_count": 16,
"metadata": {
"collapsed": false
},
@@ -1514,7 +1515,7 @@
},
{
"cell_type": "code",
- "execution_count": 18,
+ "execution_count": 17,
"metadata": {
"collapsed": false
},
diff --git a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz
index 0fc16f294..3b0146384 100644
Binary files a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz and b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz differ
diff --git a/doc/pub/DimRed/pdf/DimRed-minted.pdf b/doc/pub/DimRed/pdf/DimRed-minted.pdf
index 4cc94eaa2..a486cd0e6 100644
Binary files a/doc/pub/DimRed/pdf/DimRed-minted.pdf and b/doc/pub/DimRed/pdf/DimRed-minted.pdf differ
diff --git a/doc/src/DimRed/DimRed.do.txt b/doc/src/DimRed/DimRed.do.txt
index 1ab6c39ac..163230d7f 100644
--- a/doc/src/DimRed/DimRed.do.txt
+++ b/doc/src/DimRed/DimRed.do.txt
@@ -877,19 +877,6 @@ chapter 12.4 and discussion therein.
-!split
-===== PCA and Scikit-Learn Functionality =====
-
-
-!bc pycod
-# Now add PCA
-from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-pca.fit(X_train_scaled)
-
-X_pca = pca.transform(X_train_scaled)
-!ec
-
!split
@@ -899,12 +886,32 @@ Principal Component Analysis (PCA) is by far the most popular dimensionality red
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
The following Python code uses NumPy’s _svd()_ function to obtain all the principal components of the
-training set, then extracts the first two principal components. First we center the data
+training set, then extracts the first two principal components. First we center the data using either _pandas_ or our own code
!bc pycod
+import numpy as np
+import pandas as pd
+from IPython.display import display
+np.random.seed(100)
+# setting up a 10 x 5 vanilla matrix
+rows = 10
+cols = 5
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
+display(df)
+
+# we center it ourselves
X_centered = X - X.mean(axis=0)
+# Then check the difference between pandas and our own set up
+print(X_centered-df)
+#Now we do an SVD
U, s, V = np.linalg.svd(X_centered)
c1 = V.T[:, 0]
c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
!ec
PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
@@ -926,15 +933,17 @@ Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we d
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
that it automatically takes care of centering the data):
!bc pycod
+#thereafter we do a PCA with Scikit-learn
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
X2D = pca.fit_transform(X)
+print(X2D)
!ec
After fitting the PCA transformer to the dataset, you can access the principal components using the
components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
principal component is equal to
!bc pycod
-pca.components_.T[:, 0]).
+pca.components_.T[:, 0].
!ec
Another very useful piece of information is the explained variance ratio of each principal component,
available via the $explained\_variance\_ratio$ variable. It indicates the proportion of the dataset’s
diff --git a/doc/src/DimRed/PCAexample.py b/doc/src/DimRed/PCAexample.py
index d1f9885a0..94cddcdb0 100644
--- a/doc/src/DimRed/PCAexample.py
+++ b/doc/src/DimRed/PCAexample.py
@@ -1,31 +1,3 @@
-from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-pca.fit(X_train_scaled)
-
-X_pca = pca.transform(X_train_scaled)
-
-X_centered = X - X.mean(axis=0)
-U, s, V = np.linalg.svd(X_centered)
-c1 = V.T[:, 0]
-c2 = V.T[:, 1]
-
-W2 = V.T[:, :2]
-X2D = X_centered.dot(W2)
-
-pca = PCA(n_components = 2)
-X2D = pca.fit_transform(X)
-
-pca.components_.T[:, 0]).
-
-pca = PCA()
-pca.fit(X)
-cumsum = np.cumsum(pca.explained_variance_ratio_)
-d = np.argmax(cumsum >= 0.95) + 1
-
-pca = PCA(n_components=0.95)
-X_reduced = pca.fit_transform(X)
-
-
import numpy as np
import pandas as pd
from IPython.display import display
@@ -33,11 +5,27 @@ np.random.seed(100)
# setting up a 10 x 5 matrix
rows = 10
cols = 5
-a = np.random.randn(rows,cols)
-df = pd.DataFrame(a)
+X = np.random.randn(rows,cols)
+df = pd.DataFrame(X)
+# Pandas does the centering for us
+df = df -df.mean()
display(df)
-print(df.mean())
-print(df.std())
-display(df**2)
+# we center it ourselves
+X_centered = X - X.mean(axis=0)
+print(X_centered-df)
+#Now we do an SVD
+U, s, V = np.linalg.svd(X_centered)
+c1 = V.T[:, 0]
+c2 = V.T[:, 1]
+W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
+print(X2D)
+#thereafter we do a PCA with Scikit-learn
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D = pca.fit_transform(X)
+print(X2D)
+
+print(pca.components_.T[:, 0])