+The eigenvalues tell us then how much we need to stretch the +corresponding eigenvectors. Dimensions with large eigenvalues have +thus large variations (large variance) and define therefore useful +dimensions. The data points are more spread out in the direction of +these eigenvectors. Smaller eigenvalues mean on the other hand that +the corresponding eigenvectors are shrunk accordingly and the data +points are tightly bunched together and there is not much variation in +these specific directions. Hopefully then we could leave it out +dimensions where the eigenvalues are very small. If \( p \) is very large, +we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \) +features/predictors. +
@@ -239,7 +254,7 @@ In the derivation of the PCA theorem we will assume that the eigenvalues are ord
+Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. + +
@@ -206,7 +237,7 @@ MathJax.Hub.Config({
@@ -205,6 +207,8 @@ MathJax.Hub.Config({
- - -
# Now add PCA
-from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-pca.fit(X_train_scaled)
-
-X_pca = pca.transform(X_train_scaled)
-
@@ -214,6 +206,7 @@ X_pca = pca.26
-Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
-First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
+
-The following Python code uses NumPy’s svd() function to obtain all the principal components of the
-training set, then extracts the first two principal components
-
-PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
-the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
-forget to center the data first.
+
-Once you have identified all the principal components, you can reduce the dimensionality of the dataset
-down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
-Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
-
-
-
-
@@ -234,6 +215,7 @@ X2D = X_centered26
+Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
+First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
-Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
-following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
-that it automatically takes care of centering the data):
+The following Python code uses NumPy’s svd() function to obtain all the principal components of the
+training set, then extracts the first two principal components
-
-After fitting the PCA transformer to the dataset, you can access the principal components using the
-components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
-principal component is equal to
+PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
+the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
+forget to center the data first.
+
+
+Once you have identified all the principal components, you can reduce the dimensionality of the dataset
+down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
+Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
-
-Another very useful piece of information is the explained variance ratio of each principal component,
-available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
-variance that lies along the axis of each principal component.
-More material to come here.
-
@@ -228,6 +235,7 @@ More material to come here.
-Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
-choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
-Unless, of course, you are reducing dimensionality for data visualization — in that case you will
-generally want to reduce the dimensionality down to 2 or 3.
-The following code computes PCA without reducing dimensionality, then computes the minimum number
-of dimensions required to preserve 95% of the training set’s variance:
+Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
+following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
+that it automatically takes care of centering the data):
-
-You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
-of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
-a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
+After fitting the PCA transformer to the dataset, you can access the principal components using the
+components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
+principal component is equal to
-
+Another very useful piece of information is the explained variance ratio of each principal component,
+available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
+variance that lies along the axis of each principal component.
+More material to come here.
+
@@ -226,6 +229,7 @@ X_reduced = pca
-One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
-memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
-been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
-at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
-instances arrive).
+Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
+choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
+Unless, of course, you are reducing dimensionality for data visualization — in that case you will
+generally want to reduce the dimensionality down to 2 or 3.
+The following code computes PCA without reducing dimensionality, then computes the minimum number
+of dimensions required to preserve 95% of the training set’s variance:
+
+
+
+You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
+of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
+a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
+
+
+
+
@@ -207,6 +227,7 @@ instances arrive).
-Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
-algorithm that quickly finds an approximation of the first d principal components. Its computational
-complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
-previous algorithms when \( d \) is much smaller than \( n \).
-
-
-
@@ -210,6 +208,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
+
-The kernel trick is a mathematical technique that implicitly maps instances into a
-very high-dimensional space (called the feature space), enabling nonlinear classification and regression
-with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
-space corresponds to a complex nonlinear decision boundary in the original space.
-It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
-projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
-preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
-twisted manifold.
-For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an
-
+Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
+algorithm that quickly finds an approximation of the first d principal components. Its computational
+complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
+previous algorithms when \( d \) is much smaller than \( n \).
-
-
-Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
-(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
-algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
-closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where
-these local relationships are best preserved (more details shortly).
+The kernel trick is a mathematical technique that implicitly maps instances into a
+very high-dimensional space (called the feature space), enabling nonlinear classification and regression
+with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
+space corresponds to a complex nonlinear decision boundary in the original space.
+It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
+projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
+preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
+twisted manifold.
+For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an
+
+
+
+
+
@@ -204,6 +224,7 @@ these local relationships are best preserved (more details shortly).
-There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
+Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
+(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
+algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
+closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where
+these local relationships are best preserved (more details shortly).
-Here are some of the most popular:
-
-
-
-
-
-Thereafter we can select specific columns only and plot final results
-
-
-
-
-We can produce a \( 4\times 4 \) matrix
-
-
-
-
-and many other operations.
-
In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
+
+
+The eigenvalues tell us then how much we need to stretch the
+corresponding eigenvectors. Dimensions with large eigenvalues have
+thus large variations (large variance) and define therefore useful
+dimensions. The data points are more spread out in the direction of
+these eigenvectors. Smaller eigenvalues mean on the other hand that
+the corresponding eigenvectors are shrunk accordingly and the data
+points are tightly bunched together and there is not much variation in
+these specific directions. Hopefully then we could leave it out
+dimensions where the eigenvalues are very small. If \( p \) is very large,
+we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
+features/predictors.
+Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.
+
+
+
+After this we ask ourselves how do we prove the link between the maximum variance and the feature reduction.
@@ -966,7 +1015,7 @@ X_pca = pca.transform(X_train_scaled)
@@ -1003,7 +1052,7 @@ X2D = X_centered.dot(W2)
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1034,7 +1083,7 @@ More material to come here.
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1065,7 +1114,7 @@ X_reduced = pca.fit_transform(X)
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1077,7 +1126,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1091,7 +1140,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -1117,7 +1166,7 @@ X_reduced = rbf_pca.fit_transform(X)
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1129,7 +1178,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/DimRed/html/DimRed-solarized.html b/doc/pub/DimRed/html/DimRed-solarized.html
index 959e73c3b..006dc5c14 100644
--- a/doc/pub/DimRed/html/DimRed-solarized.html
+++ b/doc/pub/DimRed/html/DimRed-solarized.html
@@ -108,17 +108,18 @@ div { text-align: justify; text-justify: inter-word; }
None,
'___sec14'),
('Towards the PCA theorem', 2, None, '___sec15'),
- ('Classical PCA Theorem', 2, None, '___sec16'),
- ('Prof of the PCA Theorem', 2, None, '___sec17'),
- ('Getting started with PCA', 2, None, '___sec18'),
- ('Principal Component Analysis', 2, None, '___sec19'),
- ('PCA and scikit-learn', 2, None, '___sec20'),
- ('More on the PCA', 2, None, '___sec21'),
- ('Incremental PCA', 2, None, '___sec22'),
- ('Randomized PCA', 2, None, '___sec23'),
- ('Kernel PCA', 2, None, '___sec24'),
- ('LLE', 2, None, '___sec25'),
- ('Other techniques', 2, None, '___sec26')]}
+ ('The Algorithm before the Theorem', 2, None, '___sec16'),
+ ('Classical PCA Theorem', 2, None, '___sec17'),
+ ('Prof of the PCA Theorem', 2, None, '___sec18'),
+ ('Getting started with PCA', 2, None, '___sec19'),
+ ('Principal Component Analysis', 2, None, '___sec20'),
+ ('PCA and scikit-learn', 2, None, '___sec21'),
+ ('More on the PCA', 2, None, '___sec22'),
+ ('Incremental PCA', 2, None, '___sec23'),
+ ('Randomized PCA', 2, None, '___sec24'),
+ ('Kernel PCA', 2, None, '___sec25'),
+ ('LLE', 2, None, '___sec26'),
+ ('Other techniques', 2, None, '___sec27')]}
end of tocinfo -->
-
+Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.
+
+
+
+
@@ -930,7 +978,7 @@ X_pca = pca.transform(X_train_scaled)
@@ -966,7 +1014,7 @@ X2D = X_centered.dot(W2)
-
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -997,7 +1045,7 @@ More material to come here.
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1027,7 +1075,7 @@ X_reduced = pca.fit_transform(X)
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1039,7 +1087,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1054,7 +1102,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -1083,7 +1131,7 @@ X_reduced = rbf_pca.fit_transform(X)
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1095,7 +1143,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/DimRed/html/DimRed.html b/doc/pub/DimRed/html/DimRed.html
index f30d112ee..f6e2efdb3 100644
--- a/doc/pub/DimRed/html/DimRed.html
+++ b/doc/pub/DimRed/html/DimRed.html
@@ -113,17 +113,18 @@ div { text-align: justify; text-justify: inter-word; }
None,
'___sec14'),
('Towards the PCA theorem', 2, None, '___sec15'),
- ('Classical PCA Theorem', 2, None, '___sec16'),
- ('Prof of the PCA Theorem', 2, None, '___sec17'),
- ('Getting started with PCA', 2, None, '___sec18'),
- ('Principal Component Analysis', 2, None, '___sec19'),
- ('PCA and scikit-learn', 2, None, '___sec20'),
- ('More on the PCA', 2, None, '___sec21'),
- ('Incremental PCA', 2, None, '___sec22'),
- ('Randomized PCA', 2, None, '___sec23'),
- ('Kernel PCA', 2, None, '___sec24'),
- ('LLE', 2, None, '___sec25'),
- ('Other techniques', 2, None, '___sec26')]}
+ ('The Algorithm before the Theorem', 2, None, '___sec16'),
+ ('Classical PCA Theorem', 2, None, '___sec17'),
+ ('Prof of the PCA Theorem', 2, None, '___sec18'),
+ ('Getting started with PCA', 2, None, '___sec19'),
+ ('Principal Component Analysis', 2, None, '___sec20'),
+ ('PCA and scikit-learn', 2, None, '___sec21'),
+ ('More on the PCA', 2, None, '___sec22'),
+ ('Incremental PCA', 2, None, '___sec23'),
+ ('Randomized PCA', 2, None, '___sec24'),
+ ('Kernel PCA', 2, None, '___sec25'),
+ ('LLE', 2, None, '___sec26'),
+ ('Other techniques', 2, None, '___sec27')]}
end of tocinfo -->
-
+Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.
+
+
+
+
@@ -935,7 +983,7 @@ X_pca = pca.
@@ -971,7 +1019,7 @@ X2D = X_centered
-
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1002,7 +1050,7 @@ More material to come here.
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1032,7 +1080,7 @@ X_reduced = pca
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1044,7 +1092,7 @@ instances arrive).
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1059,7 +1107,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
@@ -1088,7 +1136,7 @@ X_reduced = rbf_pcaLLE
+
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1100,7 +1148,7 @@ these local relationships are best preserved (more details shortly).
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
diff --git a/doc/pub/DimRed/ipynb/DimRed.ipynb b/doc/pub/DimRed/ipynb/DimRed.ipynb
index f2c182c39..538fd7b6f 100644
--- a/doc/pub/DimRed/ipynb/DimRed.ipynb
+++ b/doc/pub/DimRed/ipynb/DimRed.ipynb
@@ -1000,6 +1000,57 @@
"In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is\n",
"$\\lambda_0 > \\lambda_1 > \\dots > \\lambda_{p-1}$. \n",
"\n",
+ "\n",
+ "The eigenvalues tell us then how much we need to stretch the\n",
+ "corresponding eigenvectors. Dimensions with large eigenvalues have\n",
+ "thus large variations (large variance) and define therefore useful\n",
+ "dimensions. The data points are more spread out in the direction of\n",
+ "these eigenvectors. Smaller eigenvalues mean on the other hand that\n",
+ "the corresponding eigenvectors are shrunk accordingly and the data\n",
+ "points are tightly bunched together and there is not much variation in\n",
+ "these specific directions. Hopefully then we could leave it out\n",
+ "dimensions where the eigenvalues are very small. If $p$ is very large,\n",
+ "we could then aim at reducing $p$ to $l << p$ and handle only $l$\n",
+ "features/predictors.\n",
+ "\n",
+ "## The Algorithm before theorem\n",
+ "\n",
+ "Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. \n",
+ "* Set up the datapoints for the design/feature matrix $\\boldsymbol{X}$ with $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors/features $p$ referring to the column numbers and the entries $n$ being the row elements."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "$$\n",
+ "\\boldsymbol{X}=\\begin{bmatrix}\n",
+ "x_{0,0} & x_{0,1} & x_{0,2}& \\dots & \\dots x_{0,p-1}\\\\\n",
+ "x_{1,0} & x_{1,1} & x_{1,2}& \\dots & \\dots x_{1,p-1}\\\\\n",
+ "x_{2,0} & x_{2,1} & x_{2,2}& \\dots & \\dots x_{2,p-1}\\\\\n",
+ "\\dots & \\dots & \\dots & \\dots \\dots & \\dots \\\\\n",
+ "x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \\dots & \\dots x_{n-2,p-1}\\\\\n",
+ "x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \\dots & \\dots x_{n-1,p-1}\\\\\n",
+ "\\end{bmatrix},\n",
+ "$$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* Center the data by subtracting the mean value for each column. This leads to a new matrix $\\boldsymbol{X}\\rightarrow \\overline{\\boldsymbol{X}}$.\n",
+ "\n",
+ "* Compute then the covariance/correlation matrix $\\mathbb{E}[\\overline{\\boldsymbol{X}}\\overline{\\boldsymbol{X}}^T].\n",
+ "\n",
+ "* Find the eigenpairs of $\\boldsymbol{C}$ with eigenvalues $[\\lambda_0,\\lambda_1,\\dots,\\lambda_{p-1}]$ and eigenvectors $[\\boldsymbol{s}_0,\\boldsymbol{s}_1,\\dots,\\boldsymbol{s}_{p-1}]$.\n",
+ "\n",
+ "* Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.\n",
+ "\n",
+ "* Keep only those $l$ eigenvalues larger than a selected threshold value, discarding thus $p-l$ features since we expect small variations in the data here.\n",
+ "\n",
+ "After this we ask ourselves how do we prove the link between the maximum variance and the feature reduction.\n",
+ "\n",
"## Classical PCA Theorem\n",
"\n",
"\n",
diff --git a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz
index 0eef6c592..554bc3eba 100644
Binary files a/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz and b/doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz differ
diff --git a/doc/pub/DimRed/pdf/DimRed-minted.pdf b/doc/pub/DimRed/pdf/DimRed-minted.pdf
index 035ad9731..6ae259989 100644
Binary files a/doc/pub/DimRed/pdf/DimRed-minted.pdf and b/doc/pub/DimRed/pdf/DimRed-minted.pdf differ
diff --git a/doc/src/DimRed/DimRed.do.txt b/doc/src/DimRed/DimRed.do.txt
index 3db103be5..a7124c982 100644
--- a/doc/src/DimRed/DimRed.do.txt
+++ b/doc/src/DimRed/DimRed.do.txt
@@ -703,6 +703,44 @@ and since $\bm{C}[\bm{y}]$ is diagonal we have for a given eigenvalue $i$ of the
In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
$\lambda_0 > \lambda_1 > \dots > \lambda_{p-1}$.
+
+The eigenvalues tell us then how much we need to stretch the
+corresponding eigenvectors. Dimensions with large eigenvalues have
+thus large variations (large variance) and define therefore useful
+dimensions. The data points are more spread out in the direction of
+these eigenvectors. Smaller eigenvalues mean on the other hand that
+the corresponding eigenvectors are shrunk accordingly and the data
+points are tightly bunched together and there is not much variation in
+these specific directions. Hopefully then we could leave it out
+dimensions where the eigenvalues are very small. If $p$ is very large,
+we could then aim at reducing $p$ to $l << p$ and handle only $l$
+features/predictors.
+
+!split
+===== The Algorithm before the Theorem =====
+
+Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.
+* Set up the datapoints for the design/feature matrix $\bm{X}$ with $\bm{X}\in {\mathbb{R}}^{n\times p}$, with the predictors/features $p$ referring to the column numbers and the entries $n$ being the row elements.
+!bt
+\[
+\bm{X}=\begin{bmatrix}
+x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
+x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
+x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
+\dots & \dots & \dots & \dots \dots & \dots \\
+x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
+x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
+\end{bmatrix},
+\]
+!et
+* Center the data by subtracting the mean value for each column. This leads to a new matrix $\bm{X}\rightarrow \overline{\bm{X}}$.
+* Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\bm{X}}\overline{\bm{X}}^T].
+* Find the eigenpairs of $\bm{C}$ with eigenvalues $[\lambda_0,\lambda_1,\dots,\lambda_{p-1}]$ and eigenvectors $[\bm{s}_0,\bm{s}_1,\dots,\bm{s}_{p-1}]$.
+* Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
+* Keep only those $l$ eigenvalues larger than a selected threshold value, discarding thus $p-l$ features since we expect small variations in the data here.
+
+After this we ask ourselves how do we prove the link between the maximum variance and the feature reduction.
+
!split
===== Classical PCA Theorem =====
diff --git a/doc/src/DimRed/mlpfranke.py b/doc/src/DimRed/mlpfranke.py
new file mode 100644
index 000000000..639e605ef
--- /dev/null
+++ b/doc/src/DimRed/mlpfranke.py
@@ -0,0 +1,68 @@
+# Common imports
+import numpy as np
+from sklearn.neural_network import MLPRegressor
+from sklearn.metrics import accuracy_score
+import seaborn as sns
+import matplotlib.pyplot as plt
+
+def FrankeFunction(x,y):
+ term1 = 0.75*np.exp(-(0.25*(9*x-2)**2) - 0.25*((9*y-2)**2))
+ term2 = 0.75*np.exp(-((9*x+1)**2)/49.0 - 0.1*(9*y+1))
+ term3 = 0.5*np.exp(-(9*x-7)**2/4.0 - 0.25*((9*y-3)**2))
+ term4 = -0.2*np.exp(-(9*x-4)**2 - (9*y-7)**2)
+ return term1 + term2 + term3 + term4
+
+
+def create_X(x, y, n ):
+ if len(x.shape) > 1:
+ x = np.ravel(x)
+ y = np.ravel(y)
+
+ N = len(x)
+ l = int((n+1)*(n+2)/2) # Number of elements in beta
+ X = np.ones((N,l))
+
+ for i in range(1,n+1):
+ q = int((i)*(i+1)/2)
+ for k in range(i+1):
+ X[:,q+k] = (x**(i-k))*(y**k)
+
+ return X
+
+
+# Making meshgrid of datapoints and compute Franke's function
+n = 4
+N = 100
+x = np.sort(np.random.uniform(0, 1, N))
+y = np.sort(np.random.uniform(0, 1, N))
+z = FrankeFunction(x, y)
+X = create_X(x, y, n=n)
+
+# only training data, no advanced splitting
+X_train = X
+Y_train = z
+# only one simple layer with 100 neurons
+n_hidden_neurons = 100
+epochs = 100
+# store models for later use
+eta_vals = np.logspace(-5, 1, 7)
+lmbd_vals = np.logspace(-5, 1, 7)
+# store the models for later use
+DNN_scikit = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)
+train_accuracy = np.zeros((len(eta_vals), len(lmbd_vals)))
+sns.set()
+for i, eta in enumerate(eta_vals):
+ for j, lmbd in enumerate(lmbd_vals):
+ dnn = MLPRegressor(hidden_layer_sizes=(n_hidden_neurons), activation='logistic',
+ alpha=lmbd, learning_rate_init=eta, max_iter=epochs)
+ dnn.fit(X_train, Y_train)
+ DNN_scikit[i][j] = dnn
+ train_accuracy[i][j] = dnn.score(X_train, Y_train)
+
+fig, ax = plt.subplots(figsize = (10, 10))
+sns.heatmap(train_accuracy, annot=True, ax=ax, cmap="viridis")
+ax.set_title("Training Accuracy")
+ax.set_ylabel("$\eta$")
+ax.set_xlabel("$\lambda$")
+plt.show()
+
Principal Component Analysis
-Getting started with PCA
-X_centered = X - X.mean(axis=0)
-U, s, V = np.linalg.svd(X_centered)
-c1 = V.T[:, 0]
-c2 = V.T[:, 1]
-
# Now add PCA
+from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+pca.fit(X_train_scaled)
-
W2 = V.T[:, :2]
-X2D = X_centered.dot(W2)
+X_pca = pca.transform(X_train_scaled)
PCA and scikit-learn
+Principal Component Analysis
+from sklearn.decomposition import PCA
-pca = PCA(n_components = 2)
-X2D = pca.fit_transform(X)
+
X_centered = X - X.mean(axis=0)
+U, s, V = np.linalg.svd(X_centered)
+c1 = V.T[:, 0]
+c2 = V.T[:, 1]
pca.components_.T[:, 0]).
+
W2 = V.T[:, :2]
+X2D = X_centered.dot(W2)
More on the PCA
+PCA and scikit-learn
pca = PCA()
-pca.fit(X)
-cumsum = np.cumsum(pca.explained_variance_ratio_)
-d = np.argmax(cumsum >= 0.95) + 1
+
from sklearn.decomposition import PCA
+pca = PCA(n_components = 2)
+X2D = pca.fit_transform(X)
pca = PCA(n_components=0.95)
-X_reduced = pca.fit_transform(X)
+
pca.components_.T[:, 0]).
Incremental PCA
+More on the PCA
pca = PCA()
+pca.fit(X)
+cumsum = np.cumsum(pca.explained_variance_ratio_)
+d = np.argmax(cumsum >= 0.95) + 1
+
pca = PCA(n_components=0.95)
+X_reduced = pca.fit_transform(X)
+
Randomized PCA
+Incremental PCA
Kernel PCA
-Randomized PCA
from sklearn.decomposition import KernelPCA
-rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
-X_reduced = rbf_pca.fit_transform(X)
-
LLE
+Kernel PCA
+from sklearn.decomposition import KernelPCA
+rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
+X_reduced = rbf_pca.fit_transform(X)
+
Other techniques
+LLE
-
-
-Here are other examples where we use the DataFrame functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix
-of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations.
-import numpy as np
-import pandas as pd
-from IPython.display import display
-np.random.seed(100)
-# setting up a 10 x 5 matrix
-rows = 10
-cols = 5
-a = np.random.randn(rows,cols)
-df = pd.DataFrame(a)
-display(df)
-print(df.mean())
-print(df.std())
-display(df**2)
-
df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']
-df.index = np.arange(10)
-
-display(df)
-print(df['Second'].mean() )
-print(df.info())
-print(df.describe())
-
-from pylab import plt, mpl
-plt.style.use('seaborn')
-mpl.rcParams['font.family'] = 'serif'
-
-df.cumsum().plot(lw=2.0, figsize=(10,6))
-plt.show()
-
-
-df.plot.bar(figsize=(10,6), rot=15)
-plt.show()
-
b = np.arange(16).reshape((4,4))
-print(b)
-df1 = pd.DataFrame(b)
-print(df1)
-
Classical PCA Theorem
+The Algorithm before theorem
+
+
+
+
+$$
+\boldsymbol{X}=\begin{bmatrix}
+x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
+x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
+x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
+\dots & \dots & \dots & \dots \dots & \dots \\
+x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
+x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
+\end{bmatrix},
+$$
+
+
+
+
+
+Prof of the PCA Theorem
+Classical PCA Theorem
Getting started with PCA
+Prof of the PCA Theorem
+Getting started with PCA
Principal Component Analysis
+Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
More on the PCA
+More on the PCA
Incremental PCA
+Incremental PCA
Randomized PCA
+Randomized PCA
Kernel PCA
+Kernel PCA
LLE
+LLE
Other techniques
+Other techniques
-
-Classical PCA Theorem
+The eigenvalues tell us then how much we need to stretch the
+corresponding eigenvectors. Dimensions with large eigenvalues have
+thus large variations (large variance) and define therefore useful
+dimensions. The data points are more spread out in the direction of
+these eigenvectors. Smaller eigenvalues mean on the other hand that
+the corresponding eigenvectors are shrunk accordingly and the data
+points are tightly bunched together and there is not much variation in
+these specific directions. Hopefully then we could leave it out
+dimensions where the eigenvalues are very small. If \( p \) is very large,
+we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
+features/predictors.
-Prof of the PCA Theorem
+The Algorithm before theorem
+
+
+
+
+$$
+\boldsymbol{X}=\begin{bmatrix}
+x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
+x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
+x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
+\dots & \dots & \dots & \dots \dots & \dots \\
+x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
+x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
+\end{bmatrix},
+$$
+
+
+
+
+
+After this we ask ourselves how do we prove the link between the maximum variance and the feature reduction.
-Getting started with PCA
+Classical PCA Theorem
+
+
+
+Prof of the PCA Theorem
+
+
+
+Getting started with PCA
-Principal Component Analysis
+Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
-More on the PCA
+More on the PCA
-Incremental PCA
+Incremental PCA
-Randomized PCA
+Randomized PCA
-Kernel PCA
+Kernel PCA
-LLE
+LLE
-Other techniques
+Other techniques
-
-Classical PCA Theorem
+The eigenvalues tell us then how much we need to stretch the
+corresponding eigenvectors. Dimensions with large eigenvalues have
+thus large variations (large variance) and define therefore useful
+dimensions. The data points are more spread out in the direction of
+these eigenvectors. Smaller eigenvalues mean on the other hand that
+the corresponding eigenvectors are shrunk accordingly and the data
+points are tightly bunched together and there is not much variation in
+these specific directions. Hopefully then we could leave it out
+dimensions where the eigenvalues are very small. If \( p \) is very large,
+we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
+features/predictors.
-Prof of the PCA Theorem
+The Algorithm before theorem
+
+
+
+
+$$
+\boldsymbol{X}=\begin{bmatrix}
+x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
+x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
+x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
+\dots & \dots & \dots & \dots \dots & \dots \\
+x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
+x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
+\end{bmatrix},
+$$
+
+
+
+
+
+After this we ask ourselves how do we prove the link between the maximum variance and the feature reduction.
-Getting started with PCA
+Classical PCA Theorem
+
+
+
+Prof of the PCA Theorem
+
+
+
+Getting started with PCA
-Principal Component Analysis
+Principal Component Analysis
PCA and scikit-learn
+PCA and scikit-learn
-More on the PCA
+More on the PCA
-Incremental PCA
+Incremental PCA
-Randomized PCA
+Randomized PCA
-Kernel PCA
+Kernel PCA
LLE
-Other techniques
+Other techniques