mhjensen
261 lines
11 KiB
Plaintext
261 lines
11 KiB
Plaintext
{
|
||
"cells": [
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<!-- dom:TITLE: Data Analysis and Machine Learning: Dimensionality Reduction -->\n",
|
||
"# Data Analysis and Machine Learning: Dimensionality Reduction\n",
|
||
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
|
||
"<!-- Author: --> \n",
|
||
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
||
"\n",
|
||
"Date: **Oct 26, 2018**\n",
|
||
"\n",
|
||
"Copyright 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Reducing the number of degrees of freedom, overarching view\n",
|
||
"\n",
|
||
"Many Machine Learning problems involve thousands or even millions of features for each training\n",
|
||
"instance. Not only does this make training extremely slow, it can also make it much harder to find a good\n",
|
||
"solution, as we will see. This problem is often referred to as the curse of dimensionality.\n",
|
||
"Fortunately, in real-world problems, it is often possible to reduce the number of features considerably,\n",
|
||
"turning an intractable problem into a tractable one.\n",
|
||
"\n",
|
||
"Here we will discuss some of the most popular dimensionality\n",
|
||
"reduction techniques: the principal component analysis PCA, Kernel PCA, and Locally Linear Embedding (LLE).\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Principal Component Analysis\n",
|
||
"Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.\n",
|
||
"First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.\n",
|
||
"\n",
|
||
"The following Python code uses NumPy’s **svd()** function to obtain all the principal components of the\n",
|
||
"training set, then extracts the first two principal components"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 1,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"X_centered = X - X.mean(axis=0)\n",
|
||
"U, s, V = np.linalg.svd(X_centered)\n",
|
||
"c1 = V.T[:, 0]\n",
|
||
"c2 = V.T[:, 1]"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering\n",
|
||
"the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t\n",
|
||
"forget to center the data first.\n",
|
||
"\n",
|
||
"Once you have identified all the principal components, you can reduce the dimensionality of the dataset\n",
|
||
"down to $d$ dimensions by projecting it onto the hyperplane defined by the first $d$ principal components.\n",
|
||
"Selecting this hyperplane ensures that the projection will preserve as much variance as possible."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 2,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"W2 = V.T[:, :2]\n",
|
||
"X2D = X_centered.dot(W2)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"<!-- !split -->\n",
|
||
"## PCA and scikit-learn\n",
|
||
"\n",
|
||
"Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The\n",
|
||
"following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note\n",
|
||
"that it automatically takes care of centering the data):"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 3,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.decomposition import PCA\n",
|
||
"pca = PCA(n_components = 2)\n",
|
||
"X2D = pca.fit_transform(X)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"After fitting the PCA transformer to the dataset, you can access the principal components using the\n",
|
||
"components variable (note that it contains the PCs as horizontal vectors, so, for example, the first\n",
|
||
"principal component is equal to"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 4,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"pca.components_.T[:, 0])."
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"Another very useful piece of information is the explained variance ratio of each principal component,\n",
|
||
"available via the $explained\\_variance\\_ratio$ variable. It indicates the proportion of the dataset’s\n",
|
||
"variance that lies along the axis of each principal component. \n",
|
||
"More material to come here.\n",
|
||
"\n",
|
||
"## More on the PCA\n",
|
||
"Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to\n",
|
||
"choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).\n",
|
||
"Unless, of course, you are reducing dimensionality for data visualization — in that case you will\n",
|
||
"generally want to reduce the dimensionality down to 2 or 3.\n",
|
||
"The following code computes PCA without reducing dimensionality, then computes the minimum number\n",
|
||
"of dimensions required to preserve 95% of the training set’s variance:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 5,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"pca = PCA()\n",
|
||
"pca.fit(X)\n",
|
||
"cumsum = np.cumsum(pca.explained_variance_ratio_)\n",
|
||
"d = np.argmax(cumsum >= 0.95) + 1"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"You could then set $n\\_components=d$ and run PCA again. However, there is a much better option: instead\n",
|
||
"of specifying the number of principal components you want to preserve, you can set $n\\_components$ to be\n",
|
||
"a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 6,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"pca = PCA(n_components=0.95)\n",
|
||
"X_reduced = pca.fit_transform(X)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## Incremental PCA\n",
|
||
"One problem with the preceding implementation of PCA is that it requires the whole training set to fit in\n",
|
||
"memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have\n",
|
||
"been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch\n",
|
||
"at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new\n",
|
||
"instances arrive).\n",
|
||
"\n",
|
||
"## Randomized PCA\n",
|
||
"\n",
|
||
"Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic\n",
|
||
"algorithm that quickly finds an approximation of the first d principal components. Its computational\n",
|
||
"complexity is $O(m \\times d^2)+O(d^3)$, instead of $O(m \\times n^2) + O(n^3)$, so it is dramatically faster than the\n",
|
||
"previous algorithms when $d$ is much smaller than $n$.\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Kernel PCA\n",
|
||
"\n",
|
||
"The kernel trick is a mathematical technique that implicitly maps instances into a\n",
|
||
"very high-dimensional space (called the feature space), enabling nonlinear classification and regression\n",
|
||
"with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature\n",
|
||
"space corresponds to a complex nonlinear decision boundary in the original space.\n",
|
||
"It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear\n",
|
||
"projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at\n",
|
||
"preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a\n",
|
||
"twisted manifold.\n",
|
||
"For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "code",
|
||
"execution_count": 7,
|
||
"metadata": {
|
||
"collapsed": false
|
||
},
|
||
"outputs": [],
|
||
"source": [
|
||
"from sklearn.decomposition import KernelPCA\n",
|
||
"rbf_pca = KernelPCA(n_components = 2, kernel=\"rbf\", gamma=0.04)\n",
|
||
"X_reduced = rbf_pca.fit_transform(X)"
|
||
]
|
||
},
|
||
{
|
||
"cell_type": "markdown",
|
||
"metadata": {},
|
||
"source": [
|
||
"## LLE\n",
|
||
"\n",
|
||
"Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction\n",
|
||
"(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous\n",
|
||
"algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its\n",
|
||
"closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where\n",
|
||
"these local relationships are best preserved (more details shortly). \n",
|
||
"\n",
|
||
"\n",
|
||
"\n",
|
||
"## Other techniques\n",
|
||
"\n",
|
||
"\n",
|
||
"There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.\n",
|
||
"\n",
|
||
"Here are some of the most popular:\n",
|
||
"* **Multidimensional Scaling (MDS)** reduces dimensionality while trying to preserve the distances between the instances.\n",
|
||
"\n",
|
||
"* **Isomap** creates a graph by connecting each instance to its nearest neighbors, then reduces dimensionality while trying to preserve the geodesic distances between the instances.\n",
|
||
"\n",
|
||
"* **t-Distributed Stochastic Neighbor Embedding** (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).\n",
|
||
"\n",
|
||
"* Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures."
|
||
]
|
||
}
|
||
],
|
||
"metadata": {},
|
||
"nbformat": 4,
|
||
"nbformat_minor": 2
|
||
}
|