mhjensen
263 lines
12 KiB
HTML
263 lines
12 KiB
HTML
<!--
|
|
Automatically generated HTML file from DocOnce source
|
|
(https://github.com/hplgit/doconce/)
|
|
-->
|
|
<html>
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
|
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
|
<meta name="description" content="Data Analysis and Machine Learning: Dimensionality Reduction">
|
|
|
|
<title>Data Analysis and Machine Learning: Dimensionality Reduction</title>
|
|
|
|
<!-- Bootstrap style: bootstrap -->
|
|
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
|
<!-- not necessary
|
|
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
|
-->
|
|
|
|
<style type="text/css">
|
|
|
|
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
|
.dropdown-menu {
|
|
height: auto;
|
|
max-height: 400px;
|
|
overflow-x: hidden;
|
|
}
|
|
|
|
/* Adds an invisible element before each target to offset for the navigation
|
|
bar */
|
|
.anchor::before {
|
|
content:"";
|
|
display:block;
|
|
height:50px; /* fixed header height for style bootstrap */
|
|
margin:-50px 0 0; /* negative fixed header height */
|
|
}
|
|
</style>
|
|
|
|
|
|
</head>
|
|
|
|
<!-- tocinfo
|
|
{'highest level': 2,
|
|
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
|
2,
|
|
None,
|
|
'___sec0'),
|
|
('Principal Component Analysis', 2, None, '___sec1'),
|
|
('Kernel PCA', 2, None, '___sec2'),
|
|
('LLE', 2, None, '___sec3'),
|
|
('Other techniques', 2, None, '___sec4')]}
|
|
end of tocinfo -->
|
|
|
|
<body>
|
|
|
|
|
|
<!-- Bootstrap navigation bar -->
|
|
<div class="navbar navbar-default navbar-fixed-top">
|
|
<div class="navbar-header">
|
|
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
<span class="icon-bar"></span>
|
|
</button>
|
|
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Dimensionality Reduction</a>
|
|
</div>
|
|
|
|
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
|
<ul class="nav navbar-nav navbar-right">
|
|
<li class="dropdown">
|
|
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
|
<ul class="dropdown-menu">
|
|
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
|
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Principal Component Analysis</a></li>
|
|
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">Kernel PCA</a></li>
|
|
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">LLE</a></li>
|
|
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Other techniques</a></li>
|
|
|
|
</ul>
|
|
</li>
|
|
</ul>
|
|
</div>
|
|
</div>
|
|
</div> <!-- end of navigation bar -->
|
|
|
|
<div class="container">
|
|
|
|
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
|
|
|
<a name="part0002"></a>
|
|
<!-- !split -->
|
|
|
|
<h2 id="___sec1" class="anchor">Principal Component Analysis </h2>
|
|
<div class="panel panel-default">
|
|
<div class="panel-body">
|
|
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
|
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
|
|
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
|
|
|
|
<p>
|
|
The following Python code uses NumPy’s svd() function to obtain all the principal components of the
|
|
training set, then extracts the first two PCs:
|
|
X_centered = X - X.mean(axis=0)
|
|
U, s, V = np.linalg.svd(X_centered)
|
|
c1 = V.T[:, 0]
|
|
c2 = V.T[:, 1]
|
|
|
|
<p>
|
|
PCA assumes that the dataset is centered around the origin. As we will see, Scikit-Learn’s PCA classes take care of centering
|
|
the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
|
|
forget to center the data first.
|
|
|
|
<p>
|
|
Once you have identified all the principal components, you can reduce the dimensionality of the dataset
|
|
down to d dimensions by projecting it onto the hyperplane defined by the first d principal components.
|
|
Selecting this hyperplane ensures that the projection will preserve as much variance as possible. For
|
|
example, in Figure 8-2 the 3D dataset is projected down to the 2D plane defined by the first two principal
|
|
components, preserving a large part of the dataset’s variance. As a result, the 2D projection looks very
|
|
much like the original 3D dataset.
|
|
|
|
<p>
|
|
W2 = V.T[:, :2]
|
|
X2D = X_centered.dot(W2)
|
|
|
|
<p>
|
|
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
|
|
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
|
|
that it automatically takes care of centering the data):
|
|
from sklearn.decomposition import PCA
|
|
pca = PCA(n_components = 2)
|
|
X2D = pca.fit_transform(X)
|
|
After fitting the PCA transformer to the dataset, you can access the principal components using the
|
|
components_ variable (note that it contains the PCs as horizontal vectors, so, for example, the first
|
|
principal component is equal to pca.components_.T[:, 0]).
|
|
|
|
<p>
|
|
Another very useful piece of information is the explained variance ratio of each principal component,
|
|
available via the explained_variance_ratio_ variable. It indicates the proportion of the dataset’s
|
|
variance that lies along the axis of each principal component. For example, let’s look at the explained
|
|
variance ratios of the first two components of the 3D dataset represented in Figure 8-2:
|
|
>>> print(pca.explained_variance_ratio_)
|
|
array([ 0.84248607, 0.14631839])
|
|
This tells you that 84.2% of the dataset’s variance lies along the first axis, and 14.6% lies along the
|
|
second axis. This leaves less than 1.2% for the third axis, so it is reasonable to assume that it probably
|
|
carries little information.
|
|
|
|
<p>
|
|
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
|
|
choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
|
|
Unless, of course, you are reducing dimensionality for data visualization — in that case you will
|
|
generally want to reduce the dimensionality down to 2 or 3.
|
|
The following code computes PCA without reducing dimensionality, then computes the minimum number
|
|
of dimensions required to preserve 95% of the training set’s variance:
|
|
pca = PCA()
|
|
pca.fit(X)
|
|
cumsum = np.cumsum(pca.explained_variance_ratio_)
|
|
d = np.argmax(cumsum >= 0.95) + 1
|
|
You could then set n_components=d and run PCA again. However, there is a much better option: instead
|
|
of specifying the number of principal components you want to preserve, you can set n_components to be
|
|
a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
|
|
pca = PCA(n_components=0.95)
|
|
X_reduced = pca.fit_transform(X)
|
|
|
|
<p>
|
|
Obviously after dimensionality reduction, the training set takes up much less space. For example, try
|
|
applying PCA to the MNIST dataset while preserving 95% of its variance. You should find that each
|
|
instance will have just over 150 features, instead of the original 784 features. So while most of the
|
|
variance is preserved, the dataset is now less than 20% of its original size! This is a reasonable
|
|
compression ratio, and you can see how this can speed up a classification algorithm (such as an SVM
|
|
classifier) tremendously.
|
|
It is also possible to decompress the reduced dataset back to 784 dimensions by applying the inverse
|
|
transformation of the PCA projection. Of course this won’t give you back the original data, since the
|
|
projection lost a bit of information (within the 5% variance that was dropped), but it will likely be quite
|
|
close to the original data. The mean squared distance between the original data and the reconstructed data
|
|
(compressed and then decompressed) is called the reconstruction error. For example, the following code
|
|
compresses the MNIST dataset down to 154 dimensions, then uses the inverse_transform() method to
|
|
decompress it back to 784 dimensions. Figure 8-9 shows a few digits from the original training set (on the
|
|
left), and the corresponding digits after compression and decompression. You can see that there is a slight
|
|
image quality loss, but the digits are still mostly intact.
|
|
pca = PCA(n_components = 154)
|
|
X_mnist_reduced = pca.fit_transform(X_mnist)
|
|
X_mnist_recovered = pca.inverse_transform(X_mnist_reduced)
|
|
Figure
|
|
|
|
<p>
|
|
Incremental PCA
|
|
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
|
|
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
|
|
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
|
|
at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
|
|
instances arrive).
|
|
The following code splits the MNIST dataset into 100 mini-batches (using NumPy’s array_split()
|
|
function) and feeds them to Scikit-Learn’s IncrementalPCA class5 to reduce the dimensionality of the
|
|
MNIST dataset down to 154 dimensions (just like before). Note that you must call the partial_fit()
|
|
method with each mini-batch rather than the fit() method with the whole training set:
|
|
from sklearn.decomposition import IncrementalPCA
|
|
n_batches = 100
|
|
inc_pca = IncrementalPCA(n_components=154)
|
|
for X_batch in np.array_split(X_mnist, n_batches):
|
|
inc_pca.partial_fit(X_batch)
|
|
X_mnist_reduced = inc_pca.transform(X_mnist)
|
|
|
|
<p>
|
|
Alternatively, you can use NumPy’s memmap class, which allows you to manipulate a large array stored in
|
|
a binary file on disk as if it were entirely in memory; the class loads only the data it needs in memory,
|
|
when it needs it. Since the IncrementalPCA class uses only a small part of the array at any given time,
|
|
the memory usage remains under control. This makes it possible to call the usual fit() method, as you
|
|
can see in the following code:
|
|
X_mm = np.memmap(filename, dtype="float32", mode="readonly", shape=(m, n))
|
|
batch_size = m // n_batches
|
|
inc_pca = IncrementalPCA(n_components=154, batch_size=batch_size)
|
|
inc_pca.fit(X_mm)
|
|
|
|
<p>
|
|
Randomized PCA
|
|
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
|
|
algorithm that quickly finds an approximation of the first d principal components. Its computational
|
|
complexity is O(m × d2) + O(d3), instead of O(m × n2) + O(n3), so it is dramatically faster than the
|
|
previous algorithms when d is much smaller than n.
|
|
rnd_pca = PCA(n_components=154, svd_solver="randomized")
|
|
X_reduced = rnd_pca.fit_transform(X_mnist)
|
|
|
|
<p>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
<p>
|
|
<p>
|
|
<!-- navigation buttons at the bottom of the page -->
|
|
<ul class="pagination">
|
|
<li><a href="._DimRed-bs001.html">«</a></li>
|
|
<li><a href="._DimRed-bs000.html">1</a></li>
|
|
<li><a href="._DimRed-bs001.html">2</a></li>
|
|
<li class="active"><a href="._DimRed-bs002.html">3</a></li>
|
|
<li><a href="._DimRed-bs003.html">4</a></li>
|
|
<li><a href="._DimRed-bs004.html">5</a></li>
|
|
<li><a href="._DimRed-bs005.html">6</a></li>
|
|
<li><a href="._DimRed-bs003.html">»</a></li>
|
|
</ul>
|
|
<!-- ------------------- end of main content --------------- -->
|
|
|
|
</div> <!-- end container -->
|
|
<!-- include javascript, jQuery *first* -->
|
|
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
|
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
|
|
|
<!-- Bootstrap footer
|
|
<footer>
|
|
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
|
</footer>
|
|
-->
|
|
|
|
|
|
<center style="font-size:80%">
|
|
<!-- copyright only on the titlepage -->
|
|
</center>
|
|
|
|
|
|
</body>
|
|
</html>
|
|
|
|
|