lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added dim red material

Browse Source
This commit is contained in:
mhjensen
2018-10-24 08:33:06 +02:00
parent ec7b073679
commit d2efb8dd0b
7 changed files with 1072 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DimRed/html/._DimRed-bs000.html
+154
View File
@@ -0,0 +1,154 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Dimensionality Reduction">
<title>Data Analysis and Machine Learning: Dimensionality Reduction</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Reducing the number of degrees of freedom, overarching view',
2,
None,
'___sec0'),
('Principal Component Analysis', 2, None, '___sec1'),
('Kernel PCA', 2, None, '___sec2'),
('LLE', 2, None, '___sec3'),
('Other techniques', 2, None, '___sec4')]}
end of tocinfo -->
<body>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Dimensionality Reduction</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0000"></a>
<!-- ------------------- main content ---------------------- -->
<div class="jumbotron">
<center><h1>Data Analysis and Machine Learning: Dimensionality Reduction</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
<center>
<b>Morten Hjorth-Jensen</b> [1, 2]
</center>
<p>
<!-- institution(s) -->
<center>[1] <b>Department of Physics, University of Oslo</b></center>
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 24, 2018</h4></center> <!-- date -->
<br>
<p>
<p><a href="._DimRed-bs001.html" class="btn btn-primary btn-lg">Read &raquo;</a></p>
</div> <!-- end jumbotron -->
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li class="active"><a href="._DimRed-bs000.html">1</a></li>
<li><a href="._DimRed-bs001.html">2</a></li>
<li><a href="._DimRed-bs002.html">3</a></li>
<li><a href="._DimRed-bs003.html">4</a></li>
<li><a href="._DimRed-bs004.html">5</a></li>
<li><a href="._DimRed-bs005.html">6</a></li>
<li><a href="._DimRed-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2018, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</body>
</html>
doc/pub/DimRed/html/._DimRed-bs001.html
+148
View File
@@ -0,0 +1,148 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Dimensionality Reduction">
<title>Data Analysis and Machine Learning: Dimensionality Reduction</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Reducing the number of degrees of freedom, overarching view',
2,
None,
'___sec0'),
('Principal Component Analysis', 2, None, '___sec1'),
('Kernel PCA', 2, None, '___sec2'),
('LLE', 2, None, '___sec3'),
('Other techniques', 2, None, '___sec4')]}
end of tocinfo -->
<body>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Dimensionality Reduction</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0001"></a>
<!-- !split -->
<h2 id="___sec0" class="anchor">Reducing the number of degrees of freedom, overarching view </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
<p>
Many Machine Learning problems involve thousands or even millions of features for each training
instance. Not only does this make training extremely slow, it can also make it much harder to find a good
solution, as we will see. This problem is often referred to as the curse of dimensionality.
Fortunately, in real-world problems, it is often possible to reduce the number of features considerably,
turning an intractable problem into a tractable one.
<p>
and we will go through three of the most popular dimensionality
reduction techniques: PCA, Kernel PCA, and LLE.
<p>
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._DimRed-bs000.html">&laquo;</a></li>
<li><a href="._DimRed-bs000.html">1</a></li>
<li class="active"><a href="._DimRed-bs001.html">2</a></li>
<li><a href="._DimRed-bs002.html">3</a></li>
<li><a href="._DimRed-bs003.html">4</a></li>
<li><a href="._DimRed-bs004.html">5</a></li>
<li><a href="._DimRed-bs005.html">6</a></li>
<li><a href="._DimRed-bs002.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/DimRed/html/._DimRed-bs002.html
+262
View File
@@ -0,0 +1,262 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Dimensionality Reduction">
<title>Data Analysis and Machine Learning: Dimensionality Reduction</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Reducing the number of degrees of freedom, overarching view',
2,
None,
'___sec0'),
('Principal Component Analysis', 2, None, '___sec1'),
('Kernel PCA', 2, None, '___sec2'),
('LLE', 2, None, '___sec3'),
('Other techniques', 2, None, '___sec4')]}
end of tocinfo -->
<body>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Dimensionality Reduction</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
<!-- navigation toc: --> <li><a href="#___sec1" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0002"></a>
<!-- !split -->
<h2 id="___sec1" class="anchor">Principal Component Analysis </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
<p>
The following Python code uses NumPy&#8217;s svd() function to obtain all the principal components of the
training set, then extracts the first two PCs:
X_centered = X - X.mean(axis=0)
U, s, V = np.linalg.svd(X_centered)
c1 = V.T[:, 0]
c2 = V.T[:, 1]
<p>
PCA assumes that the dataset is centered around the origin. As we will see, Scikit-Learn&#8217;s PCA classes take care of centering
the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don&#8217;t
forget to center the data first.
<p>
Once you have identified all the principal components, you can reduce the dimensionality of the dataset
down to d dimensions by projecting it onto the hyperplane defined by the first d principal components.
Selecting this hyperplane ensures that the projection will preserve as much variance as possible. For
example, in Figure 8-2 the 3D dataset is projected down to the 2D plane defined by the first two principal
components, preserving a large part of the dataset&#8217;s variance. As a result, the 2D projection looks very
much like the original 3D dataset.
<p>
W2 = V.T[:, :2]
X2D = X_centered.dot(W2)
<p>
Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
that it automatically takes care of centering the data):
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
X2D = pca.fit_transform(X)
After fitting the PCA transformer to the dataset, you can access the principal components using the
components_ variable (note that it contains the PCs as horizontal vectors, so, for example, the first
principal component is equal to pca.components_.T[:, 0]).
<p>
Another very useful piece of information is the explained variance ratio of each principal component,
available via the explained_variance_ratio_ variable. It indicates the proportion of the dataset&#8217;s
variance that lies along the axis of each principal component. For example, let&#8217;s look at the explained
variance ratios of the first two components of the 3D dataset represented in Figure 8-2:
>>> print(pca.explained_variance_ratio_)
array([ 0.84248607, 0.14631839])
This tells you that 84.2% of the dataset&#8217;s variance lies along the first axis, and 14.6% lies along the
second axis. This leaves less than 1.2% for the third axis, so it is reasonable to assume that it probably
carries little information.
<p>
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
Unless, of course, you are reducing dimensionality for data visualization &#8212; in that case you will
generally want to reduce the dimensionality down to 2 or 3.
The following code computes PCA without reducing dimensionality, then computes the minimum number
of dimensions required to preserve 95% of the training set&#8217;s variance:
pca = PCA()
pca.fit(X)
cumsum = np.cumsum(pca.explained_variance_ratio_)
d = np.argmax(cumsum >= 0.95) + 1
You could then set n_components=d and run PCA again. However, there is a much better option: instead
of specifying the number of principal components you want to preserve, you can set n_components to be
a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)
<p>
Obviously after dimensionality reduction, the training set takes up much less space. For example, try
applying PCA to the MNIST dataset while preserving 95% of its variance. You should find that each
instance will have just over 150 features, instead of the original 784 features. So while most of the
variance is preserved, the dataset is now less than 20% of its original size! This is a reasonable
compression ratio, and you can see how this can speed up a classification algorithm (such as an SVM
classifier) tremendously.
It is also possible to decompress the reduced dataset back to 784 dimensions by applying the inverse
transformation of the PCA projection. Of course this won&#8217;t give you back the original data, since the
projection lost a bit of information (within the 5% variance that was dropped), but it will likely be quite
close to the original data. The mean squared distance between the original data and the reconstructed data
(compressed and then decompressed) is called the reconstruction error. For example, the following code
compresses the MNIST dataset down to 154 dimensions, then uses the inverse_transform() method to
decompress it back to 784 dimensions. Figure 8-9 shows a few digits from the original training set (on the
left), and the corresponding digits after compression and decompression. You can see that there is a slight
image quality loss, but the digits are still mostly intact.
pca = PCA(n_components = 154)
X_mnist_reduced = pca.fit_transform(X_mnist)
X_mnist_recovered = pca.inverse_transform(X_mnist_reduced)
Figure
<p>
Incremental PCA
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
instances arrive).
The following code splits the MNIST dataset into 100 mini-batches (using NumPy&#8217;s array_split()
function) and feeds them to Scikit-Learn&#8217;s IncrementalPCA class5 to reduce the dimensionality of the
MNIST dataset down to 154 dimensions (just like before). Note that you must call the partial_fit()
method with each mini-batch rather than the fit() method with the whole training set:
from sklearn.decomposition import IncrementalPCA
n_batches = 100
inc_pca = IncrementalPCA(n_components=154)
for X_batch in np.array_split(X_mnist, n_batches):
inc_pca.partial_fit(X_batch)
X_mnist_reduced = inc_pca.transform(X_mnist)
<p>
Alternatively, you can use NumPy&#8217;s memmap class, which allows you to manipulate a large array stored in
a binary file on disk as if it were entirely in memory; the class loads only the data it needs in memory,
when it needs it. Since the IncrementalPCA class uses only a small part of the array at any given time,
the memory usage remains under control. This makes it possible to call the usual fit() method, as you
can see in the following code:
X_mm = np.memmap(filename, dtype="float32", mode="readonly", shape=(m, n))
batch_size = m // n_batches
inc_pca = IncrementalPCA(n_components=154, batch_size=batch_size)
inc_pca.fit(X_mm)
<p>
Randomized PCA
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
algorithm that quickly finds an approximation of the first d principal components. Its computational
complexity is O(m &#215; d2) + O(d3), instead of O(m &#215; n2) + O(n3), so it is dramatically faster than the
previous algorithms when d is much smaller than n.
rnd_pca = PCA(n_components=154, svd_solver="randomized")
X_reduced = rnd_pca.fit_transform(X_mnist)
<p>
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._DimRed-bs001.html">&laquo;</a></li>
<li><a href="._DimRed-bs000.html">1</a></li>
<li><a href="._DimRed-bs001.html">2</a></li>
<li class="active"><a href="._DimRed-bs002.html">3</a></li>
<li><a href="._DimRed-bs003.html">4</a></li>
<li><a href="._DimRed-bs004.html">5</a></li>
<li><a href="._DimRed-bs005.html">6</a></li>
<li><a href="._DimRed-bs003.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/DimRed/html/._DimRed-bs003.html
+153
View File
@@ -0,0 +1,153 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Dimensionality Reduction">
<title>Data Analysis and Machine Learning: Dimensionality Reduction</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Reducing the number of degrees of freedom, overarching view',
2,
None,
'___sec0'),
('Principal Component Analysis', 2, None, '___sec1'),
('Kernel PCA', 2, None, '___sec2'),
('LLE', 2, None, '___sec3'),
('Other techniques', 2, None, '___sec4')]}
end of tocinfo -->
<body>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Dimensionality Reduction</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="#___sec2" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0003"></a>
<!-- !split -->
<h2 id="___sec2" class="anchor">Kernel PCA </h2>
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
<p>
Kernel PCA
The kernel trick is a mathematical technique that implicitly maps instances into a
very high-dimensional space (called the feature space), enabling nonlinear classification and regression
with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
space corresponds to a complex nonlinear decision boundary in the original space.
It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
twisted manifold.
For example, the following code uses Scikit-Learn&#8217;s KernelPCA class to perform kPCA with an
from sklearn.decomposition import KernelPCA
rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
X_reduced = rbf_pca.fit_transform(X)
Figure 8-
<p>
</div>
</div>
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._DimRed-bs002.html">&laquo;</a></li>
<li><a href="._DimRed-bs000.html">1</a></li>
<li><a href="._DimRed-bs001.html">2</a></li>
<li><a href="._DimRed-bs002.html">3</a></li>
<li class="active"><a href="._DimRed-bs003.html">4</a></li>
<li><a href="._DimRed-bs004.html">5</a></li>
<li><a href="._DimRed-bs005.html">6</a></li>
<li><a href="._DimRed-bs004.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/DimRed/html/._DimRed-bs004.html
+137
View File
@@ -0,0 +1,137 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Dimensionality Reduction">
<title>Data Analysis and Machine Learning: Dimensionality Reduction</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Reducing the number of degrees of freedom, overarching view',
2,
None,
'___sec0'),
('Principal Component Analysis', 2, None, '___sec1'),
('Kernel PCA', 2, None, '___sec2'),
('LLE', 2, None, '___sec3'),
('Other techniques', 2, None, '___sec4')]}
end of tocinfo -->
<body>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Dimensionality Reduction</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="#___sec3" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0004"></a>
<!-- !split -->
<h2 id="___sec3" class="anchor">LLE </h2>
<p>
Locally Linear Embedding (LLE)8 is another very powerful nonlinear dimensionality reduction
(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where
these local relationships are best preserved (more details shortly). This makes it particularly good at
unrolling twisted manifolds, especially when there is not too much noise.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._DimRed-bs003.html">&laquo;</a></li>
<li><a href="._DimRed-bs000.html">1</a></li>
<li><a href="._DimRed-bs001.html">2</a></li>
<li><a href="._DimRed-bs002.html">3</a></li>
<li><a href="._DimRed-bs003.html">4</a></li>
<li class="active"><a href="._DimRed-bs004.html">5</a></li>
<li><a href="._DimRed-bs005.html">6</a></li>
<li><a href="._DimRed-bs005.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>
doc/pub/DimRed/html/._DimRed-bs005.html
+144
View File
@@ -0,0 +1,144 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="description" content="Data Analysis and Machine Learning: Dimensionality Reduction">
<title>Data Analysis and Machine Learning: Dimensionality Reduction</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
<!-- not necessary
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
-->
<style type="text/css">
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
.dropdown-menu {
height: auto;
max-height: 400px;
overflow-x: hidden;
}
/* Adds an invisible element before each target to offset for the navigation
bar */
.anchor::before {
content:"";
display:block;
height:50px; /* fixed header height for style bootstrap */
margin:-50px 0 0; /* negative fixed header height */
}
</style>
</head>
<!-- tocinfo
{'highest level': 2,
'sections': [('Reducing the number of degrees of freedom, overarching view',
2,
None,
'___sec0'),
('Principal Component Analysis', 2, None, '___sec1'),
('Kernel PCA', 2, None, '___sec2'),
('LLE', 2, None, '___sec3'),
('Other techniques', 2, None, '___sec4')]}
end of tocinfo -->
<body>
<!-- Bootstrap navigation bar -->
<div class="navbar navbar-default navbar-fixed-top">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Dimensionality Reduction</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="#___sec4" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
</ul>
</div>
</div>
</div> <!-- end of navigation bar -->
<div class="container">
<p>&nbsp;</p><p>&nbsp;</p><p>&nbsp;</p> <!-- add vertical space -->
<a name="part0005"></a>
<!-- !split -->
<h2 id="___sec4" class="anchor">Other techniques </h2>
<p>
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
Here are some of the most popular:
Multidimensional Scaling (MDS) reduces dimensionality while trying to preserve the distances
between the instances (see Figure 8-13).
Isomap creates a graph by connecting each instance to its nearest neighbors, then reduces
dimensionality while trying to preserve the geodesic distances9 between the instances.
t-Distributed Stochastic Neighbor Embedding (t-SNE) reduces dimensionality while trying to keep
similar instances close and dissimilar instances apart. It is mostly used for visualization, in
particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST
images in 2D).
Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it
learns the most discriminative axes between the classes, and these axes can then be used to define a
hyperplane onto which to project the data. The benefit is that the projection will keep classes as far
apart as possible, so LDA is a good technique to reduce dimensionality before running another
classification algorithm such as an SVM classifier
<p>
<!-- navigation buttons at the bottom of the page -->
<ul class="pagination">
<li><a href="._DimRed-bs004.html">&laquo;</a></li>
<li><a href="._DimRed-bs000.html">1</a></li>
<li><a href="._DimRed-bs001.html">2</a></li>
<li><a href="._DimRed-bs002.html">3</a></li>
<li><a href="._DimRed-bs003.html">4</a></li>
<li><a href="._DimRed-bs004.html">5</a></li>
<li class="active"><a href="._DimRed-bs005.html">6</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
</div> <!-- end container -->
<!-- include javascript, jQuery *first* -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright only on the titlepage -->
</center>
</body>
</html>