lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating PCA material

Browse Source
This commit is contained in:
mhjensen
2019-10-19 20:43:05 +02:00
parent 68f3c821e8
commit 477cdd57be
18 changed files with 725 additions and 552 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DimRed/html/DimRed.html
+76 -35
View File
@@ -89,15 +89,25 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec6'),
('Getting started with PCA', 2, None, '___sec7'),
('Principal Component Analysis', 2, None, '___sec8'),
('PCA and scikit-learn', 2, None, '___sec9'),
('More on the PCA', 2, None, '___sec10'),
('Incremental PCA', 2, None, '___sec11'),
('Randomized PCA', 2, None, '___sec12'),
('Kernel PCA', 2, None, '___sec13'),
('LLE', 2, None, '___sec14'),
('Other techniques', 2, None, '___sec15')]}
('Basic ideas of the Principal Component Analysis (PCA)',
2,
None,
'___sec7'),
('Introducing the Covariance and Correlation functions',
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
end of tocinfo -->
<body>
@@ -139,7 +149,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 17, 2019</h4></center> <!-- date -->
<center><h4>Oct 19, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -150,16 +160,22 @@ MathJax.Hub.Config({
<p>
<p>
Many Machine Learning problems involve thousands or even millions of features for each training
instance. Not only does this make training extremely slow, it can also make it much harder to find a good
solution, as we will see. This problem is often referred to as the curse of dimensionality.
Fortunately, in real-world problems, it is often possible to reduce the number of features considerably,
turning an intractable problem into a tractable one.
Many Machine Learning problems involve thousands or even millions of
features for each training instance. Not only does this make training
extremely slow, it can also make it much harder to find a good
solution, as we will see. This problem is often referred to as the
curse of dimensionality. Fortunately, in real-world problems, it is
often possible to reduce the number of features considerably, turning
an intractable problem into a tractable one.
<p>
Here we will discuss some of the most popular dimensionality
reduction techniques: the principal component analysis PCA, Kernel PCA, and Locally Linear Embedding (LLE).
Furthermore, we will start by looking at some simple preprocessing of the data which allow us to rescale the data.
Here we will discuss some of the most popular dimensionality reduction
techniques: the principal component analysis PCA, Kernel PCA, and
Locally Linear Embedding (LLE). Furthermore, we will start by looking
at some simple preprocessing of the data which allow us to rescale the
data.
</div>
@@ -173,11 +189,11 @@ Furthermore, we will start by looking at some simple preprocessing of the data w
<p>
Before we proceed however, we will discuss how to preprocess our
data. Till now and in connection with our previous examples we have not met so many cases
where we are too sensitive to the scaling of our data. Normally the
data may need a rescaling and/or may be sensitive to extreme
values. Scaling the data renders our inputs much more suitable for the
algorithms we want to employ.
data. Till now and in connection with our previous examples we have
not met so many cases where we are too sensitive to the scaling of our
data. Normally the data may need a rescaling and/or may be sensitive
to extreme values. Scaling the data renders our inputs much more
suitable for the algorithms we want to employ.
<p>
<b>Scikit-Learn</b> has several functions which allow us to rescale the
@@ -471,12 +487,33 @@ logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Getting started with PCA </h2>
<!-- todo: add more text in order to explain what is done here, discuss the correlation matrix -->
<p>
This material is being finalized, not yet ready.
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec7">Basic ideas of the Principal Component Analysis (PCA) </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Introducing the Covariance and Correlation functions </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Classical PCA Theorem </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Prof of the PCA Theorem </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Getting started with PCA </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -490,7 +527,7 @@ X_pca <span style="color: #666666">=</span> pca<span style="color: #666666">.</s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Principal Component Analysis </h2>
<h2 id="___sec12">Principal Component Analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -526,7 +563,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
<p>
<!-- !split -->
<h2 id="___sec9">PCA and scikit-learn </h2>
<h2 id="___sec13">PCA and scikit-learn </h2>
<p>
Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -557,7 +594,9 @@ More material to come here.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">More on the PCA </h2>
<h2 id="___sec14">More on the PCA </h2>
<p>
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
Unless, of course, you are reducing dimensionality for data visualization &#8212; in that case you will
@@ -585,7 +624,9 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Incremental PCA </h2>
<h2 id="___sec15">Incremental PCA </h2>
<p>
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
@@ -595,7 +636,7 @@ instances arrive).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Randomized PCA </h2>
<h2 id="___sec16">Randomized PCA </h2>
<p>
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -610,7 +651,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Kernel PCA </h2>
<h2 id="___sec17">Kernel PCA </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -639,7 +680,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">LLE </h2>
<h2 id="___sec18">LLE </h2>
<p>
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -651,7 +692,7 @@ these local relationships are best preserved (more details shortly).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Other techniques </h2>
<h2 id="___sec19">Other techniques </h2>
<p>
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.