lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

new update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 14:05:16 +02:00
parent 30d664a82b
commit ad92592adb
76 changed files with 5646 additions and 2878 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter8.html
+44 -44
View File
@@ -5,7 +5,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>10. Basic ideas of the Principal Component Analysis (PCA) &#8212; Applied Data Analysis and Machine Learning</title>
<title>11. Basic ideas of the Principal Component Analysis (PCA) &#8212; Applied Data Analysis and Machine Learning</title>
<link href="_static/css/theme.css" rel="stylesheet" />
<link href="_static/css/index.c5995385ac14fb8791e8eb36b4908be2.css" rel="stylesheet" />
@@ -54,8 +54,8 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="11. Clustering Analysis" href="Clustering.html" />
<link rel="prev" title="9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods" href="chapter7.html" />
<link rel="next" title="12. Clustering Analysis" href="Clustering.html" />
<link rel="prev" title="10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods" href="chapter7.html" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta name="docsearch:language" content="en" />
@@ -139,17 +139,17 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter1.html">
3. Linear Regression, basic Elements
3. Linear Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html">
4. Resampling Methods
4. Ridge and Lasso Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
5. Ridge and Lasso Regression
5. Resampling Methods
</a>
</li>
<li class="toctree-l1">
@@ -159,7 +159,7 @@
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
7. Support Vector Machines, overarching aims
8. Support Vector Machines, overarching aims
</a>
</li>
</ul>
@@ -171,12 +171,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
8. Decision trees, overarching aims
9. Decision trees, overarching aims
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
</a>
</li>
</ul>
@@ -188,12 +188,12 @@
<ul class="current nav bd-sidenav">
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
10. Basic ideas of the Principal Component Analysis (PCA)
11. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="Clustering.html">
11. Clustering Analysis
12. Clustering Analysis
</a>
</li>
</ul>
@@ -205,12 +205,12 @@
<ul class="nav bd-sidenav">
<li class="toctree-l1">
<a class="reference internal" href="chapter9.html">
12. Neural networks
13. Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter10.html">
13. Building a Feed Forward Neural Network
14. Building a Feed Forward Neural Network
</a>
</li>
</ul>
@@ -286,76 +286,76 @@
<ul class="visible nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#introducing-the-covariance-and-correlation-functions">
10.1. Introducing the Covariance and Correlation functions
11.1. Introducing the Covariance and Correlation functions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-matrix">
10.2. Correlation Matrix
11.2. Correlation Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#towards-the-pca-theorem">
10.3. Towards the PCA theorem
11.3. Towards the PCA theorem
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-algorithm-before-theorem">
10.3.1. The Algorithm before theorem
11.3.1. The Algorithm before theorem
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#writing-our-own-pca-code">
10.3.2. Writing our own PCA code
11.3.2. Writing our own PCA code
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">
10.3.3. Diagonalize the sample covariance matrix to obtain the principal components
11.3.3. Diagonalize the sample covariance matrix to obtain the principal components
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#classical-pca-theorem">
10.4. Classical PCA Theorem
11.4. Classical PCA Theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#geometric-interpretation-and-link-with-singular-value-decomposition">
10.5. Geometric Interpretation and link with Singular Value Decomposition
11.5. Geometric Interpretation and link with Singular Value Decomposition
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pca-and-scikit-learn">
10.6. PCA and scikit-learn
11.6. PCA and scikit-learn
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#back-to-the-cancer-data">
10.7. Back to the Cancer Data
11.7. Back to the Cancer Data
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#incremental-pca">
10.7.1. Incremental PCA
11.7.1. Incremental PCA
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#randomized-pca">
10.7.2. Randomized PCA
11.7.2. Randomized PCA
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#kernel-pca">
10.7.3. Kernel PCA
11.7.3. Kernel PCA
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-techniques">
10.8. Other techniques
11.8. Other techniques
</a>
</li>
</ul>
@@ -370,7 +370,7 @@
<div>
<div class="section" id="basic-ideas-of-the-principal-component-analysis-pca">
<h1><span class="section-number">10. </span>Basic ideas of the Principal Component Analysis (PCA)<a class="headerlink" href="#basic-ideas-of-the-principal-component-analysis-pca" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">11. </span>Basic ideas of the Principal Component Analysis (PCA)<a class="headerlink" href="#basic-ideas-of-the-principal-component-analysis-pca" title="Permalink to this headline"></a></h1>
<p>The principal component analysis deals with the problem of fitting a
low-dimensional affine subspace <span class="math notranslate nohighlight">\(S\)</span> of dimension <span class="math notranslate nohighlight">\(d\)</span> much smaller than
the total dimension <span class="math notranslate nohighlight">\(D\)</span> of the problem at hand (our data
@@ -386,7 +386,7 @@ Historically, the PCA was first formulated in a statistical setting in order to
</ul>
<p>A good read is for example <a class="reference external" href="https://www.springer.com/gp/book/9780387878102">Vidal, Ma and Sastry</a>.</p>
<div class="section" id="introducing-the-covariance-and-correlation-functions">
<h2><span class="section-number">10.1. </span>Introducing the Covariance and Correlation functions<a class="headerlink" href="#introducing-the-covariance-and-correlation-functions" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">11.1. </span>Introducing the Covariance and Correlation functions<a class="headerlink" href="#introducing-the-covariance-and-correlation-functions" title="Permalink to this headline"></a></h2>
<p>Before we discuss the PCA theorem, we need to remind ourselves about
the definition of the covariance and the correlation function. These are quantities</p>
<p>Suppose we have defined two vectors
@@ -532,7 +532,7 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</div>
</div>
<div class="section" id="correlation-matrix">
<h2><span class="section-number">10.2. </span>Correlation Matrix<a class="headerlink" href="#correlation-matrix" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">11.2. </span>Correlation Matrix<a class="headerlink" href="#correlation-matrix" title="Permalink to this headline"></a></h2>
<p>The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
subtract the mean values for each column. This leads to the following
@@ -752,7 +752,7 @@ x_{10}x_{00}+x_{11}x_{01} &amp; x_{10}^2+x_{11}^2\\
<p>It is easy to generalize this to a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>.</p>
</div>
<div class="section" id="towards-the-pca-theorem">
<h2><span class="section-number">10.3. </span>Towards the PCA theorem<a class="headerlink" href="#towards-the-pca-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">11.3. </span>Towards the PCA theorem<a class="headerlink" href="#towards-the-pca-theorem" title="Permalink to this headline"></a></h2>
<p>We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as</p>
<div class="math notranslate nohighlight">
\[
@@ -790,7 +790,7 @@ dimensions where the eigenvalues are very small. If <span class="math notranslat
we could then aim at reducing <span class="math notranslate nohighlight">\(p\)</span> to <span class="math notranslate nohighlight">\(l &lt;&lt; p\)</span> and handle only <span class="math notranslate nohighlight">\(l\)</span>
features/predictors.</p>
<div class="section" id="the-algorithm-before-theorem">
<h3><span class="section-number">10.3.1. </span>The Algorithm before theorem<a class="headerlink" href="#the-algorithm-before-theorem" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">11.3.1. </span>The Algorithm before theorem<a class="headerlink" href="#the-algorithm-before-theorem" title="Permalink to this headline"></a></h3>
<p>Here’s how we would proceed in setting up the algorithm for the PCA, see also discussion below here.</p>
<ul class="simple">
<li><p>Set up the datapoints for the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> with <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>, with the predictors/features <span class="math notranslate nohighlight">\(p\)</span> referring to the column numbers and the entries <span class="math notranslate nohighlight">\(n\)</span> being the row elements.</p></li>
@@ -815,7 +815,7 @@ x_{n-1,0} &amp; x_{n-1,1} &amp; x_{n-1,2}&amp; \dots &amp; \dots x_{n-1,p-1}\\
</ul>
</div>
<div class="section" id="writing-our-own-pca-code">
<h3><span class="section-number">10.3.2. </span>Writing our own PCA code<a class="headerlink" href="#writing-our-own-pca-code" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">11.3.2. </span>Writing our own PCA code<a class="headerlink" href="#writing-our-own-pca-code" title="Permalink to this headline"></a></h3>
<p>We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below):</p>
<div class="math notranslate nohighlight">
@@ -937,7 +937,7 @@ Our own code here is not very elegant and asks for obvious improvements. It is t
The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.</p>
</div>
<div class="section" id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">
<h3><span class="section-number">10.3.3. </span>Diagonalize the sample covariance matrix to obtain the principal components<a class="headerlink" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">11.3.3. </span>Diagonalize the sample covariance matrix to obtain the principal components<a class="headerlink" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" title="Permalink to this headline"></a></h3>
<p>Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix <span class="math notranslate nohighlight">\(\Sigma\)</span>. We can use the
function <strong>np.linalg.eig</strong> to do so. It will return the eigenvalues and
@@ -1005,7 +1005,7 @@ Second eigenvector
</div>
</div>
<div class="section" id="classical-pca-theorem">
<h2><span class="section-number">10.4. </span>Classical PCA Theorem<a class="headerlink" href="#classical-pca-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">11.4. </span>Classical PCA Theorem<a class="headerlink" href="#classical-pca-theorem" title="Permalink to this headline"></a></h2>
<p>We assume now that we have a design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which has been
centered as discussed above. For the sake of simplicity we skip the
overline symbol. The matrix is defined in terms of the various column
@@ -1065,7 +1065,7 @@ chapter 12.4 and discussion therein.</p>
<p>For more details, see for example <a class="reference external" href="https://www.springer.com/gp/book/9780387878102">Vidal, Ma and Sastry, chapter 2</a>.</p>
</div>
<div class="section" id="geometric-interpretation-and-link-with-singular-value-decomposition">
<h2><span class="section-number">10.5. </span>Geometric Interpretation and link with Singular Value Decomposition<a class="headerlink" href="#geometric-interpretation-and-link-with-singular-value-decomposition" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">11.5. </span>Geometric Interpretation and link with Singular Value Decomposition<a class="headerlink" href="#geometric-interpretation-and-link-with-singular-value-decomposition" title="Permalink to this headline"></a></h2>
<p>For a detailed demonstration of the geometric interpretation, see <a class="reference external" href="https://www.springer.com/gp/book/9780387878102">Vidal, Ma and Sastry, section 2.1.2</a>.</p>
<p>Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.</p>
@@ -1250,7 +1250,7 @@ Selecting this hyperplane ensures that the projection will preserve as much vari
</div>
</div>
<div class="section" id="pca-and-scikit-learn">
<h2><span class="section-number">10.6. </span>PCA and scikit-learn<a class="headerlink" href="#pca-and-scikit-learn" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">11.6. </span>PCA and scikit-learn<a class="headerlink" href="#pca-and-scikit-learn" title="Permalink to this headline"></a></h2>
<p>Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
that it automatically takes care of centering the data):</p>
@@ -1299,7 +1299,7 @@ available via the <span class="math notranslate nohighlight">\(explained\_varian
variance that lies along the axis of each principal component.</p>
</div>
<div class="section" id="back-to-the-cancer-data">
<h2><span class="section-number">10.7. </span>Back to the Cancer Data<a class="headerlink" href="#back-to-the-cancer-data" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">11.7. </span>Back to the Cancer Data<a class="headerlink" href="#back-to-the-cancer-data" title="Permalink to this headline"></a></h2>
<p>We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.</p>
<div class="cell docutils container">
@@ -1382,7 +1382,7 @@ a float between 0.0 and 1.0, indicating the ratio of variance you wish to preser
</div>
</div>
<div class="section" id="incremental-pca">
<h3><span class="section-number">10.7.1. </span>Incremental PCA<a class="headerlink" href="#incremental-pca" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">11.7.1. </span>Incremental PCA<a class="headerlink" href="#incremental-pca" title="Permalink to this headline"></a></h3>
<p>One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
@@ -1390,14 +1390,14 @@ at a time. This is useful for large training sets, and also to apply PCA online
instances arrive).</p>
</div>
<div class="section" id="randomized-pca">
<h3><span class="section-number">10.7.2. </span>Randomized PCA<a class="headerlink" href="#randomized-pca" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">11.7.2. </span>Randomized PCA<a class="headerlink" href="#randomized-pca" title="Permalink to this headline"></a></h3>
<p>Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
algorithm that quickly finds an approximation of the first d principal components. Its computational
complexity is <span class="math notranslate nohighlight">\(O(m \times d^2)+O(d^3)\)</span>, instead of <span class="math notranslate nohighlight">\(O(m \times n^2) + O(n^3)\)</span>, so it is dramatically faster than the
previous algorithms when <span class="math notranslate nohighlight">\(d\)</span> is much smaller than <span class="math notranslate nohighlight">\(n\)</span>.</p>
</div>
<div class="section" id="kernel-pca">
<h3><span class="section-number">10.7.3. </span>Kernel PCA<a class="headerlink" href="#kernel-pca" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">11.7.3. </span>Kernel PCA<a class="headerlink" href="#kernel-pca" title="Permalink to this headline"></a></h3>
<p>The kernel trick is a mathematical technique that implicitly maps instances into a
very high-dimensional space (called the feature space), enabling nonlinear classification and regression
with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
@@ -1419,7 +1419,7 @@ For example, the following code uses Scikit-Learn’s KernelPCA class to perform
</div>
</div>
<div class="section" id="other-techniques">
<h2><span class="section-number">10.8. </span>Other techniques<a class="headerlink" href="#other-techniques" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">11.8. </span>Other techniques<a class="headerlink" href="#other-techniques" title="Permalink to this headline"></a></h2>
<p>There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.</p>
<p>Here are some of the most popular:</p>
<ul class="simple">
@@ -1456,8 +1456,8 @@ For example, the following code uses Scikit-Learn’s KernelPCA class to perform
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter7.html" title="previous page"><span class="section-number">9. </span>Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a>
<a class='right-next' id="next-link" href="Clustering.html" title="next page"><span class="section-number">11. </span>Clustering Analysis</a>
<a class='left-prev' id="prev-link" href="chapter7.html" title="previous page"><span class="section-number">10. </span>Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods</a>
<a class='right-next' id="next-link" href="Clustering.html" title="next page"><span class="section-number">12. </span>Clustering Analysis</a>
</div>