lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update of lecture notes

Browse Source
This commit is contained in:
mhjensen
2020-12-20 23:07:22 +01:00
parent cb4dcd41ba
commit c001c2964f
23 changed files with 973 additions and 1406 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/src/LectureNotes/_build/html/chapter8.html
+80 -145
View File
@@ -6,7 +6,7 @@
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>20. Dimensionality Reduction &#8212; Applied Machine Learning and Data Analysis</title>
<title>7. Dimensionality Reduction &#8212; Applied Machine Learning and Data Analysis</title>
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.11.2/css/all.min.css" integrity="sha384-KA6wR/X5RY4zFAHpv/CnoG2UW1uogYfdnP67Uv7eULvTveboZJg0qUpmJZb5VqzN" crossorigin="anonymous">
<link href="_static/css/index.css" rel="stylesheet">
<link rel="stylesheet" href="_static/sphinx-book-theme.css" type="text/css" />
@@ -38,7 +38,7 @@
<script async="async" src="_static/sphinx-thebe.js"></script>
<link rel="index" title="Index" href="genindex.html" />
<link rel="search" title="Search" href="search.html" />
<link rel="prev" title="19. Support Vector Machines, overarching aims" href="chapter7.html" />
<link rel="prev" title="6. Support Vector Machines, overarching aims" href="chapter7.html" />
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="docsearch:language" content="en">
@@ -89,99 +89,34 @@
1. Elements of Probability Theory and Statistical Data Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers">
2. Random Numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-numbers-better-name-pseudo-random-numbers">
3. Random Numbers, better name: pseudo random numbers
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng">
4. Random number generator RNG
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-periodic-outputs">
5. Random number generator RNG and periodic outputs
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-and-its-period">
6. Random number generator RNG and its period
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-other-examples">
7. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id9">
8. Random number generator RNG, other examples
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0">
9. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id10">
10. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id11">
11. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#id12">
12. Random number generator RNG, RAN0
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#random-number-generator-rng-ran0-code">
13. Random number generator RNG, RAN0 code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter2.html#which-rng-should-i-use">
14. Which RNG should I use?
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter3.html">
15. Getting started, our first data and Machine Learning encounters
2. Getting started, our first data and Machine Learning encounters
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter4.html">
16. Linear Regression and more Advanced Regression Analysis
3. Linear Regression and more Advanced Regression Analysis
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
17. Logistic Regression
4. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter6.html">
18. Neural networks, from the simple perceptron to deep learning
5. Neural networks, from the simple perceptron to deep learning
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter7.html">
19. Support Vector Machines, overarching aims
6. Support Vector Machines, overarching aims
</a>
</li>
<li class="toctree-l1 current active">
<a class="current reference internal" href="#">
20. Dimensionality Reduction
7. Dimensionality Reduction
</a>
</li>
</ul>
@@ -256,179 +191,179 @@
<ul class="nav section-nav flex-column">
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#reducing-the-number-of-degrees-of-freedom-overarching-view">
20.1. Reducing the number of degrees of freedom, overarching view
7.1. Reducing the number of degrees of freedom, overarching view
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#preprocessing-our-data">
20.2. Preprocessing our data
7.2. Preprocessing our data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-preprocessing">
20.3. More preprocessing
7.3. More preprocessing
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-preprocessing-examples-franke-function-and-regression">
20.4. Simple preprocessing examples, Franke function and regression
7.4. Simple preprocessing examples, Franke function and regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#simple-preprocessing-examples-breast-cancer-data-and-classification-support-vector-machines">
20.5. Simple preprocessing examples, breast cancer data and classification, Support Vector Machines
7.5. Simple preprocessing examples, breast cancer data and classification, Support Vector Machines
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-on-cancer-data-now-with-logistic-regression">
20.6. More on Cancer Data, now with Logistic Regression
7.6. More on Cancer Data, now with Logistic Regression
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#why-should-we-think-of-reducing-the-dimensionality">
20.7. Why should we think of reducing the dimensionality
7.7. Why should we think of reducing the dimensionality
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#basic-ideas-of-the-principal-component-analysis-pca">
20.8. Basic ideas of the Principal Component Analysis (PCA)
7.8. Basic ideas of the Principal Component Analysis (PCA)
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#introducing-the-covariance-and-correlation-functions">
20.9. Introducing the Covariance and Correlation functions
7.9. Introducing the Covariance and Correlation functions
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-function-and-design-feature-matrix">
20.10. Correlation Function and Design/Feature Matrix
7.10. Correlation Function and Design/Feature Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#covariance-matrix-examples">
20.11. Covariance Matrix Examples
7.11. Covariance Matrix Examples
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-matrix">
20.12. Correlation Matrix
7.12. Correlation Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-matrix-with-pandas">
20.13. Correlation Matrix with Pandas
7.13. Correlation Matrix with Pandas
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#correlation-matrix-with-pandas-and-the-franke-function">
20.14. Correlation Matrix with Pandas and the Franke function
7.14. Correlation Matrix with Pandas and the Franke function
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#rewriting-the-covariance-and-or-correlation-matrix">
20.15. Rewriting the Covariance and/or Correlation Matrix
7.15. Rewriting the Covariance and/or Correlation Matrix
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#towards-the-pca-theorem">
20.16. Towards the PCA theorem
7.16. Towards the PCA theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-algorithm-before-theorem">
20.17. The Algorithm before theorem
7.17. The Algorithm before theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#writing-our-own-pca-code">
20.18. Writing our own PCA code
7.18. Writing our own PCA code
</a>
<ul class="nav section-nav flex-column">
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#compute-the-sample-mean-and-center-the-data">
20.18.1. Compute the sample mean and center the data
7.18.1. Compute the sample mean and center the data
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#compute-the-sample-covariance">
20.18.2. Compute the sample covariance
7.18.2. Compute the sample covariance
</a>
</li>
<li class="toc-h3 nav-item toc-entry">
<a class="reference internal nav-link" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">
20.18.3. Diagonalize the sample covariance matrix to obtain the principal components
7.18.3. Diagonalize the sample covariance matrix to obtain the principal components
</a>
</li>
</ul>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#classical-pca-theorem">
20.19. Classical PCA Theorem
7.19. Classical PCA Theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#proof-of-the-pca-theorem">
20.20. Proof of the PCA Theorem
7.20. Proof of the PCA Theorem
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pca-proof-continued">
20.21. PCA Proof continued
7.21. PCA Proof continued
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#the-final-step">
20.22. The final step
7.22. The final step
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#geometric-interpretation-and-link-with-singular-value-decomposition">
20.23. Geometric Interpretation and link with Singular Value Decomposition
7.23. Geometric Interpretation and link with Singular Value Decomposition
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#principal-component-analysis">
20.24. Principal Component Analysis
7.24. Principal Component Analysis
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#pca-and-scikit-learn">
20.25. PCA and scikit-learn
7.25. PCA and scikit-learn
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#back-to-the-cancer-data">
20.26. Back to the Cancer Data
7.26. Back to the Cancer Data
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#more-on-the-pca">
20.27. More on the PCA
7.27. More on the PCA
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#incremental-pca">
20.28. Incremental PCA
7.28. Incremental PCA
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#randomized-pca">
20.29. Randomized PCA
7.29. Randomized PCA
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#kernel-pca">
20.30. Kernel PCA
7.30. Kernel PCA
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#lle">
20.31. LLE
7.31. LLE
</a>
</li>
<li class="toc-h2 nav-item toc-entry">
<a class="reference internal nav-link" href="#other-techniques">
20.32. Other techniques
7.32. Other techniques
</a>
</li>
</ul>
@@ -442,9 +377,9 @@
<div>
<div class="section" id="dimensionality-reduction">
<h1><span class="section-number">20. </span>Dimensionality Reduction<a class="headerlink" href="#dimensionality-reduction" title="Permalink to this headline"></a></h1>
<h1><span class="section-number">7. </span>Dimensionality Reduction<a class="headerlink" href="#dimensionality-reduction" title="Permalink to this headline"></a></h1>
<div class="section" id="reducing-the-number-of-degrees-of-freedom-overarching-view">
<h2><span class="section-number">20.1. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.1. </span>Reducing the number of degrees of freedom, overarching view<a class="headerlink" href="#reducing-the-number-of-degrees-of-freedom-overarching-view" title="Permalink to this headline"></a></h2>
<p>Many Machine Learning problems involve thousands or even millions of
features for each training instance. Not only does this make training
extremely slow, it can also make it much harder to find a good
@@ -465,7 +400,7 @@ is one of the most used tools in data modeling, compression and
visualization.</p>
</div>
<div class="section" id="preprocessing-our-data">
<h2><span class="section-number">20.2. </span>Preprocessing our data<a class="headerlink" href="#preprocessing-our-data" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.2. </span>Preprocessing our data<a class="headerlink" href="#preprocessing-our-data" title="Permalink to this headline"></a></h2>
<p>Before we proceed however, we will discuss how to preprocess our
data. Till now and in connection with our previous examples we have
not met so many cases where we are too sensitive to the scaling of our
@@ -483,7 +418,7 @@ function included in <strong>Scikit-Learn</strong> is the <strong>MinMaxScaler</
ensures that all features are exactly between <span class="math notranslate nohighlight">\(0\)</span> and <span class="math notranslate nohighlight">\(1\)</span>. The</p>
</div>
<div class="section" id="more-preprocessing">
<h2><span class="section-number">20.3. </span>More preprocessing<a class="headerlink" href="#more-preprocessing" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.3. </span>More preprocessing<a class="headerlink" href="#more-preprocessing" title="Permalink to this headline"></a></h2>
<p>The <strong>Normalizer</strong> scales each data
point such that the feature vector has a euclidean length of one. In other words, it
projects a data point on the circle (or sphere in the case of higher dimensions) with a
@@ -501,7 +436,7 @@ outliers, and might often lead to trouble for other scaling
techniques.</p>
</div>
<div class="section" id="simple-preprocessing-examples-franke-function-and-regression">
<h2><span class="section-number">20.4. </span>Simple preprocessing examples, Franke function and regression<a class="headerlink" href="#simple-preprocessing-examples-franke-function-and-regression" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.4. </span>Simple preprocessing examples, Franke function and regression<a class="headerlink" href="#simple-preprocessing-examples-franke-function-and-regression" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
@@ -636,7 +571,7 @@ R2 score for scaled data: 0.97
</div>
</div>
<div class="section" id="simple-preprocessing-examples-breast-cancer-data-and-classification-support-vector-machines">
<h2><span class="section-number">20.5. </span>Simple preprocessing examples, breast cancer data and classification, Support Vector Machines<a class="headerlink" href="#simple-preprocessing-examples-breast-cancer-data-and-classification-support-vector-machines" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.5. </span>Simple preprocessing examples, breast cancer data and classification, Support Vector Machines<a class="headerlink" href="#simple-preprocessing-examples-breast-cancer-data-and-classification-support-vector-machines" title="Permalink to this headline"></a></h2>
<p>We show here how we can use a simple regression case on the breast
cancer data using support vector machines (SVM) as algorithm for
classification.</p>
@@ -715,7 +650,7 @@ Test set accuracy scaled data with Standar Scaler: 0.96
</div>
</div>
<div class="section" id="more-on-cancer-data-now-with-logistic-regression">
<h2><span class="section-number">20.6. </span>More on Cancer Data, now with Logistic Regression<a class="headerlink" href="#more-on-cancer-data-now-with-logistic-regression" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.6. </span>More on Cancer Data, now with Logistic Regression<a class="headerlink" href="#more-on-cancer-data-now-with-logistic-regression" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
@@ -761,7 +696,7 @@ Please also refer to the documentation for alternative solver options:
</div>
</div>
<div class="section" id="why-should-we-think-of-reducing-the-dimensionality">
<h2><span class="section-number">20.7. </span>Why should we think of reducing the dimensionality<a class="headerlink" href="#why-should-we-think-of-reducing-the-dimensionality" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.7. </span>Why should we think of reducing the dimensionality<a class="headerlink" href="#why-should-we-think-of-reducing-the-dimensionality" title="Permalink to this headline"></a></h2>
<p>In addition to the plot of the features, we study now also the covariance (and the correlation matrix).
We use also <strong>Pandas</strong> to compute the correlation matrix.</p>
<div class="cell docutils container">
@@ -854,7 +789,7 @@ the classical Principal Component Analysis (PCA) theorem with
applications.</p>
</div>
<div class="section" id="basic-ideas-of-the-principal-component-analysis-pca">
<h2><span class="section-number">20.8. </span>Basic ideas of the Principal Component Analysis (PCA)<a class="headerlink" href="#basic-ideas-of-the-principal-component-analysis-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.8. </span>Basic ideas of the Principal Component Analysis (PCA)<a class="headerlink" href="#basic-ideas-of-the-principal-component-analysis-pca" title="Permalink to this headline"></a></h2>
<p>The principal component analysis deals with the problem of fitting a
low-dimensional affine subspace <span class="math notranslate nohighlight">\(S\)</span> of dimension <span class="math notranslate nohighlight">\(d\)</span> much smaller than
the totaldimension <span class="math notranslate nohighlight">\(D\)</span> of the problem at hand (our data
@@ -870,7 +805,7 @@ what set the scene historically which for the PCA.</p>
</ul>
</div>
<div class="section" id="introducing-the-covariance-and-correlation-functions">
<h2><span class="section-number">20.9. </span>Introducing the Covariance and Correlation functions<a class="headerlink" href="#introducing-the-covariance-and-correlation-functions" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.9. </span>Introducing the Covariance and Correlation functions<a class="headerlink" href="#introducing-the-covariance-and-correlation-functions" title="Permalink to this headline"></a></h2>
<p>Before we discuss the PCA theorem, we need to remind ourselves about
the definition of the covariance and the correlation function. These are quantities</p>
<p>Suppose we have defined two vectors
@@ -920,7 +855,7 @@ and <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> as</p>
<p>In the above example this is the function we constructed using <strong>pandas</strong>.</p>
</div>
<div class="section" id="correlation-function-and-design-feature-matrix">
<h2><span class="section-number">20.10. </span>Correlation Function and Design/Feature Matrix<a class="headerlink" href="#correlation-function-and-design-feature-matrix" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.10. </span>Correlation Function and Design/Feature Matrix<a class="headerlink" href="#correlation-function-and-design-feature-matrix" title="Permalink to this headline"></a></h2>
<p>In our derivation of the various regression algorithms like <strong>Ordinary Least Squares</strong> or <strong>Ridge regression</strong>
we defined the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> as</p>
<div class="math notranslate nohighlight">
@@ -975,7 +910,7 @@ covariance matrix for the vectors <span class="math notranslate nohighlight">\(\
\end{split}\]</div>
</div>
<div class="section" id="covariance-matrix-examples">
<h2><span class="section-number">20.11. </span>Covariance Matrix Examples<a class="headerlink" href="#covariance-matrix-examples" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.11. </span>Covariance Matrix Examples<a class="headerlink" href="#covariance-matrix-examples" title="Permalink to this headline"></a></h2>
<p>The Numpy function <strong>np.cov</strong> calculates the covariance elements using
the factor <span class="math notranslate nohighlight">\(1/(n-1)\)</span> instead of <span class="math notranslate nohighlight">\(1/n\)</span> since it assumes we do not have
the exact mean values. The following simple function uses the
@@ -1022,7 +957,7 @@ covariance matrix through the <strong>np.linalg.eig()</strong> function.</p>
</div>
</div>
<div class="section" id="correlation-matrix">
<h2><span class="section-number">20.12. </span>Correlation Matrix<a class="headerlink" href="#correlation-matrix" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.12. </span>Correlation Matrix<a class="headerlink" href="#correlation-matrix" title="Permalink to this headline"></a></h2>
<p>The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
subtract the mean values for each column. This leads to the following
@@ -1069,7 +1004,7 @@ this matrix we easily see that it is a positive definite matrix.</p>
<p>The above procedure with <strong>numpy</strong> can be made more compact if we use <strong>pandas</strong>.</p>
</div>
<div class="section" id="correlation-matrix-with-pandas">
<h2><span class="section-number">20.13. </span>Correlation Matrix with Pandas<a class="headerlink" href="#correlation-matrix-with-pandas" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.13. </span>Correlation Matrix with Pandas<a class="headerlink" href="#correlation-matrix-with-pandas" title="Permalink to this headline"></a></h2>
<p>We whow here how we can set up the correlation matrix using <strong>pandas</strong>, as done in this simple code</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -1121,7 +1056,7 @@ this matrix we easily see that it is a positive definite matrix.</p>
<p>We expand this model to the Franke function discussed above.</p>
</div>
<div class="section" id="correlation-matrix-with-pandas-and-the-franke-function">
<h2><span class="section-number">20.14. </span>Correlation Matrix with Pandas and the Franke function<a class="headerlink" href="#correlation-matrix-with-pandas-and-the-franke-function" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.14. </span>Correlation Matrix with Pandas and the Franke function<a class="headerlink" href="#correlation-matrix-with-pandas-and-the-franke-function" title="Permalink to this headline"></a></h2>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="c1"># Common imports</span>
@@ -1217,7 +1152,7 @@ drop these elements and construct a correlation
matrix without these elements.</p>
</div>
<div class="section" id="rewriting-the-covariance-and-or-correlation-matrix">
<h2><span class="section-number">20.15. </span>Rewriting the Covariance and/or Correlation Matrix<a class="headerlink" href="#rewriting-the-covariance-and-or-correlation-matrix" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.15. </span>Rewriting the Covariance and/or Correlation Matrix<a class="headerlink" href="#rewriting-the-covariance-and-or-correlation-matrix" title="Permalink to this headline"></a></h2>
<p>We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> as</p>
<div class="math notranslate nohighlight">
\[
@@ -1252,7 +1187,7 @@ x_{10}x_{00}+x_{11}x_{01} &amp; x_{10}^2+x_{11}^2\\
<p>It is easy to generalize this to a matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>.</p>
</div>
<div class="section" id="towards-the-pca-theorem">
<h2><span class="section-number">20.16. </span>Towards the PCA theorem<a class="headerlink" href="#towards-the-pca-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.16. </span>Towards the PCA theorem<a class="headerlink" href="#towards-the-pca-theorem" title="Permalink to this headline"></a></h2>
<p>We have that the covariance matrix (the correlation matrix involves a simple rescaling) is given as</p>
<div class="math notranslate nohighlight">
\[
@@ -1291,7 +1226,7 @@ we could then aim at reducing <span class="math notranslate nohighlight">\(p\)</
features/predictors.</p>
</div>
<div class="section" id="the-algorithm-before-theorem">
<h2><span class="section-number">20.17. </span>The Algorithm before theorem<a class="headerlink" href="#the-algorithm-before-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.17. </span>The Algorithm before theorem<a class="headerlink" href="#the-algorithm-before-theorem" title="Permalink to this headline"></a></h2>
<p>Here’s how we would proceed in setting up the algorithm for the PCA, see also discussion below here.</p>
<ul class="simple">
<li><p>Set up the datapoints for the design/feature matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> with <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\mathbb{R}}^{n\times p}\)</span>, with the predictors/features <span class="math notranslate nohighlight">\(p\)</span> referring to the column numbers and the entries <span class="math notranslate nohighlight">\(n\)</span> being the row elements.</p></li>
@@ -1316,7 +1251,7 @@ x_{n-1,0} &amp; x_{n-1,1} &amp; x_{n-1,2}&amp; \dots &amp; \dots x_{n-1,p-1}\\
</ul>
</div>
<div class="section" id="writing-our-own-pca-code">
<h2><span class="section-number">20.18. </span>Writing our own PCA code<a class="headerlink" href="#writing-our-own-pca-code" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.18. </span>Writing our own PCA code<a class="headerlink" href="#writing-our-own-pca-code" title="Permalink to this headline"></a></h2>
<p>We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix:</p>
<div class="math notranslate nohighlight">
@@ -1346,7 +1281,7 @@ Note that the function <strong>multivariate</strong> returns also the covariance
</div>
<p>Now we are going to implement the PCA algorithm. We will break it down into various substeps.</p>
<div class="section" id="compute-the-sample-mean-and-center-the-data">
<h3><span class="section-number">20.18.1. </span>Compute the sample mean and center the data<a class="headerlink" href="#compute-the-sample-mean-and-center-the-data" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">7.18.1. </span>Compute the sample mean and center the data<a class="headerlink" href="#compute-the-sample-mean-and-center-the-data" title="Permalink to this headline"></a></h3>
<p>The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is</p>
<div class="math notranslate nohighlight">
\[
@@ -1383,7 +1318,7 @@ while the non-diagonal ones need to be divided by <span class="math notranslate
specific case.</p>
</div>
<div class="section" id="compute-the-sample-covariance">
<h3><span class="section-number">20.18.2. </span>Compute the sample covariance<a class="headerlink" href="#compute-the-sample-covariance" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">7.18.2. </span>Compute the sample covariance<a class="headerlink" href="#compute-the-sample-covariance" title="Permalink to this headline"></a></h3>
<p>Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation</p>
<div class="math notranslate nohighlight">
\[
@@ -1441,7 +1376,7 @@ Our own code here is not very elegant and asks for obvious improvements. It is t
The plot shows how the data are clustered around a line with slope close to one. Is this expected?</p>
</div>
<div class="section" id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">
<h3><span class="section-number">20.18.3. </span>Diagonalize the sample covariance matrix to obtain the principal components<a class="headerlink" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" title="Permalink to this headline"></a></h3>
<h3><span class="section-number">7.18.3. </span>Diagonalize the sample covariance matrix to obtain the principal components<a class="headerlink" href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" title="Permalink to this headline"></a></h3>
<p>Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix <span class="math notranslate nohighlight">\(\Sigma\)</span>. We can use the
function <strong>np.linalg.eig</strong> to do so. It will return the eigenvalues and
@@ -1507,7 +1442,7 @@ Eigenvector of largest eigenvalue
</div>
</div>
<div class="section" id="classical-pca-theorem">
<h2><span class="section-number">20.19. </span>Classical PCA Theorem<a class="headerlink" href="#classical-pca-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.19. </span>Classical PCA Theorem<a class="headerlink" href="#classical-pca-theorem" title="Permalink to this headline"></a></h2>
<p>We assume now that we have a design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span> which has been
centered as discussed above. For the sake of simplicity we skip the
overline symbol. The matrix is defined in terms of the various column
@@ -1530,7 +1465,7 @@ eigenvectors of the covariance(correlations matrix).</p>
<p>The proof which follows will be updated by mid January 2020.</p>
</div>
<div class="section" id="proof-of-the-pca-theorem">
<h2><span class="section-number">20.20. </span>Proof of the PCA Theorem<a class="headerlink" href="#proof-of-the-pca-theorem" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.20. </span>Proof of the PCA Theorem<a class="headerlink" href="#proof-of-the-pca-theorem" title="Permalink to this headline"></a></h2>
<p>To show the PCA theorem let us start with the assumption that there is one vector <span class="math notranslate nohighlight">\(\boldsymbol{w}_0\)</span> which corresponds to a solution which minimized the reconstruction error <span class="math notranslate nohighlight">\(J\)</span>. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of <span class="math notranslate nohighlight">\(\boldsymbol{w}_0\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{z}_0\)</span> as</p>
<div class="math notranslate nohighlight">
\[
@@ -1549,7 +1484,7 @@ z_{i0}=\boldsymbol{w}_0^T\boldsymbol{x}_i,
<p>where the vectors on the rhs are known.</p>
</div>
<div class="section" id="pca-proof-continued">
<h2><span class="section-number">20.21. </span>PCA Proof continued<a class="headerlink" href="#pca-proof-continued" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.21. </span>PCA Proof continued<a class="headerlink" href="#pca-proof-continued" title="Permalink to this headline"></a></h2>
<p>We have now found the unknown parameters <span class="math notranslate nohighlight">\(z_{i0}\)</span>. These correspond to the projected coordinates and we can write</p>
<div class="math notranslate nohighlight">
\[
@@ -1582,7 +1517,7 @@ matrix. Minimizing the error is equivalent to maximizing the variance
of the projected data.</p>
</div>
<div class="section" id="the-final-step">
<h2><span class="section-number">20.22. </span>The final step<a class="headerlink" href="#the-final-step" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.22. </span>The final step<a class="headerlink" href="#the-final-step" title="Permalink to this headline"></a></h2>
<p>We could trivially maximize the variance of the projection (and
thereby minimize the error in the reconstruction function) by letting
the norm-2 of <span class="math notranslate nohighlight">\(\boldsymbol{w}_0\)</span> go to infinity. However, this norm since we
@@ -1625,11 +1560,11 @@ chapter 12.4 and discussion therein.</p>
<p>Additional part of the proof for the other eigenvectors will be added by mid January 2020.</p>
</div>
<div class="section" id="geometric-interpretation-and-link-with-singular-value-decomposition">
<h2><span class="section-number">20.23. </span>Geometric Interpretation and link with Singular Value Decomposition<a class="headerlink" href="#geometric-interpretation-and-link-with-singular-value-decomposition" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.23. </span>Geometric Interpretation and link with Singular Value Decomposition<a class="headerlink" href="#geometric-interpretation-and-link-with-singular-value-decomposition" title="Permalink to this headline"></a></h2>
<p>This material will be added by mid January 2020.</p>
</div>
<div class="section" id="principal-component-analysis">
<h2><span class="section-number">20.24. </span>Principal Component Analysis<a class="headerlink" href="#principal-component-analysis" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.24. </span>Principal Component Analysis<a class="headerlink" href="#principal-component-analysis" title="Permalink to this headline"></a></h2>
<p>Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.</p>
<p>The following Python code uses NumPy’s <strong>svd()</strong> function to obtain all the principal components of the
@@ -1813,7 +1748,7 @@ Selecting this hyperplane ensures that the projection will preserve as much vari
</div>
</div>
<div class="section" id="pca-and-scikit-learn">
<h2><span class="section-number">20.25. </span>PCA and scikit-learn<a class="headerlink" href="#pca-and-scikit-learn" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.25. </span>PCA and scikit-learn<a class="headerlink" href="#pca-and-scikit-learn" title="Permalink to this headline"></a></h2>
<p>Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
that it automatically takes care of centering the data):</p>
@@ -1865,7 +1800,7 @@ available via the <span class="math notranslate nohighlight">\(explained\_varian
variance that lies along the axis of each principal component.</p>
</div>
<div class="section" id="back-to-the-cancer-data">
<h2><span class="section-number">20.26. </span>Back to the Cancer Data<a class="headerlink" href="#back-to-the-cancer-data" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.26. </span>Back to the Cancer Data<a class="headerlink" href="#back-to-the-cancer-data" title="Permalink to this headline"></a></h2>
<p>We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.</p>
<div class="cell docutils container">
@@ -1905,7 +1840,7 @@ Here we compute performance scores on the training data using logistic regressio
<p>We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.</p>
</div>
<div class="section" id="more-on-the-pca">
<h2><span class="section-number">20.27. </span>More on the PCA<a class="headerlink" href="#more-on-the-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.27. </span>More on the PCA<a class="headerlink" href="#more-on-the-pca" title="Permalink to this headline"></a></h2>
<p>Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
Unless, of course, you are reducing dimensionality for data visualization — in that case you will
@@ -1935,7 +1870,7 @@ a float between 0.0 and 1.0, indicating the ratio of variance you wish to preser
</div>
</div>
<div class="section" id="incremental-pca">
<h2><span class="section-number">20.28. </span>Incremental PCA<a class="headerlink" href="#incremental-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.28. </span>Incremental PCA<a class="headerlink" href="#incremental-pca" title="Permalink to this headline"></a></h2>
<p>One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
@@ -1943,14 +1878,14 @@ at a time. This is useful for large training sets, and also to apply PCA online
instances arrive).</p>
</div>
<div class="section" id="randomized-pca">
<h2><span class="section-number">20.29. </span>Randomized PCA<a class="headerlink" href="#randomized-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.29. </span>Randomized PCA<a class="headerlink" href="#randomized-pca" title="Permalink to this headline"></a></h2>
<p>Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
algorithm that quickly finds an approximation of the first d principal components. Its computational
complexity is <span class="math notranslate nohighlight">\(O(m \times d^2)+O(d^3)\)</span>, instead of <span class="math notranslate nohighlight">\(O(m \times n^2) + O(n^3)\)</span>, so it is dramatically faster than the
previous algorithms when <span class="math notranslate nohighlight">\(d\)</span> is much smaller than <span class="math notranslate nohighlight">\(n\)</span>.</p>
</div>
<div class="section" id="kernel-pca">
<h2><span class="section-number">20.30. </span>Kernel PCA<a class="headerlink" href="#kernel-pca" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.30. </span>Kernel PCA<a class="headerlink" href="#kernel-pca" title="Permalink to this headline"></a></h2>
<p>The kernel trick is a mathematical technique that implicitly maps instances into a
very high-dimensional space (called the feature space), enabling nonlinear classification and regression
with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
@@ -1971,7 +1906,7 @@ For example, the following code uses Scikit-Learn’s KernelPCA class to perform
</div>
</div>
<div class="section" id="lle">
<h2><span class="section-number">20.31. </span>LLE<a class="headerlink" href="#lle" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.31. </span>LLE<a class="headerlink" href="#lle" title="Permalink to this headline"></a></h2>
<p>Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
@@ -1979,7 +1914,7 @@ closest neighbors (c.n.), and then looking for a low-dimensional representation
these local relationships are best preserved (more details shortly).</p>
</div>
<div class="section" id="other-techniques">
<h2><span class="section-number">20.32. </span>Other techniques<a class="headerlink" href="#other-techniques" title="Permalink to this headline"></a></h2>
<h2><span class="section-number">7.32. </span>Other techniques<a class="headerlink" href="#other-techniques" title="Permalink to this headline"></a></h2>
<p>There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.</p>
<p>Here are some of the most popular:</p>
<ul class="simple">
@@ -2019,7 +1954,7 @@ these local relationships are best preserved (more details shortly).</p>
<div class='prev-next-bottom'>
<a class='left-prev' id="prev-link" href="chapter7.html" title="previous page"><span class="section-number">19. </span>Support Vector Machines, overarching aims</a>
<a class='left-prev' id="prev-link" href="chapter7.html" title="previous page"><span class="section-number">6. </span>Support Vector Machines, overarching aims</a>
</div>
<footer class="footer mt-5 mt-md-0">