update and correcting typos
This commit is contained in:
mhjensen
@@ -190,7 +190,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Oct 21, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
|
||||
@@ -177,9 +177,9 @@ MathJax.Hub.Config({
|
||||
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
|
||||
|
||||
<ul>
|
||||
<li> So each data point is determined by \( p \) extrinsic (measurement) variables</li>
|
||||
<li> Each data point is determined by \( p \) extrinsic (measurement) variables</li>
|
||||
<li> We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?</li>
|
||||
<li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do</li>
|
||||
<li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.</li>
|
||||
</ul>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -173,29 +173,32 @@ MathJax.Hub.Config({
|
||||
|
||||
<h2 id="___sec8" class="anchor">Introducing the Covariance and Correlation functions </h2>
|
||||
|
||||
<p>
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about the definition of the covariance and the correlation function.
|
||||
|
||||
<p>
|
||||
Suppose we have defined two vectors
|
||||
\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} cov[\boldsymbol{x},\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
cov[\boldsymbol{y},\boldsymbol{x}] & cov[\boldsymbol{y},\boldsymbol{y}] \\
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
where for example
|
||||
$$
|
||||
cov[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
$$
|
||||
|
||||
With this definition and recalling that the variance is
|
||||
With this definition and recalling that the variance is defined as
|
||||
$$
|
||||
var[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
$$
|
||||
|
||||
we can rewrite the covariance matrix in this case as
|
||||
we can rewrite the covariance matrix as
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} var[\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{y}] \\
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
@@ -207,18 +210,18 @@ introducing instead the correlation matrix defined via the so-called
|
||||
correlation function
|
||||
|
||||
$$
|
||||
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var[\boldsymbol{y}]}}.
|
||||
\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}]
|
||||
The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]
|
||||
\in [-1,1] \). This avoids eventual problems with too large values. We
|
||||
can then define the correlation matrix for the two vectors \( \boldsymbol{x} \)
|
||||
and \( \boldsymbol{y} \) as
|
||||
|
||||
$$
|
||||
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
|
||||
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
|
||||
@@ -0,0 +1,277 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0010"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec9" class="anchor">Correlation Function and Design/Feature Matrix </h2>
|
||||
|
||||
<p>
|
||||
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
$$
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
|
||||
\dots & \dots & \dots & \dots \dots & \dots \\
|
||||
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the
|
||||
entries \( n \) being the row elements.
|
||||
We can rewrite the design/feature matrix in terms of its column vectors as
|
||||
$$
|
||||
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_0 & \boldsymbol{x}_0 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
|
||||
$$
|
||||
|
||||
with a given vector
|
||||
$$
|
||||
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
and the correlation matrix
|
||||
$$
|
||||
\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs009.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs002.html">3</a></li>
|
||||
<li><a href="._DimRed-bs003.html">4</a></li>
|
||||
<li><a href="._DimRed-bs004.html">5</a></li>
|
||||
<li><a href="._DimRed-bs005.html">6</a></li>
|
||||
<li><a href="._DimRed-bs006.html">7</a></li>
|
||||
<li><a href="._DimRed-bs007.html">8</a></li>
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li class="active"><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs011.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,265 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0011"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec10" class="anchor">Covariance Matrix Examples </h2>
|
||||
|
||||
<p>
|
||||
The Numpy function <b>np.cov</b> calculates the covariance elements using
|
||||
the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have
|
||||
the exact mean values. The following simple function uses the
|
||||
<b>np.vstack</b> function which takes each vector of dimension \( 1\times n \)
|
||||
and produces a \( 2\times n \) matrix \( \boldsymbol{W} \)
|
||||
|
||||
$$
|
||||
\boldsymbol{W} = \begin{bmatrix} x_0 & y_0 \\
|
||||
x_1 & y_1 \\
|
||||
x_2 & y_2\\
|
||||
\dots & \dots \\
|
||||
x_{n-2} & y_{n-2}\\
|
||||
x_{n-1} & y_{n-1} &
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
<p>
|
||||
which in turn is converted into into the \( 2\times 2 \) covariance matrix
|
||||
\( \boldsymbol{C} \) via the Numpy function <b>np.cov()</b>. We note that we can also calculate
|
||||
the mean value of each set of samples \( \boldsymbol{x} \) etc using the Numpy
|
||||
function <b>np.mean(x)</b>. We can also extract the eigenvalues of the
|
||||
covariance matrix through the <b>np.linalg.eig()</b> function.
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>mean(x))
|
||||
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>mean(y))
|
||||
W <span style="color: #666666">=</span> np<span style="color: #666666">.</span>vstack((x, y))
|
||||
C <span style="color: #666666">=</span> np<span style="color: #666666">.</span>cov(W)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(C)
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs010.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs003.html">4</a></li>
|
||||
<li><a href="._DimRed-bs004.html">5</a></li>
|
||||
<li><a href="._DimRed-bs005.html">6</a></li>
|
||||
<li><a href="._DimRed-bs006.html">7</a></li>
|
||||
<li><a href="._DimRed-bs007.html">8</a></li>
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li class="active"><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs012.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,267 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0012"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec11" class="anchor">Correlation Matrix </h2>
|
||||
|
||||
<p>
|
||||
The previous example can be converted into the correlation matrix by
|
||||
simply scaling the matrix elements with the variances. We should also
|
||||
subtract the mean values for each column. This leads to the following
|
||||
code which sets up the correlations matrix for the previous example in
|
||||
a more brute force way. Here we scale the mean values for each column of the design matrix, calculate the relevant mean values and variances and then finally set up the \( 2\times 2 \) correlation matrix (since we have only two vectors).
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
|
||||
<span style="color: #408080; font-style: italic"># define two vectors </span>
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>random(size<span style="color: #666666">=</span>n)
|
||||
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
|
||||
<span style="color: #408080; font-style: italic">#scaling the x and y vectors </span>
|
||||
x <span style="color: #666666">=</span> x <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(x)
|
||||
y <span style="color: #666666">=</span> y <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y)
|
||||
variance_x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(x<span style="color: #AA22FF">@x</span>)<span style="color: #666666">/</span>n
|
||||
variance_y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(y<span style="color: #AA22FF">@y</span>)<span style="color: #666666">/</span>n
|
||||
<span style="color: #008000; font-weight: bold">print</span>(variance_x)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(variance_y)
|
||||
cov_xy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(x<span style="color: #AA22FF">@y</span>)<span style="color: #666666">/</span>n
|
||||
cov_xx <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(x<span style="color: #AA22FF">@x</span>)<span style="color: #666666">/</span>n
|
||||
cov_yy <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sum(y<span style="color: #AA22FF">@y</span>)<span style="color: #666666">/</span>n
|
||||
C <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #666666">2</span>,<span style="color: #666666">2</span>))
|
||||
C[<span style="color: #666666">0</span>,<span style="color: #666666">0</span>]<span style="color: #666666">=</span> cov_xx<span style="color: #666666">/</span>variance_x
|
||||
C[<span style="color: #666666">1</span>,<span style="color: #666666">1</span>]<span style="color: #666666">=</span> cov_yy<span style="color: #666666">/</span>variance_y
|
||||
C[<span style="color: #666666">0</span>,<span style="color: #666666">1</span>]<span style="color: #666666">=</span> cov_xy<span style="color: #666666">/</span>np<span style="color: #666666">.</span>sqrt(variance_y<span style="color: #666666">*</span>variance_x)
|
||||
C[<span style="color: #666666">1</span>,<span style="color: #666666">0</span>]<span style="color: #666666">=</span> C[<span style="color: #666666">0</span>,<span style="color: #666666">1</span>]
|
||||
<span style="color: #008000; font-weight: bold">print</span>(C)
|
||||
</pre></div>
|
||||
<p>
|
||||
We see that the matrix elements along the diagonal are one as they
|
||||
should be and that the matrix is symmetric. Furthermore, diagonalizing
|
||||
this matrix we easily see that it is a positive definite matrix.
|
||||
|
||||
<p>
|
||||
The above procedure with <b>numpy</b> can be made more compact if we use <b>pandas</b>.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs011.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs004.html">5</a></li>
|
||||
<li><a href="._DimRed-bs005.html">6</a></li>
|
||||
<li><a href="._DimRed-bs006.html">7</a></li>
|
||||
<li><a href="._DimRed-bs007.html">8</a></li>
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li class="active"><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs013.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0013"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec12" class="anchor">Correlation Matrix with Pandas </h2>
|
||||
|
||||
<p>
|
||||
We whow here how we can set up the correlation matrix using <b>pandas</b>, as done in this simple code
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
n <span style="color: #666666">=</span> <span style="color: #666666">10</span>
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
|
||||
x <span style="color: #666666">=</span> x <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(x)
|
||||
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
|
||||
y <span style="color: #666666">=</span> y <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y)
|
||||
X <span style="color: #666666">=</span> (np<span style="color: #666666">.</span>vstack((x, y)))<span style="color: #666666">.</span>T
|
||||
<span style="color: #008000; font-weight: bold">print</span>(X)
|
||||
Xpd <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(X)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(Xpd)
|
||||
correlation_matrix <span style="color: #666666">=</span> Xpd<span style="color: #666666">.</span>corr()
|
||||
<span style="color: #008000; font-weight: bold">print</span>(correlation_matrix)
|
||||
</pre></div>
|
||||
<p>
|
||||
We expand this model to the Franke function discussed above.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs012.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs005.html">6</a></li>
|
||||
<li><a href="._DimRed-bs006.html">7</a></li>
|
||||
<li><a href="._DimRed-bs007.html">8</a></li>
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li class="active"><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs014.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,285 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0014"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec13" class="anchor">Correlation Matrix with Pandas and the Franke function </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Common imports</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">FrankeFunction</span>(x,y):
|
||||
term1 <span style="color: #666666">=</span> <span style="color: #666666">0.75*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">0.25*</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>) <span style="color: #666666">-</span> <span style="color: #666666">0.25*</span>((<span style="color: #666666">9*</span>y<span style="color: #666666">-2</span>)<span style="color: #666666">**2</span>))
|
||||
term2 <span style="color: #666666">=</span> <span style="color: #666666">0.75*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>((<span style="color: #666666">9*</span>x<span style="color: #666666">+1</span>)<span style="color: #666666">**2</span>)<span style="color: #666666">/49.0</span> <span style="color: #666666">-</span> <span style="color: #666666">0.1*</span>(<span style="color: #666666">9*</span>y<span style="color: #666666">+1</span>))
|
||||
term3 <span style="color: #666666">=</span> <span style="color: #666666">0.5*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-7</span>)<span style="color: #666666">**2/4.0</span> <span style="color: #666666">-</span> <span style="color: #666666">0.25*</span>((<span style="color: #666666">9*</span>y<span style="color: #666666">-3</span>)<span style="color: #666666">**2</span>))
|
||||
term4 <span style="color: #666666">=</span> <span style="color: #666666">-0.2*</span>np<span style="color: #666666">.</span>exp(<span style="color: #666666">-</span>(<span style="color: #666666">9*</span>x<span style="color: #666666">-4</span>)<span style="color: #666666">**2</span> <span style="color: #666666">-</span> (<span style="color: #666666">9*</span>y<span style="color: #666666">-7</span>)<span style="color: #666666">**2</span>)
|
||||
<span style="color: #008000; font-weight: bold">return</span> term1 <span style="color: #666666">+</span> term2 <span style="color: #666666">+</span> term3 <span style="color: #666666">+</span> term4
|
||||
|
||||
|
||||
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">create_X</span>(x, y, n ):
|
||||
<span style="color: #008000; font-weight: bold">if</span> <span style="color: #008000">len</span>(x<span style="color: #666666">.</span>shape) <span style="color: #666666">></span> <span style="color: #666666">1</span>:
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ravel(x)
|
||||
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ravel(y)
|
||||
|
||||
N <span style="color: #666666">=</span> <span style="color: #008000">len</span>(x)
|
||||
l <span style="color: #666666">=</span> <span style="color: #008000">int</span>((n<span style="color: #666666">+1</span>)<span style="color: #666666">*</span>(n<span style="color: #666666">+2</span>)<span style="color: #666666">/2</span>) <span style="color: #408080; font-style: italic"># Number of elements in beta</span>
|
||||
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>ones((N,l))
|
||||
|
||||
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(<span style="color: #666666">1</span>,n<span style="color: #666666">+1</span>):
|
||||
q <span style="color: #666666">=</span> <span style="color: #008000">int</span>((i)<span style="color: #666666">*</span>(i<span style="color: #666666">+1</span>)<span style="color: #666666">/2</span>)
|
||||
<span style="color: #008000; font-weight: bold">for</span> k <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(i<span style="color: #666666">+1</span>):
|
||||
X[:,q<span style="color: #666666">+</span>k] <span style="color: #666666">=</span> (x<span style="color: #666666">**</span>(i<span style="color: #666666">-</span>k))<span style="color: #666666">*</span>(y<span style="color: #666666">**</span>k)
|
||||
|
||||
<span style="color: #008000; font-weight: bold">return</span> X
|
||||
|
||||
|
||||
<span style="color: #408080; font-style: italic"># Making meshgrid of datapoints and compute Franke's function</span>
|
||||
n <span style="color: #666666">=</span> <span style="color: #666666">4</span>
|
||||
N <span style="color: #666666">=</span> <span style="color: #666666">100</span>
|
||||
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sort(np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>uniform(<span style="color: #666666">0</span>, <span style="color: #666666">1</span>, N))
|
||||
y <span style="color: #666666">=</span> np<span style="color: #666666">.</span>sort(np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>uniform(<span style="color: #666666">0</span>, <span style="color: #666666">1</span>, N))
|
||||
z <span style="color: #666666">=</span> FrankeFunction(x, y)
|
||||
X <span style="color: #666666">=</span> create_X(x, y, n<span style="color: #666666">=</span>n)
|
||||
|
||||
Xpd <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(X)
|
||||
<span style="color: #408080; font-style: italic"># subtract the mean values and set up the covariance matrix</span>
|
||||
Xpd <span style="color: #666666">=</span> Xpd <span style="color: #666666">-</span> Xpd<span style="color: #666666">.</span>mean()
|
||||
covariance_matrix <span style="color: #666666">=</span> Xpd<span style="color: #666666">.</span>cov()
|
||||
<span style="color: #008000; font-weight: bold">print</span>(covariance_matrix)
|
||||
</pre></div>
|
||||
<p>
|
||||
We note here that the covariance is zero for the first rows and
|
||||
columns since all matrix elements in the design matrix were set to one
|
||||
(we are fitting the function in terms of a polynomial of degree \( n \)).
|
||||
|
||||
<p>
|
||||
This means that the variance for these elements will be zero and will
|
||||
cause problems when we set up the correlation matrix. We can simply
|
||||
drop these elements as follows and then construct the correlation
|
||||
matrix.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs013.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs006.html">7</a></li>
|
||||
<li><a href="._DimRed-bs007.html">8</a></li>
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li class="active"><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs015.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0015"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec14" class="anchor">Classical PCA Theorem </h2>
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs014.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs007.html">8</a></li>
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li class="active"><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs016.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,225 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0016"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec15" class="anchor">Prof of the PCA Theorem </h2>
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs015.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li class="active"><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs017.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,234 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0017"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec16" class="anchor">Getting started with PCA </h2>
|
||||
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Now add PCA</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> PCA
|
||||
pca <span style="color: #666666">=</span> PCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>)
|
||||
pca<span style="color: #666666">.</span>fit(X_train_scaled)
|
||||
|
||||
X_pca <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>transform(X_train_scaled)
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs016.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li class="active"><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs018.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0018"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec17" class="anchor">Principal Component Analysis </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
|
||||
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
|
||||
|
||||
<p>
|
||||
The following Python code uses NumPy’s <b>svd()</b> function to obtain all the principal components of the
|
||||
training set, then extracts the first two principal components
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>X_centered <span style="color: #666666">=</span> X <span style="color: #666666">-</span> X<span style="color: #666666">.</span>mean(axis<span style="color: #666666">=0</span>)
|
||||
U, s, V <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>svd(X_centered)
|
||||
c1 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, <span style="color: #666666">0</span>]
|
||||
c2 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, <span style="color: #666666">1</span>]
|
||||
</pre></div>
|
||||
<p>
|
||||
PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
|
||||
the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
|
||||
forget to center the data first.
|
||||
|
||||
<p>
|
||||
Once you have identified all the principal components, you can reduce the dimensionality of the dataset
|
||||
down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
|
||||
Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>W2 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, :<span style="color: #666666">2</span>]
|
||||
X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666">.</span>dot(W2)
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs017.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li class="active"><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs019.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,248 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0019"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec18" class="anchor">PCA and scikit-learn </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
|
||||
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
|
||||
that it automatically takes care of centering the data):
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> PCA
|
||||
pca <span style="color: #666666">=</span> PCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>)
|
||||
X2D <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
</pre></div>
|
||||
<p>
|
||||
After fitting the PCA transformer to the dataset, you can access the principal components using the
|
||||
components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
|
||||
principal component is equal to
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca<span style="color: #666666">.</span>components_<span style="color: #666666">.</span>T[:, <span style="color: #666666">0</span>])<span style="color: #666666">.</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
Another very useful piece of information is the explained variance ratio of each principal component,
|
||||
available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
|
||||
variance that lies along the axis of each principal component.
|
||||
More material to come here.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs018.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li class="active"><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs020.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,246 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0020"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec19" class="anchor">More on the PCA </h2>
|
||||
|
||||
<p>
|
||||
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
|
||||
choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
|
||||
Unless, of course, you are reducing dimensionality for data visualization — in that case you will
|
||||
generally want to reduce the dimensionality down to 2 or 3.
|
||||
The following code computes PCA without reducing dimensionality, then computes the minimum number
|
||||
of dimensions required to preserve 95% of the training set’s variance:
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca <span style="color: #666666">=</span> PCA()
|
||||
pca<span style="color: #666666">.</span>fit(X)
|
||||
cumsum <span style="color: #666666">=</span> np<span style="color: #666666">.</span>cumsum(pca<span style="color: #666666">.</span>explained_variance_ratio_)
|
||||
d <span style="color: #666666">=</span> np<span style="color: #666666">.</span>argmax(cumsum <span style="color: #666666">>=</span> <span style="color: #666666">0.95</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
|
||||
of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
|
||||
a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca <span style="color: #666666">=</span> PCA(n_components<span style="color: #666666">=0.95</span>)
|
||||
X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs019.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li class="active"><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs021.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,227 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0021"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec20" class="anchor">Incremental PCA </h2>
|
||||
|
||||
<p>
|
||||
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
|
||||
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
|
||||
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
|
||||
at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
|
||||
instances arrive).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs020.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li class="active"><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs022.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,230 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0022"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec21" class="anchor">Randomized PCA </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
|
||||
algorithm that quickly finds an approximation of the first d principal components. Its computational
|
||||
complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
|
||||
previous algorithms when \( d \) is much smaller than \( n \).
|
||||
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs021.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li class="active"><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs023.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0023"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec22" class="anchor">Kernel PCA </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
|
||||
<p>
|
||||
The kernel trick is a mathematical technique that implicitly maps instances into a
|
||||
very high-dimensional space (called the feature space), enabling nonlinear classification and regression
|
||||
with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
|
||||
space corresponds to a complex nonlinear decision boundary in the original space.
|
||||
It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
|
||||
projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
|
||||
preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
|
||||
twisted manifold.
|
||||
For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> KernelPCA
|
||||
rbf_pca <span style="color: #666666">=</span> KernelPCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>, kernel<span style="color: #666666">=</span><span style="color: #BA2121">"rbf"</span>, gamma<span style="color: #666666">=0.04</span>)
|
||||
X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
</pre></div>
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs022.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li class="active"><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs024.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,224 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0024"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec23" class="anchor">LLE </h2>
|
||||
|
||||
<p>
|
||||
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
|
||||
(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
|
||||
algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
|
||||
closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where
|
||||
these local relationships are best preserved (more details shortly).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs023.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li class="active"><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs025.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
<!--
|
||||
Automatically generated HTML file from DocOnce source
|
||||
(https://github.com/hplgit/doconce/)
|
||||
-->
|
||||
<html>
|
||||
<head>
|
||||
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
|
||||
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<meta name="description" content="Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction">
|
||||
|
||||
<title>Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</title>
|
||||
|
||||
<!-- Bootstrap style: bootstrap -->
|
||||
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
|
||||
<!-- not necessary
|
||||
<link href="https://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css" rel="stylesheet">
|
||||
-->
|
||||
|
||||
<style type="text/css">
|
||||
|
||||
/* Add scrollbar to dropdown menus in bootstrap navigation bar */
|
||||
.dropdown-menu {
|
||||
height: auto;
|
||||
max-height: 400px;
|
||||
overflow-x: hidden;
|
||||
}
|
||||
|
||||
/* Adds an invisible element before each target to offset for the navigation
|
||||
bar */
|
||||
.anchor::before {
|
||||
content:"";
|
||||
display:block;
|
||||
height:50px; /* fixed header height for style bootstrap */
|
||||
margin:-50px 0 0; /* negative fixed header height */
|
||||
}
|
||||
</style>
|
||||
|
||||
|
||||
</head>
|
||||
|
||||
<!-- tocinfo
|
||||
{'highest level': 2,
|
||||
'sections': [('Reducing the number of degrees of freedom, overarching view',
|
||||
2,
|
||||
None,
|
||||
'___sec0'),
|
||||
('Preprocessing our data', 2, None, '___sec1'),
|
||||
('More preprocessing', 2, None, '___sec2'),
|
||||
('Simple preprocessing examples, Franke function and regression',
|
||||
2,
|
||||
None,
|
||||
'___sec3'),
|
||||
('Simple preprocessing examples, breast cancer data and '
|
||||
'classification, Support Vector Machines',
|
||||
2,
|
||||
None,
|
||||
'___sec4'),
|
||||
('More on Cancer Data, now with Logistic Regression',
|
||||
2,
|
||||
None,
|
||||
'___sec5'),
|
||||
('Why should we think of reducing the dimensionality',
|
||||
2,
|
||||
None,
|
||||
'___sec6'),
|
||||
('Basic ideas of the Principal Component Analysis (PCA)',
|
||||
2,
|
||||
None,
|
||||
'___sec7'),
|
||||
('Introducing the Covariance and Correlation functions',
|
||||
2,
|
||||
None,
|
||||
'___sec8'),
|
||||
('Correlation Function and Design/Feature Matrix',
|
||||
2,
|
||||
None,
|
||||
'___sec9'),
|
||||
('Covariance Matrix Examples', 2, None, '___sec10'),
|
||||
('Correlation Matrix', 2, None, '___sec11'),
|
||||
('Correlation Matrix with Pandas', 2, None, '___sec12'),
|
||||
('Correlation Matrix with Pandas and the Franke function',
|
||||
2,
|
||||
None,
|
||||
'___sec13'),
|
||||
('Classical PCA Theorem', 2, None, '___sec14'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec15'),
|
||||
('Getting started with PCA', 2, None, '___sec16'),
|
||||
('Principal Component Analysis', 2, None, '___sec17'),
|
||||
('PCA and scikit-learn', 2, None, '___sec18'),
|
||||
('More on the PCA', 2, None, '___sec19'),
|
||||
('Incremental PCA', 2, None, '___sec20'),
|
||||
('Randomized PCA', 2, None, '___sec21'),
|
||||
('Kernel PCA', 2, None, '___sec22'),
|
||||
('LLE', 2, None, '___sec23'),
|
||||
('Other techniques', 2, None, '___sec24')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
|
||||
|
||||
|
||||
<script type="text/x-mathjax-config">
|
||||
MathJax.Hub.Config({
|
||||
TeX: {
|
||||
equationNumbers: { autoNumber: "none" },
|
||||
extensions: ["AMSmath.js", "AMSsymbols.js", "autobold.js", "color.js"]
|
||||
}
|
||||
});
|
||||
</script>
|
||||
<script type="text/javascript" async
|
||||
src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_HTMLorMML">
|
||||
</script>
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Bootstrap navigation bar -->
|
||||
<div class="navbar navbar-default navbar-fixed-top">
|
||||
<div class="navbar-header">
|
||||
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".navbar-responsive-collapse">
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
<span class="icon-bar"></span>
|
||||
</button>
|
||||
<a class="navbar-brand" href="DimRed-bs.html">Data Analysis and Machine Learning: Preprocessing and Dimensionality Reduction</a>
|
||||
</div>
|
||||
|
||||
<div class="navbar-collapse collapse navbar-responsive-collapse">
|
||||
<ul class="nav navbar-nav navbar-right">
|
||||
<li class="dropdown">
|
||||
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
|
||||
<ul class="dropdown-menu">
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs001.html#___sec0" style="font-size: 80%;">Reducing the number of degrees of freedom, overarching view</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs002.html#___sec1" style="font-size: 80%;">Preprocessing our data</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs003.html#___sec2" style="font-size: 80%;">More preprocessing</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs004.html#___sec3" style="font-size: 80%;">Simple preprocessing examples, Franke function and regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs005.html#___sec4" style="font-size: 80%;">Simple preprocessing examples, breast cancer data and classification, Support Vector Machines</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs006.html#___sec5" style="font-size: 80%;">More on Cancer Data, now with Logistic Regression</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Covariance Matrix Examples</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Correlation Matrix</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Correlation Matrix with Pandas</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Correlation Matrix with Pandas and the Franke function</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">Classical PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Getting started with PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Principal Component Analysis</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">PCA and scikit-learn</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">More on the PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Incremental PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec21" style="font-size: 80%;">Randomized PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec22" style="font-size: 80%;">Kernel PCA</a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec23" style="font-size: 80%;">LLE</a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;">Other techniques</a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div> <!-- end of navigation bar -->
|
||||
|
||||
<div class="container">
|
||||
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0025"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec24" class="anchor">Other techniques </h2>
|
||||
|
||||
<p>
|
||||
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
|
||||
|
||||
<p>
|
||||
Here are some of the most popular:
|
||||
|
||||
<ul>
|
||||
<li> <b>Multidimensional Scaling (MDS)</b> reduces dimensionality while trying to preserve the distances between the instances.</li>
|
||||
<li> <b>Isomap</b> creates a graph by connecting each instance to its nearest neighbors, then reduces dimensionality while trying to preserve the geodesic distances between the instances.</li>
|
||||
<li> <b>t-Distributed Stochastic Neighbor Embedding</b> (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).</li>
|
||||
<li> Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.</li>
|
||||
</ul>
|
||||
|
||||
Here are other examples where we use the <b>DataFrame</b> functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix
|
||||
of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
|
||||
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">100</span>)
|
||||
<span style="color: #408080; font-style: italic"># setting up a 10 x 5 matrix</span>
|
||||
rows <span style="color: #666666">=</span> <span style="color: #666666">10</span>
|
||||
cols <span style="color: #666666">=</span> <span style="color: #666666">5</span>
|
||||
a <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(rows,cols)
|
||||
df <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(a)
|
||||
display(df)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>mean())
|
||||
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>std())
|
||||
display(df<span style="color: #666666">**2</span>)
|
||||
</pre></div>
|
||||
<p>
|
||||
Thereafter we can select specific columns only and plot final results
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>df<span style="color: #666666">.</span>columns <span style="color: #666666">=</span> [<span style="color: #BA2121">'First'</span>, <span style="color: #BA2121">'Second'</span>, <span style="color: #BA2121">'Third'</span>, <span style="color: #BA2121">'Fourth'</span>, <span style="color: #BA2121">'Fifth'</span>]
|
||||
df<span style="color: #666666">.</span>index <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">10</span>)
|
||||
|
||||
display(df)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(df[<span style="color: #BA2121">'Second'</span>]<span style="color: #666666">.</span>mean() )
|
||||
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>info())
|
||||
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>describe())
|
||||
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">pylab</span> <span style="color: #008000; font-weight: bold">import</span> plt, mpl
|
||||
plt<span style="color: #666666">.</span>style<span style="color: #666666">.</span>use(<span style="color: #BA2121">'seaborn'</span>)
|
||||
mpl<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">'font.family'</span>] <span style="color: #666666">=</span> <span style="color: #BA2121">'serif'</span>
|
||||
|
||||
df<span style="color: #666666">.</span>cumsum()<span style="color: #666666">.</span>plot(lw<span style="color: #666666">=2.0</span>, figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>))
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
|
||||
|
||||
df<span style="color: #666666">.</span>plot<span style="color: #666666">.</span>bar(figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>), rot<span style="color: #666666">=15</span>)
|
||||
plt<span style="color: #666666">.</span>show()
|
||||
</pre></div>
|
||||
<p>
|
||||
We can produce a \( 4\times 4 \) matrix
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">16</span>)<span style="color: #666666">.</span>reshape((<span style="color: #666666">4</span>,<span style="color: #666666">4</span>))
|
||||
<span style="color: #008000; font-weight: bold">print</span>(b)
|
||||
df1 <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(b)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(df1)
|
||||
</pre></div>
|
||||
<p>
|
||||
and many other operations.
|
||||
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
<ul class="pagination">
|
||||
<li><a href="._DimRed-bs024.html">«</a></li>
|
||||
<li><a href="._DimRed-bs000.html">1</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li class="active"><a href="._DimRed-bs025.html">26</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
</div> <!-- end container -->
|
||||
<!-- include javascript, jQuery *first* -->
|
||||
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js"></script>
|
||||
<script src="https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
|
||||
<!-- Bootstrap footer
|
||||
<footer>
|
||||
<a href="http://..."><img width="250" align=right src="http://..."></a>
|
||||
</footer>
|
||||
-->
|
||||
|
||||
|
||||
<center style="font-size:80%">
|
||||
<!-- copyright only on the titlepage -->
|
||||
</center>
|
||||
|
||||
|
||||
</body>
|
||||
</html>
|
||||
|
||||
|
||||
@@ -190,7 +190,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Oct 21, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
|
||||
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p> <br>
|
||||
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Oct 21, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -514,9 +514,9 @@ applications.
|
||||
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
|
||||
|
||||
<ul>
|
||||
<p><li> So each data point is determined by \( p \) extrinsic (measurement) variables</li>
|
||||
<p><li> Each data point is determined by \( p \) extrinsic (measurement) variables</li>
|
||||
<p><li> We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?</li>
|
||||
<p><li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do</li>
|
||||
<p><li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.</li>
|
||||
</ul>
|
||||
</section>
|
||||
|
||||
@@ -524,13 +524,16 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
|
||||
<section>
|
||||
<h2 id="___sec8">Introducing the Covariance and Correlation functions </h2>
|
||||
|
||||
<p>
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about the definition of the covariance and the correlation function.
|
||||
|
||||
<p>
|
||||
Suppose we have defined two vectors
|
||||
\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} cov[\boldsymbol{x},\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
cov[\boldsymbol{y},\boldsymbol{x}] & cov[\boldsymbol{y},\boldsymbol{y}] \\
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
<p> <br>
|
||||
@@ -538,22 +541,22 @@ $$
|
||||
where for example
|
||||
<p> <br>
|
||||
$$
|
||||
cov[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
With this definition and recalling that the variance is
|
||||
With this definition and recalling that the variance is defined as
|
||||
<p> <br>
|
||||
$$
|
||||
var[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
we can rewrite the covariance matrix in this case as
|
||||
we can rewrite the covariance matrix as
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} var[\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{y}] \\
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
<p> <br>
|
||||
@@ -567,20 +570,20 @@ correlation function
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var[\boldsymbol{y}]}}.
|
||||
\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}.
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}]
|
||||
The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]
|
||||
\in [-1,1] \). This avoids eventual problems with too large values. We
|
||||
can then define the correlation matrix for the two vectors \( \boldsymbol{x} \)
|
||||
and \( \boldsymbol{y} \) as
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
|
||||
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
<p> <br>
|
||||
@@ -629,12 +632,12 @@ With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covar
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] & cov[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
cov[\boldsymbol{x}_1,\boldsymbol{x}_0] & var[\boldsymbol{x}_1] & cov[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
cov[\boldsymbol{x}_2,\boldsymbol{x}_0] & cov[\boldsymbol{x}_2,\boldsymbol{x}_1] & var[\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & var[\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
<p> <br>
|
||||
@@ -643,12 +646,12 @@ and the correlation matrix
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
1 & corr[\boldsymbol{x}_0,\boldsymbol{x}_1] & corr[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
corr[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & corr[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
corr[\boldsymbol{x}_2,\boldsymbol{x}_0] & corr[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & corr[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\
|
||||
\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
<p> <br>
|
||||
|
||||
@@ -155,7 +155,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Oct 21, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -518,38 +518,41 @@ applications.
|
||||
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
|
||||
|
||||
<ul>
|
||||
<li> So each data point is determined by \( p \) extrinsic (measurement) variables</li>
|
||||
<li> Each data point is determined by \( p \) extrinsic (measurement) variables</li>
|
||||
<li> We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?</li>
|
||||
<li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do</li>
|
||||
<li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.</li>
|
||||
</ul>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec8">Introducing the Covariance and Correlation functions </h2>
|
||||
|
||||
<p>
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about the definition of the covariance and the correlation function.
|
||||
|
||||
<p>
|
||||
Suppose we have defined two vectors
|
||||
\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} cov[\boldsymbol{x},\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
cov[\boldsymbol{y},\boldsymbol{x}] & cov[\boldsymbol{y},\boldsymbol{y}] \\
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
where for example
|
||||
$$
|
||||
cov[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
$$
|
||||
|
||||
With this definition and recalling that the variance is
|
||||
With this definition and recalling that the variance is defined as
|
||||
$$
|
||||
var[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
$$
|
||||
|
||||
we can rewrite the covariance matrix in this case as
|
||||
we can rewrite the covariance matrix as
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} var[\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{y}] \\
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
@@ -561,18 +564,18 @@ introducing instead the correlation matrix defined via the so-called
|
||||
correlation function
|
||||
|
||||
$$
|
||||
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var[\boldsymbol{y}]}}.
|
||||
\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}]
|
||||
The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]
|
||||
\in [-1,1] \). This avoids eventual problems with too large values. We
|
||||
can then define the correlation matrix for the two vectors \( \boldsymbol{x} \)
|
||||
and \( \boldsymbol{y} \) as
|
||||
|
||||
$$
|
||||
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
|
||||
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
@@ -613,24 +616,24 @@ $$
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] & cov[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
cov[\boldsymbol{x}_1,\boldsymbol{x}_0] & var[\boldsymbol{x}_1] & cov[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
cov[\boldsymbol{x}_2,\boldsymbol{x}_0] & cov[\boldsymbol{x}_2,\boldsymbol{x}_1] & var[\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & var[\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
and the correlation matrix
|
||||
$$
|
||||
\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
1 & corr[\boldsymbol{x}_0,\boldsymbol{x}_1] & corr[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
corr[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & corr[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
corr[\boldsymbol{x}_2,\boldsymbol{x}_0] & corr[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & corr[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\
|
||||
\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
|
||||
@@ -160,7 +160,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Oct 21, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -523,38 +523,41 @@ applications.
|
||||
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
|
||||
|
||||
<ul>
|
||||
<li> So each data point is determined by \( p \) extrinsic (measurement) variables</li>
|
||||
<li> Each data point is determined by \( p \) extrinsic (measurement) variables</li>
|
||||
<li> We may want to ask the following question: Are there fewer intrinsic variables (say \( d < < p \)) that still approximately describe the data?</li>
|
||||
<li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do</li>
|
||||
<li> If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.</li>
|
||||
</ul>
|
||||
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec8">Introducing the Covariance and Correlation functions </h2>
|
||||
|
||||
<p>
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about the definition of the covariance and the correlation function.
|
||||
|
||||
<p>
|
||||
Suppose we have defined two vectors
|
||||
\( \hat{x} \) and \( \hat{y} \) with \( n \) elements each. The covariance matrix \( \boldsymbol{C} \) is defined as
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} cov[\boldsymbol{x},\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
cov[\boldsymbol{y},\boldsymbol{x}] & cov[\boldsymbol{y},\boldsymbol{y}] \\
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{cov}[\boldsymbol{x},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{cov}[\boldsymbol{y},\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{y},\boldsymbol{y}] \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
where for example
|
||||
$$
|
||||
cov[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
$$
|
||||
|
||||
With this definition and recalling that the variance is
|
||||
With this definition and recalling that the variance is defined as
|
||||
$$
|
||||
var[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
\mathrm{var}[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
$$
|
||||
|
||||
we can rewrite the covariance matrix in this case as
|
||||
we can rewrite the covariance matrix as
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} var[\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{y}] \\
|
||||
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} \mathrm{var}[\boldsymbol{x}] & \mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}] & \mathrm{var}[\boldsymbol{y}] \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
@@ -566,18 +569,18 @@ introducing instead the correlation matrix defined via the so-called
|
||||
correlation function
|
||||
|
||||
$$
|
||||
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var[\boldsymbol{y}]}}.
|
||||
\mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]=\frac{\mathrm{cov}[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{\mathrm{var}[\boldsymbol{x}] \mathrm{var}[\boldsymbol{y}]}}.
|
||||
$$
|
||||
|
||||
<p>
|
||||
The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}]
|
||||
The correlation function is then given by values \( \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}]
|
||||
\in [-1,1] \). This avoids eventual problems with too large values. We
|
||||
can then define the correlation matrix for the two vectors \( \boldsymbol{x} \)
|
||||
and \( \boldsymbol{y} \) as
|
||||
|
||||
$$
|
||||
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
|
||||
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\boldsymbol{x},\boldsymbol{y}] \\
|
||||
\mathrm{corr}[\boldsymbol{y},\boldsymbol{x}] & 1 \\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
@@ -618,24 +621,24 @@ $$
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] & cov[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
cov[\boldsymbol{x}_1,\boldsymbol{x}_0] & var[\boldsymbol{x}_1] & cov[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
cov[\boldsymbol{x}_2,\boldsymbol{x}_0] & cov[\boldsymbol{x}_2,\boldsymbol{x}_1] & var[\boldsymbol{x}_2] & \dots & \dots & cov[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_0] & \mathrm{var}[\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_1] & \mathrm{var}[\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & cov[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & var[\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & \mathrm{var}[\boldsymbol{x}_{p-1}]\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
and the correlation matrix
|
||||
$$
|
||||
\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
1 & corr[\boldsymbol{x}_0,\boldsymbol{x}_1] & corr[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
corr[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & corr[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & corr[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
corr[\boldsymbol{x}_2,\boldsymbol{x}_0] & corr[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & corr[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
1 & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_0] & 1 & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_2] & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_1,\boldsymbol{x}_{p-1}]\\
|
||||
\mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\boldsymbol{x}_2,\boldsymbol{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & corr[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\
|
||||
\mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_0] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_1] & \mathrm{corr}[\boldsymbol{x}_{p-1},\boldsymbol{x}_{2}] & \dots & \dots & 1\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
"<!-- Author: --> \n",
|
||||
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
||||
"\n",
|
||||
"Date: **Oct 20, 2019**\n",
|
||||
"Date: **Oct 21, 2019**\n",
|
||||
"\n",
|
||||
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
||||
"\n",
|
||||
@@ -403,14 +403,16 @@
|
||||
"## Basic ideas of the Principal Component Analysis (PCA)\n",
|
||||
"\n",
|
||||
"We have a data set defined by a design/feature matrix $\\boldsymbol{X}$ (see below for its definition) \n",
|
||||
"* So each data point is determined by $p$ extrinsic (measurement) variables\n",
|
||||
"* Each data point is determined by $p$ extrinsic (measurement) variables\n",
|
||||
"\n",
|
||||
"* We may want to ask the following question: Are there fewer intrinsic variables (say $d << p$) that still approximately describe the data?\n",
|
||||
"\n",
|
||||
"* If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do\n",
|
||||
"* If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do. \n",
|
||||
"\n",
|
||||
"## Introducing the Covariance and Correlation functions\n",
|
||||
"\n",
|
||||
"Before we discuss the PCA theorem, we need to remind ourselves about the definition of the covariance and the correlation function.\n",
|
||||
"\n",
|
||||
"Suppose we have defined two vectors\n",
|
||||
"$\\hat{x}$ and $\\hat{y}$ with $n$ elements each. The covariance matrix $\\boldsymbol{C}$ is defined as"
|
||||
]
|
||||
@@ -420,8 +422,8 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} cov[\\boldsymbol{x},\\boldsymbol{x}] & cov[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
|
||||
" cov[\\boldsymbol{y},\\boldsymbol{x}] & cov[\\boldsymbol{y},\\boldsymbol{y}] \\\\\n",
|
||||
"\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
|
||||
" \\mathrm{cov}[\\boldsymbol{y},\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{y},\\boldsymbol{y}] \\\\\n",
|
||||
" \\end{bmatrix},\n",
|
||||
"$$"
|
||||
]
|
||||
@@ -438,7 +440,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"cov[\\boldsymbol{x},\\boldsymbol{y}] =\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})(y_i- \\overline{y}).\n",
|
||||
"\\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] =\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})(y_i- \\overline{y}).\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -446,7 +448,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"With this definition and recalling that the variance is"
|
||||
"With this definition and recalling that the variance is defined as"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -454,7 +456,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"var[\\boldsymbol{x}]=\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})^2,\n",
|
||||
"\\mathrm{var}[\\boldsymbol{x}]=\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})^2,\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -462,7 +464,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"we can rewrite the covariance matrix in this case as"
|
||||
"we can rewrite the covariance matrix as"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -470,8 +472,8 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} var[\\boldsymbol{x}] & cov[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
|
||||
" cov[\\boldsymbol{x},\\boldsymbol{y}] & var[\\boldsymbol{y}] \\\\\n",
|
||||
"\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} \\mathrm{var}[\\boldsymbol{x}] & \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
|
||||
" \\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}] & \\mathrm{var}[\\boldsymbol{y}] \\\\\n",
|
||||
" \\end{bmatrix},\n",
|
||||
"$$"
|
||||
]
|
||||
@@ -492,7 +494,7 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"corr[\\boldsymbol{x},\\boldsymbol{y}]=\\frac{cov[\\boldsymbol{x},\\boldsymbol{y}]}{\\sqrt{var[\\boldsymbol{x}]\\var[\\boldsymbol{y}]}}.\n",
|
||||
"\\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}]=\\frac{\\mathrm{cov}[\\boldsymbol{x},\\boldsymbol{y}]}{\\sqrt{\\mathrm{var}[\\boldsymbol{x}] \\mathrm{var}[\\boldsymbol{y}]}}.\n",
|
||||
"$$"
|
||||
]
|
||||
},
|
||||
@@ -500,7 +502,7 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"The correlation function is then given by values $corr[\\boldsymbol{x},\\boldsymbol{y}]\n",
|
||||
"The correlation function is then given by values $\\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}]\n",
|
||||
"\\in [-1,1]$. This avoids eventual problems with too large values. We\n",
|
||||
"can then define the correlation matrix for the two vectors $\\boldsymbol{x}$\n",
|
||||
"and $\\boldsymbol{y}$ as"
|
||||
@@ -511,8 +513,8 @@
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{K}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} 1 & corr[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
|
||||
" corr[\\boldsymbol{y},\\boldsymbol{x}] & 1 \\\\\n",
|
||||
"\\boldsymbol{K}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} 1 & \\mathrm{corr}[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
|
||||
" \\mathrm{corr}[\\boldsymbol{y},\\boldsymbol{x}] & 1 \\\\\n",
|
||||
" \\end{bmatrix},\n",
|
||||
"$$"
|
||||
]
|
||||
@@ -591,12 +593,12 @@
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{C}[\\boldsymbol{x}] = \\begin{bmatrix}\n",
|
||||
"var[\\boldsymbol{x}_0] & cov[\\boldsymbol{x}_0,\\boldsymbol{x}_1] & cov[\\boldsymbol{x}_0,\\boldsymbol{x}_2] & \\dots & \\dots & cov[\\boldsymbol{x}_0,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"cov[\\boldsymbol{x}_1,\\boldsymbol{x}_0] & var[\\boldsymbol{x}_1] & cov[\\boldsymbol{x}_1,\\boldsymbol{x}_2] & \\dots & \\dots & cov[\\boldsymbol{x}_1,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"cov[\\boldsymbol{x}_2,\\boldsymbol{x}_0] & cov[\\boldsymbol{x}_2,\\boldsymbol{x}_1] & var[\\boldsymbol{x}_2] & \\dots & \\dots & cov[\\boldsymbol{x}_2,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\mathrm{var}[\\boldsymbol{x}_0] & \\mathrm{cov}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] & \\mathrm{cov}[\\boldsymbol{x}_0,\\boldsymbol{x}_2] & \\dots & \\dots & \\mathrm{cov}[\\boldsymbol{x}_0,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\mathrm{cov}[\\boldsymbol{x}_1,\\boldsymbol{x}_0] & \\mathrm{var}[\\boldsymbol{x}_1] & \\mathrm{cov}[\\boldsymbol{x}_1,\\boldsymbol{x}_2] & \\dots & \\dots & \\mathrm{cov}[\\boldsymbol{x}_1,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\mathrm{cov}[\\boldsymbol{x}_2,\\boldsymbol{x}_0] & \\mathrm{cov}[\\boldsymbol{x}_2,\\boldsymbol{x}_1] & \\mathrm{var}[\\boldsymbol{x}_2] & \\dots & \\dots & \\mathrm{cov}[\\boldsymbol{x}_2,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
||||
"cov[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_0] & cov[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_1] & cov[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_{2}] & \\dots & \\dots & var[\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\mathrm{cov}[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_0] & \\mathrm{cov}[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_1] & \\mathrm{cov}[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_{2}] & \\dots & \\dots & \\mathrm{var}[\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\end{bmatrix},\n",
|
||||
"$$"
|
||||
]
|
||||
@@ -614,12 +616,12 @@
|
||||
"source": [
|
||||
"$$\n",
|
||||
"\\boldsymbol{K}[\\boldsymbol{x}] = \\begin{bmatrix}\n",
|
||||
"1 & corr[\\boldsymbol{x}_0,\\boldsymbol{x}_1] & corr[\\boldsymbol{x}_0,\\boldsymbol{x}_2] & \\dots & \\dots & corr[\\boldsymbol{x}_0,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"corr[\\boldsymbol{x}_1,\\boldsymbol{x}_0] & 1 & corr[\\boldsymbol{x}_1,\\boldsymbol{x}_2] & \\dots & \\dots & corr[\\boldsymbol{x}_1,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"corr[\\boldsymbol{x}_2,\\boldsymbol{x}_0] & corr[\\boldsymbol{x}_2,\\boldsymbol{x}_1] & 1 & \\dots & \\dots & corr[\\boldsymbol{x}_2,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"1 & \\mathrm{corr}[\\boldsymbol{x}_0,\\boldsymbol{x}_1] & \\mathrm{corr}[\\boldsymbol{x}_0,\\boldsymbol{x}_2] & \\dots & \\dots & \\mathrm{corr}[\\boldsymbol{x}_0,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\mathrm{corr}[\\boldsymbol{x}_1,\\boldsymbol{x}_0] & 1 & \\mathrm{corr}[\\boldsymbol{x}_1,\\boldsymbol{x}_2] & \\dots & \\dots & \\mathrm{corr}[\\boldsymbol{x}_1,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\mathrm{corr}[\\boldsymbol{x}_2,\\boldsymbol{x}_0] & \\mathrm{corr}[\\boldsymbol{x}_2,\\boldsymbol{x}_1] & 1 & \\dots & \\dots & \\mathrm{corr}[\\boldsymbol{x}_2,\\boldsymbol{x}_{p-1}]\\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
||||
"\\dots & \\dots & \\dots & \\dots & \\dots & \\dots \\\\\n",
|
||||
"corr[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_0] & corr[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_1] & corr[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_{2}] & \\dots & \\dots & 1\\\\\n",
|
||||
"\\mathrm{corr}[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_0] & \\mathrm{corr}[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_1] & \\mathrm{corr}[\\boldsymbol{x}_{p-1},\\boldsymbol{x}_{2}] & \\dots & \\dots & 1\\\\\n",
|
||||
"\\end{bmatrix},\n",
|
||||
"$$"
|
||||
]
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -332,40 +332,42 @@ applications.
|
||||
===== Basic ideas of the Principal Component Analysis (PCA) =====
|
||||
|
||||
We have a data set defined by a design/feature matrix $\bm{X}$ (see below for its definition)
|
||||
* So each data point is determined by $p$ extrinsic (measurement) variables
|
||||
* Each data point is determined by $p$ extrinsic (measurement) variables
|
||||
* We may want to ask the following question: Are there fewer intrinsic variables (say $d << p$) that still approximately describe the data?
|
||||
* If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do
|
||||
* If so, these intrinsic variables may tell us something important and finding these intrinsic variables is what dimension reduction methods do.
|
||||
|
||||
|
||||
!split
|
||||
===== Introducing the Covariance and Correlation functions =====
|
||||
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about the definition of the covariance and the correlation function.
|
||||
|
||||
Suppose we have defined two vectors
|
||||
$\hat{x}$ and $\hat{y}$ with $n$ elements each. The covariance matrix $\bm{C}$ is defined as
|
||||
!bt
|
||||
\[
|
||||
\bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} cov[\bm{x},\bm{x}] & cov[\bm{x},\bm{y}] \\
|
||||
cov[\bm{y},\bm{x}] & cov[\bm{y},\bm{y}] \\
|
||||
\bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} \mathrm{cov}[\bm{x},\bm{x}] & \mathrm{cov}[\bm{x},\bm{y}] \\
|
||||
\mathrm{cov}[\bm{y},\bm{x}] & \mathrm{cov}[\bm{y},\bm{y}] \\
|
||||
\end{bmatrix},
|
||||
\]
|
||||
!et
|
||||
where for example
|
||||
!bt
|
||||
\[
|
||||
cov[\bm{x},\bm{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
\mathrm{cov}[\bm{x},\bm{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
|
||||
\]
|
||||
!et
|
||||
With this definition and recalling that the variance is
|
||||
With this definition and recalling that the variance is defined as
|
||||
!bt
|
||||
\[
|
||||
var[\bm{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
\mathrm{var}[\bm{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
|
||||
\]
|
||||
!et
|
||||
we can rewrite the covariance matrix in this case as
|
||||
we can rewrite the covariance matrix as
|
||||
!bt
|
||||
\[
|
||||
\bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} var[\bm{x}] & cov[\bm{x},\bm{y}] \\
|
||||
cov[\bm{x},\bm{y}] & var[\bm{y}] \\
|
||||
\bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} \mathrm{var}[\bm{x}] & \mathrm{cov}[\bm{x},\bm{y}] \\
|
||||
\mathrm{cov}[\bm{x},\bm{y}] & \mathrm{var}[\bm{y}] \\
|
||||
\end{bmatrix},
|
||||
\]
|
||||
!et
|
||||
@@ -378,19 +380,19 @@ correlation function
|
||||
|
||||
!bt
|
||||
\[
|
||||
corr[\bm{x},\bm{y}]=\frac{cov[\bm{x},\bm{y}]}{\sqrt{var[\bm{x}]\var[\bm{y}]}}.
|
||||
\mathrm{corr}[\bm{x},\bm{y}]=\frac{\mathrm{cov}[\bm{x},\bm{y}]}{\sqrt{\mathrm{var}[\bm{x}] \mathrm{var}[\bm{y}]}}.
|
||||
\]
|
||||
!et
|
||||
|
||||
The correlation function is then given by values $corr[\bm{x},\bm{y}]
|
||||
The correlation function is then given by values $\mathrm{corr}[\bm{x},\bm{y}]
|
||||
\in [-1,1]$. This avoids eventual problems with too large values. We
|
||||
can then define the correlation matrix for the two vectors $\bm{x}$
|
||||
and $\bm{y}$ as
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{K}[\bm{x},\bm{y}] = \begin{bmatrix} 1 & corr[\bm{x},\bm{y}] \\
|
||||
corr[\bm{y},\bm{x}] & 1 \\
|
||||
\bm{K}[\bm{x},\bm{y}] = \begin{bmatrix} 1 & \mathrm{corr}[\bm{x},\bm{y}] \\
|
||||
\mathrm{corr}[\bm{y},\bm{x}] & 1 \\
|
||||
\end{bmatrix},
|
||||
\]
|
||||
!et
|
||||
@@ -432,12 +434,12 @@ With these definitions, we can now rewrite our $2\times 2$ correaltion/covarianc
|
||||
!bt
|
||||
\[
|
||||
\bm{C}[\bm{x}] = \begin{bmatrix}
|
||||
var[\bm{x}_0] & cov[\bm{x}_0,\bm{x}_1] & cov[\bm{x}_0,\bm{x}_2] & \dots & \dots & cov[\bm{x}_0,\bm{x}_{p-1}]\\
|
||||
cov[\bm{x}_1,\bm{x}_0] & var[\bm{x}_1] & cov[\bm{x}_1,\bm{x}_2] & \dots & \dots & cov[\bm{x}_1,\bm{x}_{p-1}]\\
|
||||
cov[\bm{x}_2,\bm{x}_0] & cov[\bm{x}_2,\bm{x}_1] & var[\bm{x}_2] & \dots & \dots & cov[\bm{x}_2,\bm{x}_{p-1}]\\
|
||||
\mathrm{var}[\bm{x}_0] & \mathrm{cov}[\bm{x}_0,\bm{x}_1] & \mathrm{cov}[\bm{x}_0,\bm{x}_2] & \dots & \dots & \mathrm{cov}[\bm{x}_0,\bm{x}_{p-1}]\\
|
||||
\mathrm{cov}[\bm{x}_1,\bm{x}_0] & \mathrm{var}[\bm{x}_1] & \mathrm{cov}[\bm{x}_1,\bm{x}_2] & \dots & \dots & \mathrm{cov}[\bm{x}_1,\bm{x}_{p-1}]\\
|
||||
\mathrm{cov}[\bm{x}_2,\bm{x}_0] & \mathrm{cov}[\bm{x}_2,\bm{x}_1] & \mathrm{var}[\bm{x}_2] & \dots & \dots & \mathrm{cov}[\bm{x}_2,\bm{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
cov[\bm{x}_{p-1},\bm{x}_0] & cov[\bm{x}_{p-1},\bm{x}_1] & cov[\bm{x}_{p-1},\bm{x}_{2}] & \dots & \dots & var[\bm{x}_{p-1}]\\
|
||||
\mathrm{cov}[\bm{x}_{p-1},\bm{x}_0] & \mathrm{cov}[\bm{x}_{p-1},\bm{x}_1] & \mathrm{cov}[\bm{x}_{p-1},\bm{x}_{2}] & \dots & \dots & \mathrm{var}[\bm{x}_{p-1}]\\
|
||||
\end{bmatrix},
|
||||
\]
|
||||
!et
|
||||
@@ -445,12 +447,12 @@ and the correlation matrix
|
||||
!bt
|
||||
\[
|
||||
\bm{K}[\bm{x}] = \begin{bmatrix}
|
||||
1 & corr[\bm{x}_0,\bm{x}_1] & corr[\bm{x}_0,\bm{x}_2] & \dots & \dots & corr[\bm{x}_0,\bm{x}_{p-1}]\\
|
||||
corr[\bm{x}_1,\bm{x}_0] & 1 & corr[\bm{x}_1,\bm{x}_2] & \dots & \dots & corr[\bm{x}_1,\bm{x}_{p-1}]\\
|
||||
corr[\bm{x}_2,\bm{x}_0] & corr[\bm{x}_2,\bm{x}_1] & 1 & \dots & \dots & corr[\bm{x}_2,\bm{x}_{p-1}]\\
|
||||
1 & \mathrm{corr}[\bm{x}_0,\bm{x}_1] & \mathrm{corr}[\bm{x}_0,\bm{x}_2] & \dots & \dots & \mathrm{corr}[\bm{x}_0,\bm{x}_{p-1}]\\
|
||||
\mathrm{corr}[\bm{x}_1,\bm{x}_0] & 1 & \mathrm{corr}[\bm{x}_1,\bm{x}_2] & \dots & \dots & \mathrm{corr}[\bm{x}_1,\bm{x}_{p-1}]\\
|
||||
\mathrm{corr}[\bm{x}_2,\bm{x}_0] & \mathrm{corr}[\bm{x}_2,\bm{x}_1] & 1 & \dots & \dots & \mathrm{corr}[\bm{x}_2,\bm{x}_{p-1}]\\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
\dots & \dots & \dots & \dots & \dots & \dots \\
|
||||
corr[\bm{x}_{p-1},\bm{x}_0] & corr[\bm{x}_{p-1},\bm{x}_1] & corr[\bm{x}_{p-1},\bm{x}_{2}] & \dots & \dots & 1\\
|
||||
\mathrm{corr}[\bm{x}_{p-1},\bm{x}_0] & \mathrm{corr}[\bm{x}_{p-1},\bm{x}_1] & \mathrm{corr}[\bm{x}_{p-1},\bm{x}_{2}] & \dots & \dots & 1\\
|
||||
\end{bmatrix},
|
||||
\]
|
||||
!et
|
||||
|
||||
Reference in New Issue
Block a user