lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

more dim red update

Browse Source
This commit is contained in:
mhjensen
2019-10-24 06:06:29 +02:00
parent 5aeb822fd8
commit 6712184d36
36 changed files with 1386 additions and 936 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DimRed/html/DimRed.html
+98 -58
View File
@@ -116,15 +116,17 @@ div { text-align: justify; text-justify: inter-word; }
('The Algorithm before the Theorem', 2, None, '___sec16'),
('Classical PCA Theorem', 2, None, '___sec17'),
('Proof of the PCA Theorem', 2, None, '___sec18'),
('Getting started with PCA', 2, None, '___sec19'),
('Principal Component Analysis', 2, None, '___sec20'),
('PCA and scikit-learn', 2, None, '___sec21'),
('More on the PCA', 2, None, '___sec22'),
('Incremental PCA', 2, None, '___sec23'),
('Randomized PCA', 2, None, '___sec24'),
('Kernel PCA', 2, None, '___sec25'),
('LLE', 2, None, '___sec26'),
('Other techniques', 2, None, '___sec27')]}
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
('PCA and Scikit-Learn Functionality', 2, None, '___sec21'),
('Principal Component Analysis', 2, None, '___sec22'),
('PCA and scikit-learn', 2, None, '___sec23'),
('More on the PCA', 2, None, '___sec24'),
('Incremental PCA', 2, None, '___sec25'),
('Randomized PCA', 2, None, '___sec26'),
('Kernel PCA', 2, None, '___sec27'),
('LLE', 2, None, '___sec28'),
('Other techniques', 2, None, '___sec29')]}
end of tocinfo -->
<body>
@@ -166,7 +168,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 22, 2019</h4></center> <!-- date -->
<center><h4>Oct 24, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -971,10 +973,10 @@ J(\boldsymbol{W},\boldsymbol{Z}) = \frac{1}{p}\sum_i (\boldsymbol{x}_i - \overli
$$
with \( \overline{\boldsymbol{x}_i} = \boldsymbol{W}\boldsymbol{z}_i \), where \( \boldsymbol{z}_i \) is a row vector with dimension \( {\mathbb{R}}^{n} \) of the matrix
\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \).
\( \boldsymbol{Z}\in {\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
<p>
The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by they eigenvectors of the covariance(correlations matrix).
The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( l \) $\boldsymbol{z}_i$, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -982,7 +984,7 @@ The PCA theorem states that minimizing the above reconstruction error correspond
<h2 id="___sec18">Proof of the PCA Theorem </h2>
<p>
To show the PCA theorem let us start with the assumption that there is a vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
$$
J(\boldsymbol{w}_0,\boldsymbol{z}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i - z_{i0}\boldsymbol{w}_0)^2=\frac{1}{p}\sum_i (\boldsymbol{x}_^T\boldsymbol{x}_i - 2z_{i0}\boldsymbol{w}_0^T\boldsymbol{x}_i+z_{i0}^2\boldsymbol{w}_0^T\boldsymbol{w}_0),
$$
@@ -1002,7 +1004,80 @@ where the vectors on the rhs are known.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Getting started with PCA </h2>
<h2 id="___sec19">PCA Proof continued </h2>
<p>
We have now found the unknown parameters \( z_{i0} \). These correspond to the project coordinates and we can write
$$
J(\boldsymbol{w}_0)= \frac{1}{p}\sum_i (\boldsymbol{x}_i^T\boldsymbol{x}_i - z_{i0}^2)=\mathrm{const}-\frac{1}{p}\sum_i z_{i0}^2.
$$
<p>
We can show that the variance of the project coordinates defined by \( \boldsymbol{w}_0^T\boldsymbol{x}_i \) are given by
$$
\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2,
$$
since the expectation value of
$$
\mathbb{E}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \mathbb{E}[z_{i0}]= \boldsymbol{w}_0^T\mathbb{E}[\boldsymbol{x}_i]=0,
$$
where we have used the fact that our data are centered.
<p>
Recalling our definition of the covariance as
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}\boldsymbol{X}^T= \mathbb{E}[\boldsymbol{X}\boldsymbol{X}^T],
$$
we have thus that
$$
\mathrm{var}[\boldsymbol{w}_0^T\boldsymbol{x}_i] = \frac{1}{p}\sum_i z_{i0}^2=\boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0.
$$
<p>
We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">The final step </h2>
<p>
We could trivially maximize the variance of the projection (and
thereby minimize the error in the reconstruction function) by letting
the norm-2 of \( \boldsymbol{w}_0 \) go to infinity. However, this norm since we
want the matrix \( \boldsymbol{W} \) to be an orthogonal matrix, is constrained by
\( $\vert\vert \boldsymbol{w}_0 \vert\vert_2^2=1 \). Imposing this condition via a
Lagrange multiplier we can then in turn maximize
$$
J(\boldsymbol{w}_0)= \boldsymbol{w}_0^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0+\lambda_0(1-\boldsymbol{w}_0^T\boldsymbol{w}_0).
$$
Taking the derivative with respect to \( \boldsymbol{w}_0 \) we obtain
$$
\frac{\partial J(\boldsymbol{w}_0)}{\partial \boldsymbol{w}_0}= 2\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0-2\lambda_0\boldsymbol{w}_0=0,
$$
meaning that
$$
\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0\boldsymbol{w}_0.
$$
<b>The direction that maximizes the variance (or minimizes the construction error) is an eigenvector of the covariance matrix</b>! If we left multiply with \( \boldsymbol{w}_0^T \) we have the variance of the projected data is
$$
\boldsymbol{w}_^T\boldsymbol{C}[\boldsymbol{x}]\boldsymbol{w}_0=\lambda_0.
$$
If we want to maximize the variance (minimize the construction error) we simply pick the eigenvector of the covariance matrix with the largest eigenvalue. This establishes the link between the minimization of the reconstruction function \( J \) in terms of an orthogonal matrix and the maximization of the variance and thereby the covariance of our observations encoded in the design/feature matrix \( \boldsymbol{X} \). The proof for the other eigenvectors \( \boldsymbol{w}_1,\boldsymbol{w}_2,\dots \) cna be established by applying the above arguments and using the fact that basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/books/machine-learning-1" target="_blank">Murphy chapter 12.2</a>. The discussion in chapter 12.2 of Murphy's text has also a nice link with the Singular Value Decomposition theorem. For categorical data, see chapter 12.4 and discussion therein.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec21">PCA and Scikit-Learn Functionality </h2>
<p>
@@ -1017,7 +1092,7 @@ X_pca <span style="color: #666666">=</span> pca<span style="color: #666666">.</s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">Principal Component Analysis </h2>
<h2 id="___sec22">Principal Component Analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1026,7 +1101,7 @@ First it identifies the hyperplane that lies closest to the data, and then it pr
<p>
The following Python code uses NumPy&#8217;s <b>svd()</b> function to obtain all the principal components of the
training set, then extracts the first two principal components
training set, then extracts the first two principal components. First we center the data
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -1053,7 +1128,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
<p>
<!-- !split -->
<h2 id="___sec21">PCA and scikit-learn </h2>
<h2 id="___sec23">PCA and scikit-learn </h2>
<p>
Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1084,7 +1159,7 @@ More material to come here.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec22">More on the PCA </h2>
<h2 id="___sec24">More on the PCA </h2>
<p>
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1114,7 +1189,7 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Incremental PCA </h2>
<h2 id="___sec25">Incremental PCA </h2>
<p>
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1126,7 +1201,7 @@ instances arrive).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Randomized PCA </h2>
<h2 id="___sec26">Randomized PCA </h2>
<p>
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1141,7 +1216,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Kernel PCA </h2>
<h2 id="___sec27">Kernel PCA </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1170,7 +1245,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">LLE </h2>
<h2 id="___sec28">LLE </h2>
<p>
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1182,7 +1257,7 @@ these local relationships are best preserved (more details shortly).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Other techniques </h2>
<h2 id="___sec29">Other techniques </h2>
<p>
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
@@ -1217,41 +1292,6 @@ display(df)
display(df<span style="color: #666666">**2</span>)
</pre></div>
<p>
Thereafter we can select specific columns only and plot final results
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>df<span style="color: #666666">.</span>columns <span style="color: #666666">=</span> [<span style="color: #BA2121">&#39;First&#39;</span>, <span style="color: #BA2121">&#39;Second&#39;</span>, <span style="color: #BA2121">&#39;Third&#39;</span>, <span style="color: #BA2121">&#39;Fourth&#39;</span>, <span style="color: #BA2121">&#39;Fifth&#39;</span>]
df<span style="color: #666666">.</span>index <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">10</span>)
display(df)
<span style="color: #008000; font-weight: bold">print</span>(df[<span style="color: #BA2121">&#39;Second&#39;</span>]<span style="color: #666666">.</span>mean() )
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>info())
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>describe())
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">pylab</span> <span style="color: #008000; font-weight: bold">import</span> plt, mpl
plt<span style="color: #666666">.</span>style<span style="color: #666666">.</span>use(<span style="color: #BA2121">&#39;seaborn&#39;</span>)
mpl<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;font.family&#39;</span>] <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;serif&#39;</span>
df<span style="color: #666666">.</span>cumsum()<span style="color: #666666">.</span>plot(lw<span style="color: #666666">=2.0</span>, figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>))
plt<span style="color: #666666">.</span>show()
df<span style="color: #666666">.</span>plot<span style="color: #666666">.</span>bar(figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>), rot<span style="color: #666666">=15</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
We can produce a \( 4\times 4 \) matrix
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">16</span>)<span style="color: #666666">.</span>reshape((<span style="color: #666666">4</span>,<span style="color: #666666">4</span>))
<span style="color: #008000; font-weight: bold">print</span>(b)
df1 <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(b)
<span style="color: #008000; font-weight: bold">print</span>(df1)
</pre></div>
<p>
and many other operations.
<!-- ------------------- end of main content --------------- -->