lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added pca code

Browse Source
This commit is contained in:
mhjensen
2019-12-26 11:06:16 +01:00
parent f170a1a9d5
commit 54fc7da531
39 changed files with 2866 additions and 1735 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DimRed/html/DimRed.html
+135 -27
View File
@@ -114,19 +114,30 @@ div { text-align: justify; text-justify: inter-word; }
'___sec14'),
('Towards the PCA theorem', 2, None, '___sec15'),
('The Algorithm before the Theorem', 2, None, '___sec16'),
('Classical PCA Theorem', 2, None, '___sec17'),
('Proof of the PCA Theorem', 2, None, '___sec18'),
('PCA Proof continued', 2, None, '___sec19'),
('The final step', 2, None, '___sec20'),
('Principal Component Analysis', 2, None, '___sec21'),
('PCA and scikit-learn', 2, None, '___sec22'),
('Back to the Cancer Data', 2, None, '___sec23'),
('More on the PCA', 2, None, '___sec24'),
('Incremental PCA', 2, None, '___sec25'),
('Randomized PCA', 2, None, '___sec26'),
('Kernel PCA', 2, None, '___sec27'),
('LLE', 2, None, '___sec28'),
('Other techniques', 2, None, '___sec29')]}
('Writing our own PCA code', 2, None, '___sec17'),
('Compute the sample mean and center the data',
3,
None,
'___sec18'),
('Compute the sample covariance', 3, None, '___sec19'),
('Diagonalize the sample covariance matrix to obtain the '
'principal components',
3,
None,
'___sec20'),
('Classical PCA Theorem', 2, None, '___sec21'),
('Proof of the PCA Theorem', 2, None, '___sec22'),
('PCA Proof continued', 2, None, '___sec23'),
('The final step', 2, None, '___sec24'),
('Principal Component Analysis', 2, None, '___sec25'),
('PCA and scikit-learn', 2, None, '___sec26'),
('Back to the Cancer Data', 2, None, '___sec27'),
('More on the PCA', 2, None, '___sec28'),
('Incremental PCA', 2, None, '___sec29'),
('Randomized PCA', 2, None, '___sec30'),
('Kernel PCA', 2, None, '___sec31'),
('LLE', 2, None, '___sec32'),
('Other techniques', 2, None, '___sec33')]}
end of tocinfo -->
<body>
@@ -168,7 +179,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 25, 2019</h4></center> <!-- date -->
<center><h4>Dec 26, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -955,12 +966,109 @@ $$
<li> Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Writing our own PCA code </h2>
<p>
We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix:
$$
\mu = (-1,2) \qquad \Sigma = \begin{bmatrix} 4 & 2 \\
2 & 2
\end{bmatrix}
$$
<p>
We will generate \( N = 1000 \) points \( X = \{ x_1, \ldots, x_N \} \) from
this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \).
<p>
The following Python code aids in setting up the data
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>N <span style="color: #666666">=</span> <span style="color: #666666">1000</span>
mean <span style="color: #666666">=</span> (<span style="color: #666666">-1</span>, <span style="color: #666666">2</span>)
cov <span style="color: #666666">=</span> [[<span style="color: #666666">4</span>, <span style="color: #666666">2</span>], [<span style="color: #666666">2</span>, <span style="color: #666666">2</span>]]
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>multivariate_normal(mean, cov, N)
</pre></div>
<p>
Make a small Python code which plots the data.
<p>
Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.
<h3 id="___sec18">Compute the sample mean and center the data </h3>
<p>
The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall the sample mean is
$$
\mu_N = \frac{1}{N} \sum_{i=1}^N x_i
$$
and the mean-centered data \( \bar{X} = \{ \bar{x}_1, \ldots, \bar{x}_N \} \) takes the form
$$
\bar{x}_i = x_i - \mu_N
$$
When you are done with these steps, print out \( \mu_N \) to verify it is
close to \( \mu \) and plot your mean centered data to verify it is
centered at the origin! Compare your code with the functionality from <b>Scikit-Learn</b> discussed above.
<h3 id="___sec19">Compute the sample covariance </h3>
<p>
Now we are going to use the mean centered data to compute the sample covariance of the data. Recall it is given by:
$$
\begin{equation*}
\Sigma_N = \frac{1}{N-1} \sum_{i=1}^N \bar{x}_i^T \bar{x}_i = \frac{1}{N-1} \sum_{i=1}^N (x_i - \mu_N)^T (x_i - \mu_N)
\end{equation*}
$$
where the data points \( x_i \in \mathbb{R}^p \) (here in this example \( p = 2 \)) are column vectors and \( x^T \) is the transpose of \( x \).
Compare the computed covariance with the answer given above.
<h3 id="___sec20">Diagonalize the sample covariance matrix to obtain the principal components </h3>
<p>
Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix \( \Sigma_N \). We can use the
function <b>np.linalg.eig</b> to do so. It will return the eigenvalues and
eigenvectors of \( \Sigma_N \). Once you have these, carry out the
following tasks:
<ul>
<li> Compute the percentage of the total variance captured by the first principal component</li>
<li> Plot the mean centered data and lines along the first and second principal components</li>
<li> Project the mean centered data onto the first and second principal components, and plot the projected data. What do you observe?</li>
<li> Approximate the data as</li>
</ul>
$$
\begin{equation*}
x_i \approx \tilde{x}_i := \mu_N + \langle x_i, v_0 \rangle v_0
\end{equation*}
$$
where \( v_0 \) is the first principal component. What do you observe?
<p>
Finally, collect all these steps and write your own PCA function and
compare this with the functionality included in <b>Scikit-Learn</b>.
Have the input be the data and have the output be the principal components and their associated eigenvalues, sorted in descending order. Can you think of a way to make it more efficient than the algorithm outlined above?
<p>
Finally, try out your own PCA function with other data sets.
<p>
After this we ask ourselves how do we prove the link between the maximum variance and the feature reduction.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Classical PCA Theorem </h2>
<h2 id="___sec21">Classical PCA Theorem </h2>
<p>
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \)
@@ -981,7 +1089,7 @@ The PCA theorem states that minimizing the above reconstruction error correspond
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Proof of the PCA Theorem </h2>
<h2 id="___sec22">Proof of the PCA Theorem </h2>
<p>
To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{w}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as
@@ -1004,7 +1112,7 @@ where the vectors on the rhs are known.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">PCA Proof continued </h2>
<h2 id="___sec23">PCA Proof continued </h2>
<p>
We have now found the unknown parameters \( z_{i0} \). These correspond to the projected coordinates and we can write
@@ -1042,7 +1150,7 @@ We are almost there, we have obtained a relation between minimizing the reconstr
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">The final step </h2>
<h2 id="___sec24">The final step </h2>
<p>
We could trivially maximize the variance of the projection (and
@@ -1093,7 +1201,7 @@ chapter 12.4 and discussion therein.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec21">Principal Component Analysis </h2>
<h2 id="___sec25">Principal Component Analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1149,7 +1257,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
<p>
<!-- !split -->
<h2 id="___sec22">PCA and scikit-learn </h2>
<h2 id="___sec26">PCA and scikit-learn </h2>
<p>
Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1181,7 +1289,7 @@ variance that lies along the axis of each principal component.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Back to the Cancer Data </h2>
<h2 id="___sec27">Back to the Cancer Data </h2>
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
<p>
@@ -1222,7 +1330,7 @@ We see that our training data after the PCA decomposition has a performance simi
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">More on the PCA </h2>
<h2 id="___sec28">More on the PCA </h2>
<p>
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1252,7 +1360,7 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Incremental PCA </h2>
<h2 id="___sec29">Incremental PCA </h2>
<p>
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1264,7 +1372,7 @@ instances arrive).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Randomized PCA </h2>
<h2 id="___sec30">Randomized PCA </h2>
<p>
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1279,7 +1387,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Kernel PCA </h2>
<h2 id="___sec31">Kernel PCA </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1308,7 +1416,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">LLE </h2>
<h2 id="___sec32">LLE </h2>
<p>
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1320,7 +1428,7 @@ these local relationships are best preserved (more details shortly).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">Other techniques </h2>
<h2 id="___sec33">Other techniques </h2>
<p>
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.