lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating

Browse Source
This commit is contained in:
mhjensen
2019-12-29 21:56:55 +01:00
parent c6fc54e6ff
commit c2f533862f
39 changed files with 1281 additions and 874 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DimRed/html/DimRed.html
+72 -29
View File
@@ -129,15 +129,20 @@ div { text-align: justify; text-justify: inter-word; }
('Proof of the PCA Theorem', 2, None, '___sec22'),
('PCA Proof continued', 2, None, '___sec23'),
('The final step', 2, None, '___sec24'),
('Principal Component Analysis', 2, None, '___sec25'),
('PCA and scikit-learn', 2, None, '___sec26'),
('Back to the Cancer Data', 2, None, '___sec27'),
('More on the PCA', 2, None, '___sec28'),
('Incremental PCA', 2, None, '___sec29'),
('Randomized PCA', 2, None, '___sec30'),
('Kernel PCA', 2, None, '___sec31'),
('LLE', 2, None, '___sec32'),
('Other techniques', 2, None, '___sec33')]}
('Geometric Interpretation and link with Singular Value '
'Decomposition',
2,
None,
'___sec25'),
('Principal Component Analysis', 2, None, '___sec26'),
('PCA and scikit-learn', 2, None, '___sec27'),
('Back to the Cancer Data', 2, None, '___sec28'),
('More on the PCA', 2, None, '___sec29'),
('Incremental PCA', 2, None, '___sec30'),
('Randomized PCA', 2, None, '___sec31'),
('Kernel PCA', 2, None, '___sec32'),
('LLE', 2, None, '___sec33'),
('Other techniques', 2, None, '___sec34')]}
end of tocinfo -->
<body>
@@ -179,7 +184,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Dec 28, 2019</h4></center> <!-- date -->
<center><h4>Dec 29, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -546,6 +551,15 @@ applications.
<h2 id="___sec7">Basic ideas of the Principal Component Analysis (PCA) </h2>
<p>
The principal component analysis deals with the problem of fitting a
low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
the totaldimension \( D \) of the problem at hand (our data
set). Mathematically it can be formulated as a statistical problem or
a geometric problem. In our discussion of the theorem for the
classical PCA, we will stay with a statistical approach. This is also
what set the scene historically which for the PCA.
<p>
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
@@ -561,7 +575,7 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
<p>
Before we discuss the PCA theorem, we need to remind ourselves about
the definition of the covariance and the correlation function.
the definition of the covariance and the correlation function. These are quantities
<p>
Suppose we have defined two vectors
@@ -621,7 +635,9 @@ In the above example this is the function we constructed using <b>pandas</b>.
<h2 id="___sec9">Correlation Function and Design/Feature Matrix </h2>
<p>
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
we defined the design/feature matrix \( \boldsymbol{X} \) as
$$
\boldsymbol{X}=\begin{bmatrix}
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
@@ -646,7 +662,11 @@ $$
$$
<p>
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
With these definitions, we can now rewrite our \( 2\times 2 \)
correaltion/covariance matrix in terms of a moe general design/feature
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
$$
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
@@ -845,8 +865,8 @@ columns since all matrix elements in the design matrix were set to one
<p>
This means that the variance for these elements will be zero and will
cause problems when we set up the correlation matrix. We can simply
drop these elements as follows and then construct the correlation
matrix.
drop these elements and construct a correlation
matrix without these elements.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1006,7 +1026,7 @@ X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>r
Make thereafter a small Python code which plots the data. Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
<p>
Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.
Now we are going to implement the PCA algorithm. We will break it down into various substeps.
<h3 id="___sec18">Compute the sample mean and center the data </h3>
@@ -1076,8 +1096,11 @@ Finally, try out your own PCA function with other data sets.
<h2 id="___sec21">Classical PCA Theorem </h2>
<p>
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \)
each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
centered as discussed above. For the sake of simplicity we skip the
overline symbol. The matrix is defined in terms of the various column
vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension
\( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
<p>
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
@@ -1089,7 +1112,13 @@ with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
<p>
The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
The PCA theorem states that minimizing the above reconstruction error
corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which
diagonalizes the empirical covariance(correlation) matrix. The optimal
low-dimensional encoding of the data is then given by a set of vectors
\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the
orthogonal projection of the data onto the columns spanned by the
eigenvectors of the covariance(correlations matrix).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1150,7 +1179,10 @@ $$
$$
<p>
We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
We are almost there, we have obtained a relation between minimizing
the reconstruction error and the variance and the covariance
matrix. Minimizing the error is equivalent to maximizing the variance
of the projected data.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -1203,10 +1235,21 @@ discussion in chapter 12.2 of Murphy's text has also a nice link with
the Singular Value Decomposition theorem. For categorical data, see
chapter 12.4 and discussion therein.
<p>
Additional part of the proof for the other eigenvectors will be added by mid January 2020.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">Principal Component Analysis </h2>
<h2 id="___sec25">Geometric Interpretation and link with Singular Value Decomposition </h2>
<p>
This material will be added by mid January 2020.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Principal Component Analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1262,7 +1305,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
<p>
<!-- !split -->
<h2 id="___sec26">PCA and scikit-learn </h2>
<h2 id="___sec27">PCA and scikit-learn </h2>
<p>
Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -1294,7 +1337,7 @@ variance that lies along the axis of each principal component.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Back to the Cancer Data </h2>
<h2 id="___sec28">Back to the Cancer Data </h2>
We can now repeat the above but applied to real data, in this case our breast cancer data.
Here we compute performance scores on the training data using logistic regression.
<p>
@@ -1335,7 +1378,7 @@ We see that our training data after the PCA decomposition has a performance simi
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec28">More on the PCA </h2>
<h2 id="___sec29">More on the PCA </h2>
<p>
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -1365,7 +1408,7 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">Incremental PCA </h2>
<h2 id="___sec30">Incremental PCA </h2>
<p>
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -1377,7 +1420,7 @@ instances arrive).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">Randomized PCA </h2>
<h2 id="___sec31">Randomized PCA </h2>
<p>
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -1392,7 +1435,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec31">Kernel PCA </h2>
<h2 id="___sec32">Kernel PCA </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -1421,7 +1464,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec32">LLE </h2>
<h2 id="___sec33">LLE </h2>
<p>
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -1433,7 +1476,7 @@ these local relationships are best preserved (more details shortly).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec33">Other techniques </h2>
<h2 id="___sec34">Other techniques </h2>
<p>
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.