further update, added also
This commit is contained in:
mhjensen
@@ -113,17 +113,18 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
None,
|
||||
'___sec14'),
|
||||
('Towards the PCA theorem', 2, None, '___sec15'),
|
||||
('Classical PCA Theorem', 2, None, '___sec16'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec17'),
|
||||
('Getting started with PCA', 2, None, '___sec18'),
|
||||
('Principal Component Analysis', 2, None, '___sec19'),
|
||||
('PCA and scikit-learn', 2, None, '___sec20'),
|
||||
('More on the PCA', 2, None, '___sec21'),
|
||||
('Incremental PCA', 2, None, '___sec22'),
|
||||
('Randomized PCA', 2, None, '___sec23'),
|
||||
('Kernel PCA', 2, None, '___sec24'),
|
||||
('LLE', 2, None, '___sec25'),
|
||||
('Other techniques', 2, None, '___sec26')]}
|
||||
('The Algorithm before the Theorem', 2, None, '___sec16'),
|
||||
('Classical PCA Theorem', 2, None, '___sec17'),
|
||||
('Prof of the PCA Theorem', 2, None, '___sec18'),
|
||||
('Getting started with PCA', 2, None, '___sec19'),
|
||||
('Principal Component Analysis', 2, None, '___sec20'),
|
||||
('PCA and scikit-learn', 2, None, '___sec21'),
|
||||
('More on the PCA', 2, None, '___sec22'),
|
||||
('Incremental PCA', 2, None, '___sec23'),
|
||||
('Randomized PCA', 2, None, '___sec24'),
|
||||
('Kernel PCA', 2, None, '___sec25'),
|
||||
('LLE', 2, None, '___sec26'),
|
||||
('Other techniques', 2, None, '___sec27')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -908,19 +909,66 @@ In the derivation of the PCA theorem we will assume that the eigenvalues are ord
|
||||
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec16">Classical PCA Theorem </h2>
|
||||
The eigenvalues tell us then how much we need to stretch the
|
||||
corresponding eigenvectors. Dimensions with large eigenvalues have
|
||||
thus large variations (large variance) and define therefore useful
|
||||
dimensions. The data points are more spread out in the direction of
|
||||
these eigenvectors. Smaller eigenvalues mean on the other hand that
|
||||
the corresponding eigenvectors are shrunk accordingly and the data
|
||||
points are tightly bunched together and there is not much variation in
|
||||
these specific directions. Hopefully then we could leave it out
|
||||
dimensions where the eigenvalues are very small. If \( p \) is very large,
|
||||
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
|
||||
features/predictors.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec17">Prof of the PCA Theorem </h2>
|
||||
<h2 id="___sec16">The Algorithm before theorem </h2>
|
||||
|
||||
<p>
|
||||
Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.
|
||||
|
||||
<ul>
|
||||
<li> Set up the datapoints for the design/feature matrix \( \boldsymbol{X} \) with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) referring to the column numbers and the entries \( n \) being the row elements.</li>
|
||||
</ul>
|
||||
|
||||
$$
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
|
||||
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
|
||||
\dots & \dots & \dots & \dots \dots & \dots \\
|
||||
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
|
||||
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
|
||||
\end{bmatrix},
|
||||
$$
|
||||
|
||||
|
||||
<ul>
|
||||
<li> Center the data by subtracting the mean value for each column. This leads to a new matrix \( \boldsymbol{X}\rightarrow \overline{\boldsymbol{X}} \).</li>
|
||||
<li> Compute then the covariance/correlation matrix $\mathbb{E}[\overline{\boldsymbol{X}}\overline{\boldsymbol{X}}^T].</li>
|
||||
<li> Find the eigenpairs of \( \boldsymbol{C} \) with eigenvalues \( [\lambda_0,\lambda_1,\dots,\lambda_{p-1}] \) and eigenvectors \( [\boldsymbol{s}_0,\boldsymbol{s}_1,\dots,\boldsymbol{s}_{p-1}] \).</li>
|
||||
<li> Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.</li>
|
||||
<li> Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.</li>
|
||||
</ul>
|
||||
|
||||
After this we ask ourselves how do we prove the link between the maximum variance and the feature reduction.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">Getting started with PCA </h2>
|
||||
<h2 id="___sec17">Classical PCA Theorem </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec18">Prof of the PCA Theorem </h2>
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec19">Getting started with PCA </h2>
|
||||
|
||||
<p>
|
||||
|
||||
@@ -935,7 +983,7 @@ X_pca <span style="color: #666666">=</span> pca<span style="color: #666666">.</s
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec19">Principal Component Analysis </h2>
|
||||
<h2 id="___sec20">Principal Component Analysis </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -971,7 +1019,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec20">PCA and scikit-learn </h2>
|
||||
<h2 id="___sec21">PCA and scikit-learn </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
|
||||
@@ -1002,7 +1050,7 @@ More material to come here.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec21">More on the PCA </h2>
|
||||
<h2 id="___sec22">More on the PCA </h2>
|
||||
|
||||
<p>
|
||||
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
|
||||
@@ -1032,7 +1080,7 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec22">Incremental PCA </h2>
|
||||
<h2 id="___sec23">Incremental PCA </h2>
|
||||
|
||||
<p>
|
||||
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
|
||||
@@ -1044,7 +1092,7 @@ instances arrive).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec23">Randomized PCA </h2>
|
||||
<h2 id="___sec24">Randomized PCA </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
|
||||
@@ -1059,7 +1107,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec24">Kernel PCA </h2>
|
||||
<h2 id="___sec25">Kernel PCA </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -1088,7 +1136,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">LLE </h2>
|
||||
<h2 id="___sec26">LLE </h2>
|
||||
|
||||
<p>
|
||||
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
|
||||
@@ -1100,7 +1148,7 @@ these local relationships are best preserved (more details shortly).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec26">Other techniques </h2>
|
||||
<h2 id="___sec27">Other techniques </h2>
|
||||
|
||||
<p>
|
||||
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
|
||||
|
||||
Reference in New Issue
Block a user