lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

small update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-10-29 06:22:31 +02:00
parent 2714dd2a7a
commit 5c0e2988af
8 changed files with 818 additions and 275 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/schedule.md
+1
View File
@@ -134,6 +134,7 @@ For the reading assignments we use the following abbreviations:
### Week 43 October 25-29
- Lab Wednesday: Work on project 2
- Lecture Thursday: Recurrent Neural Networks
- Video of Lecture at https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureOctober28.mp4?vrtx=view-as-webpage
- Lecture Friday: Recurrent Neural Networks and principal component analysis (PCA)
- Reading recommendations:
- See lecture notes for week 43 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
doc/pub/week43/html/week43-bs.html
+137 -18
View File
@@ -97,19 +97,48 @@ doconce format html week43.do.txt --html_style=bootstrap --pygments_html_style=d
2,
None,
'introducing-the-covariance-and-correlation-functions'),
('Correlation Matrix', 2, None, 'correlation-matrix'),
('More on the covariance', 2, None, 'more-on-the-covariance'),
('Reminding ourselves about Linear Regression',
2,
None,
'reminding-ourselves-about-linear-regression'),
('Simple Example', 2, None, 'simple-example'),
('The Correlation Matrix', 2, None, 'the-correlation-matrix'),
('Numpy Functionality', 2, None, 'numpy-functionality'),
('Correlation Matrix again', 2, None, 'correlation-matrix-again'),
('Using Pandas', 2, None, 'using-pandas'),
('And then the Franke Function',
2,
None,
'and-then-the-franke-function'),
('Lnks with the Design Matrix',
2,
None,
'lnks-with-the-design-matrix'),
('Computing the Expectation Values',
2,
None,
'computing-the-expectation-values'),
('Towards the PCA theorem', 2, None, 'towards-the-pca-theorem'),
('More on the PCA Theorem', 2, None, 'more-on-the-pca-theorem'),
('The Algorithm before the Theorem',
3,
2,
None,
'the-algorithm-before-the-theorem'),
('Writing our own PCA code', 3, None, 'writing-our-own-pca-code'),
('Writing our own PCA code', 2, None, 'writing-our-own-pca-code'),
('Implementing it', 2, None, 'implementing-it'),
('First Step', 2, None, 'first-step'),
('Scaling', 2, None, 'scaling'),
('Centered Data', 2, None, 'centered-data'),
('Exploring', 2, None, 'exploring'),
('Diagonalize the sample covariance matrix to obtain the '
'principal components',
3,
2,
None,
'diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components'),
('Collecting all Steps', 2, None, 'collecting-all-steps'),
('Classical PCA Theorem', 2, None, 'classical-pca-theorem'),
('The PCA Theorem', 2, None, 'the-pca-theorem'),
('Geometric Interpretation and link with Singular Value '
'Decomposition',
2,
@@ -117,7 +146,7 @@ doconce format html week43.do.txt --html_style=bootstrap --pygments_html_style=d
'geometric-interpretation-and-link-with-singular-value-decomposition'),
('PCA and scikit-learn', 2, None, 'pca-and-scikit-learn'),
('Back to the Cancer Data', 2, None, 'back-to-the-cancer-data'),
('Incremental PCA', 3, None, 'incremental-pca'),
('Incremental PCA', 2, None, 'incremental-pca'),
('Randomized PCA', 3, None, 'randomized-pca'),
('Kernel PCA', 3, None, 'kernel-pca'),
('Other techniques', 2, None, 'other-techniques')]}
@@ -183,16 +212,33 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="#interpolating-between-mnist-digits" style="font-size: 80%;"><b>Interpolating Between MNIST Digits</b></a></li>
<!-- navigation toc: --> <li><a href="#basic-ideas-of-the-principal-component-analysis-pca" style="font-size: 80%;"><b>Basic ideas of the Principal Component Analysis (PCA)</b></a></li>
<!-- navigation toc: --> <li><a href="#introducing-the-covariance-and-correlation-functions" style="font-size: 80%;"><b>Introducing the Covariance and Correlation functions</b></a></li>
<!-- navigation toc: --> <li><a href="#correlation-matrix" style="font-size: 80%;"><b>Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="#more-on-the-covariance" style="font-size: 80%;"><b>More on the covariance</b></a></li>
<!-- navigation toc: --> <li><a href="#reminding-ourselves-about-linear-regression" style="font-size: 80%;"><b>Reminding ourselves about Linear Regression</b></a></li>
<!-- navigation toc: --> <li><a href="#simple-example" style="font-size: 80%;"><b>Simple Example</b></a></li>
<!-- navigation toc: --> <li><a href="#the-correlation-matrix" style="font-size: 80%;"><b>The Correlation Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="#numpy-functionality" style="font-size: 80%;"><b>Numpy Functionality</b></a></li>
<!-- navigation toc: --> <li><a href="#correlation-matrix-again" style="font-size: 80%;"><b>Correlation Matrix again</b></a></li>
<!-- navigation toc: --> <li><a href="#using-pandas" style="font-size: 80%;"><b>Using Pandas</b></a></li>
<!-- navigation toc: --> <li><a href="#and-then-the-franke-function" style="font-size: 80%;"><b>And then the Franke Function</b></a></li>
<!-- navigation toc: --> <li><a href="#lnks-with-the-design-matrix" style="font-size: 80%;"><b>Lnks with the Design Matrix</b></a></li>
<!-- navigation toc: --> <li><a href="#computing-the-expectation-values" style="font-size: 80%;"><b>Computing the Expectation Values</b></a></li>
<!-- navigation toc: --> <li><a href="#towards-the-pca-theorem" style="font-size: 80%;"><b>Towards the PCA theorem</b></a></li>
<!-- navigation toc: --> <li><a href="#the-algorithm-before-the-theorem" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;The Algorithm before the Theorem</a></li>
<!-- navigation toc: --> <li><a href="#writing-our-own-pca-code" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Writing our own PCA code</a></li>
<!-- navigation toc: --> <li><a href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Diagonalize the sample covariance matrix to obtain the principal components</a></li>
<!-- navigation toc: --> <li><a href="#more-on-the-pca-theorem" style="font-size: 80%;"><b>More on the PCA Theorem</b></a></li>
<!-- navigation toc: --> <li><a href="#the-algorithm-before-the-theorem" style="font-size: 80%;"><b>The Algorithm before the Theorem</b></a></li>
<!-- navigation toc: --> <li><a href="#writing-our-own-pca-code" style="font-size: 80%;"><b>Writing our own PCA code</b></a></li>
<!-- navigation toc: --> <li><a href="#implementing-it" style="font-size: 80%;"><b>Implementing it</b></a></li>
<!-- navigation toc: --> <li><a href="#first-step" style="font-size: 80%;"><b>First Step</b></a></li>
<!-- navigation toc: --> <li><a href="#scaling" style="font-size: 80%;"><b>Scaling</b></a></li>
<!-- navigation toc: --> <li><a href="#centered-data" style="font-size: 80%;"><b>Centered Data</b></a></li>
<!-- navigation toc: --> <li><a href="#exploring" style="font-size: 80%;"><b>Exploring</b></a></li>
<!-- navigation toc: --> <li><a href="#diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" style="font-size: 80%;"><b>Diagonalize the sample covariance matrix to obtain the principal components</b></a></li>
<!-- navigation toc: --> <li><a href="#collecting-all-steps" style="font-size: 80%;"><b>Collecting all Steps</b></a></li>
<!-- navigation toc: --> <li><a href="#classical-pca-theorem" style="font-size: 80%;"><b>Classical PCA Theorem</b></a></li>
<!-- navigation toc: --> <li><a href="#the-pca-theorem" style="font-size: 80%;"><b>The PCA Theorem</b></a></li>
<!-- navigation toc: --> <li><a href="#geometric-interpretation-and-link-with-singular-value-decomposition" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
<!-- navigation toc: --> <li><a href="#pca-and-scikit-learn" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
<!-- navigation toc: --> <li><a href="#back-to-the-cancer-data" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
<!-- navigation toc: --> <li><a href="#incremental-pca" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="#incremental-pca" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
<!-- navigation toc: --> <li><a href="#randomized-pca" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="#kernel-pca" style="font-size: 80%;">&nbsp;&nbsp;&nbsp;Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="#other-techniques" style="font-size: 80%;"><b>Other techniques</b></a></li>
@@ -224,7 +270,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Oct 28, 2021</h4>
<h4>Oct 29, 2021</h4>
</center> <!-- date -->
<br>
@@ -2229,6 +2275,9 @@ $$
\end{bmatrix}.
$$
<!-- !split -->
<h2 id="more-on-the-covariance" class="anchor">More on the covariance </h2>
<p>The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
large values. It is common to scale the covariance matrix by
@@ -2254,6 +2303,8 @@ $$
<p>In the above example this is the function we constructed using <b>pandas</b>.</p>
<!-- !split -->
<h2 id="reminding-ourselves-about-linear-regression" class="anchor">Reminding ourselves about Linear Regression </h2>
<p>In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
we defined the design/feature matrix \( \boldsymbol{X} \) as
</p>
@@ -2282,8 +2333,11 @@ $$
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
$$
<!-- !split -->
<h2 id="simple-example" class="anchor">Simple Example </h2>
<p>With these definitions, we can now rewrite our \( 2\times 2 \)
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
</p>
@@ -2299,6 +2353,10 @@ $$
\end{bmatrix},
$$
<!-- !split -->
<h2 id="the-correlation-matrix" class="anchor">The Correlation Matrix </h2>
<p>and the correlation matrix</p>
$$
\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
@@ -2311,6 +2369,10 @@ $$
\end{bmatrix},
$$
<!-- !split -->
<h2 id="numpy-functionality" class="anchor">Numpy Functionality </h2>
<p>The Numpy function <b>np.cov</b> calculates the covariance elements using
the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have
the exact mean values. The following simple function uses the
@@ -2369,7 +2431,7 @@ C <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c
<!-- !split -->
<h2 id="correlation-matrix" class="anchor">Correlation Matrix </h2>
<h2 id="correlation-matrix-again" class="anchor">Correlation Matrix again </h2>
<p>The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
@@ -2428,6 +2490,9 @@ this matrix we easily see that it is a positive definite matrix.
<p>The above procedure with <b>numpy</b> can be made more compact if we use <b>pandas</b>.</p>
<!-- !split -->
<h2 id="using-pandas" class="anchor">Using Pandas </h2>
<p>We whow here how we can set up the correlation matrix using <b>pandas</b>, as done in this simple code</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -2464,6 +2529,10 @@ correlation_matrix <span style="color: #666666">=</span> Xpd<span style="color:
</div>
</div>
<!-- !split -->
<h2 id="and-then-the-franke-function" class="anchor">And then the Franke Function </h2>
<p>We expand this model to the Franke function discussed above.</p>
@@ -2536,9 +2605,12 @@ columns since all matrix elements in the design matrix were set to one
(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept
and wee can simply
drop these elements and construct a correlation
matrix without them.
matrix without them by centering our matrix elements by subtracting the mean of each column.
</p>
<!-- !split -->
<h2 id="lnks-with-the-design-matrix" class="anchor">Lnks with the Design Matrix </h2>
<p>We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as </p>
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}].
@@ -2554,6 +2626,10 @@ x_{10} & x_{11}\\
\end{bmatrix}.
$$
<!-- !split -->
<h2 id="computing-the-expectation-values" class="anchor">Computing the Expectation Values </h2>
<p>If we then compute the expectation value</p>
$$
\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix}
@@ -2603,6 +2679,10 @@ $$
\boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i.
$$
<!-- !split -->
<h2 id="more-on-the-pca-theorem" class="anchor">More on the PCA Theorem </h2>
<p>In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
</p>
@@ -2619,7 +2699,9 @@ dimensions where the eigenvalues are very small. If \( p \) is very large,
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
features/predictors.
</p>
<h3 id="the-algorithm-before-theorem" class="anchor">The Algorithm before theorem </h3>
<!-- !split -->
<h2 id="the-algorithm-before-theorem" class="anchor">The Algorithm before theorem </h2>
<p>Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. </p>
<ul>
@@ -2643,7 +2725,8 @@ $$
<li> Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.</li>
<li> Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.</li>
</ul>
<h3 id="writing-our-own-pca-code" class="anchor">Writing our own PCA code </h3>
<!-- !split -->
<h2 id="writing-our-own-pca-code" class="anchor">Writing our own PCA code </h2>
<p>We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below):
@@ -2659,6 +2742,8 @@ We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from
this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.
</p>
<!-- !split -->
<h2 id="implementing-it" class="anchor">Implementing it </h2>
<p>The following Python code aids in setting up the data and writing out the design matrix.
Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
</p>
@@ -2694,6 +2779,9 @@ X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>r
<p>Now we are going to implement the PCA algorithm. We will break it down into various substeps.</p>
<!-- !split -->
<h2 id="first-step" class="anchor">First Step </h2>
<p>The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is</p>
$$
\mu_n = \frac{1}{n} \sum_{i=1}^n x_i
@@ -2736,6 +2824,9 @@ X_centered <span style="color: #666666">=</span> X <span style="color: #666666">
</div>
</div>
<!-- !split -->
<h2 id="scaling" class="anchor">Scaling </h2>
<p>Alternatively, we could use the functions we discussed
earlier for scaling the data set. That is, we could have used the
<b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which ensures
@@ -2747,6 +2838,9 @@ while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our
specific case.
</p>
<!-- !split -->
<h2 id="centered-data" class="anchor">Centered Data </h2>
<p>Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation</p>
$$
\begin{equation*}
@@ -2819,10 +2913,16 @@ plt<span style="color: #666666">.</span>show()
</div>
</div>
<!-- !split -->
<h2 id="exploring" class="anchor">Exploring </h2>
<p>Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.
</p>
<h3 id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" class="anchor">Diagonalize the sample covariance matrix to obtain the principal components </h3>
<!-- !split -->
<h2 id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components" class="anchor">Diagonalize the sample covariance matrix to obtain the principal components </h2>
<p>Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix \( \Sigma \). We can use the
@@ -2845,6 +2945,9 @@ $$
<p>where \( v_0 \) is the first principal component. </p>
<!-- !split -->
<h2 id="collecting-all-steps" class="anchor">Collecting all Steps </h2>
<p>Collecting all these steps we can write our own PCA function and
compare this with the functionality included in <b>Scikit-Learn</b>.
</p>
@@ -2898,6 +3001,8 @@ X2Dsl <span style="color: #666666">=</span> pca<span style="color: #666666">.</s
</div>
<p>This code does not contain all the above elements, but it shows how we can use <b>Scikit-Learn</b> to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then? </p>
<!-- !split -->
<h2 id="classical-pca-theorem" class="anchor">Classical PCA Theorem </h2>
<p>We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
@@ -2916,6 +3021,9 @@ orthogonal projection of the data onto the columns spanned by the
eigenvectors of the covariance(correlations matrix).
</p>
<!-- !split -->
<h2 id="the-pca-theorem" class="anchor">The PCA Theorem </h2>
<p>To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as</p>
<p>We are almost there, we have obtained a relation between minimizing
@@ -2971,6 +3079,8 @@ chapter 12.4 and discussion therein.
</p>
<p>For more details, see for example <a href="https://www.springer.com/gp/book/9780387878102" target="_self">Vidal, Ma and Sastry, chapter 2</a>.</p>
<!-- !split -->
<h2 id="geometric-interpretation-and-link-with-singular-value-decomposition" class="anchor">Geometric Interpretation and link with Singular Value Decomposition </h2>
<p>For a detailed demonstration of the geometric interpretation, see <a href="https://www.springer.com/gp/book/9780387878102" target="_self">Vidal, Ma and Sastry, section 2.1.2</a>.</p>
@@ -3060,6 +3170,9 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
</div>
</div>
</div>
<!-- !split -->
<h2 id="pca-and-scikit-learn" class="anchor">PCA and scikit-learn </h2>
<p>Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -3242,7 +3355,10 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
</div>
</div>
</div>
<h3 id="incremental-pca" class="anchor">Incremental PCA </h3>
<!-- !split -->
<h2 id="incremental-pca" class="anchor">Incremental PCA </h2>
<p>One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
@@ -3293,6 +3409,9 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
</div>
</div>
</div>
<!-- !split -->
<h2 id="other-techniques" class="anchor">Other techniques </h2>
<p>There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.</p>
doc/pub/week43/html/week43-reveal.html
+95 -9
View File
@@ -184,7 +184,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Oct 28, 2021</h4>
<h4>Oct 29, 2021</h4>
</center> <!-- date -->
<br>
@@ -2228,7 +2228,10 @@ $$
\end{bmatrix}.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="more-on-the-covariance">More on the covariance </h2>
<p>The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
large values. It is common to scale the covariance matrix by
@@ -2257,7 +2260,10 @@ $$
<p>&nbsp;<br>
<p>In the above example this is the function we constructed using <b>pandas</b>.</p>
</section>
<section>
<h2 id="reminding-ourselves-about-linear-regression">Reminding ourselves about Linear Regression </h2>
<p>In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
we defined the design/feature matrix \( \boldsymbol{X} \) as
</p>
@@ -2291,9 +2297,12 @@ $$
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="simple-example">Simple Example </h2>
<p>With these definitions, we can now rewrite our \( 2\times 2 \)
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
</p>
@@ -2310,6 +2319,10 @@ $$
\end{bmatrix},
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="the-correlation-matrix">The Correlation Matrix </h2>
<p>and the correlation matrix</p>
<p>&nbsp;<br>
@@ -2324,6 +2337,10 @@ $$
\end{bmatrix},
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="numpy-functionality">Numpy Functionality </h2>
<p>The Numpy function <b>np.cov</b> calculates the covariance elements using
the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have
@@ -2385,7 +2402,7 @@ C = np.cov(W)
</section>
<section>
<h2 id="correlation-matrix">Correlation Matrix </h2>
<h2 id="correlation-matrix-again">Correlation Matrix again </h2>
<p>The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
@@ -2443,6 +2460,10 @@ this matrix we easily see that it is a positive definite matrix.
</p>
<p>The above procedure with <b>numpy</b> can be made more compact if we use <b>pandas</b>.</p>
</section>
<section>
<h2 id="using-pandas">Using Pandas </h2>
<p>We whow here how we can set up the correlation matrix using <b>pandas</b>, as done in this simple code</p>
@@ -2479,6 +2500,10 @@ correlation_matrix = Xpd.corr()
</div>
</div>
</div>
</section>
<section>
<h2 id="and-then-the-franke-function">And then the Franke Function </h2>
<p>We expand this model to the Franke function discussed above.</p>
@@ -2552,8 +2577,12 @@ columns since all matrix elements in the design matrix were set to one
(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept
and wee can simply
drop these elements and construct a correlation
matrix without them.
matrix without them by centering our matrix elements by subtracting the mean of each column.
</p>
</section>
<section>
<h2 id="lnks-with-the-design-matrix">Lnks with the Design Matrix </h2>
<p>We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as </p>
<p>&nbsp;<br>
@@ -2573,6 +2602,10 @@ x_{10} & x_{11}\\
\end{bmatrix}.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="computing-the-expectation-values">Computing the Expectation Values </h2>
<p>If we then compute the expectation value</p>
<p>&nbsp;<br>
@@ -2637,6 +2670,10 @@ $$
\boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i.
$$
<p>&nbsp;<br>
</section>
<section>
<h2 id="more-on-the-pca-theorem">More on the PCA Theorem </h2>
<p>In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
@@ -2654,7 +2691,10 @@ dimensions where the eigenvalues are very small. If \( p \) is very large,
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
features/predictors.
</p>
<h3 id="the-algorithm-before-theorem">The Algorithm before theorem </h3>
</section>
<section>
<h2 id="the-algorithm-before-theorem">The Algorithm before theorem </h2>
<p>Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. </p>
<ul>
@@ -2681,8 +2721,10 @@ $$
<p><li> Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.</li>
<p><li> Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.</li>
</ul>
<p>
<h3 id="writing-our-own-pca-code">Writing our own PCA code </h3>
</section>
<section>
<h2 id="writing-our-own-pca-code">Writing our own PCA code </h2>
<p>We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below):
@@ -2699,7 +2741,10 @@ $$
We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from
this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.
</p>
</section>
<section>
<h2 id="implementing-it">Implementing it </h2>
<p>The following Python code aids in setting up the data and writing out the design matrix.
Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
</p>
@@ -2734,6 +2779,10 @@ X = np.random.multivariate_normal(mean, cov, n)
</div>
<p>Now we are going to implement the PCA algorithm. We will break it down into various substeps.</p>
</section>
<section>
<h2 id="first-step">First Step </h2>
<p>The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is</p>
<p>&nbsp;<br>
@@ -2780,7 +2829,10 @@ X_centered = X - X.mean(axis=<span style="color: #B452CD">0</span>)
</div>
</div>
</div>
</section>
<section>
<h2 id="scaling">Scaling </h2>
<p>Alternatively, we could use the functions we discussed
earlier for scaling the data set. That is, we could have used the
<b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which ensures
@@ -2791,6 +2843,10 @@ variance. The diagonal covariance matrix elements will then be one,
while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our
specific case.
</p>
</section>
<section>
<h2 id="centered-data">Centered Data </h2>
<p>Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation</p>
<p>&nbsp;<br>
@@ -2865,11 +2921,18 @@ plt.show()
</div>
</div>
</div>
</section>
<section>
<h2 id="exploring">Exploring </h2>
<p>Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.
</p>
<h3 id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">Diagonalize the sample covariance matrix to obtain the principal components </h3>
</section>
<section>
<h2 id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">Diagonalize the sample covariance matrix to obtain the principal components </h2>
<p>Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix \( \Sigma \). We can use the
@@ -2894,6 +2957,10 @@ $$
<p>&nbsp;<br>
<p>where \( v_0 \) is the first principal component. </p>
</section>
<section>
<h2 id="collecting-all-steps">Collecting all Steps </h2>
<p>Collecting all these steps we can write our own PCA function and
compare this with the functionality included in <b>Scikit-Learn</b>.
@@ -2948,6 +3015,9 @@ X2Dsl = pca.fit_transform(X)
</div>
<p>This code does not contain all the above elements, but it shows how we can use <b>Scikit-Learn</b> to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then? </p>
</section>
<section>
<h2 id="classical-pca-theorem">Classical PCA Theorem </h2>
<p>We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
@@ -2965,6 +3035,10 @@ low-dimensional encoding of the data is then given by a set of vectors
orthogonal projection of the data onto the columns spanned by the
eigenvectors of the covariance(correlations matrix).
</p>
</section>
<section>
<h2 id="the-pca-theorem">The PCA Theorem </h2>
<p>To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as</p>
@@ -3029,6 +3103,9 @@ chapter 12.4 and discussion therein.
</p>
<p>For more details, see for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, chapter 2</a>.</p>
</section>
<section>
<h2 id="geometric-interpretation-and-link-with-singular-value-decomposition">Geometric Interpretation and link with Singular Value Decomposition </h2>
<p>For a detailed demonstration of the geometric interpretation, see <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, section 2.1.2</a>.</p>
@@ -3118,6 +3195,9 @@ X2D = X_centered.dot(W2)
</div>
</div>
</div>
</section>
<section>
<h2 id="pca-and-scikit-learn">PCA and scikit-learn </h2>
<p>Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -3301,7 +3381,10 @@ X_reduced = pca.fit_transform(X)
</div>
</div>
</div>
<h3 id="incremental-pca">Incremental PCA </h3>
</section>
<section>
<h2 id="incremental-pca">Incremental PCA </h2>
<p>One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
@@ -3352,6 +3435,9 @@ X_reduced = rbf_pca.fit_transform(X)
</div>
</div>
</div>
</section>
<section>
<h2 id="other-techniques">Other techniques </h2>
<p>There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.</p>
doc/pub/week43/html/week43-solarized.html
+115 -13
View File
@@ -124,19 +124,48 @@ div.toc p,a {
2,
None,
'introducing-the-covariance-and-correlation-functions'),
('Correlation Matrix', 2, None, 'correlation-matrix'),
('More on the covariance', 2, None, 'more-on-the-covariance'),
('Reminding ourselves about Linear Regression',
2,
None,
'reminding-ourselves-about-linear-regression'),
('Simple Example', 2, None, 'simple-example'),
('The Correlation Matrix', 2, None, 'the-correlation-matrix'),
('Numpy Functionality', 2, None, 'numpy-functionality'),
('Correlation Matrix again', 2, None, 'correlation-matrix-again'),
('Using Pandas', 2, None, 'using-pandas'),
('And then the Franke Function',
2,
None,
'and-then-the-franke-function'),
('Lnks with the Design Matrix',
2,
None,
'lnks-with-the-design-matrix'),
('Computing the Expectation Values',
2,
None,
'computing-the-expectation-values'),
('Towards the PCA theorem', 2, None, 'towards-the-pca-theorem'),
('More on the PCA Theorem', 2, None, 'more-on-the-pca-theorem'),
('The Algorithm before the Theorem',
3,
2,
None,
'the-algorithm-before-the-theorem'),
('Writing our own PCA code', 3, None, 'writing-our-own-pca-code'),
('Writing our own PCA code', 2, None, 'writing-our-own-pca-code'),
('Implementing it', 2, None, 'implementing-it'),
('First Step', 2, None, 'first-step'),
('Scaling', 2, None, 'scaling'),
('Centered Data', 2, None, 'centered-data'),
('Exploring', 2, None, 'exploring'),
('Diagonalize the sample covariance matrix to obtain the '
'principal components',
3,
2,
None,
'diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components'),
('Collecting all Steps', 2, None, 'collecting-all-steps'),
('Classical PCA Theorem', 2, None, 'classical-pca-theorem'),
('The PCA Theorem', 2, None, 'the-pca-theorem'),
('Geometric Interpretation and link with Singular Value '
'Decomposition',
2,
@@ -144,7 +173,7 @@ div.toc p,a {
'geometric-interpretation-and-link-with-singular-value-decomposition'),
('PCA and scikit-learn', 2, None, 'pca-and-scikit-learn'),
('Back to the Cancer Data', 2, None, 'back-to-the-cancer-data'),
('Incremental PCA', 3, None, 'incremental-pca'),
('Incremental PCA', 2, None, 'incremental-pca'),
('Randomized PCA', 3, None, 'randomized-pca'),
('Kernel PCA', 3, None, 'kernel-pca'),
('Other techniques', 2, None, 'other-techniques')]}
@@ -185,7 +214,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Oct 28, 2021</h4>
<h4>Oct 29, 2021</h4>
</center> <!-- date -->
<br>
@@ -2185,6 +2214,9 @@ $$
\end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-the-covariance">More on the covariance </h2>
<p>The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
large values. It is common to scale the covariance matrix by
@@ -2210,6 +2242,8 @@ $$
<p>In the above example this is the function we constructed using <b>pandas</b>.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="reminding-ourselves-about-linear-regression">Reminding ourselves about Linear Regression </h2>
<p>In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
we defined the design/feature matrix \( \boldsymbol{X} \) as
</p>
@@ -2238,8 +2272,11 @@ $$
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="simple-example">Simple Example </h2>
<p>With these definitions, we can now rewrite our \( 2\times 2 \)
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
</p>
@@ -2255,6 +2292,10 @@ $$
\end{bmatrix},
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-correlation-matrix">The Correlation Matrix </h2>
<p>and the correlation matrix</p>
$$
\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
@@ -2267,6 +2308,10 @@ $$
\end{bmatrix},
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="numpy-functionality">Numpy Functionality </h2>
<p>The Numpy function <b>np.cov</b> calculates the covariance elements using
the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have
the exact mean values. The following simple function uses the
@@ -2325,7 +2370,7 @@ C = np.cov(W)
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="correlation-matrix">Correlation Matrix </h2>
<h2 id="correlation-matrix-again">Correlation Matrix again </h2>
<p>The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
@@ -2384,6 +2429,9 @@ this matrix we easily see that it is a positive definite matrix.
<p>The above procedure with <b>numpy</b> can be made more compact if we use <b>pandas</b>.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="using-pandas">Using Pandas </h2>
<p>We whow here how we can set up the correlation matrix using <b>pandas</b>, as done in this simple code</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
@@ -2420,6 +2468,10 @@ correlation_matrix = Xpd.corr()
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="and-then-the-franke-function">And then the Franke Function </h2>
<p>We expand this model to the Franke function discussed above.</p>
@@ -2492,9 +2544,12 @@ columns since all matrix elements in the design matrix were set to one
(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept
and wee can simply
drop these elements and construct a correlation
matrix without them.
matrix without them by centering our matrix elements by subtracting the mean of each column.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="lnks-with-the-design-matrix">Lnks with the Design Matrix </h2>
<p>We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as </p>
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}].
@@ -2510,6 +2565,10 @@ x_{10} & x_{11}\\
\end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="computing-the-expectation-values">Computing the Expectation Values </h2>
<p>If we then compute the expectation value</p>
$$
\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix}
@@ -2559,6 +2618,10 @@ $$
\boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-the-pca-theorem">More on the PCA Theorem </h2>
<p>In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
</p>
@@ -2575,7 +2638,9 @@ dimensions where the eigenvalues are very small. If \( p \) is very large,
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
features/predictors.
</p>
<h3 id="the-algorithm-before-theorem">The Algorithm before theorem </h3>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-algorithm-before-theorem">The Algorithm before theorem </h2>
<p>Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. </p>
<ul>
@@ -2599,7 +2664,8 @@ $$
<li> Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.</li>
<li> Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.</li>
</ul>
<h3 id="writing-our-own-pca-code">Writing our own PCA code </h3>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="writing-our-own-pca-code">Writing our own PCA code </h2>
<p>We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below):
@@ -2615,6 +2681,8 @@ We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from
this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="implementing-it">Implementing it </h2>
<p>The following Python code aids in setting up the data and writing out the design matrix.
Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
</p>
@@ -2650,6 +2718,9 @@ X = np.random.multivariate_normal(mean, cov, n)
<p>Now we are going to implement the PCA algorithm. We will break it down into various substeps.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="first-step">First Step </h2>
<p>The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is</p>
$$
\mu_n = \frac{1}{n} \sum_{i=1}^n x_i
@@ -2692,6 +2763,9 @@ X_centered = X - X.mean(axis=<span style="color: #B452CD">0</span>)
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="scaling">Scaling </h2>
<p>Alternatively, we could use the functions we discussed
earlier for scaling the data set. That is, we could have used the
<b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which ensures
@@ -2703,6 +2777,9 @@ while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our
specific case.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="centered-data">Centered Data </h2>
<p>Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation</p>
$$
\begin{equation*}
@@ -2775,10 +2852,16 @@ plt.show()
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="exploring">Exploring </h2>
<p>Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.
</p>
<h3 id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">Diagonalize the sample covariance matrix to obtain the principal components </h3>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">Diagonalize the sample covariance matrix to obtain the principal components </h2>
<p>Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix \( \Sigma \). We can use the
@@ -2801,6 +2884,9 @@ $$
<p>where \( v_0 \) is the first principal component. </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="collecting-all-steps">Collecting all Steps </h2>
<p>Collecting all these steps we can write our own PCA function and
compare this with the functionality included in <b>Scikit-Learn</b>.
</p>
@@ -2854,6 +2940,8 @@ X2Dsl = pca.fit_transform(X)
</div>
<p>This code does not contain all the above elements, but it shows how we can use <b>Scikit-Learn</b> to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then? </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="classical-pca-theorem">Classical PCA Theorem </h2>
<p>We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
@@ -2872,6 +2960,9 @@ orthogonal projection of the data onto the columns spanned by the
eigenvectors of the covariance(correlations matrix).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-pca-theorem">The PCA Theorem </h2>
<p>To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as</p>
<p>We are almost there, we have obtained a relation between minimizing
@@ -2927,6 +3018,8 @@ chapter 12.4 and discussion therein.
</p>
<p>For more details, see for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, chapter 2</a>.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="geometric-interpretation-and-link-with-singular-value-decomposition">Geometric Interpretation and link with Singular Value Decomposition </h2>
<p>For a detailed demonstration of the geometric interpretation, see <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, section 2.1.2</a>.</p>
@@ -3016,6 +3109,9 @@ X2D = X_centered.dot(W2)
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="pca-and-scikit-learn">PCA and scikit-learn </h2>
<p>Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -3198,7 +3294,10 @@ X_reduced = pca.fit_transform(X)
</div>
</div>
</div>
<h3 id="incremental-pca">Incremental PCA </h3>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="incremental-pca">Incremental PCA </h2>
<p>One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
@@ -3249,6 +3348,9 @@ X_reduced = rbf_pca.fit_transform(X)
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="other-techniques">Other techniques </h2>
<p>There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.</p>
doc/pub/week43/html/week43.html
+115 -13
View File
@@ -201,19 +201,48 @@ div.toc p,a {
2,
None,
'introducing-the-covariance-and-correlation-functions'),
('Correlation Matrix', 2, None, 'correlation-matrix'),
('More on the covariance', 2, None, 'more-on-the-covariance'),
('Reminding ourselves about Linear Regression',
2,
None,
'reminding-ourselves-about-linear-regression'),
('Simple Example', 2, None, 'simple-example'),
('The Correlation Matrix', 2, None, 'the-correlation-matrix'),
('Numpy Functionality', 2, None, 'numpy-functionality'),
('Correlation Matrix again', 2, None, 'correlation-matrix-again'),
('Using Pandas', 2, None, 'using-pandas'),
('And then the Franke Function',
2,
None,
'and-then-the-franke-function'),
('Lnks with the Design Matrix',
2,
None,
'lnks-with-the-design-matrix'),
('Computing the Expectation Values',
2,
None,
'computing-the-expectation-values'),
('Towards the PCA theorem', 2, None, 'towards-the-pca-theorem'),
('More on the PCA Theorem', 2, None, 'more-on-the-pca-theorem'),
('The Algorithm before the Theorem',
3,
2,
None,
'the-algorithm-before-the-theorem'),
('Writing our own PCA code', 3, None, 'writing-our-own-pca-code'),
('Writing our own PCA code', 2, None, 'writing-our-own-pca-code'),
('Implementing it', 2, None, 'implementing-it'),
('First Step', 2, None, 'first-step'),
('Scaling', 2, None, 'scaling'),
('Centered Data', 2, None, 'centered-data'),
('Exploring', 2, None, 'exploring'),
('Diagonalize the sample covariance matrix to obtain the '
'principal components',
3,
2,
None,
'diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components'),
('Collecting all Steps', 2, None, 'collecting-all-steps'),
('Classical PCA Theorem', 2, None, 'classical-pca-theorem'),
('The PCA Theorem', 2, None, 'the-pca-theorem'),
('Geometric Interpretation and link with Singular Value '
'Decomposition',
2,
@@ -221,7 +250,7 @@ div.toc p,a {
'geometric-interpretation-and-link-with-singular-value-decomposition'),
('PCA and scikit-learn', 2, None, 'pca-and-scikit-learn'),
('Back to the Cancer Data', 2, None, 'back-to-the-cancer-data'),
('Incremental PCA', 3, None, 'incremental-pca'),
('Incremental PCA', 2, None, 'incremental-pca'),
('Randomized PCA', 3, None, 'randomized-pca'),
('Kernel PCA', 3, None, 'kernel-pca'),
('Other techniques', 2, None, 'other-techniques')]}
@@ -262,7 +291,7 @@ MathJax.Hub.Config({
</center>
<br>
<center>
<h4>Oct 28, 2021</h4>
<h4>Oct 29, 2021</h4>
</center> <!-- date -->
<br>
@@ -2262,6 +2291,9 @@ $$
\end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-the-covariance">More on the covariance </h2>
<p>The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
large values. It is common to scale the covariance matrix by
@@ -2287,6 +2319,8 @@ $$
<p>In the above example this is the function we constructed using <b>pandas</b>.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="reminding-ourselves-about-linear-regression">Reminding ourselves about Linear Regression </h2>
<p>In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
we defined the design/feature matrix \( \boldsymbol{X} \) as
</p>
@@ -2315,8 +2349,11 @@ $$
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="simple-example">Simple Example </h2>
<p>With these definitions, we can now rewrite our \( 2\times 2 \)
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
</p>
@@ -2332,6 +2369,10 @@ $$
\end{bmatrix},
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-correlation-matrix">The Correlation Matrix </h2>
<p>and the correlation matrix</p>
$$
\boldsymbol{K}[\boldsymbol{x}] = \begin{bmatrix}
@@ -2344,6 +2385,10 @@ $$
\end{bmatrix},
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="numpy-functionality">Numpy Functionality </h2>
<p>The Numpy function <b>np.cov</b> calculates the covariance elements using
the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have
the exact mean values. The following simple function uses the
@@ -2402,7 +2447,7 @@ C <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="correlation-matrix">Correlation Matrix </h2>
<h2 id="correlation-matrix-again">Correlation Matrix again </h2>
<p>The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
@@ -2461,6 +2506,9 @@ this matrix we easily see that it is a positive definite matrix.
<p>The above procedure with <b>numpy</b> can be made more compact if we use <b>pandas</b>.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="using-pandas">Using Pandas </h2>
<p>We whow here how we can set up the correlation matrix using <b>pandas</b>, as done in this simple code</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
@@ -2497,6 +2545,10 @@ correlation_matrix <span style="color: #666666">=</span> Xpd<span style="color:
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="and-then-the-franke-function">And then the Franke Function </h2>
<p>We expand this model to the Franke function discussed above.</p>
@@ -2569,9 +2621,12 @@ columns since all matrix elements in the design matrix were set to one
(we are fitting the function in terms of a polynomial of degree \( n \)). We would however not include the intercept
and wee can simply
drop these elements and construct a correlation
matrix without them.
matrix without them by centering our matrix elements by subtracting the mean of each column.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="lnks-with-the-design-matrix">Lnks with the Design Matrix </h2>
<p>We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix \( \boldsymbol{X} \) as </p>
$$
\boldsymbol{C}[\boldsymbol{x}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}= \mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}].
@@ -2587,6 +2642,10 @@ x_{10} & x_{11}\\
\end{bmatrix}.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="computing-the-expectation-values">Computing the Expectation Values </h2>
<p>If we then compute the expectation value</p>
$$
\mathbb{E}[\boldsymbol{X}^T\boldsymbol{X}] = \frac{1}{n}\boldsymbol{X}^T\boldsymbol{X}=\begin{bmatrix}
@@ -2636,6 +2695,10 @@ $$
\boldsymbol{S}_i\lambda_i = \boldsymbol{C}[\boldsymbol{x}]\boldsymbol{S}_i.
$$
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="more-on-the-pca-theorem">More on the PCA Theorem </h2>
<p>In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
\( \lambda_0 > \lambda_1 > \dots > \lambda_{p-1} \).
</p>
@@ -2652,7 +2715,9 @@ dimensions where the eigenvalues are very small. If \( p \) is very large,
we could then aim at reducing \( p \) to \( l < < p \) and handle only \( l \)
features/predictors.
</p>
<h3 id="the-algorithm-before-theorem">The Algorithm before theorem </h3>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-algorithm-before-theorem">The Algorithm before theorem </h2>
<p>Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here. </p>
<ul>
@@ -2676,7 +2741,8 @@ $$
<li> Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.</li>
<li> Keep only those \( l \) eigenvalues larger than a selected threshold value, discarding thus \( p-l \) features since we expect small variations in the data here.</li>
</ul>
<h3 id="writing-our-own-pca-code">Writing our own PCA code </h3>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="writing-our-own-pca-code">Writing our own PCA code </h2>
<p>We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below):
@@ -2692,6 +2758,8 @@ We will generate \( n = 10000 \) points \( X = \{ x_1, \ldots, x_N \} \) from
this distribution, and store them in the \( 1000 \times 2 \) matrix \( \boldsymbol{X} \). This is our design matrix where we have forced the covariance and mean values to take specific values.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="implementing-it">Implementing it </h2>
<p>The following Python code aids in setting up the data and writing out the design matrix.
Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
</p>
@@ -2727,6 +2795,9 @@ X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>r
<p>Now we are going to implement the PCA algorithm. We will break it down into various substeps.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="first-step">First Step </h2>
<p>The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is</p>
$$
\mu_n = \frac{1}{n} \sum_{i=1}^n x_i
@@ -2769,6 +2840,9 @@ X_centered <span style="color: #666666">=</span> X <span style="color: #666666">
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="scaling">Scaling </h2>
<p>Alternatively, we could use the functions we discussed
earlier for scaling the data set. That is, we could have used the
<b>StandardScaler</b> function in <b>Scikit-Learn</b>, a function which ensures
@@ -2780,6 +2854,9 @@ while the non-diagonal ones need to be divided by \( 2\sqrt{2} \) for our
specific case.
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="centered-data">Centered Data </h2>
<p>Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation</p>
$$
\begin{equation*}
@@ -2852,10 +2929,16 @@ plt<span style="color: #666666">.</span>show()
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="exploring">Exploring </h2>
<p>Depending on the number of points \( n \), we will get results that are close to the covariance values defined above.
The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.
</p>
<h3 id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">Diagonalize the sample covariance matrix to obtain the principal components </h3>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="diagonalize-the-sample-covariance-matrix-to-obtain-the-principal-components">Diagonalize the sample covariance matrix to obtain the principal components </h2>
<p>Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix \( \Sigma \). We can use the
@@ -2878,6 +2961,9 @@ $$
<p>where \( v_0 \) is the first principal component. </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="collecting-all-steps">Collecting all Steps </h2>
<p>Collecting all these steps we can write our own PCA function and
compare this with the functionality included in <b>Scikit-Learn</b>.
</p>
@@ -2931,6 +3017,8 @@ X2Dsl <span style="color: #666666">=</span> pca<span style="color: #666666">.</s
</div>
<p>This code does not contain all the above elements, but it shows how we can use <b>Scikit-Learn</b> to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then? </p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="classical-pca-theorem">Classical PCA Theorem </h2>
<p>We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
@@ -2949,6 +3037,9 @@ orthogonal projection of the data onto the columns spanned by the
eigenvectors of the covariance(correlations matrix).
</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="the-pca-theorem">The PCA Theorem </h2>
<p>To show the PCA theorem let us start with the assumption that there is one vector \( \boldsymbol{s}_0 \) which corresponds to a solution which minimized the reconstruction error \( J \). This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of \( \boldsymbol{w}_0 \) and \( \boldsymbol{z}_0 \) as</p>
<p>We are almost there, we have obtained a relation between minimizing
@@ -3004,6 +3095,8 @@ chapter 12.4 and discussion therein.
</p>
<p>For more details, see for example <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, chapter 2</a>.</p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="geometric-interpretation-and-link-with-singular-value-decomposition">Geometric Interpretation and link with Singular Value Decomposition </h2>
<p>For a detailed demonstration of the geometric interpretation, see <a href="https://www.springer.com/gp/book/9780387878102" target="_blank">Vidal, Ma and Sastry, section 2.1.2</a>.</p>
@@ -3093,6 +3186,9 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="pca-and-scikit-learn">PCA and scikit-learn </h2>
<p>Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -3275,7 +3371,10 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
</div>
</div>
</div>
<h3 id="incremental-pca">Incremental PCA </h3>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="incremental-pca">Incremental PCA </h2>
<p>One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
@@ -3326,6 +3425,9 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
</div>
</div>
</div>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="other-techniques">Other techniques </h2>
<p>There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.</p>
doc/pub/week43/ipynb/ipynb-week43-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week43/ipynb/week43.ipynb
+300 -207
View File
File diff suppressed because it is too large Load Diff
doc/src/week43/week43.do.txt
+55 -15
View File
@@ -1390,6 +1390,8 @@ we can rewrite the covariance matrix as
\]
!et
!split
===== More on the covariance =====
The covariance takes values between zero and infinity and may thus
lead to problems with loss of numerical precision for particularly
large values. It is common to scale the covariance matrix by
@@ -1417,7 +1419,8 @@ and $\bm{y}$ as
In the above example this is the function we constructed using _pandas_.
!split
===== Reminding ourselves about Linear Regression =====
In our derivation of the various regression algorithms like _Ordinary Least Squares_ or _Ridge regression_
we defined the design/feature matrix $\bm{X}$ as
@@ -1448,8 +1451,10 @@ with a given vector
\]
!et
!split
===== Simple Example =====
With these definitions, we can now rewrite our $2\times 2$
correaltion/covariance matrix in terms of a moe general design/feature
correlation/covariance matrix in terms of a moe general design/feature
matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$
covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$
@@ -1465,6 +1470,10 @@ covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$
\end{bmatrix},
\]
!et
!split
===== The Correlation Matrix =====
and the correlation matrix
!bt
\[
@@ -1480,7 +1489,8 @@ and the correlation matrix
!et
!split
===== Numpy Functionality =====
The Numpy function _np.cov_ calculates the covariance elements using
the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have
@@ -1522,7 +1532,7 @@ print(C)
!split
===== Correlation Matrix =====
===== Correlation Matrix again =====
The previous example can be converted into the correlation matrix by
simply scaling the matrix elements with the variances. We should also
@@ -1560,6 +1570,8 @@ this matrix we easily see that it is a positive definite matrix.
The above procedure with _numpy_ can be made more compact if we use _pandas_.
!split
===== Using Pandas =====
We whow here how we can set up the correlation matrix using _pandas_, as done in this simple code
!bc pycod
@@ -1578,6 +1590,8 @@ correlation_matrix = Xpd.corr()
print(correlation_matrix)
!ec
!split
===== And then the Franke Function =====
We expand this model to the Franke function discussed above.
@@ -1633,9 +1647,10 @@ columns since all matrix elements in the design matrix were set to one
(we are fitting the function in terms of a polynomial of degree $n$). We would however not include the intercept
and wee can simply
drop these elements and construct a correlation
matrix without them.
matrix without them by centering our matrix elements by subtracting the mean of each column.
!split
===== Lnks with the Design Matrix =====
We can rewrite the covariance matrix in a more compact form in terms of the design/feature matrix $\bm{X}$ as
!bt
@@ -1656,6 +1671,9 @@ x_{10} & x_{11}\\
\]
!et
!split
===== Computing the Expectation Values =====
If we then compute the expectation value
!bt
\[
@@ -1712,6 +1730,9 @@ and since $\bm{C}[\bm{y}]$ is diagonal we have for a given eigenvalue $i$ of the
\]
!et
!split
===== More on the PCA Theorem =====
In the derivation of the PCA theorem we will assume that the eigenvalues are ordered in descending order, that is
$\lambda_0 > \lambda_1 > \dots > \lambda_{p-1}$.
@@ -1728,7 +1749,8 @@ dimensions where the eigenvalues are very small. If $p$ is very large,
we could then aim at reducing $p$ to $l << p$ and handle only $l$
features/predictors.
=== The Algorithm before the Theorem ===
!split
===== The Algorithm before the Theorem =====
Here's how we would proceed in setting up the algorithm for the PCA, see also discussion below here.
* Set up the datapoints for the design/feature matrix $\bm{X}$ with $\bm{X}\in {\mathbb{R}}^{n\times p}$, with the predictors/features $p$ referring to the column numbers and the entries $n$ being the row elements.
@@ -1750,8 +1772,8 @@ x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
* Order the eigenvalue (and the eigenvectors accordingly) in order of decreasing eigenvalues.
* Keep only those $l$ eigenvalues larger than a selected threshold value, discarding thus $p-l$ features since we expect small variations in the data here.
=== Writing our own PCA code ===
!split
===== Writing our own PCA code =====
We will use a simple example first with two-dimensional data
drawn from a multivariate normal distribution with the following mean and covariance matrix (we have fixed these quantities but will play around with them below):
@@ -1766,6 +1788,8 @@ Note that the mean refers to each column of data.
We will generate $n = 10000$ points $X = \{ x_1, \ldots, x_N \}$ from
this distribution, and store them in the $1000 \times 2$ matrix $\bm{X}$. This is our design matrix where we have forced the covariance and mean values to take specific values.
!split
===== Implementing it =====
The following Python code aids in setting up the data and writing out the design matrix.
Note that the function _multivariate_ returns also the covariance discussed above and that it is defined by dividing by $n-1$ instead of $n$.
!bc pycod
@@ -1781,6 +1805,8 @@ X = np.random.multivariate_normal(mean, cov, n)
Now we are going to implement the PCA algorithm. We will break it down into various substeps.
!split
===== First Step =====
The first step of PCA is to compute the sample mean of the data and use it to center the data. Recall that the sample mean is
!bt
@@ -1806,6 +1832,8 @@ df = df -df.mean()
X_centered = X - X.mean(axis=0)
!ec
!split
===== Scaling =====
Alternatively, we could use the functions we discussed
earlier for scaling the data set. That is, we could have used the
_StandardScaler_ function in _Scikit-Learn_, a function which ensures
@@ -1816,6 +1844,8 @@ variance. The diagonal covariance matrix elements will then be one,
while the non-diagonal ones need to be divided by $2\sqrt{2}$ for our
specific case.
!split
===== Centered Data =====
Now we are going to use the mean centered data to compute the sample covariance of the data by using the following equation
!bt
@@ -1847,10 +1877,14 @@ plt.axis('equal')
plt.show()
!ec
!split
===== Exploring =====
Depending on the number of points $n$, we will get results that are close to the covariance values defined above.
The plot shows how the data are clustered around a line with slope close to one. Is this expected? Try to change the covariance and the mean values. For example, try to make the variance of the first element much larger than that of the second diagonal element. Try also to shrink the covariance (the non-diagonal elements) and see how the data points are distributed.
=== Diagonalize the sample covariance matrix to obtain the principal components ===
!split
===== Diagonalize the sample covariance matrix to obtain the principal components =====
Now we are ready to solve for the principal components! To do so we
diagonalize the sample covariance matrix $\Sigma$. We can use the
@@ -1870,6 +1904,9 @@ x_i \approx \tilde{x}_i = \mu_n + \langle x_i, v_0 \rangle v_0
!et
where $v_0$ is the first principal component.
!split
===== Collecting all Steps =====
Collecting all these steps we can write our own PCA function and
compare this with the functionality included in _Scikit-Learn_.
@@ -1903,7 +1940,7 @@ print(pca.components_.T[:, 0])
!ec
This code does not contain all the above elements, but it shows how we can use _Scikit-Learn_ to extract the eigenvector which corresponds to the largest eigenvalue. Try to address the questions we pose before the above code. Try also to change the values of the covariance matrix by making one of the diagonal elements much larger than the other. What do you observe then?
!split
===== Classical PCA Theorem =====
We assume now that we have a design matrix $\bm{X}$ which has been
@@ -1924,6 +1961,8 @@ eigenvectors of the covariance(correlations matrix).
!split
===== The PCA Theorem =====
To show the PCA theorem let us start with the assumption that there is one vector $\bm{s}_0$ which corresponds to a solution which minimized the reconstruction error $J$. This is an orthogonal vector. It means that we now approximate the reconstruction error in terms of $\bm{w}_0$ and $\bm{z}_0$ as
@@ -1985,7 +2024,7 @@ chapter 12.4 and discussion therein.
For more details, see for example "Vidal, Ma and Sastry, chapter 2":"https://www.springer.com/gp/book/9780387878102".
!split
===== Geometric Interpretation and link with Singular Value Decomposition =====
For a detailed demonstration of the geometric interpretation, see "Vidal, Ma and Sastry, section 2.1.2":"https://www.springer.com/gp/book/9780387878102".
@@ -2035,6 +2074,7 @@ W2 = V.T[:, :2]
X2D = X_centered.dot(W2)
!ec
!split
===== PCA and scikit-learn =====
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -2116,8 +2156,8 @@ pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X)
!ec
=== Incremental PCA ===
!split
===== Incremental PCA =====
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
@@ -2152,7 +2192,7 @@ rbf_pca = KernelPCA(n_components = 2, kernel="rbf", gamma=0.04)
X_reduced = rbf_pca.fit_transform(X)
!ec
!split
===== Other techniques =====