updating
This commit is contained in:
mhjensen
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -218,7 +224,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Dec 28, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Dec 29, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -242,7 +248,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -250,7 +256,7 @@ visualization.
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs002.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -241,7 +247,7 @@ ensures that all features are exactly between \( 0 \) and \( 1 \). The
|
||||
<li><a href="._DimRed-bs010.html">11</a></li>
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs003.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -244,7 +250,7 @@ techniques.
|
||||
<li><a href="._DimRed-bs011.html">12</a></li>
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs004.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -319,7 +325,7 @@ svm<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
|
||||
<li><a href="._DimRed-bs012.html">13</a></li>
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs005.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -269,7 +275,7 @@ svm<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
|
||||
<li><a href="._DimRed-bs013.html">14</a></li>
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs006.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -248,7 +254,7 @@ logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
|
||||
<li><a href="._DimRed-bs014.html">15</a></li>
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs007.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -303,7 +309,7 @@ applications.
|
||||
<li><a href="._DimRed-bs015.html">16</a></li>
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs008.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -201,6 +207,15 @@ MathJax.Hub.Config({
|
||||
|
||||
<h2 id="___sec7" class="anchor">Basic ideas of the Principal Component Analysis (PCA) </h2>
|
||||
|
||||
<p>
|
||||
The principal component analysis deals with the problem of fitting a
|
||||
low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
|
||||
the totaldimension \( D \) of the problem at hand (our data
|
||||
set). Mathematically it can be formulated as a statistical problem or
|
||||
a geometric problem. In our discussion of the theorem for the
|
||||
classical PCA, we will stay with a statistical approach. This is also
|
||||
what set the scene historically which for the PCA.
|
||||
|
||||
<p>
|
||||
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
|
||||
|
||||
@@ -233,7 +248,7 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
|
||||
<li><a href="._DimRed-bs016.html">17</a></li>
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs009.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -203,7 +209,7 @@ MathJax.Hub.Config({
|
||||
|
||||
<p>
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about
|
||||
the definition of the covariance and the correlation function.
|
||||
the definition of the covariance and the correlation function. These are quantities
|
||||
|
||||
<p>
|
||||
Suppose we have defined two vectors
|
||||
@@ -282,7 +288,7 @@ In the above example this is the function we constructed using <b>pandas</b>.
|
||||
<li><a href="._DimRed-bs017.html">18</a></li>
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs010.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -202,7 +208,9 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec9" class="anchor">Correlation Function and Design/Feature Matrix </h2>
|
||||
|
||||
<p>
|
||||
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
|
||||
we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
|
||||
$$
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
@@ -227,7 +235,11 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \)
|
||||
correaltion/covariance matrix in terms of a moe general design/feature
|
||||
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
|
||||
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
|
||||
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
@@ -277,7 +289,7 @@ $$
|
||||
<li><a href="._DimRed-bs018.html">19</a></li>
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs011.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -265,7 +271,7 @@ C <span style="color: #666666">=</span> np<span style="color: #666666">.</span>c
|
||||
<li><a href="._DimRed-bs019.html">20</a></li>
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs012.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -267,7 +273,7 @@ The above procedure with <b>numpy</b> can be made more compact if we use <b>pand
|
||||
<li><a href="._DimRed-bs020.html">21</a></li>
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs013.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -249,7 +255,7 @@ We expand this model to the Franke function discussed above.
|
||||
<li><a href="._DimRed-bs021.html">22</a></li>
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs014.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -256,8 +262,8 @@ columns since all matrix elements in the design matrix were set to one
|
||||
<p>
|
||||
This means that the variance for these elements will be zero and will
|
||||
cause problems when we set up the correlation matrix. We can simply
|
||||
drop these elements as follows and then construct the correlation
|
||||
matrix.
|
||||
drop these elements and construct a correlation
|
||||
matrix without these elements.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -285,7 +291,7 @@ matrix.
|
||||
<li><a href="._DimRed-bs022.html">23</a></li>
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs015.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -265,7 +271,7 @@ It is easy to generalize this to a matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\t
|
||||
<li><a href="._DimRed-bs023.html">24</a></li>
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs016.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -273,7 +279,7 @@ features/predictors.
|
||||
<li><a href="._DimRed-bs024.html">25</a></li>
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs017.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -253,7 +259,7 @@ $$
|
||||
<li><a href="._DimRed-bs025.html">26</a></li>
|
||||
<li><a href="._DimRed-bs026.html">27</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs018.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -229,7 +235,7 @@ X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>r
|
||||
Make thereafter a small Python code which plots the data. Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
|
||||
|
||||
<p>
|
||||
Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.
|
||||
Now we are going to implement the PCA algorithm. We will break it down into various substeps.
|
||||
|
||||
<h3 id="___sec18" class="anchor">Compute the sample mean and center the data </h3>
|
||||
|
||||
@@ -319,7 +325,7 @@ Finally, try out your own PCA function with other data sets.
|
||||
<li><a href="._DimRed-bs026.html">27</a></li>
|
||||
<li><a href="._DimRed-bs027.html">28</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs019.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -202,8 +208,11 @@ MathJax.Hub.Config({
|
||||
<h2 id="___sec21" class="anchor">Classical PCA Theorem </h2>
|
||||
|
||||
<p>
|
||||
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \)
|
||||
each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
|
||||
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
|
||||
centered as discussed above. For the sake of simplicity we skip the
|
||||
overline symbol. The matrix is defined in terms of the various column
|
||||
vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension
|
||||
\( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
|
||||
|
||||
<p>
|
||||
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
|
||||
@@ -215,7 +224,13 @@ with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \
|
||||
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
|
||||
|
||||
<p>
|
||||
The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
|
||||
The PCA theorem states that minimizing the above reconstruction error
|
||||
corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which
|
||||
diagonalizes the empirical covariance(correlation) matrix. The optimal
|
||||
low-dimensional encoding of the data is then given by a set of vectors
|
||||
\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the
|
||||
orthogonal projection of the data onto the columns spanned by the
|
||||
eigenvectors of the covariance(correlations matrix).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -243,7 +258,7 @@ The PCA theorem states that minimizing the above reconstruction error correspond
|
||||
<li><a href="._DimRed-bs027.html">28</a></li>
|
||||
<li><a href="._DimRed-bs028.html">29</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs020.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -245,7 +251,7 @@ where the vectors on the rhs are known.
|
||||
<li><a href="._DimRed-bs028.html">29</a></li>
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs021.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -232,7 +238,10 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
|
||||
We are almost there, we have obtained a relation between minimizing
|
||||
the reconstruction error and the variance and the covariance
|
||||
matrix. Minimizing the error is equivalent to maximizing the variance
|
||||
of the projected data.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -260,7 +269,7 @@ We are almost there, we have obtained a relation between minimizing the reconstr
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs022.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -247,6 +253,9 @@ discussion in chapter 12.2 of Murphy's text has also a nice link with
|
||||
the Singular Value Decomposition theorem. For categorical data, see
|
||||
chapter 12.4 and discussion therein.
|
||||
|
||||
<p>
|
||||
Additional part of the proof for the other eigenvectors will be added by mid January 2020.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -272,6 +281,8 @@ chapter 12.4 and discussion therein.
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs023.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -199,59 +205,11 @@ MathJax.Hub.Config({
|
||||
<a name="part0023"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec25" class="anchor">Principal Component Analysis </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
|
||||
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
|
||||
<h2 id="___sec25" class="anchor">Geometric Interpretation and link with Singular Value Decomposition </h2>
|
||||
|
||||
<p>
|
||||
The following Python code uses NumPy’s <b>svd()</b> function to obtain all the principal components of the
|
||||
training set, then extracts the first two principal components. First we center the data using either <b>pandas</b> or our own code
|
||||
<p>
|
||||
This material will be added by mid January 2020.
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
|
||||
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">100</span>)
|
||||
<span style="color: #408080; font-style: italic"># setting up a 10 x 5 vanilla matrix </span>
|
||||
rows <span style="color: #666666">=</span> <span style="color: #666666">10</span>
|
||||
cols <span style="color: #666666">=</span> <span style="color: #666666">5</span>
|
||||
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(rows,cols)
|
||||
df <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(X)
|
||||
<span style="color: #408080; font-style: italic"># Pandas does the centering for us</span>
|
||||
df <span style="color: #666666">=</span> df <span style="color: #666666">-</span>df<span style="color: #666666">.</span>mean()
|
||||
display(df)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># we center it ourselves</span>
|
||||
X_centered <span style="color: #666666">=</span> X <span style="color: #666666">-</span> X<span style="color: #666666">.</span>mean(axis<span style="color: #666666">=0</span>)
|
||||
<span style="color: #408080; font-style: italic"># Then check the difference between pandas and our own set up</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(X_centered<span style="color: #666666">-</span>df)
|
||||
<span style="color: #408080; font-style: italic">#Now we do an SVD</span>
|
||||
U, s, V <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>svd(X_centered)
|
||||
c1 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, <span style="color: #666666">0</span>]
|
||||
c2 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, <span style="color: #666666">1</span>]
|
||||
W2 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, :<span style="color: #666666">2</span>]
|
||||
X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666">.</span>dot(W2)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(X2D)
|
||||
</pre></div>
|
||||
<p>
|
||||
PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
|
||||
the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
|
||||
forget to center the data first.
|
||||
|
||||
<p>
|
||||
Once you have identified all the principal components, you can reduce the dimensionality of the dataset
|
||||
down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
|
||||
Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>W2 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, :<span style="color: #666666">2</span>]
|
||||
X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666">.</span>dot(W2)
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -276,6 +234,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs024.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -197,37 +203,61 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0024"></a>
|
||||
<!-- !split -->
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec26" class="anchor">PCA and scikit-learn </h2>
|
||||
<h2 id="___sec26" class="anchor">Principal Component Analysis </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
Principal Component Analysis (PCA) is by far the most popular dimensionality reduction algorithm.
|
||||
First it identifies the hyperplane that lies closest to the data, and then it projects the data onto it.
|
||||
|
||||
<p>
|
||||
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
|
||||
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
|
||||
that it automatically takes care of centering the data):
|
||||
The following Python code uses NumPy’s <b>svd()</b> function to obtain all the principal components of the
|
||||
training set, then extracts the first two principal components. First we center the data using either <b>pandas</b> or our own code
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic">#thereafter we do a PCA with Scikit-learn</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> PCA
|
||||
pca <span style="color: #666666">=</span> PCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>)
|
||||
X2D <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
|
||||
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">100</span>)
|
||||
<span style="color: #408080; font-style: italic"># setting up a 10 x 5 vanilla matrix </span>
|
||||
rows <span style="color: #666666">=</span> <span style="color: #666666">10</span>
|
||||
cols <span style="color: #666666">=</span> <span style="color: #666666">5</span>
|
||||
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(rows,cols)
|
||||
df <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(X)
|
||||
<span style="color: #408080; font-style: italic"># Pandas does the centering for us</span>
|
||||
df <span style="color: #666666">=</span> df <span style="color: #666666">-</span>df<span style="color: #666666">.</span>mean()
|
||||
display(df)
|
||||
|
||||
<span style="color: #408080; font-style: italic"># we center it ourselves</span>
|
||||
X_centered <span style="color: #666666">=</span> X <span style="color: #666666">-</span> X<span style="color: #666666">.</span>mean(axis<span style="color: #666666">=0</span>)
|
||||
<span style="color: #408080; font-style: italic"># Then check the difference between pandas and our own set up</span>
|
||||
<span style="color: #008000; font-weight: bold">print</span>(X_centered<span style="color: #666666">-</span>df)
|
||||
<span style="color: #408080; font-style: italic">#Now we do an SVD</span>
|
||||
U, s, V <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>svd(X_centered)
|
||||
c1 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, <span style="color: #666666">0</span>]
|
||||
c2 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, <span style="color: #666666">1</span>]
|
||||
W2 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, :<span style="color: #666666">2</span>]
|
||||
X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666">.</span>dot(W2)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(X2D)
|
||||
</pre></div>
|
||||
<p>
|
||||
After fitting the PCA transformer to the dataset, you can access the principal components using the
|
||||
components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
|
||||
principal component is equal to
|
||||
PCA assumes that the dataset is centered around the origin. Scikit-Learn’s PCA classes take care of centering
|
||||
the data for you. However, if you implement PCA yourself (as in the preceding example), or if you use other libraries, don’t
|
||||
forget to center the data first.
|
||||
|
||||
<p>
|
||||
Once you have identified all the principal components, you can reduce the dimensionality of the dataset
|
||||
down to \( d \) dimensions by projecting it onto the hyperplane defined by the first \( d \) principal components.
|
||||
Selecting this hyperplane ensures that the projection will preserve as much variance as possible.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca<span style="color: #666666">.</span>components_<span style="color: #666666">.</span>T[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>W2 <span style="color: #666666">=</span> V<span style="color: #666666">.</span>T[:, :<span style="color: #666666">2</span>]
|
||||
X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666">.</span>dot(W2)
|
||||
</pre></div>
|
||||
<p>
|
||||
Another very useful piece of information is the explained variance ratio of each principal component,
|
||||
available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
|
||||
variance that lies along the axis of each principal component.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -251,6 +281,7 @@ variance that lies along the axis of each principal component.
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs025.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -197,45 +203,36 @@ MathJax.Hub.Config({
|
||||
<p> </p><p> </p><p> </p> <!-- add vertical space -->
|
||||
|
||||
<a name="part0025"></a>
|
||||
<!-- !split -->
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec27" class="anchor">Back to the Cancer Data </h2>
|
||||
We can now repeat the above but applied to real data, in this case our breast cancer data.
|
||||
Here we compute performance scores on the training data using logistic regression.
|
||||
<h2 id="___sec27" class="anchor">PCA and scikit-learn </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
|
||||
following code applies PCA to reduce the dimensionality of the dataset down to two dimensions (note
|
||||
that it automatically takes care of centering the data):
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> load_breast_cancer
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LogisticRegression
|
||||
cancer <span style="color: #666666">=</span> load_breast_cancer()
|
||||
|
||||
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
|
||||
|
||||
logreg <span style="color: #666666">=</span> LogisticRegression()
|
||||
logreg<span style="color: #666666">.</span>fit(X_train, y_train)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Train set accuracy from Logistic Regression: {:.2f}"</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_train,y_train)))
|
||||
<span style="color: #408080; font-style: italic"># We scale the data</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
|
||||
scaler <span style="color: #666666">=</span> StandardScaler()
|
||||
scaler<span style="color: #666666">.</span>fit(X_train)
|
||||
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
|
||||
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
|
||||
<span style="color: #408080; font-style: italic"># Then perform again a log reg fit</span>
|
||||
logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Train set accuracy scaled data: {:.2f}"</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_train_scaled,y_train)))
|
||||
<span style="color: #408080; font-style: italic">#thereafter we do a PCA with Scikit-learn</span>
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic">#thereafter we do a PCA with Scikit-learn</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> PCA
|
||||
pca <span style="color: #666666">=</span> PCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>)
|
||||
X2D_train <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X_train_scaled)
|
||||
<span style="color: #408080; font-style: italic"># and finally compute the log reg fit and the score on the training data </span>
|
||||
logreg<span style="color: #666666">.</span>fit(X2D_train,y_train)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Train set accuracy scaled and PCA data: {:.2f}"</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X2D_train,y_train)))
|
||||
X2D <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(X2D)
|
||||
</pre></div>
|
||||
<p>
|
||||
We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.
|
||||
After fitting the PCA transformer to the dataset, you can access the principal components using the
|
||||
components variable (note that it contains the PCs as horizontal vectors, so, for example, the first
|
||||
principal component is equal to
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca<span style="color: #666666">.</span>components_<span style="color: #666666">.</span>T[:, <span style="color: #666666">0</span>]<span style="color: #666666">.</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
Another very useful piece of information is the explained variance ratio of each principal component,
|
||||
available via the \( explained\_variance\_ratio \) variable. It indicates the proportion of the dataset’s
|
||||
variance that lies along the axis of each principal component.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -259,6 +256,7 @@ We see that our training data after the PCA decomposition has a performance simi
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs026.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -199,33 +205,44 @@ MathJax.Hub.Config({
|
||||
<a name="part0026"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec28" class="anchor">More on the PCA </h2>
|
||||
|
||||
<p>
|
||||
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
|
||||
choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
|
||||
Unless, of course, you are reducing dimensionality for data visualization — in that case you will
|
||||
generally want to reduce the dimensionality down to 2 or 3.
|
||||
The following code computes PCA without reducing dimensionality, then computes the minimum number
|
||||
of dimensions required to preserve 95% of the training set’s variance:
|
||||
<h2 id="___sec28" class="anchor">Back to the Cancer Data </h2>
|
||||
We can now repeat the above but applied to real data, in this case our breast cancer data.
|
||||
Here we compute performance scores on the training data using logistic regression.
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca <span style="color: #666666">=</span> PCA()
|
||||
pca<span style="color: #666666">.</span>fit(X)
|
||||
cumsum <span style="color: #666666">=</span> np<span style="color: #666666">.</span>cumsum(pca<span style="color: #666666">.</span>explained_variance_ratio_)
|
||||
d <span style="color: #666666">=</span> np<span style="color: #666666">.</span>argmax(cumsum <span style="color: #666666">>=</span> <span style="color: #666666">0.95</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
|
||||
of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
|
||||
a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
|
||||
<p>
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
|
||||
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.datasets</span> <span style="color: #008000; font-weight: bold">import</span> load_breast_cancer
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LogisticRegression
|
||||
cancer <span style="color: #666666">=</span> load_breast_cancer()
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca <span style="color: #666666">=</span> PCA(n_components<span style="color: #666666">=0.95</span>)
|
||||
X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
|
||||
|
||||
logreg <span style="color: #666666">=</span> LogisticRegression()
|
||||
logreg<span style="color: #666666">.</span>fit(X_train, y_train)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Train set accuracy from Logistic Regression: {:.2f}"</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_train,y_train)))
|
||||
<span style="color: #408080; font-style: italic"># We scale the data</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
|
||||
scaler <span style="color: #666666">=</span> StandardScaler()
|
||||
scaler<span style="color: #666666">.</span>fit(X_train)
|
||||
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
|
||||
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
|
||||
<span style="color: #408080; font-style: italic"># Then perform again a log reg fit</span>
|
||||
logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Train set accuracy scaled data: {:.2f}"</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_train_scaled,y_train)))
|
||||
<span style="color: #408080; font-style: italic">#thereafter we do a PCA with Scikit-learn</span>
|
||||
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> PCA
|
||||
pca <span style="color: #666666">=</span> PCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>)
|
||||
X2D_train <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X_train_scaled)
|
||||
<span style="color: #408080; font-style: italic"># and finally compute the log reg fit and the score on the training data </span>
|
||||
logreg<span style="color: #666666">.</span>fit(X2D_train,y_train)
|
||||
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">"Train set accuracy scaled and PCA data: {:.2f}"</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X2D_train,y_train)))
|
||||
</pre></div>
|
||||
<p>
|
||||
We see that our training data after the PCA decomposition has a performance similar to the non-scaled data.
|
||||
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -247,6 +264,7 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs027.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -199,15 +205,33 @@ MathJax.Hub.Config({
|
||||
<a name="part0027"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec29" class="anchor">Incremental PCA </h2>
|
||||
<h2 id="___sec29" class="anchor">More on the PCA </h2>
|
||||
|
||||
<p>
|
||||
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
|
||||
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
|
||||
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
|
||||
at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
|
||||
instances arrive).
|
||||
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
|
||||
choose the number of dimensions that add up to a sufficiently large portion of the variance (e.g., 95%).
|
||||
Unless, of course, you are reducing dimensionality for data visualization — in that case you will
|
||||
generally want to reduce the dimensionality down to 2 or 3.
|
||||
The following code computes PCA without reducing dimensionality, then computes the minimum number
|
||||
of dimensions required to preserve 95% of the training set’s variance:
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca <span style="color: #666666">=</span> PCA()
|
||||
pca<span style="color: #666666">.</span>fit(X)
|
||||
cumsum <span style="color: #666666">=</span> np<span style="color: #666666">.</span>cumsum(pca<span style="color: #666666">.</span>explained_variance_ratio_)
|
||||
d <span style="color: #666666">=</span> np<span style="color: #666666">.</span>argmax(cumsum <span style="color: #666666">>=</span> <span style="color: #666666">0.95</span>) <span style="color: #666666">+</span> <span style="color: #666666">1</span>
|
||||
</pre></div>
|
||||
<p>
|
||||
You could then set \( n\_components=d \) and run PCA again. However, there is a much better option: instead
|
||||
of specifying the number of principal components you want to preserve, you can set \( n\_components \) to be
|
||||
a float between 0.0 and 1.0, indicating the ratio of variance you wish to preserve:
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>pca <span style="color: #666666">=</span> PCA(n_components<span style="color: #666666">=0.95</span>)
|
||||
X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
</pre></div>
|
||||
<p>
|
||||
<p>
|
||||
<!-- navigation buttons at the bottom of the page -->
|
||||
@@ -228,6 +252,7 @@ instances arrive).
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs028.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -199,18 +205,14 @@ MathJax.Hub.Config({
|
||||
<a name="part0028"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec30" class="anchor">Randomized PCA </h2>
|
||||
<h2 id="___sec30" class="anchor">Incremental PCA </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
|
||||
algorithm that quickly finds an approximation of the first d principal components. Its computational
|
||||
complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
|
||||
previous algorithms when \( d \) is much smaller than \( n \).
|
||||
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
|
||||
memory in order for the SVD algorithm to run. Fortunately, Incremental PCA (IPCA) algorithms have
|
||||
been developed: you can split the training set into mini-batches and feed an IPCA algorithm one minibatch
|
||||
at a time. This is useful for large training sets, and also to apply PCA online (i.e., on the fly, as new
|
||||
instances arrive).
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -231,6 +233,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs029.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -199,28 +205,14 @@ MathJax.Hub.Config({
|
||||
<a name="part0029"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec31" class="anchor">Kernel PCA </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
<h2 id="___sec31" class="anchor">Randomized PCA </h2>
|
||||
|
||||
<p>
|
||||
The kernel trick is a mathematical technique that implicitly maps instances into a
|
||||
very high-dimensional space (called the feature space), enabling nonlinear classification and regression
|
||||
with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
|
||||
space corresponds to a complex nonlinear decision boundary in the original space.
|
||||
It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
|
||||
projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
|
||||
preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
|
||||
twisted manifold.
|
||||
For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an
|
||||
<p>
|
||||
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
|
||||
algorithm that quickly finds an approximation of the first d principal components. Its computational
|
||||
complexity is \( O(m \times d^2)+O(d^3) \), instead of \( O(m \times n^2) + O(n^3) \), so it is dramatically faster than the
|
||||
previous algorithms when \( d \) is much smaller than \( n \).
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> KernelPCA
|
||||
rbf_pca <span style="color: #666666">=</span> KernelPCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>, kernel<span style="color: #666666">=</span><span style="color: #BA2121">"rbf"</span>, gamma<span style="color: #666666">=0.04</span>)
|
||||
X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
</pre></div>
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
@@ -244,6 +236,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
|
||||
<li class="active"><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs030.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -199,14 +205,32 @@ MathJax.Hub.Config({
|
||||
<a name="part0030"></a>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec32" class="anchor">LLE </h2>
|
||||
<h2 id="___sec32" class="anchor">Kernel PCA </h2>
|
||||
<div class="panel panel-default">
|
||||
<div class="panel-body">
|
||||
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
|
||||
|
||||
<p>
|
||||
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
|
||||
(NLDR) technique. It is a Manifold Learning technique that does not rely on projections like the previous
|
||||
algorithms. In a nutshell, LLE works by first measuring how each training instance linearly relates to its
|
||||
closest neighbors (c.n.), and then looking for a low-dimensional representation of the training set where
|
||||
these local relationships are best preserved (more details shortly).
|
||||
The kernel trick is a mathematical technique that implicitly maps instances into a
|
||||
very high-dimensional space (called the feature space), enabling nonlinear classification and regression
|
||||
with Support Vector Machines. Recall that a linear decision boundary in the high-dimensional feature
|
||||
space corresponds to a complex nonlinear decision boundary in the original space.
|
||||
It turns out that the same trick can be applied to PCA, making it possible to perform complex nonlinear
|
||||
projections for dimensionality reduction. This is called Kernel PCA (kPCA). It is often good at
|
||||
preserving clusters of instances after projection, or sometimes even unrolling datasets that lie close to a
|
||||
twisted manifold.
|
||||
For example, the following code uses Scikit-Learn’s KernelPCA class to perform kPCA with an
|
||||
<p>
|
||||
|
||||
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
|
||||
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.decomposition</span> <span style="color: #008000; font-weight: bold">import</span> KernelPCA
|
||||
rbf_pca <span style="color: #666666">=</span> KernelPCA(n_components <span style="color: #666666">=</span> <span style="color: #666666">2</span>, kernel<span style="color: #666666">=</span><span style="color: #BA2121">"rbf"</span>, gamma<span style="color: #666666">=0.04</span>)
|
||||
X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #666666">.</span>fit_transform(X)
|
||||
</pre></div>
|
||||
<p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
<p>
|
||||
<p>
|
||||
@@ -225,6 +249,7 @@ these local relationships are best preserved (more details shortly).
|
||||
<li><a href="._DimRed-bs029.html">30</a></li>
|
||||
<li class="active"><a href="._DimRed-bs030.html">31</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs031.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -104,15 +104,20 @@ Automatically generated HTML file from DocOnce source
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -175,15 +180,16 @@ MathJax.Hub.Config({
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec22" style="font-size: 80%;"><b>Proof of the PCA Theorem</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec23" style="font-size: 80%;"><b>PCA Proof continued</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs022.html#___sec24" style="font-size: 80%;"><b>The final step</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs023.html#___sec25" style="font-size: 80%;"><b>Geometric Interpretation and link with Singular Value Decomposition</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs024.html#___sec26" style="font-size: 80%;"><b>Principal Component Analysis</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs025.html#___sec27" style="font-size: 80%;"><b>PCA and scikit-learn</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs026.html#___sec28" style="font-size: 80%;"><b>Back to the Cancer Data</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs027.html#___sec29" style="font-size: 80%;"><b>More on the PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs028.html#___sec30" style="font-size: 80%;"><b>Incremental PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs029.html#___sec31" style="font-size: 80%;"><b>Randomized PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs030.html#___sec32" style="font-size: 80%;"><b>Kernel PCA</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs031.html#___sec33" style="font-size: 80%;"><b>LLE</b></a></li>
|
||||
<!-- navigation toc: --> <li><a href="._DimRed-bs032.html#___sec34" style="font-size: 80%;"><b>Other techniques</b></a></li>
|
||||
|
||||
</ul>
|
||||
</li>
|
||||
@@ -218,7 +224,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Dec 28, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Dec 29, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -242,7 +248,7 @@ MathJax.Hub.Config({
|
||||
<li><a href="._DimRed-bs008.html">9</a></li>
|
||||
<li><a href="._DimRed-bs009.html">10</a></li>
|
||||
<li><a href="">...</a></li>
|
||||
<li><a href="._DimRed-bs031.html">32</a></li>
|
||||
<li><a href="._DimRed-bs032.html">33</a></li>
|
||||
<li><a href="._DimRed-bs001.html">»</a></li>
|
||||
</ul>
|
||||
<!-- ------------------- end of main content --------------- -->
|
||||
|
||||
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p> <br>
|
||||
<center><h4>Dec 28, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Dec 29, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
|
||||
@@ -518,6 +518,15 @@ applications.
|
||||
<section>
|
||||
<h2 id="___sec7">Basic ideas of the Principal Component Analysis (PCA) </h2>
|
||||
|
||||
<p>
|
||||
The principal component analysis deals with the problem of fitting a
|
||||
low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
|
||||
the totaldimension \( D \) of the problem at hand (our data
|
||||
set). Mathematically it can be formulated as a statistical problem or
|
||||
a geometric problem. In our discussion of the theorem for the
|
||||
classical PCA, we will stay with a statistical approach. This is also
|
||||
what set the scene historically which for the PCA.
|
||||
|
||||
<p>
|
||||
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
|
||||
|
||||
@@ -534,7 +543,7 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
|
||||
|
||||
<p>
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about
|
||||
the definition of the covariance and the correlation function.
|
||||
the definition of the covariance and the correlation function. These are quantities
|
||||
|
||||
<p>
|
||||
Suppose we have defined two vectors
|
||||
@@ -606,7 +615,9 @@ In the above example this is the function we constructed using <b>pandas</b>.
|
||||
<h2 id="___sec9">Correlation Function and Design/Feature Matrix </h2>
|
||||
|
||||
<p>
|
||||
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
|
||||
we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
@@ -637,7 +648,11 @@ $$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \)
|
||||
correaltion/covariance matrix in terms of a moe general design/feature
|
||||
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
|
||||
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
|
||||
|
||||
<p> <br>
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
@@ -843,8 +858,8 @@ columns since all matrix elements in the design matrix were set to one
|
||||
<p>
|
||||
This means that the variance for these elements will be zero and will
|
||||
cause problems when we set up the correlation matrix. We can simply
|
||||
drop these elements as follows and then construct the correlation
|
||||
matrix.
|
||||
drop these elements and construct a correlation
|
||||
matrix without these elements.
|
||||
</section>
|
||||
|
||||
|
||||
@@ -1026,7 +1041,7 @@ X = np.random.multivariate_normal(mean, cov, n)
|
||||
Make thereafter a small Python code which plots the data. Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
|
||||
|
||||
<p>
|
||||
Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.
|
||||
Now we are going to implement the PCA algorithm. We will break it down into various substeps.
|
||||
|
||||
<h3 id="___sec18">Compute the sample mean and center the data </h3>
|
||||
|
||||
@@ -1103,8 +1118,11 @@ Finally, try out your own PCA function with other data sets.
|
||||
<h2 id="___sec21">Classical PCA Theorem </h2>
|
||||
|
||||
<p>
|
||||
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \)
|
||||
each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
|
||||
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
|
||||
centered as discussed above. For the sake of simplicity we skip the
|
||||
overline symbol. The matrix is defined in terms of the various column
|
||||
vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension
|
||||
\( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
|
||||
|
||||
<p>
|
||||
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
|
||||
@@ -1118,7 +1136,13 @@ with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \
|
||||
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
|
||||
|
||||
<p>
|
||||
The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
|
||||
The PCA theorem states that minimizing the above reconstruction error
|
||||
corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which
|
||||
diagonalizes the empirical covariance(correlation) matrix. The optimal
|
||||
low-dimensional encoding of the data is then given by a set of vectors
|
||||
\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the
|
||||
orthogonal projection of the data onto the columns spanned by the
|
||||
eigenvectors of the covariance(correlations matrix).
|
||||
</section>
|
||||
|
||||
|
||||
@@ -1195,7 +1219,10 @@ $$
|
||||
<p> <br>
|
||||
|
||||
<p>
|
||||
We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
|
||||
We are almost there, we have obtained a relation between minimizing
|
||||
the reconstruction error and the variance and the covariance
|
||||
matrix. Minimizing the error is equivalent to maximizing the variance
|
||||
of the projected data.
|
||||
</section>
|
||||
|
||||
|
||||
@@ -1255,11 +1282,22 @@ our basis of eigenvectors is orthogonal, see <a href="https://mitpress.mit.edu/b
|
||||
discussion in chapter 12.2 of Murphy's text has also a nice link with
|
||||
the Singular Value Decomposition theorem. For categorical data, see
|
||||
chapter 12.4 and discussion therein.
|
||||
|
||||
<p>
|
||||
Additional part of the proof for the other eigenvectors will be added by mid January 2020.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec25">Principal Component Analysis </h2>
|
||||
<h2 id="___sec25">Geometric Interpretation and link with Singular Value Decomposition </h2>
|
||||
|
||||
<p>
|
||||
This material will be added by mid January 2020.
|
||||
</section>
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec26">Principal Component Analysis </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -1316,7 +1354,7 @@ X2D = X_centered.dot(W2)
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec26">PCA and scikit-learn </h2>
|
||||
<h2 id="___sec27">PCA and scikit-learn </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
|
||||
@@ -1348,7 +1386,7 @@ variance that lies along the axis of each principal component.
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec27">Back to the Cancer Data </h2>
|
||||
<h2 id="___sec28">Back to the Cancer Data </h2>
|
||||
We can now repeat the above but applied to real data, in this case our breast cancer data.
|
||||
Here we compute performance scores on the training data using logistic regression.
|
||||
<p>
|
||||
@@ -1389,7 +1427,7 @@ We see that our training data after the PCA decomposition has a performance simi
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec28">More on the PCA </h2>
|
||||
<h2 id="___sec29">More on the PCA </h2>
|
||||
|
||||
<p>
|
||||
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
|
||||
@@ -1420,7 +1458,7 @@ X_reduced = pca.fit_transform(X)
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec29">Incremental PCA </h2>
|
||||
<h2 id="___sec30">Incremental PCA </h2>
|
||||
|
||||
<p>
|
||||
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
|
||||
@@ -1432,7 +1470,7 @@ instances arrive).
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec30">Randomized PCA </h2>
|
||||
<h2 id="___sec31">Randomized PCA </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
|
||||
@@ -1446,7 +1484,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec31">Kernel PCA </h2>
|
||||
<h2 id="___sec32">Kernel PCA </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -1472,7 +1510,7 @@ X_reduced = rbf_pca.fit_transform(X)
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec32">LLE </h2>
|
||||
<h2 id="___sec33">LLE </h2>
|
||||
|
||||
<p>
|
||||
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
|
||||
@@ -1484,7 +1522,7 @@ these local relationships are best preserved (more details shortly).
|
||||
|
||||
|
||||
<section>
|
||||
<h2 id="___sec33">Other techniques </h2>
|
||||
<h2 id="___sec34">Other techniques </h2>
|
||||
|
||||
<p>
|
||||
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
|
||||
|
||||
@@ -124,15 +124,20 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -174,7 +179,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Dec 28, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Dec 29, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -541,6 +546,15 @@ applications.
|
||||
|
||||
<h2 id="___sec7">Basic ideas of the Principal Component Analysis (PCA) </h2>
|
||||
|
||||
<p>
|
||||
The principal component analysis deals with the problem of fitting a
|
||||
low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
|
||||
the totaldimension \( D \) of the problem at hand (our data
|
||||
set). Mathematically it can be formulated as a statistical problem or
|
||||
a geometric problem. In our discussion of the theorem for the
|
||||
classical PCA, we will stay with a statistical approach. This is also
|
||||
what set the scene historically which for the PCA.
|
||||
|
||||
<p>
|
||||
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
|
||||
|
||||
@@ -556,7 +570,7 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
|
||||
|
||||
<p>
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about
|
||||
the definition of the covariance and the correlation function.
|
||||
the definition of the covariance and the correlation function. These are quantities
|
||||
|
||||
<p>
|
||||
Suppose we have defined two vectors
|
||||
@@ -616,7 +630,9 @@ In the above example this is the function we constructed using <b>pandas</b>.
|
||||
<h2 id="___sec9">Correlation Function and Design/Feature Matrix </h2>
|
||||
|
||||
<p>
|
||||
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
|
||||
we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
|
||||
$$
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
@@ -641,7 +657,11 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \)
|
||||
correaltion/covariance matrix in terms of a moe general design/feature
|
||||
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
|
||||
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
|
||||
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
@@ -840,8 +860,8 @@ columns since all matrix elements in the design matrix were set to one
|
||||
<p>
|
||||
This means that the variance for these elements will be zero and will
|
||||
cause problems when we set up the correlation matrix. We can simply
|
||||
drop these elements as follows and then construct the correlation
|
||||
matrix.
|
||||
drop these elements and construct a correlation
|
||||
matrix without these elements.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1001,7 +1021,7 @@ X = np.random.multivariate_normal(mean, cov, n)
|
||||
Make thereafter a small Python code which plots the data. Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
|
||||
|
||||
<p>
|
||||
Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.
|
||||
Now we are going to implement the PCA algorithm. We will break it down into various substeps.
|
||||
|
||||
<h3 id="___sec18">Compute the sample mean and center the data </h3>
|
||||
|
||||
@@ -1071,8 +1091,11 @@ Finally, try out your own PCA function with other data sets.
|
||||
<h2 id="___sec21">Classical PCA Theorem </h2>
|
||||
|
||||
<p>
|
||||
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \)
|
||||
each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
|
||||
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
|
||||
centered as discussed above. For the sake of simplicity we skip the
|
||||
overline symbol. The matrix is defined in terms of the various column
|
||||
vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension
|
||||
\( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
|
||||
|
||||
<p>
|
||||
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
|
||||
@@ -1084,7 +1107,13 @@ with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \
|
||||
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
|
||||
|
||||
<p>
|
||||
The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
|
||||
The PCA theorem states that minimizing the above reconstruction error
|
||||
corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which
|
||||
diagonalizes the empirical covariance(correlation) matrix. The optimal
|
||||
low-dimensional encoding of the data is then given by a set of vectors
|
||||
\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the
|
||||
orthogonal projection of the data onto the columns spanned by the
|
||||
eigenvectors of the covariance(correlations matrix).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1145,7 +1174,10 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
|
||||
We are almost there, we have obtained a relation between minimizing
|
||||
the reconstruction error and the variance and the covariance
|
||||
matrix. Minimizing the error is equivalent to maximizing the variance
|
||||
of the projected data.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1198,10 +1230,21 @@ discussion in chapter 12.2 of Murphy's text has also a nice link with
|
||||
the Singular Value Decomposition theorem. For categorical data, see
|
||||
chapter 12.4 and discussion therein.
|
||||
|
||||
<p>
|
||||
Additional part of the proof for the other eigenvectors will be added by mid January 2020.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">Principal Component Analysis </h2>
|
||||
<h2 id="___sec25">Geometric Interpretation and link with Singular Value Decomposition </h2>
|
||||
|
||||
<p>
|
||||
This material will be added by mid January 2020.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec26">Principal Component Analysis </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -1257,7 +1300,7 @@ X2D = X_centered.dot(W2)
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec26">PCA and scikit-learn </h2>
|
||||
<h2 id="___sec27">PCA and scikit-learn </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
|
||||
@@ -1289,7 +1332,7 @@ variance that lies along the axis of each principal component.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec27">Back to the Cancer Data </h2>
|
||||
<h2 id="___sec28">Back to the Cancer Data </h2>
|
||||
We can now repeat the above but applied to real data, in this case our breast cancer data.
|
||||
Here we compute performance scores on the training data using logistic regression.
|
||||
<p>
|
||||
@@ -1330,7 +1373,7 @@ We see that our training data after the PCA decomposition has a performance simi
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec28">More on the PCA </h2>
|
||||
<h2 id="___sec29">More on the PCA </h2>
|
||||
|
||||
<p>
|
||||
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
|
||||
@@ -1360,7 +1403,7 @@ X_reduced = pca.fit_transform(X)
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec29">Incremental PCA </h2>
|
||||
<h2 id="___sec30">Incremental PCA </h2>
|
||||
|
||||
<p>
|
||||
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
|
||||
@@ -1372,7 +1415,7 @@ instances arrive).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec30">Randomized PCA </h2>
|
||||
<h2 id="___sec31">Randomized PCA </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
|
||||
@@ -1387,7 +1430,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec31">Kernel PCA </h2>
|
||||
<h2 id="___sec32">Kernel PCA </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -1416,7 +1459,7 @@ X_reduced = rbf_pca.fit_transform(X)
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec32">LLE </h2>
|
||||
<h2 id="___sec33">LLE </h2>
|
||||
|
||||
<p>
|
||||
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
|
||||
@@ -1428,7 +1471,7 @@ these local relationships are best preserved (more details shortly).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec33">Other techniques </h2>
|
||||
<h2 id="___sec34">Other techniques </h2>
|
||||
|
||||
<p>
|
||||
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
|
||||
|
||||
@@ -129,15 +129,20 @@ div { text-align: justify; text-justify: inter-word; }
|
||||
('Proof of the PCA Theorem', 2, None, '___sec22'),
|
||||
('PCA Proof continued', 2, None, '___sec23'),
|
||||
('The final step', 2, None, '___sec24'),
|
||||
('Principal Component Analysis', 2, None, '___sec25'),
|
||||
('PCA and scikit-learn', 2, None, '___sec26'),
|
||||
('Back to the Cancer Data', 2, None, '___sec27'),
|
||||
('More on the PCA', 2, None, '___sec28'),
|
||||
('Incremental PCA', 2, None, '___sec29'),
|
||||
('Randomized PCA', 2, None, '___sec30'),
|
||||
('Kernel PCA', 2, None, '___sec31'),
|
||||
('LLE', 2, None, '___sec32'),
|
||||
('Other techniques', 2, None, '___sec33')]}
|
||||
('Geometric Interpretation and link with Singular Value '
|
||||
'Decomposition',
|
||||
2,
|
||||
None,
|
||||
'___sec25'),
|
||||
('Principal Component Analysis', 2, None, '___sec26'),
|
||||
('PCA and scikit-learn', 2, None, '___sec27'),
|
||||
('Back to the Cancer Data', 2, None, '___sec28'),
|
||||
('More on the PCA', 2, None, '___sec29'),
|
||||
('Incremental PCA', 2, None, '___sec30'),
|
||||
('Randomized PCA', 2, None, '___sec31'),
|
||||
('Kernel PCA', 2, None, '___sec32'),
|
||||
('LLE', 2, None, '___sec33'),
|
||||
('Other techniques', 2, None, '___sec34')]}
|
||||
end of tocinfo -->
|
||||
|
||||
<body>
|
||||
@@ -179,7 +184,7 @@ MathJax.Hub.Config({
|
||||
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
|
||||
<br>
|
||||
<p>
|
||||
<center><h4>Dec 28, 2019</h4></center> <!-- date -->
|
||||
<center><h4>Dec 29, 2019</h4></center> <!-- date -->
|
||||
<br>
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -546,6 +551,15 @@ applications.
|
||||
|
||||
<h2 id="___sec7">Basic ideas of the Principal Component Analysis (PCA) </h2>
|
||||
|
||||
<p>
|
||||
The principal component analysis deals with the problem of fitting a
|
||||
low-dimensional affine subspace \( S \) of dimension \( d \) much smaller than
|
||||
the totaldimension \( D \) of the problem at hand (our data
|
||||
set). Mathematically it can be formulated as a statistical problem or
|
||||
a geometric problem. In our discussion of the theorem for the
|
||||
classical PCA, we will stay with a statistical approach. This is also
|
||||
what set the scene historically which for the PCA.
|
||||
|
||||
<p>
|
||||
We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see below for its definition)
|
||||
|
||||
@@ -561,7 +575,7 @@ We have a data set defined by a design/feature matrix \( \boldsymbol{X} \) (see
|
||||
|
||||
<p>
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about
|
||||
the definition of the covariance and the correlation function.
|
||||
the definition of the covariance and the correlation function. These are quantities
|
||||
|
||||
<p>
|
||||
Suppose we have defined two vectors
|
||||
@@ -621,7 +635,9 @@ In the above example this is the function we constructed using <b>pandas</b>.
|
||||
<h2 id="___sec9">Correlation Function and Design/Feature Matrix </h2>
|
||||
|
||||
<p>
|
||||
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
In our derivation of the various regression algorithms like <b>Ordinary Least Squares</b> or <b>Ridge regression</b>
|
||||
we defined the design/feature matrix \( \boldsymbol{X} \) as
|
||||
|
||||
$$
|
||||
\boldsymbol{X}=\begin{bmatrix}
|
||||
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
|
||||
@@ -646,7 +662,11 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
|
||||
With these definitions, we can now rewrite our \( 2\times 2 \)
|
||||
correaltion/covariance matrix in terms of a moe general design/feature
|
||||
matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \)
|
||||
covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i=0,1,\dots,p-1 \)
|
||||
|
||||
$$
|
||||
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix}
|
||||
\mathrm{var}[\boldsymbol{x}_0] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_1] & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_2] & \dots & \dots & \mathrm{cov}[\boldsymbol{x}_0,\boldsymbol{x}_{p-1}]\\
|
||||
@@ -845,8 +865,8 @@ columns since all matrix elements in the design matrix were set to one
|
||||
<p>
|
||||
This means that the variance for these elements will be zero and will
|
||||
cause problems when we set up the correlation matrix. We can simply
|
||||
drop these elements as follows and then construct the correlation
|
||||
matrix.
|
||||
drop these elements and construct a correlation
|
||||
matrix without these elements.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1006,7 +1026,7 @@ X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>r
|
||||
Make thereafter a small Python code which plots the data. Note that the function <b>multivariate</b> returns also the covariance discussed above and that it is defined by dividing by \( n-1 \) instead of \( n \).
|
||||
|
||||
<p>
|
||||
Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.
|
||||
Now we are going to implement the PCA algorithm. We will break it down into various substeps.
|
||||
|
||||
<h3 id="___sec18">Compute the sample mean and center the data </h3>
|
||||
|
||||
@@ -1076,8 +1096,11 @@ Finally, try out your own PCA function with other data sets.
|
||||
<h2 id="___sec21">Classical PCA Theorem </h2>
|
||||
|
||||
<p>
|
||||
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \)
|
||||
each with dimension \( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
|
||||
We assume now that we have a design matrix \( \boldsymbol{X} \) which has been
|
||||
centered as discussed above. For the sake of simplicity we skip the
|
||||
overline symbol. The matrix is defined in terms of the various column
|
||||
vectors \( [\boldsymbol{x}_0,\boldsymbol{x}_1,\dots, \boldsymbol{x}_{p-1}] \) each with dimension
|
||||
\( \boldsymbol{x}\in {\mathbb{R}}^{n} \).
|
||||
|
||||
<p>
|
||||
We assume also that we have an orthogonal transformation \( \boldsymbol{W}\in {\mathbb{R}}^{p\times p} \). We define the reconstruction error (which is similar to the mean squared error we have seen before) as
|
||||
@@ -1089,7 +1112,13 @@ with \( \overline{\boldsymbol{x}}_i = \boldsymbol{W}\boldsymbol{z}_i \), where \
|
||||
\( \boldsymbol{Z}\in{\mathbb{R}}^{p\times n} \). When doing PCA we want to reduce this dimensionality.
|
||||
|
||||
<p>
|
||||
The PCA theorem states that minimizing the above reconstruction error corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors \( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
|
||||
The PCA theorem states that minimizing the above reconstruction error
|
||||
corresponds to setting \( \boldsymbol{W}=\boldsymbol{S} \), the orthogonal matrix which
|
||||
diagonalizes the empirical covariance(correlation) matrix. The optimal
|
||||
low-dimensional encoding of the data is then given by a set of vectors
|
||||
\( \boldsymbol{z}_i \) with at most \( l \) vectors, with \( l < < p \), defined by the
|
||||
orthogonal projection of the data onto the columns spanned by the
|
||||
eigenvectors of the covariance(correlations matrix).
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1150,7 +1179,10 @@ $$
|
||||
$$
|
||||
|
||||
<p>
|
||||
We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
|
||||
We are almost there, we have obtained a relation between minimizing
|
||||
the reconstruction error and the variance and the covariance
|
||||
matrix. Minimizing the error is equivalent to maximizing the variance
|
||||
of the projected data.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
@@ -1203,10 +1235,21 @@ discussion in chapter 12.2 of Murphy's text has also a nice link with
|
||||
the Singular Value Decomposition theorem. For categorical data, see
|
||||
chapter 12.4 and discussion therein.
|
||||
|
||||
<p>
|
||||
Additional part of the proof for the other eigenvectors will be added by mid January 2020.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec25">Principal Component Analysis </h2>
|
||||
<h2 id="___sec25">Geometric Interpretation and link with Singular Value Decomposition </h2>
|
||||
|
||||
<p>
|
||||
This material will be added by mid January 2020.
|
||||
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec26">Principal Component Analysis </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -1262,7 +1305,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
|
||||
<p>
|
||||
<!-- !split -->
|
||||
|
||||
<h2 id="___sec26">PCA and scikit-learn </h2>
|
||||
<h2 id="___sec27">PCA and scikit-learn </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn’s PCA class implements PCA using SVD decomposition just like we did before. The
|
||||
@@ -1294,7 +1337,7 @@ variance that lies along the axis of each principal component.
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec27">Back to the Cancer Data </h2>
|
||||
<h2 id="___sec28">Back to the Cancer Data </h2>
|
||||
We can now repeat the above but applied to real data, in this case our breast cancer data.
|
||||
Here we compute performance scores on the training data using logistic regression.
|
||||
<p>
|
||||
@@ -1335,7 +1378,7 @@ We see that our training data after the PCA decomposition has a performance simi
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec28">More on the PCA </h2>
|
||||
<h2 id="___sec29">More on the PCA </h2>
|
||||
|
||||
<p>
|
||||
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
|
||||
@@ -1365,7 +1408,7 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec29">Incremental PCA </h2>
|
||||
<h2 id="___sec30">Incremental PCA </h2>
|
||||
|
||||
<p>
|
||||
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
|
||||
@@ -1377,7 +1420,7 @@ instances arrive).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec30">Randomized PCA </h2>
|
||||
<h2 id="___sec31">Randomized PCA </h2>
|
||||
|
||||
<p>
|
||||
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
|
||||
@@ -1392,7 +1435,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec31">Kernel PCA </h2>
|
||||
<h2 id="___sec32">Kernel PCA </h2>
|
||||
<div class="alert alert-block alert-block alert-text-normal">
|
||||
<b></b>
|
||||
<p>
|
||||
@@ -1421,7 +1464,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec32">LLE </h2>
|
||||
<h2 id="___sec33">LLE </h2>
|
||||
|
||||
<p>
|
||||
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
|
||||
@@ -1433,7 +1476,7 @@ these local relationships are best preserved (more details shortly).
|
||||
<p>
|
||||
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
|
||||
|
||||
<h2 id="___sec33">Other techniques </h2>
|
||||
<h2 id="___sec34">Other techniques </h2>
|
||||
|
||||
<p>
|
||||
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
"<!-- Author: --> \n",
|
||||
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
|
||||
"\n",
|
||||
"Date: **Dec 28, 2019**\n",
|
||||
"Date: **Dec 29, 2019**\n",
|
||||
"\n",
|
||||
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
|
||||
"\n",
|
||||
@@ -409,6 +409,14 @@
|
||||
"\n",
|
||||
"## Basic ideas of the Principal Component Analysis (PCA)\n",
|
||||
"\n",
|
||||
"The principal component analysis deals with the problem of fitting a\n",
|
||||
"low-dimensional affine subspace $S$ of dimension $d$ much smaller than\n",
|
||||
"the totaldimension $D$ of the problem at hand (our data\n",
|
||||
"set). Mathematically it can be formulated as a statistical problem or\n",
|
||||
"a geometric problem. In our discussion of the theorem for the\n",
|
||||
"classical PCA, we will stay with a statistical approach. This is also\n",
|
||||
"what set the scene historically which for the PCA.\n",
|
||||
"\n",
|
||||
"We have a data set defined by a design/feature matrix $\\boldsymbol{X}$ (see below for its definition) \n",
|
||||
"* Each data point is determined by $p$ extrinsic (measurement) variables\n",
|
||||
"\n",
|
||||
@@ -419,7 +427,7 @@
|
||||
"## Introducing the Covariance and Correlation functions\n",
|
||||
"\n",
|
||||
"Before we discuss the PCA theorem, we need to remind ourselves about\n",
|
||||
"the definition of the covariance and the correlation function.\n",
|
||||
"the definition of the covariance and the correlation function. These are quantities \n",
|
||||
"\n",
|
||||
"Suppose we have defined two vectors\n",
|
||||
"$\\hat{x}$ and $\\hat{y}$ with $n$ elements each. The covariance matrix $\\boldsymbol{C}$ is defined as"
|
||||
@@ -535,7 +543,8 @@
|
||||
"\n",
|
||||
"## Correlation Function and Design/Feature Matrix\n",
|
||||
"\n",
|
||||
"In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix $\\boldsymbol{X}$ as"
|
||||
"In our derivation of the various regression algorithms like **Ordinary Least Squares** or **Ridge regression**\n",
|
||||
"we defined the design/feature matrix $\\boldsymbol{X}$ as"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -592,7 +601,10 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"With these definitions, we can now rewrite our $2\\times 2$ correaltion/covariance matrix in terms of a moe general design/feature matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. This leads to a $p\\times p$ covariance matrix for the vectors $\\boldsymbol{x}_i$ with $i =0,1,\\dots,p-1$"
|
||||
"With these definitions, we can now rewrite our $2\\times 2$\n",
|
||||
"correaltion/covariance matrix in terms of a moe general design/feature\n",
|
||||
"matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. This leads to a $p\\times p$\n",
|
||||
"covariance matrix for the vectors $\\boldsymbol{x}_i$ with $i=0,1,\\dots,p-1$"
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -848,8 +860,8 @@
|
||||
"\n",
|
||||
"This means that the variance for these elements will be zero and will\n",
|
||||
"cause problems when we set up the correlation matrix. We can simply\n",
|
||||
"drop these elements as follows and then construct the correlation\n",
|
||||
"matrix. \n",
|
||||
"drop these elements and construct a correlation\n",
|
||||
"matrix without these elements. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Rewriting the Covariance and/or Correlation Matrix\n",
|
||||
@@ -1104,7 +1116,7 @@
|
||||
"source": [
|
||||
"Make thereafter a small Python code which plots the data. Note that the function **multivariate** returns also the covariance discussed above and that it is defined by dividing by $n-1$ instead of $n$.\n",
|
||||
"\n",
|
||||
"Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.\n",
|
||||
"Now we are going to implement the PCA algorithm. We will break it down into various substeps.\n",
|
||||
"\n",
|
||||
"### Compute the sample mean and center the data\n",
|
||||
"\n",
|
||||
@@ -1208,8 +1220,11 @@
|
||||
"\n",
|
||||
"## Classical PCA Theorem\n",
|
||||
"\n",
|
||||
"We assume now that we have a design matrix $\\boldsymbol{X}$ which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors $[\\boldsymbol{x}_0,\\boldsymbol{x}_1,\\dots, \\boldsymbol{x}_{p-1}]$\n",
|
||||
"each with dimension $\\boldsymbol{x}\\in {\\mathbb{R}}^{n}$.\n",
|
||||
"We assume now that we have a design matrix $\\boldsymbol{X}$ which has been\n",
|
||||
"centered as discussed above. For the sake of simplicity we skip the\n",
|
||||
"overline symbol. The matrix is defined in terms of the various column\n",
|
||||
"vectors $[\\boldsymbol{x}_0,\\boldsymbol{x}_1,\\dots, \\boldsymbol{x}_{p-1}]$ each with dimension\n",
|
||||
"$\\boldsymbol{x}\\in {\\mathbb{R}}^{n}$.\n",
|
||||
"\n",
|
||||
"We assume also that we have an orthogonal transformation $\\boldsymbol{W}\\in {\\mathbb{R}}^{p\\times p}$. We define the reconstruction error (which is similar to the mean squared error we have seen before) as"
|
||||
]
|
||||
@@ -1230,7 +1245,13 @@
|
||||
"with $\\overline{\\boldsymbol{x}}_i = \\boldsymbol{W}\\boldsymbol{z}_i$, where $\\boldsymbol{z}_i$ is a row vector with dimension ${\\mathbb{R}}^{n}$ of the matrix\n",
|
||||
"$\\boldsymbol{Z}\\in{\\mathbb{R}}^{p\\times n}$. When doing PCA we want to reduce this dimensionality. \n",
|
||||
"\n",
|
||||
"The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\\boldsymbol{W}=\\boldsymbol{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $\\boldsymbol{z}_i$ with at most $l$ vectors, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).\n",
|
||||
"The PCA theorem states that minimizing the above reconstruction error\n",
|
||||
"corresponds to setting $\\boldsymbol{W}=\\boldsymbol{S}$, the orthogonal matrix which\n",
|
||||
"diagonalizes the empirical covariance(correlation) matrix. The optimal\n",
|
||||
"low-dimensional encoding of the data is then given by a set of vectors\n",
|
||||
"$\\boldsymbol{z}_i$ with at most $l$ vectors, with $l << p$, defined by the\n",
|
||||
"orthogonal projection of the data onto the columns spanned by the\n",
|
||||
"eigenvectors of the covariance(correlations matrix).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -1371,7 +1392,10 @@
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data. \n",
|
||||
"We are almost there, we have obtained a relation between minimizing\n",
|
||||
"the reconstruction error and the variance and the covariance\n",
|
||||
"matrix. Minimizing the error is equivalent to maximizing the variance\n",
|
||||
"of the projected data.\n",
|
||||
"\n",
|
||||
"## The final step\n",
|
||||
"\n",
|
||||
@@ -1460,9 +1484,11 @@
|
||||
"the Singular Value Decomposition theorem. For categorical data, see\n",
|
||||
"chapter 12.4 and discussion therein.\n",
|
||||
"\n",
|
||||
"Additional part of the proof for the other eigenvectors will be added by mid January 2020.\n",
|
||||
"\n",
|
||||
"## Geometric Interpretation and link with Singular Value Decomposition\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"This material will be added by mid January 2020.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"## Principal Component Analysis\n",
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -337,6 +337,14 @@ applications.
|
||||
!split
|
||||
===== Basic ideas of the Principal Component Analysis (PCA) =====
|
||||
|
||||
The principal component analysis deals with the problem of fitting a
|
||||
low-dimensional affine subspace $S$ of dimension $d$ much smaller than
|
||||
the totaldimension $D$ of the problem at hand (our data
|
||||
set). Mathematically it can be formulated as a statistical problem or
|
||||
a geometric problem. In our discussion of the theorem for the
|
||||
classical PCA, we will stay with a statistical approach. This is also
|
||||
what set the scene historically which for the PCA.
|
||||
|
||||
We have a data set defined by a design/feature matrix $\bm{X}$ (see below for its definition)
|
||||
* Each data point is determined by $p$ extrinsic (measurement) variables
|
||||
* We may want to ask the following question: Are there fewer intrinsic variables (say $d << p$) that still approximately describe the data?
|
||||
@@ -347,7 +355,7 @@ We have a data set defined by a design/feature matrix $\bm{X}$ (see below for it
|
||||
===== Introducing the Covariance and Correlation functions =====
|
||||
|
||||
Before we discuss the PCA theorem, we need to remind ourselves about
|
||||
the definition of the covariance and the correlation function.
|
||||
the definition of the covariance and the correlation function. These are quantities
|
||||
|
||||
Suppose we have defined two vectors
|
||||
$\hat{x}$ and $\hat{y}$ with $n$ elements each. The covariance matrix $\bm{C}$ is defined as
|
||||
@@ -409,7 +417,9 @@ In the above example this is the function we constructed using _pandas_.
|
||||
!split
|
||||
===== Correlation Function and Design/Feature Matrix =====
|
||||
|
||||
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix $\bm{X}$ as
|
||||
In our derivation of the various regression algorithms like _Ordinary Least Squares_ or _Ridge regression_
|
||||
we defined the design/feature matrix $\bm{X}$ as
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{X}=\begin{bmatrix}
|
||||
@@ -437,7 +447,11 @@ with a given vector
|
||||
\]
|
||||
!et
|
||||
|
||||
With these definitions, we can now rewrite our $2\times 2$ correaltion/covariance matrix in terms of a moe general design/feature matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$ covariance matrix for the vectors $\bm{x}_i$ with $i =0,1,\dots,p-1$
|
||||
With these definitions, we can now rewrite our $2\times 2$
|
||||
correaltion/covariance matrix in terms of a moe general design/feature
|
||||
matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$
|
||||
covariance matrix for the vectors $\bm{x}_i$ with $i=0,1,\dots,p-1$
|
||||
|
||||
!bt
|
||||
\[
|
||||
\bm{C}[\bm{x}] = \begin{bmatrix}
|
||||
@@ -624,8 +638,8 @@ columns since all matrix elements in the design matrix were set to one
|
||||
|
||||
This means that the variance for these elements will be zero and will
|
||||
cause problems when we set up the correlation matrix. We can simply
|
||||
drop these elements as follows and then construct the correlation
|
||||
matrix.
|
||||
drop these elements and construct a correlation
|
||||
matrix without these elements.
|
||||
|
||||
|
||||
!split
|
||||
@@ -773,7 +787,7 @@ X = np.random.multivariate_normal(mean, cov, n)
|
||||
|
||||
Make thereafter a small Python code which plots the data. Note that the function _multivariate_ returns also the covariance discussed above and that it is defined by dividing by $n-1$ instead of $n$.
|
||||
|
||||
Now we are going to implement the PCA algorithm. We will break it down into sub-steps and across multiple cells.
|
||||
Now we are going to implement the PCA algorithm. We will break it down into various substeps.
|
||||
|
||||
=== Compute the sample mean and center the data ===
|
||||
|
||||
@@ -835,8 +849,11 @@ Finally, try out your own PCA function with other data sets.
|
||||
!split
|
||||
===== Classical PCA Theorem =====
|
||||
|
||||
We assume now that we have a design matrix $\bm{X}$ which has been centered as discussed above. For the sake of simplicity we skip the overline symbol. The matrix is defined in terms of the various column vectors $[\bm{x}_0,\bm{x}_1,\dots, \bm{x}_{p-1}]$
|
||||
each with dimension $\bm{x}\in {\mathbb{R}}^{n}$.
|
||||
We assume now that we have a design matrix $\bm{X}$ which has been
|
||||
centered as discussed above. For the sake of simplicity we skip the
|
||||
overline symbol. The matrix is defined in terms of the various column
|
||||
vectors $[\bm{x}_0,\bm{x}_1,\dots, \bm{x}_{p-1}]$ each with dimension
|
||||
$\bm{x}\in {\mathbb{R}}^{n}$.
|
||||
|
||||
We assume also that we have an orthogonal transformation $\bm{W}\in {\mathbb{R}}^{p\times p}$. We define the reconstruction error (which is similar to the mean squared error we have seen before) as
|
||||
!bt
|
||||
@@ -847,7 +864,13 @@ J(\bm{W},\bm{Z}) = \frac{1}{n}\sum_i (\bm{x}_i - \overline{\bm{x}}_i)^2,
|
||||
with $\overline{\bm{x}}_i = \bm{W}\bm{z}_i$, where $\bm{z}_i$ is a row vector with dimension ${\mathbb{R}}^{n}$ of the matrix
|
||||
$\bm{Z}\in{\mathbb{R}}^{p\times n}$. When doing PCA we want to reduce this dimensionality.
|
||||
|
||||
The PCA theorem states that minimizing the above reconstruction error corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which diagonalizes the empirical covariance(correlation) matrix. The optimal low-dimensional encoding of the data is then given by a set of vectors $\bm{z}_i$ with at most $l$ vectors, with $l << p$, defined by the orthogonal projection of the data onto the columns spanned by the eigenvectors of the covariance(correlations matrix).
|
||||
The PCA theorem states that minimizing the above reconstruction error
|
||||
corresponds to setting $\bm{W}=\bm{S}$, the orthogonal matrix which
|
||||
diagonalizes the empirical covariance(correlation) matrix. The optimal
|
||||
low-dimensional encoding of the data is then given by a set of vectors
|
||||
$\bm{z}_i$ with at most $l$ vectors, with $l << p$, defined by the
|
||||
orthogonal projection of the data onto the columns spanned by the
|
||||
eigenvectors of the covariance(correlations matrix).
|
||||
|
||||
|
||||
|
||||
@@ -912,7 +935,10 @@ we have thus that
|
||||
\]
|
||||
!et
|
||||
|
||||
We are almost there, we have obtained a relation between minimizing the reconstruction error and the variance and the covariance matrix. Minimizing the error is equivalent to maximizing the variance of the projected data.
|
||||
We are almost there, we have obtained a relation between minimizing
|
||||
the reconstruction error and the variance and the covariance
|
||||
matrix. Minimizing the error is equivalent to maximizing the variance
|
||||
of the projected data.
|
||||
|
||||
!split
|
||||
===== The final step =====
|
||||
@@ -965,9 +991,12 @@ discussion in chapter 12.2 of Murphy's text has also a nice link with
|
||||
the Singular Value Decomposition theorem. For categorical data, see
|
||||
chapter 12.4 and discussion therein.
|
||||
|
||||
Additional part of the proof for the other eigenvectors will be added by mid January 2020.
|
||||
|
||||
!split
|
||||
===== Geometric Interpretation and link with Singular Value Decomposition =====
|
||||
|
||||
|
||||
This material will be added by mid January 2020.
|
||||
|
||||
|
||||
!split
|
||||
|
||||
Reference in New Issue
Block a user