lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating dim red material

Browse Source
This commit is contained in:
mhjensen
2019-10-20 21:42:36 +02:00
parent 57d6efae1c
commit c6c476bd42
18 changed files with 1560 additions and 427 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/DimRed/html/._DimRed-bs000.html
+29 -24
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -174,7 +179,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 19, 2019</h4></center> <!-- date -->
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
<br>
<p>
@@ -198,7 +203,7 @@ MathJax.Hub.Config({
<li><a href="._DimRed-bs008.html">9</a></li>
<li><a href="._DimRed-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs001.html
+28 -23
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -198,7 +203,7 @@ data.
<li><a href="._DimRed-bs009.html">10</a></li>
<li><a href="._DimRed-bs010.html">11</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs002.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs002.html
+28 -23
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -197,7 +202,7 @@ ensures that all features are exactly between \( 0 \) and \( 1 \). The
<li><a href="._DimRed-bs010.html">11</a></li>
<li><a href="._DimRed-bs011.html">12</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs003.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs003.html
+28 -23
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -200,7 +205,7 @@ techniques.
<li><a href="._DimRed-bs011.html">12</a></li>
<li><a href="._DimRed-bs012.html">13</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs004.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs004.html
+28 -23
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -275,7 +280,7 @@ svm<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<li><a href="._DimRed-bs012.html">13</a></li>
<li><a href="._DimRed-bs013.html">14</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs005.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs005.html
+28 -23
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -225,7 +230,7 @@ svm<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<li><a href="._DimRed-bs013.html">14</a></li>
<li><a href="._DimRed-bs014.html">15</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs006.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs006.html
+28 -23
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -204,7 +209,7 @@ logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<li><a href="._DimRed-bs014.html">15</a></li>
<li><a href="._DimRed-bs015.html">16</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs007.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs007.html
+61 -42
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -200,27 +205,41 @@ plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic">#print eigvalues of correlation matrix</span>
EigValues, EigVectors <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>eig(correlation_matrix)
<span style="color: #008000; font-weight: bold">print</span>(EigValues)
<span style="color: #408080; font-style: italic">#split into train and test and then scale thereafter</span>
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000; font-weight: bold">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000; font-weight: bold">print</span>(X_test<span style="color: #666666">.</span>shape)
logreg <span style="color: #666666">=</span> LogisticRegression()
logreg<span style="color: #666666">.</span>fit(X_train, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy from Logistic Regression: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test,y_test)))
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> MinMaxScaler, StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
</pre></div>
<p>
<!-- todo: add more text in order to explain what is done here, discuss the correlation matrix -->
In the above example we note two things. In the first plot we display
the overlap of benign and malignant tumors as functions of the various
features in the Wisconsing breast cancer data set. We see that for
some of the features we can distinguish clearly the benign and
malignant cases while for other features we cannot. This can point to
us which features may be of greater interest when we wish to classify
a benign or not benign tumour.
<p>
In the second figure we have computed the so-called correlation
matrix, which in our case with thirty features becomes a \( 30\times 30 \)
matrix.
<p>
We constructed this matrix using <b>pandas</b> via the statements
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>cancerpd <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(cancer<span style="color: #666666">.</span>data, columns<span style="color: #666666">=</span>cancer<span style="color: #666666">.</span>feature_names)
</pre></div>
<p>
and then
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>correlation_matrix <span style="color: #666666">=</span> cancerpd<span style="color: #666666">.</span>corr()<span style="color: #666666">.</span>round(<span style="color: #666666">1</span>)
</pre></div>
<p>
Diagonalizing this matrix we can in turn say something about which
features are of relevance and which are not. But before we proceed we
need to define covariance and correlation matrices. This leads us to
the classical Principal Component Analysis (PCA) theorem with
applications.
<p>
<p>
@@ -245,7 +264,7 @@ logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<li><a href="._DimRed-bs015.html">16</a></li>
<li><a href="._DimRed-bs016.html">17</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs008.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs008.html
+28 -23
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -181,7 +186,7 @@ MathJax.Hub.Config({
<li><a href="._DimRed-bs016.html">17</a></li>
<li><a href="._DimRed-bs017.html">18</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs009.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/._DimRed-bs009.html
+70 -23
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -157,6 +162,48 @@ MathJax.Hub.Config({
<h2 id="___sec8" class="anchor">Introducing the Covariance and Correlation functions </h2>
<p>
Suppose we have defined two vectors
$\hat{x} and \hat{y} with \( n \) elements each. The covariance matrix $\boldsymbol{C}is defined as
$$
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} cov[\boldsymbol{x},\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
cov[\boldsymbol{y},\boldsymbol{x}] & cov[\boldsymbol{y},\boldsymbol{y}] \\
\end{bmatrix},
$$
where for example
$$
cov[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
$$
With this definition and recalling that the variance is
$$
var[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
$$
we can rewrite the covariance matrix in this case as
$$
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} var[\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{y}] \\
\end{bmatrix},
$$
<p>
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function
$$
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var\boldsymbol{y}]}}.
$$
The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as
$$
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
\end{bmatrix},
$$
<p>
In the above example this is the function we constructed using <b>pandas</b>.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
@@ -182,7 +229,7 @@ MathJax.Hub.Config({
<li><a href="._DimRed-bs017.html">18</a></li>
<li><a href="._DimRed-bs018.html">19</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs010.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/DimRed-bs.html
+29 -24
View File
@@ -72,17 +72,21 @@ Automatically generated HTML file from DocOnce source
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -129,17 +133,18 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._DimRed-bs007.html#___sec6" style="font-size: 80%;">Why should we think of reducing the dimensionality</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs008.html#___sec7" style="font-size: 80%;">Basic ideas of the Principal Component Analysis (PCA)</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs009.html#___sec8" style="font-size: 80%;">Introducing the Covariance and Correlation functions</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">Other techniques</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs010.html#___sec9" style="font-size: 80%;">Correlation Function and Design/Feature Matrix</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs011.html#___sec10" style="font-size: 80%;">Classical PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs012.html#___sec11" style="font-size: 80%;">Prof of the PCA Theorem</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs013.html#___sec12" style="font-size: 80%;">Getting started with PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs014.html#___sec13" style="font-size: 80%;">Principal Component Analysis</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs015.html#___sec14" style="font-size: 80%;">PCA and scikit-learn</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs016.html#___sec15" style="font-size: 80%;">More on the PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs017.html#___sec16" style="font-size: 80%;">Incremental PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs018.html#___sec17" style="font-size: 80%;">Randomized PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs019.html#___sec18" style="font-size: 80%;">Kernel PCA</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs020.html#___sec19" style="font-size: 80%;">LLE</a></li>
<!-- navigation toc: --> <li><a href="._DimRed-bs021.html#___sec20" style="font-size: 80%;">Other techniques</a></li>
</ul>
</li>
@@ -174,7 +179,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 19, 2019</h4></center> <!-- date -->
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
<br>
<p>
@@ -198,7 +203,7 @@ MathJax.Hub.Config({
<li><a href="._DimRed-bs008.html">9</a></li>
<li><a href="._DimRed-bs009.html">10</a></li>
<li><a href="">...</a></li>
<li><a href="._DimRed-bs020.html">21</a></li>
<li><a href="._DimRed-bs021.html">22</a></li>
<li><a href="._DimRed-bs001.html">&raquo;</a></li>
</ul>
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/DimRed-reveal.html
+244 -31
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Oct 19, 2019</h4></center> <!-- date -->
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
<br>
<p>
@@ -469,27 +469,41 @@ plt.show()
<span style="color: #228B22">#print eigvalues of correlation matrix</span>
EigValues, EigVectors = np.linalg.eig(correlation_matrix)
<span style="color: #8B008B; font-weight: bold">print</span>(EigValues)
<span style="color: #228B22">#split into train and test and then scale thereafter</span>
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=<span style="color: #B452CD">0</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(X_train.shape)
<span style="color: #8B008B; font-weight: bold">print</span>(X_test.shape)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy from Logistic Regression: {:.2f}&quot;</span>.format(logreg.score(X_test,y_test)))
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> MinMaxScaler, StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
logreg.fit(X_train_scaled, y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy scaled data: {:.2f}&quot;</span>.format(logreg.score(X_test_scaled,y_test)))
</pre></div>
<p>
<!-- todo: add more text in order to explain what is done here, discuss the correlation matrix -->
In the above example we note two things. In the first plot we display
the overlap of benign and malignant tumors as functions of the various
features in the Wisconsing breast cancer data set. We see that for
some of the features we can distinguish clearly the benign and
malignant cases while for other features we cannot. This can point to
us which features may be of greater interest when we wish to classify
a benign or not benign tumour.
<p>
In the second figure we have computed the so-called correlation
matrix, which in our case with thirty features becomes a \( 30\times 30 \)
matrix.
<p>
We constructed this matrix using <b>pandas</b> via the statements
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)
</pre></div>
<p>
and then
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>correlation_matrix = cancerpd.corr().round(<span style="color: #B452CD">1</span>)
</pre></div>
<p>
Diagonalizing this matrix we can in turn say something about which
features are of relevance and which are not. But before we proceed we
need to define covariance and correlation matrices. This leads us to
the classical Principal Component Analysis (PCA) theorem with
applications.
</section>
@@ -500,21 +514,163 @@ logreg.fit(X_train_scaled, y_train)
<section>
<h2 id="___sec8">Introducing the Covariance and Correlation functions </h2>
<p>
Suppose we have defined two vectors
$\hat{x} and \hat{y} with \( n \) elements each. The covariance matrix $\boldsymbol{C}is defined as
<p>&nbsp;<br>
$$
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} cov[\boldsymbol{x},\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
cov[\boldsymbol{y},\boldsymbol{x}] & cov[\boldsymbol{y},\boldsymbol{y}] \\
\end{bmatrix},
$$
<p>&nbsp;<br>
where for example
<p>&nbsp;<br>
$$
cov[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
$$
<p>&nbsp;<br>
With this definition and recalling that the variance is
<p>&nbsp;<br>
$$
var[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
$$
<p>&nbsp;<br>
we can rewrite the covariance matrix in this case as
<p>&nbsp;<br>
$$
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} var[\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{y}] \\
\end{bmatrix},
$$
<p>&nbsp;<br>
<p>
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function
<p>&nbsp;<br>
$$
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var\boldsymbol{y}]}}.
$$
<p>&nbsp;<br>
The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as
<p>&nbsp;<br>
$$
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
\end{bmatrix},
$$
<p>&nbsp;<br>
<p>
In the above example this is the function we constructed using <b>pandas</b>.
</section>
<section>
<h2 id="___sec9">Classical PCA Theorem </h2>
<h2 id="___sec9">Correlation Function and Design/Feature Matrix </h2>
<p>
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
<p>&nbsp;<br>
$$
\boldsymbol{X}=\begin{bmatrix}
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
\dots & \dots & \dots & \dots \dots & \dots \\
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
\end{bmatrix},
$$
<p>&nbsp;<br>
with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the
entries \( n \) being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
<p>&nbsp;<br>
$$
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_0 & \boldsymbol{x}_0 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
$$
<p>&nbsp;<br>
with a given vector
<p>&nbsp;<br>
$$
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
$$
<p>&nbsp;<br>
<p>
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
<p>&nbsp;<br>
$$
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] \\
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{x}_{p-1}] \\
\end{bmatrix},
$$
<p>&nbsp;<br>
<p>
The Numpy function <b>np.cov</b> calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values.
The following simple function uses the <b>np.vstack</b> function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \hat{W} \)
<p>&nbsp;<br>
$$
\hat{W} = \begin{bmatrix} x_0 & y_0 \\
x_1 & y_1 \\
x_2 & y_2\\
\dots & \dots \\
x_{n-2} & y_{n-2}\\
x_{n-1} & y_{n-1} &
\end{bmatrix},
$$
<p>&nbsp;<br>
<p>
which in turn is converted into into the \( 3\times 3 \) covariance matrix
\( \hat{\Sigma} \) via the Numpy function <b>np.cov()</b>. We note that we can also calculate
the mean value of each set of samples \( \hat{x} \) etc using the Numpy
function <b>np.mean(x)</b>. We can also extract the eigenvalues of the
covariance matrix through the <b>np.linalg.eig()</b> function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
n = <span style="color: #B452CD">100</span>
x = np.random.normal(size=n)
<span style="color: #8B008B; font-weight: bold">print</span>(np.mean(x))
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.normal(size=n)
<span style="color: #8B008B; font-weight: bold">print</span>(np.mean(y))
z = x**<span style="color: #B452CD">3</span>+np.random.normal(size=n)
<span style="color: #8B008B; font-weight: bold">print</span>(np.mean(z))
W = np.vstack((x, y, z))
Sigma = np.cov(W)
<span style="color: #8B008B; font-weight: bold">print</span>(Sigma)
Eigvals, Eigvecs = np.linalg.eig(Sigma)
<span style="color: #8B008B; font-weight: bold">print</span>(Eigvals)
</pre></div>
</section>
<section>
<h2 id="___sec10">Prof of the PCA Theorem </h2>
<h2 id="___sec10">Classical PCA Theorem </h2>
</section>
<section>
<h2 id="___sec11">Getting started with PCA </h2>
<h2 id="___sec11">Prof of the PCA Theorem </h2>
</section>
<section>
<h2 id="___sec12">Getting started with PCA </h2>
<p>
@@ -530,7 +686,7 @@ X_pca = pca.transform(X_train_scaled)
<section>
<h2 id="___sec12">Principal Component Analysis </h2>
<h2 id="___sec13">Principal Component Analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -567,7 +723,7 @@ X2D = X_centered.dot(W2)
<section>
<h2 id="___sec13">PCA and scikit-learn </h2>
<h2 id="___sec14">PCA and scikit-learn </h2>
<p>
Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -598,7 +754,7 @@ More material to come here.
<section>
<h2 id="___sec14">More on the PCA </h2>
<h2 id="___sec15">More on the PCA </h2>
<p>
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -629,7 +785,7 @@ X_reduced = pca.fit_transform(X)
<section>
<h2 id="___sec15">Incremental PCA </h2>
<h2 id="___sec16">Incremental PCA </h2>
<p>
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -641,7 +797,7 @@ instances arrive).
<section>
<h2 id="___sec16">Randomized PCA </h2>
<h2 id="___sec17">Randomized PCA </h2>
<p>
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -655,7 +811,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
<section>
<h2 id="___sec17">Kernel PCA </h2>
<h2 id="___sec18">Kernel PCA </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -681,7 +837,7 @@ X_reduced = rbf_pca.fit_transform(X)
<section>
<h2 id="___sec18">LLE </h2>
<h2 id="___sec19">LLE </h2>
<p>
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -693,7 +849,7 @@ these local relationships are best preserved (more details shortly).
<section>
<h2 id="___sec19">Other techniques </h2>
<h2 id="___sec20">Other techniques </h2>
<p>
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
@@ -707,6 +863,63 @@ Here are some of the most popular:
<p><li> <b>t-Distributed Stochastic Neighbor Embedding</b> (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).</li>
<p><li> Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.</li>
</ul>
<p>
Here are other examples where we use the <b>DataFrame</b> functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix
of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">IPython.display</span> <span style="color: #8B008B; font-weight: bold">import</span> display
np.random.seed(<span style="color: #B452CD">100</span>)
<span style="color: #228B22"># setting up a 10 x 5 matrix</span>
rows = <span style="color: #B452CD">10</span>
cols = <span style="color: #B452CD">5</span>
a = np.random.randn(rows,cols)
df = pd.DataFrame(a)
display(df)
<span style="color: #8B008B; font-weight: bold">print</span>(df.mean())
<span style="color: #8B008B; font-weight: bold">print</span>(df.std())
display(df**<span style="color: #B452CD">2</span>)
</pre></div>
<p>
Thereafter we can select specific columns only and plot final results
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>df.columns = [<span style="color: #CD5555">&#39;First&#39;</span>, <span style="color: #CD5555">&#39;Second&#39;</span>, <span style="color: #CD5555">&#39;Third&#39;</span>, <span style="color: #CD5555">&#39;Fourth&#39;</span>, <span style="color: #CD5555">&#39;Fifth&#39;</span>]
df.index = np.arange(<span style="color: #B452CD">10</span>)
display(df)
<span style="color: #8B008B; font-weight: bold">print</span>(df[<span style="color: #CD5555">&#39;Second&#39;</span>].mean() )
<span style="color: #8B008B; font-weight: bold">print</span>(df.info())
<span style="color: #8B008B; font-weight: bold">print</span>(df.describe())
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">pylab</span> <span style="color: #8B008B; font-weight: bold">import</span> plt, mpl
plt.style.use(<span style="color: #CD5555">&#39;seaborn&#39;</span>)
mpl.rcParams[<span style="color: #CD5555">&#39;font.family&#39;</span>] = <span style="color: #CD5555">&#39;serif&#39;</span>
df.cumsum().plot(lw=<span style="color: #B452CD">2.0</span>, figsize=(<span style="color: #B452CD">10</span>,<span style="color: #B452CD">6</span>))
plt.show()
df.plot.bar(figsize=(<span style="color: #B452CD">10</span>,<span style="color: #B452CD">6</span>), rot=<span style="color: #B452CD">15</span>)
plt.show()
</pre></div>
<p>
We can produce a \( 4\times 4 \) matrix
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span>b = np.arange(<span style="color: #B452CD">16</span>).reshape((<span style="color: #B452CD">4</span>,<span style="color: #B452CD">4</span>))
<span style="color: #8B008B; font-weight: bold">print</span>(b)
df1 = pd.DataFrame(b)
<span style="color: #8B008B; font-weight: bold">print</span>(df1)
</pre></div>
<p>
and many other operations.
</section>
doc/pub/DimRed/html/DimRed-solarized.html
+235 -43
View File
@@ -92,17 +92,21 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -144,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 19, 2019</h4></center> <!-- date -->
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -462,27 +466,41 @@ plt.show()
<span style="color: #228B22">#print eigvalues of correlation matrix</span>
EigValues, EigVectors = np.linalg.eig(correlation_matrix)
<span style="color: #8B008B; font-weight: bold">print</span>(EigValues)
<span style="color: #228B22">#split into train and test and then scale thereafter</span>
X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=<span style="color: #B452CD">0</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(X_train.shape)
<span style="color: #8B008B; font-weight: bold">print</span>(X_test.shape)
logreg = LogisticRegression()
logreg.fit(X_train, y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy from Logistic Regression: {:.2f}&quot;</span>.format(logreg.score(X_test,y_test)))
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> MinMaxScaler, StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
logreg.fit(X_train_scaled, y_train)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Test set accuracy scaled data: {:.2f}&quot;</span>.format(logreg.score(X_test_scaled,y_test)))
</pre></div>
<p>
<!-- todo: add more text in order to explain what is done here, discuss the correlation matrix -->
In the above example we note two things. In the first plot we display
the overlap of benign and malignant tumors as functions of the various
features in the Wisconsing breast cancer data set. We see that for
some of the features we can distinguish clearly the benign and
malignant cases while for other features we cannot. This can point to
us which features may be of greater interest when we wish to classify
a benign or not benign tumour.
<p>
In the second figure we have computed the so-called correlation
matrix, which in our case with thirty features becomes a \( 30\times 30 \)
matrix.
<p>
We constructed this matrix using <b>pandas</b> via the statements
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)
</pre></div>
<p>
and then
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>correlation_matrix = cancerpd.corr().round(<span style="color: #B452CD">1</span>)
</pre></div>
<p>
Diagonalizing this matrix we can in turn say something about which
features are of relevance and which are not. But before we proceed we
need to define covariance and correlation matrices. This leads us to
the classical Principal Component Analysis (PCA) theorem with
applications.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -495,19 +513,138 @@ logreg.fit(X_train_scaled, y_train)
<h2 id="___sec8">Introducing the Covariance and Correlation functions </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
Suppose we have defined two vectors
$\hat{x} and \hat{y} with \( n \) elements each. The covariance matrix $\boldsymbol{C}is defined as
$$
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} cov[\boldsymbol{x},\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
cov[\boldsymbol{y},\boldsymbol{x}] & cov[\boldsymbol{y},\boldsymbol{y}] \\
\end{bmatrix},
$$
<h2 id="___sec9">Classical PCA Theorem </h2>
where for example
$$
cov[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
$$
With this definition and recalling that the variance is
$$
var[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
$$
we can rewrite the covariance matrix in this case as
$$
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} var[\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{y}] \\
\end{bmatrix},
$$
<p>
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function
$$
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var\boldsymbol{y}]}}.
$$
The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as
$$
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
\end{bmatrix},
$$
<p>
In the above example this is the function we constructed using <b>pandas</b>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Prof of the PCA Theorem </h2>
<h2 id="___sec9">Correlation Function and Design/Feature Matrix </h2>
<p>
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
$$
\boldsymbol{X}=\begin{bmatrix}
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
\dots & \dots & \dots & \dots \dots & \dots \\
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
\end{bmatrix},
$$
with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the
entries \( n \) being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
$$
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_0 & \boldsymbol{x}_0 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
$$
with a given vector
$$
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
$$
<p>
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
$$
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] \\
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{x}_{p-1}] \\
\end{bmatrix},
$$
<p>
The Numpy function <b>np.cov</b> calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values.
The following simple function uses the <b>np.vstack</b> function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \hat{W} \)
$$
\hat{W} = \begin{bmatrix} x_0 & y_0 \\
x_1 & y_1 \\
x_2 & y_2\\
\dots & \dots \\
x_{n-2} & y_{n-2}\\
x_{n-1} & y_{n-1} &
\end{bmatrix},
$$
<p>
which in turn is converted into into the \( 3\times 3 \) covariance matrix
\( \hat{\Sigma} \) via the Numpy function <b>np.cov()</b>. We note that we can also calculate
the mean value of each set of samples \( \hat{x} \) etc using the Numpy
function <b>np.mean(x)</b>. We can also extract the eigenvalues of the
covariance matrix through the <b>np.linalg.eig()</b> function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># Importing various packages</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
n = <span style="color: #B452CD">100</span>
x = np.random.normal(size=n)
<span style="color: #8B008B; font-weight: bold">print</span>(np.mean(x))
y = <span style="color: #B452CD">4</span>+<span style="color: #B452CD">3</span>*x+np.random.normal(size=n)
<span style="color: #8B008B; font-weight: bold">print</span>(np.mean(y))
z = x**<span style="color: #B452CD">3</span>+np.random.normal(size=n)
<span style="color: #8B008B; font-weight: bold">print</span>(np.mean(z))
W = np.vstack((x, y, z))
Sigma = np.cov(W)
<span style="color: #8B008B; font-weight: bold">print</span>(Sigma)
Eigvals, Eigvecs = np.linalg.eig(Sigma)
<span style="color: #8B008B; font-weight: bold">print</span>(Eigvals)
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Classical PCA Theorem </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Getting started with PCA </h2>
<h2 id="___sec11">Prof of the PCA Theorem </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Getting started with PCA </h2>
<p>
@@ -522,7 +659,7 @@ X_pca = pca.transform(X_train_scaled)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Principal Component Analysis </h2>
<h2 id="___sec13">Principal Component Analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -558,7 +695,7 @@ X2D = X_centered.dot(W2)
<p>
<!-- !split -->
<h2 id="___sec13">PCA and scikit-learn </h2>
<h2 id="___sec14">PCA and scikit-learn </h2>
<p>
Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -589,7 +726,7 @@ More material to come here.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">More on the PCA </h2>
<h2 id="___sec15">More on the PCA </h2>
<p>
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -619,7 +756,7 @@ X_reduced = pca.fit_transform(X)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Incremental PCA </h2>
<h2 id="___sec16">Incremental PCA </h2>
<p>
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -631,7 +768,7 @@ instances arrive).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Randomized PCA </h2>
<h2 id="___sec17">Randomized PCA </h2>
<p>
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -646,7 +783,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Kernel PCA </h2>
<h2 id="___sec18">Kernel PCA </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -675,7 +812,7 @@ X_reduced = rbf_pca.fit_transform(X)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">LLE </h2>
<h2 id="___sec19">LLE </h2>
<p>
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -687,7 +824,7 @@ these local relationships are best preserved (more details shortly).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Other techniques </h2>
<h2 id="___sec20">Other techniques </h2>
<p>
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
@@ -702,6 +839,61 @@ Here are some of the most popular:
<li> Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.</li>
</ul>
Here are other examples where we use the <b>DataFrame</b> functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix
of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">IPython.display</span> <span style="color: #8B008B; font-weight: bold">import</span> display
np.random.seed(<span style="color: #B452CD">100</span>)
<span style="color: #228B22"># setting up a 10 x 5 matrix</span>
rows = <span style="color: #B452CD">10</span>
cols = <span style="color: #B452CD">5</span>
a = np.random.randn(rows,cols)
df = pd.DataFrame(a)
display(df)
<span style="color: #8B008B; font-weight: bold">print</span>(df.mean())
<span style="color: #8B008B; font-weight: bold">print</span>(df.std())
display(df**<span style="color: #B452CD">2</span>)
</pre></div>
<p>
Thereafter we can select specific columns only and plot final results
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>df.columns = [<span style="color: #CD5555">&#39;First&#39;</span>, <span style="color: #CD5555">&#39;Second&#39;</span>, <span style="color: #CD5555">&#39;Third&#39;</span>, <span style="color: #CD5555">&#39;Fourth&#39;</span>, <span style="color: #CD5555">&#39;Fifth&#39;</span>]
df.index = np.arange(<span style="color: #B452CD">10</span>)
display(df)
<span style="color: #8B008B; font-weight: bold">print</span>(df[<span style="color: #CD5555">&#39;Second&#39;</span>].mean() )
<span style="color: #8B008B; font-weight: bold">print</span>(df.info())
<span style="color: #8B008B; font-weight: bold">print</span>(df.describe())
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">pylab</span> <span style="color: #8B008B; font-weight: bold">import</span> plt, mpl
plt.style.use(<span style="color: #CD5555">&#39;seaborn&#39;</span>)
mpl.rcParams[<span style="color: #CD5555">&#39;font.family&#39;</span>] = <span style="color: #CD5555">&#39;serif&#39;</span>
df.cumsum().plot(lw=<span style="color: #B452CD">2.0</span>, figsize=(<span style="color: #B452CD">10</span>,<span style="color: #B452CD">6</span>))
plt.show()
df.plot.bar(figsize=(<span style="color: #B452CD">10</span>,<span style="color: #B452CD">6</span>), rot=<span style="color: #B452CD">15</span>)
plt.show()
</pre></div>
<p>
We can produce a \( 4\times 4 \) matrix
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span>b = np.arange(<span style="color: #B452CD">16</span>).reshape((<span style="color: #B452CD">4</span>,<span style="color: #B452CD">4</span>))
<span style="color: #8B008B; font-weight: bold">print</span>(b)
df1 = pd.DataFrame(b)
<span style="color: #8B008B; font-weight: bold">print</span>(df1)
</pre></div>
<p>
and many other operations.
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/html/DimRed.html
+235 -43
View File
@@ -97,17 +97,21 @@ div { text-align: justify; text-justify: inter-word; }
2,
None,
'___sec8'),
('Classical PCA Theorem', 2, None, '___sec9'),
('Prof of the PCA Theorem', 2, None, '___sec10'),
('Getting started with PCA', 2, None, '___sec11'),
('Principal Component Analysis', 2, None, '___sec12'),
('PCA and scikit-learn', 2, None, '___sec13'),
('More on the PCA', 2, None, '___sec14'),
('Incremental PCA', 2, None, '___sec15'),
('Randomized PCA', 2, None, '___sec16'),
('Kernel PCA', 2, None, '___sec17'),
('LLE', 2, None, '___sec18'),
('Other techniques', 2, None, '___sec19')]}
('Correlation Function and Design/Feature Matrix',
2,
None,
'___sec9'),
('Classical PCA Theorem', 2, None, '___sec10'),
('Prof of the PCA Theorem', 2, None, '___sec11'),
('Getting started with PCA', 2, None, '___sec12'),
('Principal Component Analysis', 2, None, '___sec13'),
('PCA and scikit-learn', 2, None, '___sec14'),
('More on the PCA', 2, None, '___sec15'),
('Incremental PCA', 2, None, '___sec16'),
('Randomized PCA', 2, None, '___sec17'),
('Kernel PCA', 2, None, '___sec18'),
('LLE', 2, None, '___sec19'),
('Other techniques', 2, None, '___sec20')]}
end of tocinfo -->
<body>
@@ -149,7 +153,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Oct 19, 2019</h4></center> <!-- date -->
<center><h4>Oct 20, 2019</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -467,27 +471,41 @@ plt<span style="color: #666666">.</span>show()
<span style="color: #408080; font-style: italic">#print eigvalues of correlation matrix</span>
EigValues, EigVectors <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>eig(correlation_matrix)
<span style="color: #008000; font-weight: bold">print</span>(EigValues)
<span style="color: #408080; font-style: italic">#split into train and test and then scale thereafter</span>
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(cancer<span style="color: #666666">.</span>data,cancer<span style="color: #666666">.</span>target,random_state<span style="color: #666666">=0</span>)
<span style="color: #008000; font-weight: bold">print</span>(X_train<span style="color: #666666">.</span>shape)
<span style="color: #008000; font-weight: bold">print</span>(X_test<span style="color: #666666">.</span>shape)
logreg <span style="color: #666666">=</span> LogisticRegression()
logreg<span style="color: #666666">.</span>fit(X_train, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy from Logistic Regression: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test,y_test)))
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> MinMaxScaler, StandardScaler
scaler <span style="color: #666666">=</span> StandardScaler()
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Test set accuracy scaled data: {:.2f}&quot;</span><span style="color: #666666">.</span>format(logreg<span style="color: #666666">.</span>score(X_test_scaled,y_test)))
</pre></div>
<p>
<!-- todo: add more text in order to explain what is done here, discuss the correlation matrix -->
In the above example we note two things. In the first plot we display
the overlap of benign and malignant tumors as functions of the various
features in the Wisconsing breast cancer data set. We see that for
some of the features we can distinguish clearly the benign and
malignant cases while for other features we cannot. This can point to
us which features may be of greater interest when we wish to classify
a benign or not benign tumour.
<p>
In the second figure we have computed the so-called correlation
matrix, which in our case with thirty features becomes a \( 30\times 30 \)
matrix.
<p>
We constructed this matrix using <b>pandas</b> via the statements
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>cancerpd <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(cancer<span style="color: #666666">.</span>data, columns<span style="color: #666666">=</span>cancer<span style="color: #666666">.</span>feature_names)
</pre></div>
<p>
and then
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>correlation_matrix <span style="color: #666666">=</span> cancerpd<span style="color: #666666">.</span>corr()<span style="color: #666666">.</span>round(<span style="color: #666666">1</span>)
</pre></div>
<p>
Diagonalizing this matrix we can in turn say something about which
features are of relevance and which are not. But before we proceed we
need to define covariance and correlation matrices. This leads us to
the classical Principal Component Analysis (PCA) theorem with
applications.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -500,19 +518,138 @@ logreg<span style="color: #666666">.</span>fit(X_train_scaled, y_train)
<h2 id="___sec8">Introducing the Covariance and Correlation functions </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
Suppose we have defined two vectors
$\hat{x} and \hat{y} with \( n \) elements each. The covariance matrix $\boldsymbol{C}is defined as
$$
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} cov[\boldsymbol{x},\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
cov[\boldsymbol{y},\boldsymbol{x}] & cov[\boldsymbol{y},\boldsymbol{y}] \\
\end{bmatrix},
$$
<h2 id="___sec9">Classical PCA Theorem </h2>
where for example
$$
cov[\boldsymbol{x},\boldsymbol{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
$$
With this definition and recalling that the variance is
$$
var[\boldsymbol{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
$$
we can rewrite the covariance matrix in this case as
$$
\boldsymbol{C}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} var[\boldsymbol{x}] & cov[\boldsymbol{x},\boldsymbol{y}] \\
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{y}] \\
\end{bmatrix},
$$
<p>
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function
$$
corr[\boldsymbol{x},\boldsymbol{y}]=\frac{cov[\boldsymbol{x},\boldsymbol{y}]}{\sqrt{var[\boldsymbol{x}]\var\boldsymbol{y}]}}.
$$
The correlation function is then given by values \( corr[\boldsymbol{x},\boldsymbol{y}] \in [-1,1] \). This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors \( \boldsymbol{x} \) and \( \boldsymbol{y} \) as
$$
\boldsymbol{K}[\boldsymbol{x},\boldsymbol{y}] = \begin{bmatrix} 1 & corr[\boldsymbol{x},\boldsymbol{y}] \\
corr[\boldsymbol{y},\boldsymbol{x}] & 1 \\
\end{bmatrix},
$$
<p>
In the above example this is the function we constructed using <b>pandas</b>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Prof of the PCA Theorem </h2>
<h2 id="___sec9">Correlation Function and Design/Feature Matrix </h2>
<p>
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix \( \boldsymbol{X} \) as
$$
\boldsymbol{X}=\begin{bmatrix}
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
\dots & \dots & \dots & \dots \dots & \dots \\
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
\end{bmatrix},
$$
with \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \), with the predictors/features \( p \) refering to the column numbers and the
entries \( n \) being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
$$
\boldsymbol{X}=\begin{bmatrix} \boldsymbol{x}_0 & \boldsymbol{x}_0 & \boldsymbol{x}_0 & \dots & \dots & \boldsymbol{x}_{p-1}\end{bmatrix},
$$
with a given vector
$$
\boldsymbol{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
$$
<p>
With these definitions, we can now rewrite our \( 2\times 2 \) correaltion/covariance matrix in terms of a moe general design/feature matrix \( \boldsymbol{X}\in {\mathbb{R}}^{n\times p} \). This leads to a \( p\times p \) covariance matrix for the vectors \( \boldsymbol{x}_i \) with \( i =0,1,\dots,p-1 \)
$$
\boldsymbol{C}[\boldsymbol{x}] = \begin{bmatrix} var[\boldsymbol{x}_0] & cov[\boldsymbol{x}_0,\boldsymbol{x}_1] \\
cov[\boldsymbol{x},\boldsymbol{y}] & var[\boldsymbol{x}_{p-1}] \\
\end{bmatrix},
$$
<p>
The Numpy function <b>np.cov</b> calculates the covariance elements using the factor \( 1/(n-1) \) instead of \( 1/n \) since it assumes we do not have the exact mean values.
The following simple function uses the <b>np.vstack</b> function which takes each vector of dimension \( 1\times n \) and produces a \( 2\times n \) matrix \( \hat{W} \)
$$
\hat{W} = \begin{bmatrix} x_0 & y_0 \\
x_1 & y_1 \\
x_2 & y_2\\
\dots & \dots \\
x_{n-2} & y_{n-2}\\
x_{n-1} & y_{n-1} &
\end{bmatrix},
$$
<p>
which in turn is converted into into the \( 3\times 3 \) covariance matrix
\( \hat{\Sigma} \) via the Numpy function <b>np.cov()</b>. We note that we can also calculate
the mean value of each set of samples \( \hat{x} \) etc using the Numpy
function <b>np.mean(x)</b>. We can also extract the eigenvalues of the
covariance matrix through the <b>np.linalg.eig()</b> function.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Importing various packages</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
<span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>mean(x))
y <span style="color: #666666">=</span> <span style="color: #666666">4+3*</span>x<span style="color: #666666">+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
<span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>mean(y))
z <span style="color: #666666">=</span> x<span style="color: #666666">**3+</span>np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>normal(size<span style="color: #666666">=</span>n)
<span style="color: #008000; font-weight: bold">print</span>(np<span style="color: #666666">.</span>mean(z))
W <span style="color: #666666">=</span> np<span style="color: #666666">.</span>vstack((x, y, z))
Sigma <span style="color: #666666">=</span> np<span style="color: #666666">.</span>cov(W)
<span style="color: #008000; font-weight: bold">print</span>(Sigma)
Eigvals, Eigvecs <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>eig(Sigma)
<span style="color: #008000; font-weight: bold">print</span>(Eigvals)
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Classical PCA Theorem </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Getting started with PCA </h2>
<h2 id="___sec11">Prof of the PCA Theorem </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Getting started with PCA </h2>
<p>
@@ -527,7 +664,7 @@ X_pca <span style="color: #666666">=</span> pca<span style="color: #666666">.</s
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Principal Component Analysis </h2>
<h2 id="___sec13">Principal Component Analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -563,7 +700,7 @@ X2D <span style="color: #666666">=</span> X_centered<span style="color: #666666"
<p>
<!-- !split -->
<h2 id="___sec13">PCA and scikit-learn </h2>
<h2 id="___sec14">PCA and scikit-learn </h2>
<p>
Scikit-Learn&#8217;s PCA class implements PCA using SVD decomposition just like we did before. The
@@ -594,7 +731,7 @@ More material to come here.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">More on the PCA </h2>
<h2 id="___sec15">More on the PCA </h2>
<p>
Instead of arbitrarily choosing the number of dimensions to reduce down to, it is generally preferable to
@@ -624,7 +761,7 @@ X_reduced <span style="color: #666666">=</span> pca<span style="color: #666666">
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Incremental PCA </h2>
<h2 id="___sec16">Incremental PCA </h2>
<p>
One problem with the preceding implementation of PCA is that it requires the whole training set to fit in
@@ -636,7 +773,7 @@ instances arrive).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Randomized PCA </h2>
<h2 id="___sec17">Randomized PCA </h2>
<p>
Scikit-Learn offers yet another option to perform PCA, called Randomized PCA. This is a stochastic
@@ -651,7 +788,7 @@ previous algorithms when \( d \) is much smaller than \( n \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Kernel PCA </h2>
<h2 id="___sec18">Kernel PCA </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -680,7 +817,7 @@ X_reduced <span style="color: #666666">=</span> rbf_pca<span style="color: #6666
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">LLE </h2>
<h2 id="___sec19">LLE </h2>
<p>
Locally Linear Embedding (LLE) is another very powerful nonlinear dimensionality reduction
@@ -692,7 +829,7 @@ these local relationships are best preserved (more details shortly).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Other techniques </h2>
<h2 id="___sec20">Other techniques </h2>
<p>
There are many other dimensionality reduction techniques, several of which are available in Scikit-Learn.
@@ -707,6 +844,61 @@ Here are some of the most popular:
<li> Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.</li>
</ul>
Here are other examples where we use the <b>DataFrame</b> functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix
of dimensionality \( 10\times 5 \) and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">IPython.display</span> <span style="color: #008000; font-weight: bold">import</span> display
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">100</span>)
<span style="color: #408080; font-style: italic"># setting up a 10 x 5 matrix</span>
rows <span style="color: #666666">=</span> <span style="color: #666666">10</span>
cols <span style="color: #666666">=</span> <span style="color: #666666">5</span>
a <span style="color: #666666">=</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(rows,cols)
df <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(a)
display(df)
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>mean())
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>std())
display(df<span style="color: #666666">**2</span>)
</pre></div>
<p>
Thereafter we can select specific columns only and plot final results
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>df<span style="color: #666666">.</span>columns <span style="color: #666666">=</span> [<span style="color: #BA2121">&#39;First&#39;</span>, <span style="color: #BA2121">&#39;Second&#39;</span>, <span style="color: #BA2121">&#39;Third&#39;</span>, <span style="color: #BA2121">&#39;Fourth&#39;</span>, <span style="color: #BA2121">&#39;Fifth&#39;</span>]
df<span style="color: #666666">.</span>index <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">10</span>)
display(df)
<span style="color: #008000; font-weight: bold">print</span>(df[<span style="color: #BA2121">&#39;Second&#39;</span>]<span style="color: #666666">.</span>mean() )
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>info())
<span style="color: #008000; font-weight: bold">print</span>(df<span style="color: #666666">.</span>describe())
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">pylab</span> <span style="color: #008000; font-weight: bold">import</span> plt, mpl
plt<span style="color: #666666">.</span>style<span style="color: #666666">.</span>use(<span style="color: #BA2121">&#39;seaborn&#39;</span>)
mpl<span style="color: #666666">.</span>rcParams[<span style="color: #BA2121">&#39;font.family&#39;</span>] <span style="color: #666666">=</span> <span style="color: #BA2121">&#39;serif&#39;</span>
df<span style="color: #666666">.</span>cumsum()<span style="color: #666666">.</span>plot(lw<span style="color: #666666">=2.0</span>, figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>))
plt<span style="color: #666666">.</span>show()
df<span style="color: #666666">.</span>plot<span style="color: #666666">.</span>bar(figsize<span style="color: #666666">=</span>(<span style="color: #666666">10</span>,<span style="color: #666666">6</span>), rot<span style="color: #666666">=15</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
We can produce a \( 4\times 4 \) matrix
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span>b <span style="color: #666666">=</span> np<span style="color: #666666">.</span>arange(<span style="color: #666666">16</span>)<span style="color: #666666">.</span>reshape((<span style="color: #666666">4</span>,<span style="color: #666666">4</span>))
<span style="color: #008000; font-weight: bold">print</span>(b)
df1 <span style="color: #666666">=</span> pd<span style="color: #666666">.</span>DataFrame(b)
<span style="color: #008000; font-weight: bold">print</span>(df1)
</pre></div>
<p>
and many other operations.
<!-- ------------------- end of main content --------------- -->
doc/pub/DimRed/ipynb/DimRed.ipynb
+395 -30
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Oct 19, 2019**\n",
"Date: **Oct 20, 2019**\n",
"\n",
"Copyright 1999-2019, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -337,32 +337,67 @@
"\n",
"#print eigvalues of correlation matrix\n",
"EigValues, EigVectors = np.linalg.eig(correlation_matrix)\n",
"print(EigValues)\n",
"\n",
"#split into train and test and then scale thereafter\n",
"X_train, X_test, y_train, y_test = train_test_split(cancer.data,cancer.target,random_state=0)\n",
"print(X_train.shape)\n",
"print(X_test.shape)\n",
"\n",
"logreg = LogisticRegression()\n",
"logreg.fit(X_train, y_train)\n",
"print(\"Test set accuracy from Logistic Regression: {:.2f}\".format(logreg.score(X_test,y_test)))\n",
"\n",
"from sklearn.preprocessing import MinMaxScaler, StandardScaler\n",
"scaler = StandardScaler()\n",
"scaler.fit(X_train)\n",
"X_train_scaled = scaler.transform(X_train)\n",
"X_test_scaled = scaler.transform(X_test)\n",
"\n",
"logreg.fit(X_train_scaled, y_train)\n",
"print(\"Test set accuracy scaled data: {:.2f}\".format(logreg.score(X_test_scaled,y_test)))"
"print(EigValues)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- todo: add more text in order to explain what is done here, discuss the correlation matrix -->\n",
"In the above example we note two things. In the first plot we display\n",
"the overlap of benign and malignant tumors as functions of the various\n",
"features in the Wisconsing breast cancer data set. We see that for\n",
"some of the features we can distinguish clearly the benign and\n",
"malignant cases while for other features we cannot. This can point to\n",
"us which features may be of greater interest when we wish to classify\n",
"a benign or not benign tumour.\n",
"\n",
"In the second figure we have computed the so-called correlation\n",
"matrix, which in our case with thirty features becomes a $30\\times 30$\n",
"matrix.\n",
"\n",
"We constructed this matrix using **pandas** via the statements"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and then"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"correlation_matrix = cancerpd.corr().round(1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Diagonalizing this matrix we can in turn say something about which\n",
"features are of relevance and which are not. But before we proceed we\n",
"need to define covariance and correlation matrices. This leads us to\n",
"the classical Principal Component Analysis (PCA) theorem with\n",
"applications.\n",
"\n",
"\n",
"\n",
"## Basic ideas of the Principal Component Analysis (PCA)\n",
@@ -370,6 +405,247 @@
"\n",
"## Introducing the Covariance and Correlation functions\n",
"\n",
"Suppose we have defined two vectors\n",
"$\\hat{x} and \\hat{y} with $n$ elements each. The covariance matrix $\\boldsymbol{C}is defined as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} cov[\\boldsymbol{x},\\boldsymbol{x}] & cov[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
" cov[\\boldsymbol{y},\\boldsymbol{x}] & cov[\\boldsymbol{y},\\boldsymbol{y}] \\\\\n",
" \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"where for example"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"cov[\\boldsymbol{x},\\boldsymbol{y}] =\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})(y_i- \\overline{y}).\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With this definition and recalling that the variance is"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"var[\\boldsymbol{x}]=\\frac{1}{n} \\sum_{i=0}^{n-1}(x_i- \\overline{x})^2,\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"we can rewrite the covariance matrix in this case as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{C}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} var[\\boldsymbol{x}] & cov[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
" cov[\\boldsymbol{x},\\boldsymbol{y}] & var[\\boldsymbol{y}] \\\\\n",
" \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"corr[\\boldsymbol{x},\\boldsymbol{y}]=\\frac{cov[\\boldsymbol{x},\\boldsymbol{y}]}{\\sqrt{var[\\boldsymbol{x}]\\var\\boldsymbol{y}]}}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The correlation function is then given by values $corr[\\boldsymbol{x},\\boldsymbol{y}] \\in [-1,1]$. This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors $\\boldsymbol{x}$ and $\\boldsymbol{y}$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{K}[\\boldsymbol{x},\\boldsymbol{y}] = \\begin{bmatrix} 1 & corr[\\boldsymbol{x},\\boldsymbol{y}] \\\\\n",
" corr[\\boldsymbol{y},\\boldsymbol{x}] & 1 \\\\\n",
" \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the above example this is the function we constructed using **pandas**.\n",
"\n",
"## Correlation Function and Design/Feature Matrix\n",
"\n",
"In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix $\\boldsymbol{X}$ as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix}\n",
"x_{0,0} & x_{0,1} & x_{0,2}& \\dots & \\dots x_{0,p-1}\\\\\n",
"x_{1,0} & x_{1,1} & x_{1,2}& \\dots & \\dots x_{1,p-1}\\\\\n",
"x_{2,0} & x_{2,1} & x_{2,2}& \\dots & \\dots x_{2,p-1}\\\\\n",
"\\dots & \\dots & \\dots & \\dots \\dots & \\dots \\\\\n",
"x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \\dots & \\dots x_{n-2,p-1}\\\\\n",
"x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \\dots & \\dots x_{n-1,p-1}\\\\\n",
"\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"with $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$, with the predictors/features $p$ refering to the column numbers and the\n",
"entries $n$ being the row elements.\n",
"We can rewrite the design/feature matrix in terms of its column vectors as"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{X}=\\begin{bmatrix} \\boldsymbol{x}_0 & \\boldsymbol{x}_0 & \\boldsymbol{x}_0 & \\dots & \\dots & \\boldsymbol{x}_{p-1}\\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"with a given vector"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{x}_i^T = \\begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \\dots & \\dots x_{n-1,i}\\end{bmatrix}.\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With these definitions, we can now rewrite our $2\\times 2$ correaltion/covariance matrix in terms of a moe general design/feature matrix $\\boldsymbol{X}\\in {\\mathbb{R}}^{n\\times p}$. This leads to a $p\\times p$ covariance matrix for the vectors $\\boldsymbol{x}_i$ with $i =0,1,\\dots,p-1$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\boldsymbol{C}[\\boldsymbol{x}] = \\begin{bmatrix} var[\\boldsymbol{x}_0] & cov[\\boldsymbol{x}_0,\\boldsymbol{x}_1] \\\\\n",
" cov[\\boldsymbol{x},\\boldsymbol{y}] & var[\\boldsymbol{x}_{p-1}] \\\\\n",
" \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The Numpy function **np.cov** calculates the covariance elements using the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have the exact mean values. \n",
"The following simple function uses the **np.vstack** function which takes each vector of dimension $1\\times n$ and produces a $2\\times n$ matrix $\\hat{W}$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"$$\n",
"\\hat{W} = \\begin{bmatrix} x_0 & y_0 \\\\\n",
" x_1 & y_1 \\\\\n",
" x_2 & y_2\\\\\n",
" \\dots & \\dots \\\\\n",
" x_{n-2} & y_{n-2}\\\\\n",
" x_{n-1} & y_{n-1} & \n",
" \\end{bmatrix},\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"which in turn is converted into into the $3\\times 3$ covariance matrix\n",
"$\\hat{\\Sigma}$ via the Numpy function **np.cov()**. We note that we can also calculate\n",
"the mean value of each set of samples $\\hat{x}$ etc using the Numpy\n",
"function **np.mean(x)**. We can also extract the eigenvalues of the\n",
"covariance matrix through the **np.linalg.eig()** function."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# Importing various packages\n",
"import numpy as np\n",
"\n",
"n = 100\n",
"x = np.random.normal(size=n)\n",
"print(np.mean(x))\n",
"y = 4+3*x+np.random.normal(size=n)\n",
"print(np.mean(y))\n",
"z = x**3+np.random.normal(size=n)\n",
"print(np.mean(z))\n",
"W = np.vstack((x, y, z))\n",
"Sigma = np.cov(W)\n",
"print(Sigma)\n",
"Eigvals, Eigvecs = np.linalg.eig(Sigma)\n",
"print(Eigvals)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Classical PCA Theorem\n",
"\n",
"\n",
@@ -385,7 +661,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 8,
"metadata": {
"collapsed": false
},
@@ -413,7 +689,7 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 9,
"metadata": {
"collapsed": false
},
@@ -440,7 +716,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 10,
"metadata": {
"collapsed": false
},
@@ -464,7 +740,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 11,
"metadata": {
"collapsed": false
},
@@ -486,7 +762,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 12,
"metadata": {
"collapsed": false
},
@@ -516,7 +792,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 13,
"metadata": {
"collapsed": false
},
@@ -539,7 +815,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 14,
"metadata": {
"collapsed": false
},
@@ -586,7 +862,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 15,
"metadata": {
"collapsed": false
},
@@ -623,7 +899,96 @@
"\n",
"* **t-Distributed Stochastic Neighbor Embedding** (t-SNE) reduces dimensionality while trying to keep similar instances close and dissimilar instances apart. It is mostly used for visualization, in particular to visualize clusters of instances in high-dimensional space (e.g., to visualize the MNIST images in 2D).\n",
"\n",
"* Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures."
"* Linear Discriminant Analysis (LDA) is actually a classification algorithm, but during training it learns the most discriminative axes between the classes, and these axes can then be used to define a hyperplane onto which to project the data. The benefit is that the projection will keep classes as far apart as possible, so LDA is a good technique to reduce dimensionality before running another classification algorithm such as a Support Vector Machine (SVM) classifier discussed in the SVM lectures.\n",
"\n",
"Here are other examples where we use the **DataFrame** functionality to handle arrays, now with more interesting features for us, namely numbers. We set up a matrix \n",
"of dimensionality $10\\times 5$ and compute the mean value and standard deviation of each column. Similarly, we can perform mathematial operations like squaring the matrix elements and many other operations."
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"from IPython.display import display\n",
"np.random.seed(100)\n",
"# setting up a 10 x 5 matrix\n",
"rows = 10\n",
"cols = 5\n",
"a = np.random.randn(rows,cols)\n",
"df = pd.DataFrame(a)\n",
"display(df)\n",
"print(df.mean())\n",
"print(df.std())\n",
"display(df**2)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Thereafter we can select specific columns only and plot final results"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"df.columns = ['First', 'Second', 'Third', 'Fourth', 'Fifth']\n",
"df.index = np.arange(10)\n",
"\n",
"display(df)\n",
"print(df['Second'].mean() )\n",
"print(df.info())\n",
"print(df.describe())\n",
"\n",
"from pylab import plt, mpl\n",
"plt.style.use('seaborn')\n",
"mpl.rcParams['font.family'] = 'serif'\n",
"\n",
"df.cumsum().plot(lw=2.0, figsize=(10,6))\n",
"plt.show()\n",
"\n",
"\n",
"df.plot.bar(figsize=(10,6), rot=15)\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can produce a $4\\times 4$ matrix"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"b = np.arange(16).reshape((4,4))\n",
"print(b)\n",
"df1 = pd.DataFrame(b)\n",
"print(df1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"and many other operations."
]
}
],
doc/pub/DimRed/ipynb/ipynb-DimRed-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/DimRed/pdf/DimRed-minted.pdf
BIN
View File
Binary file not shown.
doc/src/DimRed/DimRed.do.txt
+66 -6
View File
@@ -339,31 +339,91 @@ Suppose we have defined two vectors
$\hat{x} and \hat{y} with $n$ elements each. The covariance matrix $\bm{C}is defined as
!bt
\[
\bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} cov[xx] & cov[xy] \\
cov[yx] & cov[yy] \\
\bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} cov[\bm{x},\bm{x}] & cov[\bm{x},\bm{y}] \\
cov[\bm{y},\bm{x}] & cov[\bm{y},\bm{y}] \\
\end{bmatrix},
\]
!et
where for example
!bt
\[
cov[xy] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
cov[\bm{x},\bm{y}] =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})(y_i- \overline{y}).
\]
!et
With this definition and recalling that the variance is
!bt
\[
var[\bm{x}]=\sigma_{xx} =\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
var[\bm{x}]=\frac{1}{n} \sum_{i=0}^{n-1}(x_i- \overline{x})^2,
\]
!et
we can rewrite the covariance matrix in this case as
!bt
\[
\bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} var[\bm{x}] & \sigma_{xy} \\
\sigma_{yx} & var[\bm{y}] \\
\bm{C}[\bm{x},\bm{y}] = \begin{bmatrix} var[\bm{x}] & cov[\bm{x},\bm{y}] \\
cov[\bm{x},\bm{y}] & var[\bm{y}] \\
\end{bmatrix},
\]
!et
The covariance takes values between zero and infinity and may thus lead to problems with loss of numerical precision for particularly large values. It is common to scale the covariance matrix by introducing instead the correlation matrix defined via the so-called correlation function
!bt
\[
corr[\bm{x},\bm{y}]=\frac{cov[\bm{x},\bm{y}]}{\sqrt{var[\bm{x}]\var\bm{y}]}}.
\]
!et
The correlation function is then given by values $corr[\bm{x},\bm{y}] \in [-1,1]$. This avoids eventual problems with too large values. We can then define the correlation matrix for the two vectors $\bm{x}$ and $\bm{y}$ as
!bt
\[
\bm{K}[\bm{x},\bm{y}] = \begin{bmatrix} 1 & corr[\bm{x},\bm{y}] \\
corr[\bm{y},\bm{x}] & 1 \\
\end{bmatrix},
\]
!et
In the above example this is the function we constructed using _pandas_.
!split
===== Correlation Function and Design/Feature Matrix =====
In our derivation of the various regression algorithms like Ordinary Least Squares or Ridge regression we defined the design/feature matrix $\bm{X}$ as
!bt
\[
\bm{X}=\begin{bmatrix}
x_{0,0} & x_{0,1} & x_{0,2}& \dots & \dots x_{0,p-1}\\
x_{1,0} & x_{1,1} & x_{1,2}& \dots & \dots x_{1,p-1}\\
x_{2,0} & x_{2,1} & x_{2,2}& \dots & \dots x_{2,p-1}\\
\dots & \dots & \dots & \dots \dots & \dots \\
x_{n-2,0} & x_{n-2,1} & x_{n-2,2}& \dots & \dots x_{n-2,p-1}\\
x_{n-1,0} & x_{n-1,1} & x_{n-1,2}& \dots & \dots x_{n-1,p-1}\\
\end{bmatrix},
\]
!et
with $\bm{X}\in {\mathbb{R}}^{n\times p}$, with the predictors/features $p$ refering to the column numbers and the
entries $n$ being the row elements.
We can rewrite the design/feature matrix in terms of its column vectors as
!bt
\[
\bm{X}=\begin{bmatrix} \bm{x}_0 & \bm{x}_0 & \bm{x}_0 & \dots & \dots & \bm{x}_{p-1}\end{bmatrix},
\]
!et
with a given vector
!bt
\[
\bm{x}_i^T = \begin{bmatrix}x_{0,i} & x_{1,i} & x_{2,i}& \dots & \dots x_{n-1,i}\end{bmatrix}.
\]
!et
With these definitions, we can now rewrite our $2\times 2$ correaltion/covariance matrix in terms of a moe general design/feature matrix $\bm{X}\in {\mathbb{R}}^{n\times p}$. This leads to a $p\times p$ covariance matrix for the vectors $\bm{x}_i$ with $i =0,1,\dots,p-1$
!bt
\[
\bm{C}[\bm{x}] = \begin{bmatrix} var[\bm{x}_0] & cov[\bm{x}_0,\bm{x}_1] \\
cov[\bm{x},\bm{y}] & var[\bm{x}_{p-1}] \\
\end{bmatrix},
\]
!et
The Numpy function _np.cov_ calculates the covariance elements using the factor $1/(n-1)$ instead of $1/n$ since it assumes we do not have the exact mean values.
The following simple function uses the _np.vstack_ function which takes each vector of dimension $1\times n$ and produces a $2\times n$ matrix $\hat{W}$