lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating week 36

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-06 22:28:54 +02:00
parent ed47ea2760
commit 331bb78d0a
7 changed files with 1110 additions and 1032 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week36/html/week36-bs.html
+119 -80
View File
@@ -1,15 +1,15 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 36: Resampling techniques and Ordinary Least Square">
<meta name="description" content="Week 36: Statistical interpretation of Linear Regression and Resampling techniques">
<title>Week 36: Resampling techniques and Ordinary Least Square</title>
<title>Week 36: Statistical interpretation of Linear Regression and Resampling techniques</title>
<!-- Bootstrap style: bootstrap -->
<link href="https://netdna.bootstrapcdn.com/bootstrap/3.1.1/css/bootstrap.min.css" rel="stylesheet">
@@ -41,72 +41,111 @@ Automatically generated HTML file from DocOnce source
<!-- tocinfo
{'highest level': 2,
'sections': [('Plans for week 36', 2, None, '___sec0'),
('Thursday September 3', 2, None, '___sec1'),
('Why resampling methods', 2, None, '___sec2'),
('Resampling methods', 2, None, '___sec3'),
('Resampling approaches can be computationally expensive',
2,
None,
'___sec4'),
('Why resampling methods ?', 2, None, '___sec5'),
('Statistical analysis', 2, None, '___sec6'),
'sections': [('Plans for week 36', 2, None, 'plans-for-week-36'),
('Thursday September 9', 2, None, 'thursday-september-9'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
None,
'___sec7'),
('Assumptions made', 2, None, '___sec8'),
('Expectation value and variance', 2, None, '___sec9'),
'linking-the-regression-analysis-with-a-statistical-interpretation'),
('Assumptions made', 2, None, 'assumptions-made'),
('Expectation value and variance',
2,
None,
'expectation-value-and-variance'),
('Expectation value and variance for $\\boldsymbol{\\beta}$',
2,
None,
'___sec10'),
('Resampling methods', 2, None, '___sec11'),
'expectation-value-and-variance-for-boldsymbol-beta'),
('Friday September 10', 2, None, 'friday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
2,
None,
'resampling-approaches-can-be-computationally-expensive'),
('Why resampling methods ?', 2, None, 'why-resampling-methods'),
('Statistical analysis', 2, None, 'statistical-analysis'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling methods: Jackknife and Bootstrap',
2,
None,
'___sec12'),
('Resampling methods: Jackknife', 2, None, '___sec13'),
('Jackknife code example', 2, None, '___sec14'),
('Resampling methods: Bootstrap', 2, None, '___sec15'),
('Resampling methods: Bootstrap background', 2, None, '___sec16'),
'resampling-methods-jackknife-and-bootstrap'),
('Resampling methods: Jackknife',
2,
None,
'resampling-methods-jackknife'),
('Jackknife code example', 2, None, 'jackknife-code-example'),
('Resampling methods: Bootstrap',
2,
None,
'resampling-methods-bootstrap'),
('Resampling methods: Bootstrap background',
2,
None,
'resampling-methods-bootstrap-background'),
('Resampling methods: More Bootstrap background',
2,
None,
'___sec17'),
('Resampling methods: Bootstrap approach', 2, None, '___sec18'),
('Resampling methods: Bootstrap steps', 2, None, '___sec19'),
('Code example for the Bootstrap method', 2, None, '___sec20'),
('Various steps in cross-validation', 2, None, '___sec21'),
'resampling-methods-more-bootstrap-background'),
('Resampling methods: Bootstrap approach',
2,
None,
'resampling-methods-bootstrap-approach'),
('Resampling methods: Bootstrap steps',
2,
None,
'resampling-methods-bootstrap-steps'),
('Code example for the Bootstrap method',
2,
None,
'code-example-for-the-bootstrap-method'),
('Various steps in cross-validation',
2,
None,
'various-steps-in-cross-validation'),
('How to set up the cross-validation for Ridge and/or Lasso',
2,
None,
'___sec22'),
('Cross-validation in brief', 2, None, '___sec23'),
'how-to-set-up-the-cross-validation-for-ridge-and-or-lasso'),
('Cross-validation in brief',
2,
None,
'cross-validation-in-brief'),
('Code Example for Cross-validation and $k$-fold '
'Cross-validation',
2,
None,
'___sec24'),
('The bias-variance tradeoff', 2, None, '___sec25'),
('Example code for Bias-Variance tradeoff', 2, None, '___sec26'),
('Understanding what happens', 2, None, '___sec27'),
('Summing up', 2, None, '___sec28'),
'code-example-for-cross-validation-and-k-fold-cross-validation'),
('The bias-variance tradeoff',
2,
None,
'the-bias-variance-tradeoff'),
('Example code for Bias-Variance tradeoff',
2,
None,
'example-code-for-bias-variance-tradeoff'),
('Understanding what happens',
2,
None,
'understanding-what-happens'),
('Summing up', 2, None, 'summing-up'),
("Another Example from Scikit-Learn's Repository",
2,
None,
'___sec29'),
'another-example-from-scikit-learn-s-repository'),
('More examples on bootstrap and cross-validation and errors',
2,
None,
'___sec30'),
'more-examples-on-bootstrap-and-cross-validation-and-errors'),
('The same example but now with cross-validation',
2,
None,
'___sec31'),
('Cross-validation with Ridge', 2, None, '___sec32'),
('Friday September 4', 2, None, '___sec33')]}
'the-same-example-but-now-with-cross-validation'),
('Cross-validation with Ridge',
2,
None,
'cross-validation-with-ridge')]}
end of tocinfo -->
<body>
@@ -136,7 +175,7 @@ MathJax.Hub.Config({
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="week36-bs.html">Week 36: Resampling techniques and Ordinary Least Square</a>
<a class="navbar-brand" href="week36-bs.html">Week 36: Statistical interpretation of Linear Regression and Resampling techniques</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
@@ -144,40 +183,40 @@ MathJax.Hub.Config({
<li class="dropdown">
<a href="#" class="dropdown-toggle" data-toggle="dropdown">Contents <b class="caret"></b></a>
<ul class="dropdown-menu">
<!-- navigation toc: --> <li><a href="._week36-bs001.html#___sec0" style="font-size: 80%;">Plans for week 36</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs002.html#___sec1" style="font-size: 80%;">Thursday September 3</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs003.html#___sec2" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs004.html#___sec3" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs005.html#___sec4" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs006.html#___sec5" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs007.html#___sec6" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs008.html#___sec7" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs009.html#___sec8" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs010.html#___sec9" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs011.html#___sec10" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs012.html#___sec11" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs013.html#___sec12" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs014.html#___sec13" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs015.html#___sec14" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs016.html#___sec15" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs017.html#___sec16" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs018.html#___sec17" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs019.html#___sec18" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs020.html#___sec19" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs021.html#___sec20" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#___sec21" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#___sec22" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#___sec23" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#___sec24" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#___sec25" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#___sec26" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#___sec27" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#___sec28" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#___sec29" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#___sec30" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#___sec31" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#___sec32" style="font-size: 80%;">Cross-validation with Ridge</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#___sec33" style="font-size: 80%;">Friday September 4</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs001.html#plans-for-week-36" style="font-size: 80%;">Plans for week 36</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs002.html#thursday-september-9" style="font-size: 80%;">Thursday September 9</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs003.html#linking-the-regression-analysis-with-a-statistical-interpretation" style="font-size: 80%;">Linking the regression analysis with a statistical interpretation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs004.html#assumptions-made" style="font-size: 80%;">Assumptions made</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs005.html#expectation-value-and-variance" style="font-size: 80%;">Expectation value and variance</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs006.html#expectation-value-and-variance-for-boldsymbol-beta" style="font-size: 80%;">Expectation value and variance for \( \boldsymbol{\beta} \)</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs007.html#friday-september-10" style="font-size: 80%;">Friday September 10</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs011.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs013.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs010.html#resampling-approaches-can-be-computationally-expensive" style="font-size: 80%;">Resampling approaches can be computationally expensive</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs011.html#why-resampling-methods" style="font-size: 80%;">Why resampling methods ?</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs012.html#statistical-analysis" style="font-size: 80%;">Statistical analysis</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs013.html#resampling-methods" style="font-size: 80%;">Resampling methods</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs014.html#resampling-methods-jackknife-and-bootstrap" style="font-size: 80%;">Resampling methods: Jackknife and Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs015.html#resampling-methods-jackknife" style="font-size: 80%;">Resampling methods: Jackknife</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs016.html#jackknife-code-example" style="font-size: 80%;">Jackknife code example</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs017.html#resampling-methods-bootstrap" style="font-size: 80%;">Resampling methods: Bootstrap</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs018.html#resampling-methods-bootstrap-background" style="font-size: 80%;">Resampling methods: Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs019.html#resampling-methods-more-bootstrap-background" style="font-size: 80%;">Resampling methods: More Bootstrap background</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs020.html#resampling-methods-bootstrap-approach" style="font-size: 80%;">Resampling methods: Bootstrap approach</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs021.html#resampling-methods-bootstrap-steps" style="font-size: 80%;">Resampling methods: Bootstrap steps</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs022.html#code-example-for-the-bootstrap-method" style="font-size: 80%;">Code example for the Bootstrap method</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs023.html#various-steps-in-cross-validation" style="font-size: 80%;">Various steps in cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs024.html#how-to-set-up-the-cross-validation-for-ridge-and-or-lasso" style="font-size: 80%;">How to set up the cross-validation for Ridge and/or Lasso</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs025.html#cross-validation-in-brief" style="font-size: 80%;">Cross-validation in brief</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs026.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs027.html#the-bias-variance-tradeoff" style="font-size: 80%;">The bias-variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs028.html#example-code-for-bias-variance-tradeoff" style="font-size: 80%;">Example code for Bias-Variance tradeoff</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs029.html#understanding-what-happens" style="font-size: 80%;">Understanding what happens</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs030.html#summing-up" style="font-size: 80%;">Summing up</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs031.html#another-example-from-scikit-learn-s-repository" style="font-size: 80%;">Another Example from Scikit-Learn's Repository</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs032.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs033.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week36-bs034.html#cross-validation-with-ridge" style="font-size: 80%;">Cross-validation with Ridge</a></li>
</ul>
</li>
@@ -196,7 +235,7 @@ MathJax.Hub.Config({
<div class="jumbotron">
<center><h1>Week 36: Resampling techniques and Ordinary Least Square</h1></center> <!-- document title -->
<center><h1>Week 36: Statistical interpretation of Linear Regression and Resampling techniques</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -212,7 +251,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 6, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -248,13 +287,13 @@ MathJax.Hub.Config({
<!-- Bootstrap footer
<footer>
<a href="http://..."><img width="250" align=right src="http://..."></a>
<a href="https://..."><img width="250" align=right src="https://..."></a>
</footer>
-->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week36/html/week36-reveal.html
+197 -209
View File
@@ -1,11 +1,11 @@
<!DOCTYPE html>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 36: Resampling techniques and Ordinary Least Square">
<meta name="description" content="Week 36: Statistical interpretation of Linear Regression and Resampling techniques">
<title>Week 36: Resampling techniques and Ordinary Least Square</title>
<title>Week 36: Statistical interpretation of Linear Regression and Resampling techniques</title>
@@ -13,7 +13,7 @@
<!-- reveal.js: http://lab.hakim.se/reveal-js/ -->
<!-- reveal.js: https://lab.hakim.se/reveal-js/ -->
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
@@ -76,10 +76,10 @@ document.getElementsByTagName( 'head' )[0].appendChild( link );
.reveal .alert-block > p, .alert-block > ul {margin-bottom:1em}
/*.reveal .alert li {margin-top: 1em}*/
.reveal .alert-block p+p {margin-top:5px}
/*.reveal .alert-notice { background-image: url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_notice.png); }
.reveal .alert-summary { background-image:url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_summary.png); }
.reveal .alert-warning { background-image: url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_warning.png); }
.reveal .alert-question {background-image:url(http://hplgit.github.io/doconce/bundled/html_images/small_gray_question.png); } */
/*.reveal .alert-notice { background-image: url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_notice.png); }
.reveal .alert-summary { background-image:url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_summary.png); }
.reveal .alert-warning { background-image: url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_warning.png); }
.reveal .alert-question {background-image:url(https://hplgit.github.io/doconce/bundled/html_images/small_gray_question.png); } */
</style>
@@ -132,7 +132,7 @@ MathJax.Hub.Config({
<center><h1 style="text-align: center;">Week 36: Resampling techniques and Ordinary Least Square</h1></center> <!-- document title -->
<center><h1 style="text-align: center;">Week 36: Statistical interpretation of Linear Regression and Resampling techniques</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -148,149 +148,33 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 6, 2021</h4></center> <!-- date -->
<br>
<p>
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
</section>
<section>
<h2 id="___sec0">Plans for week 36 </h2>
<h2 id="plans-for-week-36">Plans for week 36 </h2>
<ul>
<p><li> Thursday: Statistics, probability theory and resampling methods</li>
<p><li> Friday: Resampling methods and motivation for Ridge Regression</li>
<p><li> Thursday: Statistics, probability theory and linear regression</li>
<p><li> Friday: Resampling methods</li>
</ul>
</section>
<section>
<h2 id="___sec1">Thursday September 3 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember3.pdf" target="_blank">handwritten notes</a>
<p>
More material will be added here, see handwritten notes also.
<h2 id="thursday-september-9">Thursday September 9 </h2>
</section>
<section>
<h2 id="___sec2">Why resampling methods </h2>
<p>
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
<ol>
<p><li> look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff</li>
<p><li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
<p>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
</section>
<section>
<h2 id="___sec3">Resampling methods </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
<p>
Two resampling methods are often used in Machine Learning analyses,
<ol>
<p><li> The <b>bootstrap method</b></li>
<p><li> and <b>Cross-Validation</b></li>
</ol>
<p>
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
</div>
</section>
<section>
<h2 id="___sec4">Resampling approaches can be computationally expensive </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model&#8217;s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
</div>
</section>
<section>
<h2 id="___sec5">Why resampling methods ? </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Statistical analysis.</b>
<ul>
<p><li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<p><li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<p><li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
</section>
<section>
<h2 id="___sec6">Statistical analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<ul>
<p><li> As in other experiments, many numerical experiments have two classes of errors:</li>
<ul>
<p><li> Statistical errors</li>
<p><li> Systematical errors</li>
</ul>
<p><li> Statistical errors can be estimated using standard tools from statistics</li>
<p><li> Systematical errors are method specific and must be treated differently from case to case.</li>
</ul>
</div>
</section>
<section>
<h2 id="___sec7">Linking the regression analysis with a statistical interpretation </h2>
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
<p>
The
@@ -329,7 +213,7 @@ row number \( i \) and perform a sum over all values \( p \).
<section>
<h2 id="___sec8">Assumptions made </h2>
<h2 id="assumptions-made">Assumptions made </h2>
<p>
The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
@@ -353,7 +237,7 @@ $$
<section>
<h2 id="___sec9">Expectation value and variance </h2>
<h2 id="expectation-value-and-variance">Expectation value and variance </h2>
<p>
We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
@@ -392,7 +276,7 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n
<section>
<h2 id="___sec10">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
<h2 id="expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
<p>
With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value
@@ -487,7 +371,122 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
<section>
<h2 id="___sec11">Resampling methods </h2>
<h2 id="friday-september-10">Friday September 10 </h2>
</section>
<section>
<h2 id="why-resampling-methods">Why resampling methods </h2>
<p>
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
<ol>
<p><li> look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff</li>
<p><li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
<p>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
</section>
<section>
<h2 id="resampling-methods">Resampling methods </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
<p>
Two resampling methods are often used in Machine Learning analyses,
<ol>
<p><li> The <b>bootstrap method</b></li>
<p><li> and <b>Cross-Validation</b></li>
</ol>
<p>
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
</div>
</section>
<section>
<h2 id="resampling-approaches-can-be-computationally-expensive">Resampling approaches can be computationally expensive </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model&#8217;s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
</div>
</section>
<section>
<h2 id="why-resampling-methods">Why resampling methods ? </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Statistical analysis</b>
<ul>
<p><li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<p><li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<p><li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
</section>
<section>
<h2 id="statistical-analysis">Statistical analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<ul>
<p><li> As in other experiments, many numerical experiments have two classes of errors:</li>
<ul>
<p><li> Statistical errors</li>
<p><li> Systematical errors</li>
</ul>
<p><li> Statistical errors can be estimated using standard tools from statistics</li>
<p><li> Systematical errors are method specific and must be treated differently from case to case.</li>
</ul>
</div>
</section>
<section>
<h2 id="resampling-methods">Resampling methods </h2>
<p>
With all these analytical equations for both the OLS and Ridge
@@ -517,7 +516,7 @@ training error reaches a saturation.
<section>
<h2 id="___sec12">Resampling methods: Jackknife and Bootstrap </h2>
<h2 id="resampling-methods-jackknife-and-bootstrap">Resampling methods: Jackknife and Bootstrap </h2>
<p>
Two famous
@@ -540,7 +539,7 @@ need for bootstrapping.
<section>
<h2 id="___sec13">Resampling methods: Jackknife </h2>
<h2 id="resampling-methods-jackknife">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
@@ -561,11 +560,11 @@ number \( i \) is left out. Using this notation, define
<section>
<h2 id="___sec14">Jackknife code example </h2>
<h2 id="jackknife-code-example">Jackknife code example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy.random</span> <span style="color: #8B008B; font-weight: bold">import</span> randint, randn
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">time</span> <span style="color: #8B008B; font-weight: bold">import</span> time
@@ -576,9 +575,9 @@ number \( i \) is left out. Using this notation, define
t[i] = stat(delete(data,i) )
<span style="color: #228B22"># analysis </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Runtime: %g sec&quot;</span> % (time()-t0)); <span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Jackknife Statistics :&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;original bias std. error&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;%8g %14g %15g&quot;</span> % (stat(data),(n-<span style="color: #B452CD">1</span>)*mean(t)/n, (n*var(t))**.<span style="color: #B452CD">5</span>))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Runtime: %g sec&quot;</span> % (time()-t0)); <span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Jackknife Statistics :&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;original bias std. error&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;%8g %14g %15g&quot;</span> % (stat(data),(n-<span style="color: #B452CD">1</span>)*mean(t)/n, (n*var(t))**.<span style="color: #B452CD">5</span>))
<span style="color: #8B008B; font-weight: bold">return</span> t
@@ -598,7 +597,7 @@ t = jackknife(x, stat)
<section>
<h2 id="___sec15">Resampling methods: Bootstrap </h2>
<h2 id="resampling-methods-bootstrap">Resampling methods: Bootstrap </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -620,7 +619,7 @@ advantages:
<section>
<h2 id="___sec16">Resampling methods: Bootstrap background </h2>
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
@@ -636,7 +635,7 @@ estimators.
<section>
<h2 id="___sec17">Resampling methods: More Bootstrap background </h2>
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\theta} \) has
@@ -659,7 +658,7 @@ idea is to use the relative frequency of \( \widehat{\theta}^* \)
<section>
<h2 id="___sec18">Resampling methods: Bootstrap approach </h2>
<h2 id="resampling-methods-bootstrap-approach">Resampling methods: Bootstrap approach </h2>
<p>
But
@@ -680,7 +679,7 @@ frequency of the observation \( X_i \), just draw the values
<section>
<h2 id="___sec19">Resampling methods: Bootstrap steps </h2>
<h2 id="resampling-methods-bootstrap-steps">Resampling methods: Bootstrap steps </h2>
<p>
The independent bootstrap works like this:
@@ -706,7 +705,7 @@ example, if you are interested in estimating the variance of \( \widehat
<section>
<h2 id="___sec20">Code example for the Bootstrap method </h2>
<h2 id="code-example-for-the-bootstrap-method">Code example for the Bootstrap method </h2>
<p>
The following code starts with a Gaussian distribution with mean value
@@ -725,7 +724,7 @@ theorem.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy.random</span> <span style="color: #8B008B; font-weight: bold">import</span> randint, randn
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">time</span> <span style="color: #8B008B; font-weight: bold">import</span> time
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.mlab</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">mlab</span>
@@ -743,9 +742,9 @@ theorem.
t[i] = statistic(data[randint(<span style="color: #B452CD">0</span>,n,n)])
<span style="color: #228B22"># analysis </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Runtime: %g sec&quot;</span> % (time()-t0)); <span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Bootstrap Statistics :&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;original bias std. error&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;%8g %8g %14g %15g&quot;</span> % (statistic(data), std(data),mean(t),std(t)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Runtime: %g sec&quot;</span> % (time()-t0)); <span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Bootstrap Statistics :&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;original bias std. error&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;%8g %8g %14g %15g&quot;</span> % (statistic(data), std(data),mean(t),std(t)))
<span style="color: #8B008B; font-weight: bold">return</span> t
@@ -763,7 +762,7 @@ lt = plt.plot(binsboot, y, <span style="color: #CD5555">&#39;r--&#39;</span>, li
plt.xlabel(<span style="color: #CD5555">&#39;Smarts&#39;</span>)
plt.ylabel(<span style="color: #CD5555">&#39;Probability&#39;</span>)
plt.axis([<span style="color: #B452CD">99.5</span>, <span style="color: #B452CD">100.6</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">3.0</span>])
plt.grid(<span style="color: #658b00">True</span>)
plt.grid(<span style="color: #8B008B; font-weight: bold">True</span>)
plt.show()
</pre></div>
@@ -771,7 +770,7 @@ plt.show()
<section>
<h2 id="___sec21">Various steps in cross-validation </h2>
<h2 id="various-steps-in-cross-validation">Various steps in cross-validation </h2>
<p>
When the repetitive splitting of the data set is done randomly,
@@ -792,7 +791,7 @@ cross-validation (LOOCV).
<section>
<h2 id="___sec22">How to set up the cross-validation for Ridge and/or Lasso </h2>
<h2 id="how-to-set-up-the-cross-validation-for-ridge-and-or-lasso">How to set up the cross-validation for Ridge and/or Lasso </h2>
<ul>
<p><li> Define a range of interest for the penalty parameter.</li>
@@ -826,7 +825,7 @@ $$
<section>
<h2 id="___sec23">Cross-validation in brief </h2>
<h2 id="cross-validation-in-brief">Cross-validation in brief </h2>
<p>
For the various values of \( k \)
@@ -848,14 +847,14 @@ For the various values of \( k \)
<section>
<h2 id="___sec24">Code Example for Cross-validation and \( k \)-fold Cross-validation </h2>
<h2 id="code-example-for-cross-validation-and-k-fold-cross-validation">Code Example for Cross-validation and \( k \)-fold Cross-validation </h2>
<p>
The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> KFold
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> Ridge
@@ -949,7 +948,7 @@ plt.show()
<section>
<h2 id="___sec25">The bias-variance tradeoff </h2>
<h2 id="the-bias-variance-tradeoff">The bias-variance tradeoff </h2>
<p>
We will discuss the bias-variance tradeoff in the context of
@@ -1027,11 +1026,11 @@ that is the rewriting in terms of the so-called bias, the variance of the model
<section>
<h2 id="___sec26">Example code for Bias-Variance tradeoff </h2>
<h2 id="example-code-for-bias-variance-tradeoff">Example code for Bias-Variance tradeoff </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression, Ridge, Lasso
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
@@ -1055,7 +1054,7 @@ x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=<span style=
<span style="color: #228B22"># Combine x transformation and model into one operation.</span>
<span style="color: #228B22"># Not neccesary, but convenient.</span>
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=<span style="color: #658b00">False</span>))
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=<span style="color: #8B008B; font-weight: bold">False</span>))
<span style="color: #228B22"># The following (m x n_bootstraps) matrix holds the column vectors y_pred</span>
<span style="color: #228B22"># for each bootstrap iteration.</span>
@@ -1072,13 +1071,13 @@ y_pred = np.empty((y_test.shape[<span style="color: #B452CD">0</span>], n_boostr
<span style="color: #228B22"># calculated per data point in the test set.</span>
<span style="color: #228B22"># Note 2: The use of keepdims=True is important in the calculation of bias as this </span>
<span style="color: #228B22"># maintains the column vector form. Dropping this yields very unexpected results.</span>
error = np.mean( np.mean((y_test - y_pred)**<span style="color: #B452CD">2</span>, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>) )
bias = np.mean( (y_test - np.mean(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>))**<span style="color: #B452CD">2</span> )
variance = np.mean( np.var(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>) )
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Error:&#39;</span>, error)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Bias^2:&#39;</span>, bias)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Var:&#39;</span>, variance)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;{} &gt;= {} + {} = {}&#39;</span>.format(error, bias, variance, bias+variance))
error = np.mean( np.mean((y_test - y_pred)**<span style="color: #B452CD">2</span>, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>) )
bias = np.mean( (y_test - np.mean(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>))**<span style="color: #B452CD">2</span> )
variance = np.mean( np.var(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>) )
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Error:&#39;</span>, error)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Bias^2:&#39;</span>, bias)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Var:&#39;</span>, variance)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;{} &gt;= {} + {} = {}&#39;</span>.format(error, bias, variance, bias+variance))
plt.plot(x[::<span style="color: #B452CD">5</span>, :], y[::<span style="color: #B452CD">5</span>, :], label=<span style="color: #CD5555">&#39;f(x)&#39;</span>)
plt.scatter(x_test, y_test, label=<span style="color: #CD5555">&#39;Data points&#39;</span>)
@@ -1090,11 +1089,11 @@ plt.show()
<section>
<h2 id="___sec27">Understanding what happens </h2>
<h2 id="understanding-what-happens">Understanding what happens </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression, Ridge, Lasso
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
@@ -1119,21 +1118,21 @@ polydegree = np.zeros(maxdegree)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #8B008B; font-weight: bold">for</span> degree <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(maxdegree):
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=<span style="color: #658b00">False</span>))
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=<span style="color: #8B008B; font-weight: bold">False</span>))
y_pred = np.empty((y_test.shape[<span style="color: #B452CD">0</span>], n_boostraps))
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(n_boostraps):
x_, y_ = resample(x_train, y_train)
y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
polydegree[degree] = degree
error[degree] = np.mean( np.mean((y_test - y_pred)**<span style="color: #B452CD">2</span>, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>) )
bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>))**<span style="color: #B452CD">2</span> )
variance[degree] = np.mean( np.var(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>) )
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Polynomial degree:&#39;</span>, degree)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Error:&#39;</span>, error[degree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;{} &gt;= {} + {} = {}&#39;</span>.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
error[degree] = np.mean( np.mean((y_test - y_pred)**<span style="color: #B452CD">2</span>, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>) )
bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>))**<span style="color: #B452CD">2</span> )
variance[degree] = np.mean( np.var(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>) )
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Polynomial degree:&#39;</span>, degree)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Error:&#39;</span>, error[degree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;{} &gt;= {} + {} = {}&#39;</span>.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
plt.plot(polydegree, error, label=<span style="color: #CD5555">&#39;Error&#39;</span>)
plt.plot(polydegree, bias, label=<span style="color: #CD5555">&#39;bias&#39;</span>)
@@ -1145,7 +1144,7 @@ plt.show()
<section>
<h2 id="___sec28">Summing up </h2>
<h2 id="summing-up">Summing up </h2>
<p>
The bias-variance tradeoff summarizes the fundamental tension in
@@ -1182,11 +1181,11 @@ You may also find this recent <a href="https://www.pnas.org/content/116/32/15849
<section>
<h2 id="___sec29">Another Example from Scikit-Learn's Repository </h2>
<h2 id="another-example-from-scikit-learn-s-repository">Another Example from Scikit-Learn's Repository </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">============================</span>
<span style="color: #CD5555">Underfitting vs. Overfitting</span>
<span style="color: #CD5555">============================</span>
@@ -1208,7 +1207,7 @@ You may also find this recent <a href="https://www.pnas.org/content/116/32/15849
<span style="color: #CD5555">training data.</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #00688B">__doc__</span>)
<span style="color: #658b00">print</span>(<span style="color: #00688B">__doc__</span>)
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
@@ -1235,7 +1234,7 @@ plt.figure(figsize=(<span style="color: #B452CD">14</span>, <span style="color:
plt.setp(ax, xticks=(), yticks=())
polynomial_features = PolynomialFeatures(degree=degrees[i],
include_bias=<span style="color: #658b00">False</span>)
include_bias=<span style="color: #8B008B; font-weight: bold">False</span>)
linear_regression = LinearRegression()
pipeline = Pipeline([(<span style="color: #CD5555">&quot;polynomial_features&quot;</span>, polynomial_features),
(<span style="color: #CD5555">&quot;linear_regression&quot;</span>, linear_regression)])
@@ -1262,12 +1261,12 @@ plt.show()
<section>
<h2 id="___sec30">More examples on bootstrap and cross-validation and errors </h2>
<h2 id="more-examples-on-bootstrap-and-cross-validation-and-errors">More examples on bootstrap and cross-validation and errors </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># Common imports</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># Common imports</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">os</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
@@ -1297,7 +1296,7 @@ DATA_ID = <span style="color: #CD5555">&quot;DataFiles/&quot;</span>
<span style="color: #8B008B; font-weight: bold">return</span> os.path.join(DATA_ID, dat_id)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">save_fig</span>(fig_id):
plt.savefig(image_path(fig_id) + <span style="color: #CD5555">&quot;.png&quot;</span>, format=<span style="color: #CD5555">&#39;png&#39;</span>)
plt.savefig(image_path(fig_id) + <span style="color: #CD5555">&quot;.png&quot;</span>, <span style="color: #658b00">format</span>=<span style="color: #CD5555">&#39;png&#39;</span>)
infile = <span style="color: #658b00">open</span>(data_path(<span style="color: #CD5555">&quot;EoS.csv&quot;</span>),<span style="color: #CD5555">&#39;r&#39;</span>)
@@ -1327,7 +1326,7 @@ trials = <span style="color: #B452CD">100</span>
trainingerror[polydegree] = <span style="color: #B452CD">0.0</span>
<span style="color: #8B008B; font-weight: bold">for</span> samples <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(trials):
x_train, x_test, y_train, y_test = train_test_split(X, Energies, test_size=<span style="color: #B452CD">0.2</span>)
model = LinearRegression(fit_intercept=<span style="color: #658b00">True</span>).fit(x_train, y_train)
model = LinearRegression(fit_intercept=<span style="color: #8B008B; font-weight: bold">True</span>).fit(x_train, y_train)
ypred = model.predict(x_train)
ytilde = model.predict(x_test)
testerror[polydegree] += mean_squared_error(y_test, ytilde)
@@ -1335,9 +1334,9 @@ trials = <span style="color: #B452CD">100</span>
testerror[polydegree] /= trials
trainingerror[polydegree] /= trials
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Degree of polynomial: %3d&quot;</span>% polynomial[polydegree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Mean squared error on training data: %.8f&quot;</span> % trainingerror[polydegree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Mean squared error on test data: %.8f&quot;</span> % testerror[polydegree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Degree of polynomial: %3d&quot;</span>% polynomial[polydegree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Mean squared error on training data: %.8f&quot;</span> % trainingerror[polydegree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Mean squared error on test data: %.8f&quot;</span> % testerror[polydegree])
plt.plot(polynomial, np.log10(trainingerror), label=<span style="color: #CD5555">&#39;Training Error&#39;</span>)
plt.plot(polynomial, np.log10(testerror), label=<span style="color: #CD5555">&#39;Test Error&#39;</span>)
@@ -1350,12 +1349,12 @@ plt.show()
<section>
<h2 id="___sec31">The same example but now with cross-validation </h2>
<h2 id="the-same-example-but-now-with-cross-validation">The same example but now with cross-validation </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #228B22"># Common imports</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #228B22"># Common imports</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">os</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
@@ -1387,7 +1386,7 @@ DATA_ID = <span style="color: #CD5555">&quot;DataFiles/&quot;</span>
<span style="color: #8B008B; font-weight: bold">return</span> os.path.join(DATA_ID, dat_id)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">save_fig</span>(fig_id):
plt.savefig(image_path(fig_id) + <span style="color: #CD5555">&quot;.png&quot;</span>, format=<span style="color: #CD5555">&#39;png&#39;</span>)
plt.savefig(image_path(fig_id) + <span style="color: #CD5555">&quot;.png&quot;</span>, <span style="color: #658b00">format</span>=<span style="color: #CD5555">&#39;png&#39;</span>)
infile = <span style="color: #658b00">open</span>(data_path(<span style="color: #CD5555">&quot;EoS.csv&quot;</span>),<span style="color: #CD5555">&#39;r&#39;</span>)
@@ -1427,11 +1426,11 @@ plt.show()
<section>
<h2 id="___sec32">Cross-validation with Ridge </h2>
<h2 id="cross-validation-with-ridge">Cross-validation with Ridge </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="font-size: 80%; line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> KFold
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> Ridge
@@ -1470,17 +1469,6 @@ plt.show()
</section>
<section>
<h2 id="___sec33">Friday September 4 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember4.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember4.pdf" target="_blank">handwritten notes</a>
<p>
More material will be added here, see handwritten notes also.
</section>
</div> <!-- class="slides" -->
</div> <!-- class="reveal" -->
doc/pub/week36/html/week36-solarized.html
+282 -255
View File
@@ -1,19 +1,19 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 36: Resampling techniques and Ordinary Least Square">
<meta name="description" content="Week 36: Statistical interpretation of Linear Regression and Resampling techniques">
<title>Week 36: Resampling techniques and Ordinary Least Square</title>
<title>Week 36: Statistical interpretation of Linear Regression and Resampling techniques</title>
<link href="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<link href="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/css/solarized_light_code.css" rel="stylesheet" type="text/css" title="light"/>
<script src="https://cdn.rawgit.com/doconce/doconce/master/bundled/html_styles/style_solarized_box/js/highlight.pack.js"></script>
<script>hljs.initHighlightingOnLoad();</script>
<link href="https://thomasf.github.io/solarized-css/solarized-light.min.css" rel="stylesheet">
@@ -48,10 +48,10 @@ pre {
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_yellow_question.png); }
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_yellow_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
@@ -61,72 +61,111 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Plans for week 36', 2, None, '___sec0'),
('Thursday September 3', 2, None, '___sec1'),
('Why resampling methods', 2, None, '___sec2'),
('Resampling methods', 2, None, '___sec3'),
('Resampling approaches can be computationally expensive',
2,
None,
'___sec4'),
('Why resampling methods ?', 2, None, '___sec5'),
('Statistical analysis', 2, None, '___sec6'),
'sections': [('Plans for week 36', 2, None, 'plans-for-week-36'),
('Thursday September 9', 2, None, 'thursday-september-9'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
None,
'___sec7'),
('Assumptions made', 2, None, '___sec8'),
('Expectation value and variance', 2, None, '___sec9'),
'linking-the-regression-analysis-with-a-statistical-interpretation'),
('Assumptions made', 2, None, 'assumptions-made'),
('Expectation value and variance',
2,
None,
'expectation-value-and-variance'),
('Expectation value and variance for $\\boldsymbol{\\beta}$',
2,
None,
'___sec10'),
('Resampling methods', 2, None, '___sec11'),
'expectation-value-and-variance-for-boldsymbol-beta'),
('Friday September 10', 2, None, 'friday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
2,
None,
'resampling-approaches-can-be-computationally-expensive'),
('Why resampling methods ?', 2, None, 'why-resampling-methods'),
('Statistical analysis', 2, None, 'statistical-analysis'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling methods: Jackknife and Bootstrap',
2,
None,
'___sec12'),
('Resampling methods: Jackknife', 2, None, '___sec13'),
('Jackknife code example', 2, None, '___sec14'),
('Resampling methods: Bootstrap', 2, None, '___sec15'),
('Resampling methods: Bootstrap background', 2, None, '___sec16'),
'resampling-methods-jackknife-and-bootstrap'),
('Resampling methods: Jackknife',
2,
None,
'resampling-methods-jackknife'),
('Jackknife code example', 2, None, 'jackknife-code-example'),
('Resampling methods: Bootstrap',
2,
None,
'resampling-methods-bootstrap'),
('Resampling methods: Bootstrap background',
2,
None,
'resampling-methods-bootstrap-background'),
('Resampling methods: More Bootstrap background',
2,
None,
'___sec17'),
('Resampling methods: Bootstrap approach', 2, None, '___sec18'),
('Resampling methods: Bootstrap steps', 2, None, '___sec19'),
('Code example for the Bootstrap method', 2, None, '___sec20'),
('Various steps in cross-validation', 2, None, '___sec21'),
'resampling-methods-more-bootstrap-background'),
('Resampling methods: Bootstrap approach',
2,
None,
'resampling-methods-bootstrap-approach'),
('Resampling methods: Bootstrap steps',
2,
None,
'resampling-methods-bootstrap-steps'),
('Code example for the Bootstrap method',
2,
None,
'code-example-for-the-bootstrap-method'),
('Various steps in cross-validation',
2,
None,
'various-steps-in-cross-validation'),
('How to set up the cross-validation for Ridge and/or Lasso',
2,
None,
'___sec22'),
('Cross-validation in brief', 2, None, '___sec23'),
'how-to-set-up-the-cross-validation-for-ridge-and-or-lasso'),
('Cross-validation in brief',
2,
None,
'cross-validation-in-brief'),
('Code Example for Cross-validation and $k$-fold '
'Cross-validation',
2,
None,
'___sec24'),
('The bias-variance tradeoff', 2, None, '___sec25'),
('Example code for Bias-Variance tradeoff', 2, None, '___sec26'),
('Understanding what happens', 2, None, '___sec27'),
('Summing up', 2, None, '___sec28'),
'code-example-for-cross-validation-and-k-fold-cross-validation'),
('The bias-variance tradeoff',
2,
None,
'the-bias-variance-tradeoff'),
('Example code for Bias-Variance tradeoff',
2,
None,
'example-code-for-bias-variance-tradeoff'),
('Understanding what happens',
2,
None,
'understanding-what-happens'),
('Summing up', 2, None, 'summing-up'),
("Another Example from Scikit-Learn's Repository",
2,
None,
'___sec29'),
'another-example-from-scikit-learn-s-repository'),
('More examples on bootstrap and cross-validation and errors',
2,
None,
'___sec30'),
'more-examples-on-bootstrap-and-cross-validation-and-errors'),
('The same example but now with cross-validation',
2,
None,
'___sec31'),
('Cross-validation with Ridge', 2, None, '___sec32'),
('Friday September 4', 2, None, '___sec33')]}
'the-same-example-but-now-with-cross-validation'),
('Cross-validation with Ridge',
2,
None,
'cross-validation-with-ridge')]}
end of tocinfo -->
<body>
@@ -152,7 +191,7 @@ MathJax.Hub.Config({
<center><h1>Week 36: Resampling techniques and Ordinary Least Square</h1></center> <!-- document title -->
<center><h1>Week 36: Statistical interpretation of Linear Regression and Resampling techniques</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -168,149 +207,26 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 6, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Plans for week 36 </h2>
<h2 id="plans-for-week-36">Plans for week 36 </h2>
<ul>
<li> Thursday: Statistics, probability theory and resampling methods</li>
<li> Friday: Resampling methods and motivation for Ridge Regression</li>
<li> Thursday: Statistics, probability theory and linear regression</li>
<li> Friday: Resampling methods</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Thursday September 3 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember3.pdf" target="_blank">handwritten notes</a>
<p>
More material will be added here, see handwritten notes also.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Why resampling methods </h2>
<p>
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
<ol>
<li> look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff</li>
<li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Resampling methods </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
<p>
Two resampling methods are often used in Machine Learning analyses,
<ol>
<li> The <b>bootstrap method</b></li>
<li> and <b>Cross-Validation</b></li>
</ol>
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Resampling approaches can be computationally expensive </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model&#8217;s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Why resampling methods ? </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Statistical analysis.</b>
<p>
<ul>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Statistical analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<ul>
<li> As in other experiments, many numerical experiments have two classes of errors:</li>
<ul>
<li> Statistical errors</li>
<li> Systematical errors</li>
</ul>
<li> Statistical errors can be estimated using standard tools from statistics</li>
<li> Systematical errors are method specific and must be treated differently from case to case.</li>
</ul>
</div>
<h2 id="thursday-september-9">Thursday September 9 </h2>
<p>
<!-- !split -->
<h2 id="___sec7">Linking the regression analysis with a statistical interpretation </h2>
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
<p>
The
@@ -347,7 +263,7 @@ row number \( i \) and perform a sum over all values \( p \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Assumptions made </h2>
<h2 id="assumptions-made">Assumptions made </h2>
<p>
The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
@@ -367,7 +283,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Expectation value and variance </h2>
<h2 id="expectation-value-and-variance">Expectation value and variance </h2>
<p>
We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
@@ -402,7 +318,7 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
<h2 id="expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
<p>
With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value
@@ -487,7 +403,129 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Resampling methods </h2>
<h2 id="friday-september-10">Friday September 10 </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="why-resampling-methods">Why resampling methods </h2>
<p>
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
<ol>
<li> look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff</li>
<li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="resampling-methods">Resampling methods </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
<p>
Two resampling methods are often used in Machine Learning analyses,
<ol>
<li> The <b>bootstrap method</b></li>
<li> and <b>Cross-Validation</b></li>
</ol>
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="resampling-approaches-can-be-computationally-expensive">Resampling approaches can be computationally expensive </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model&#8217;s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="why-resampling-methods">Why resampling methods ? </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Statistical analysis</b>
<p>
<ul>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="statistical-analysis">Statistical analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<ul>
<li> As in other experiments, many numerical experiments have two classes of errors:</li>
<ul>
<li> Statistical errors</li>
<li> Systematical errors</li>
</ul>
<li> Statistical errors can be estimated using standard tools from statistics</li>
<li> Systematical errors are method specific and must be treated differently from case to case.</li>
</ul>
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="resampling-methods">Resampling methods </h2>
<p>
With all these analytical equations for both the OLS and Ridge
@@ -516,7 +554,7 @@ training error reaches a saturation.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Resampling methods: Jackknife and Bootstrap </h2>
<h2 id="resampling-methods-jackknife-and-bootstrap">Resampling methods: Jackknife and Bootstrap </h2>
<p>
Two famous
@@ -539,7 +577,7 @@ need for bootstrapping.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Resampling methods: Jackknife </h2>
<h2 id="resampling-methods-jackknife">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
@@ -558,11 +596,11 @@ number \( i \) is left out. Using this notation, define
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Jackknife code example </h2>
<h2 id="jackknife-code-example">Jackknife code example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy.random</span> <span style="color: #8B008B; font-weight: bold">import</span> randint, randn
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">time</span> <span style="color: #8B008B; font-weight: bold">import</span> time
@@ -573,9 +611,9 @@ number \( i \) is left out. Using this notation, define
t[i] = stat(delete(data,i) )
<span style="color: #228B22"># analysis </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Runtime: %g sec&quot;</span> % (time()-t0)); <span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Jackknife Statistics :&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;original bias std. error&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;%8g %14g %15g&quot;</span> % (stat(data),(n-<span style="color: #B452CD">1</span>)*mean(t)/n, (n*var(t))**.<span style="color: #B452CD">5</span>))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Runtime: %g sec&quot;</span> % (time()-t0)); <span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Jackknife Statistics :&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;original bias std. error&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;%8g %14g %15g&quot;</span> % (stat(data),(n-<span style="color: #B452CD">1</span>)*mean(t)/n, (n*var(t))**.<span style="color: #B452CD">5</span>))
<span style="color: #8B008B; font-weight: bold">return</span> t
@@ -594,7 +632,7 @@ t = jackknife(x, stat)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Resampling methods: Bootstrap </h2>
<h2 id="resampling-methods-bootstrap">Resampling methods: Bootstrap </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -615,7 +653,7 @@ advantages:
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Resampling methods: Bootstrap background </h2>
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
@@ -631,7 +669,7 @@ estimators.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Resampling methods: More Bootstrap background </h2>
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\theta} \) has
@@ -653,7 +691,7 @@ idea is to use the relative frequency of \( \widehat{\theta}^* \)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Resampling methods: Bootstrap approach </h2>
<h2 id="resampling-methods-bootstrap-approach">Resampling methods: Bootstrap approach </h2>
<p>
But
@@ -674,7 +712,7 @@ frequency of the observation \( X_i \), just draw the values
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Resampling methods: Bootstrap steps </h2>
<h2 id="resampling-methods-bootstrap-steps">Resampling methods: Bootstrap steps </h2>
<p>
The independent bootstrap works like this:
@@ -699,7 +737,7 @@ example, if you are interested in estimating the variance of \( \widehat
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">Code example for the Bootstrap method </h2>
<h2 id="code-example-for-the-bootstrap-method">Code example for the Bootstrap method </h2>
<p>
The following code starts with a Gaussian distribution with mean value
@@ -718,7 +756,7 @@ theorem.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">import</span> *
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">numpy.random</span> <span style="color: #8B008B; font-weight: bold">import</span> randint, randn
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">time</span> <span style="color: #8B008B; font-weight: bold">import</span> time
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.mlab</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">mlab</span>
@@ -736,9 +774,9 @@ theorem.
t[i] = statistic(data[randint(<span style="color: #B452CD">0</span>,n,n)])
<span style="color: #228B22"># analysis </span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Runtime: %g sec&quot;</span> % (time()-t0)); <span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Bootstrap Statistics :&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;original bias std. error&quot;</span>)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;%8g %8g %14g %15g&quot;</span> % (statistic(data), std(data),mean(t),std(t)))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Runtime: %g sec&quot;</span> % (time()-t0)); <span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Bootstrap Statistics :&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;original bias std. error&quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;%8g %8g %14g %15g&quot;</span> % (statistic(data), std(data),mean(t),std(t)))
<span style="color: #8B008B; font-weight: bold">return</span> t
@@ -756,14 +794,14 @@ lt = plt.plot(binsboot, y, <span style="color: #CD5555">&#39;r--&#39;</span>, li
plt.xlabel(<span style="color: #CD5555">&#39;Smarts&#39;</span>)
plt.ylabel(<span style="color: #CD5555">&#39;Probability&#39;</span>)
plt.axis([<span style="color: #B452CD">99.5</span>, <span style="color: #B452CD">100.6</span>, <span style="color: #B452CD">0</span>, <span style="color: #B452CD">3.0</span>])
plt.grid(<span style="color: #658b00">True</span>)
plt.grid(<span style="color: #8B008B; font-weight: bold">True</span>)
plt.show()
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec21">Various steps in cross-validation </h2>
<h2 id="various-steps-in-cross-validation">Various steps in cross-validation </h2>
<p>
When the repetitive splitting of the data set is done randomly,
@@ -784,7 +822,7 @@ cross-validation (LOOCV).
<p>
<!-- !split -->
<h2 id="___sec22">How to set up the cross-validation for Ridge and/or Lasso </h2>
<h2 id="how-to-set-up-the-cross-validation-for-ridge-and-or-lasso">How to set up the cross-validation for Ridge and/or Lasso </h2>
<ul>
<li> Define a range of interest for the penalty parameter.</li>
@@ -816,7 +854,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Cross-validation in brief </h2>
<h2 id="cross-validation-in-brief">Cross-validation in brief </h2>
<p>
For the various values of \( k \)
@@ -838,14 +876,14 @@ For the various values of \( k \)
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Code Example for Cross-validation and \( k \)-fold Cross-validation </h2>
<h2 id="code-example-for-cross-validation-and-k-fold-cross-validation">Code Example for Cross-validation and \( k \)-fold Cross-validation </h2>
<p>
The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> KFold
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> Ridge
@@ -938,7 +976,7 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">The bias-variance tradeoff </h2>
<h2 id="the-bias-variance-tradeoff">The bias-variance tradeoff </h2>
<p>
We will discuss the bias-variance tradeoff in the context of
@@ -1004,11 +1042,11 @@ that is the rewriting in terms of the so-called bias, the variance of the model
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Example code for Bias-Variance tradeoff </h2>
<h2 id="example-code-for-bias-variance-tradeoff">Example code for Bias-Variance tradeoff </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression, Ridge, Lasso
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
@@ -1032,7 +1070,7 @@ x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=<span style=
<span style="color: #228B22"># Combine x transformation and model into one operation.</span>
<span style="color: #228B22"># Not neccesary, but convenient.</span>
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=<span style="color: #658b00">False</span>))
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=<span style="color: #8B008B; font-weight: bold">False</span>))
<span style="color: #228B22"># The following (m x n_bootstraps) matrix holds the column vectors y_pred</span>
<span style="color: #228B22"># for each bootstrap iteration.</span>
@@ -1049,13 +1087,13 @@ y_pred = np.empty((y_test.shape[<span style="color: #B452CD">0</span>], n_boostr
<span style="color: #228B22"># calculated per data point in the test set.</span>
<span style="color: #228B22"># Note 2: The use of keepdims=True is important in the calculation of bias as this </span>
<span style="color: #228B22"># maintains the column vector form. Dropping this yields very unexpected results.</span>
error = np.mean( np.mean((y_test - y_pred)**<span style="color: #B452CD">2</span>, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>) )
bias = np.mean( (y_test - np.mean(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>))**<span style="color: #B452CD">2</span> )
variance = np.mean( np.var(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>) )
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Error:&#39;</span>, error)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Bias^2:&#39;</span>, bias)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Var:&#39;</span>, variance)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;{} &gt;= {} + {} = {}&#39;</span>.format(error, bias, variance, bias+variance))
error = np.mean( np.mean((y_test - y_pred)**<span style="color: #B452CD">2</span>, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>) )
bias = np.mean( (y_test - np.mean(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>))**<span style="color: #B452CD">2</span> )
variance = np.mean( np.var(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>) )
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Error:&#39;</span>, error)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Bias^2:&#39;</span>, bias)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Var:&#39;</span>, variance)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;{} &gt;= {} + {} = {}&#39;</span>.format(error, bias, variance, bias+variance))
plt.plot(x[::<span style="color: #B452CD">5</span>, :], y[::<span style="color: #B452CD">5</span>, :], label=<span style="color: #CD5555">&#39;f(x)&#39;</span>)
plt.scatter(x_test, y_test, label=<span style="color: #CD5555">&#39;Data points&#39;</span>)
@@ -1066,11 +1104,11 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Understanding what happens </h2>
<h2 id="understanding-what-happens">Understanding what happens </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression, Ridge, Lasso
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
@@ -1095,21 +1133,21 @@ polydegree = np.zeros(maxdegree)
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #8B008B; font-weight: bold">for</span> degree <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(maxdegree):
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=<span style="color: #658b00">False</span>))
model = make_pipeline(PolynomialFeatures(degree=degree), LinearRegression(fit_intercept=<span style="color: #8B008B; font-weight: bold">False</span>))
y_pred = np.empty((y_test.shape[<span style="color: #B452CD">0</span>], n_boostraps))
<span style="color: #8B008B; font-weight: bold">for</span> i <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(n_boostraps):
x_, y_ = resample(x_train, y_train)
y_pred[:, i] = model.fit(x_, y_).predict(x_test).ravel()
polydegree[degree] = degree
error[degree] = np.mean( np.mean((y_test - y_pred)**<span style="color: #B452CD">2</span>, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>) )
bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>))**<span style="color: #B452CD">2</span> )
variance[degree] = np.mean( np.var(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #658b00">True</span>) )
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Polynomial degree:&#39;</span>, degree)
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Error:&#39;</span>, error[degree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&#39;{} &gt;= {} + {} = {}&#39;</span>.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
error[degree] = np.mean( np.mean((y_test - y_pred)**<span style="color: #B452CD">2</span>, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>) )
bias[degree] = np.mean( (y_test - np.mean(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>))**<span style="color: #B452CD">2</span> )
variance[degree] = np.mean( np.var(y_pred, axis=<span style="color: #B452CD">1</span>, keepdims=<span style="color: #8B008B; font-weight: bold">True</span>) )
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Polynomial degree:&#39;</span>, degree)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Error:&#39;</span>, error[degree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&#39;{} &gt;= {} + {} = {}&#39;</span>.format(error[degree], bias[degree], variance[degree], bias[degree]+variance[degree]))
plt.plot(polydegree, error, label=<span style="color: #CD5555">&#39;Error&#39;</span>)
plt.plot(polydegree, bias, label=<span style="color: #CD5555">&#39;bias&#39;</span>)
@@ -1120,7 +1158,7 @@ plt.show()
<p>
<!-- !split -->
<h2 id="___sec28">Summing up </h2>
<h2 id="summing-up">Summing up </h2>
<p>
The bias-variance tradeoff summarizes the fundamental tension in
@@ -1157,11 +1195,11 @@ You may also find this recent <a href="https://www.pnas.org/content/116/32/15849
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">Another Example from Scikit-Learn's Repository </h2>
<h2 id="another-example-from-scikit-learn-s-repository">Another Example from Scikit-Learn's Repository </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #CD5555">============================</span>
<span style="color: #CD5555">Underfitting vs. Overfitting</span>
<span style="color: #CD5555">============================</span>
@@ -1183,7 +1221,7 @@ You may also find this recent <a href="https://www.pnas.org/content/116/32/15849
<span style="color: #CD5555">training data.</span>
<span style="color: #CD5555">&quot;&quot;&quot;</span>
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #00688B">__doc__</span>)
<span style="color: #658b00">print</span>(<span style="color: #00688B">__doc__</span>)
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
@@ -1210,7 +1248,7 @@ plt.figure(figsize=(<span style="color: #B452CD">14</span>, <span style="color:
plt.setp(ax, xticks=(), yticks=())
polynomial_features = PolynomialFeatures(degree=degrees[i],
include_bias=<span style="color: #658b00">False</span>)
include_bias=<span style="color: #8B008B; font-weight: bold">False</span>)
linear_regression = LinearRegression()
pipeline = Pipeline([(<span style="color: #CD5555">&quot;polynomial_features&quot;</span>, polynomial_features),
(<span style="color: #CD5555">&quot;linear_regression&quot;</span>, linear_regression)])
@@ -1236,12 +1274,12 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">More examples on bootstrap and cross-validation and errors </h2>
<h2 id="more-examples-on-bootstrap-and-cross-validation-and-errors">More examples on bootstrap and cross-validation and errors </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># Common imports</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># Common imports</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">os</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
@@ -1271,7 +1309,7 @@ DATA_ID = <span style="color: #CD5555">&quot;DataFiles/&quot;</span>
<span style="color: #8B008B; font-weight: bold">return</span> os.path.join(DATA_ID, dat_id)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">save_fig</span>(fig_id):
plt.savefig(image_path(fig_id) + <span style="color: #CD5555">&quot;.png&quot;</span>, format=<span style="color: #CD5555">&#39;png&#39;</span>)
plt.savefig(image_path(fig_id) + <span style="color: #CD5555">&quot;.png&quot;</span>, <span style="color: #658b00">format</span>=<span style="color: #CD5555">&#39;png&#39;</span>)
infile = <span style="color: #658b00">open</span>(data_path(<span style="color: #CD5555">&quot;EoS.csv&quot;</span>),<span style="color: #CD5555">&#39;r&#39;</span>)
@@ -1301,7 +1339,7 @@ trials = <span style="color: #B452CD">100</span>
trainingerror[polydegree] = <span style="color: #B452CD">0.0</span>
<span style="color: #8B008B; font-weight: bold">for</span> samples <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(trials):
x_train, x_test, y_train, y_test = train_test_split(X, Energies, test_size=<span style="color: #B452CD">0.2</span>)
model = LinearRegression(fit_intercept=<span style="color: #658b00">True</span>).fit(x_train, y_train)
model = LinearRegression(fit_intercept=<span style="color: #8B008B; font-weight: bold">True</span>).fit(x_train, y_train)
ypred = model.predict(x_train)
ytilde = model.predict(x_test)
testerror[polydegree] += mean_squared_error(y_test, ytilde)
@@ -1309,9 +1347,9 @@ trials = <span style="color: #B452CD">100</span>
testerror[polydegree] /= trials
trainingerror[polydegree] /= trials
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Degree of polynomial: %3d&quot;</span>% polynomial[polydegree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Mean squared error on training data: %.8f&quot;</span> % trainingerror[polydegree])
<span style="color: #8B008B; font-weight: bold">print</span>(<span style="color: #CD5555">&quot;Mean squared error on test data: %.8f&quot;</span> % testerror[polydegree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Degree of polynomial: %3d&quot;</span>% polynomial[polydegree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Mean squared error on training data: %.8f&quot;</span> % trainingerror[polydegree])
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;Mean squared error on test data: %.8f&quot;</span> % testerror[polydegree])
plt.plot(polynomial, np.log10(trainingerror), label=<span style="color: #CD5555">&#39;Training Error&#39;</span>)
plt.plot(polynomial, np.log10(testerror), label=<span style="color: #CD5555">&#39;Test Error&#39;</span>)
@@ -1323,12 +1361,12 @@ plt.show()
<p>
<!-- !split -->
<h2 id="___sec31">The same example but now with cross-validation </h2>
<h2 id="the-same-example-but-now-with-cross-validation">The same example but now with cross-validation </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #228B22"># Common imports</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #228B22"># Common imports</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">os</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">pandas</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">pd</span>
@@ -1360,7 +1398,7 @@ DATA_ID = <span style="color: #CD5555">&quot;DataFiles/&quot;</span>
<span style="color: #8B008B; font-weight: bold">return</span> os.path.join(DATA_ID, dat_id)
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">save_fig</span>(fig_id):
plt.savefig(image_path(fig_id) + <span style="color: #CD5555">&quot;.png&quot;</span>, format=<span style="color: #CD5555">&#39;png&#39;</span>)
plt.savefig(image_path(fig_id) + <span style="color: #CD5555">&quot;.png&quot;</span>, <span style="color: #658b00">format</span>=<span style="color: #CD5555">&#39;png&#39;</span>)
infile = <span style="color: #658b00">open</span>(data_path(<span style="color: #CD5555">&quot;EoS.csv&quot;</span>),<span style="color: #CD5555">&#39;r&#39;</span>)
@@ -1399,11 +1437,11 @@ plt.show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec32">Cross-validation with Ridge </h2>
<h2 id="cross-validation-with-ridge">Cross-validation with Ridge </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<div class="highlight" style="background: #eeeedd"><pre style="line-height: 125%;"><span></span><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> KFold
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> Ridge
@@ -1439,24 +1477,13 @@ plt.ylabel(<span style="color: #CD5555">&#39;MSE&#39;</span>)
plt.legend()
plt.show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec33">Friday September 4 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember4.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember4.pdf" target="_blank">handwritten notes</a>
<p>
More material will be added here, see handwritten notes also.
<p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week36/html/week36.html
+281 -254
View File
@@ -1,15 +1,15 @@
<!--
Automatically generated HTML file from DocOnce source
(https://github.com/hplgit/doconce/)
(https://github.com/doconce/doconce/)
-->
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="DocOnce: https://github.com/hplgit/doconce/" />
<meta name="generator" content="DocOnce: https://github.com/doconce/doconce/" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="description" content="Week 36: Resampling techniques and Ordinary Least Square">
<meta name="description" content="Week 36: Statistical interpretation of Linear Regression and Resampling techniques">
<title>Week 36: Resampling techniques and Ordinary Least Square</title>
<title>Week 36: Statistical interpretation of Linear Regression and Resampling techniques</title>
<style type="text/css">
@@ -53,10 +53,10 @@ hr.figure { border: 0; width: 80%; border-bottom: 1px solid #aaa}
.alert-block > p, .alert-block > ul {margin-bottom:1em}
.alert li {margin-top: 1em}
.alert-block p+p {margin-top:5px}
.alert-notice { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/hplgit/doconce/master/bundled/html_images/small_gray_question.png); }
.alert-notice { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_notice.png); }
.alert-summary { background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_summary.png); }
.alert-warning { background-image: url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_warning.png); }
.alert-question {background-image:url(https://cdn.rawgit.com/doconce/doconce/master/bundled/html_images/small_gray_question.png); }
div { text-align: justify; text-justify: inter-word; }
</style>
@@ -66,72 +66,111 @@ div { text-align: justify; text-justify: inter-word; }
<!-- tocinfo
{'highest level': 2,
'sections': [('Plans for week 36', 2, None, '___sec0'),
('Thursday September 3', 2, None, '___sec1'),
('Why resampling methods', 2, None, '___sec2'),
('Resampling methods', 2, None, '___sec3'),
('Resampling approaches can be computationally expensive',
2,
None,
'___sec4'),
('Why resampling methods ?', 2, None, '___sec5'),
('Statistical analysis', 2, None, '___sec6'),
'sections': [('Plans for week 36', 2, None, 'plans-for-week-36'),
('Thursday September 9', 2, None, 'thursday-september-9'),
('Linking the regression analysis with a statistical '
'interpretation',
2,
None,
'___sec7'),
('Assumptions made', 2, None, '___sec8'),
('Expectation value and variance', 2, None, '___sec9'),
'linking-the-regression-analysis-with-a-statistical-interpretation'),
('Assumptions made', 2, None, 'assumptions-made'),
('Expectation value and variance',
2,
None,
'expectation-value-and-variance'),
('Expectation value and variance for $\\boldsymbol{\\beta}$',
2,
None,
'___sec10'),
('Resampling methods', 2, None, '___sec11'),
'expectation-value-and-variance-for-boldsymbol-beta'),
('Friday September 10', 2, None, 'friday-september-10'),
('Why resampling methods', 2, None, 'why-resampling-methods'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling approaches can be computationally expensive',
2,
None,
'resampling-approaches-can-be-computationally-expensive'),
('Why resampling methods ?', 2, None, 'why-resampling-methods'),
('Statistical analysis', 2, None, 'statistical-analysis'),
('Resampling methods', 2, None, 'resampling-methods'),
('Resampling methods: Jackknife and Bootstrap',
2,
None,
'___sec12'),
('Resampling methods: Jackknife', 2, None, '___sec13'),
('Jackknife code example', 2, None, '___sec14'),
('Resampling methods: Bootstrap', 2, None, '___sec15'),
('Resampling methods: Bootstrap background', 2, None, '___sec16'),
'resampling-methods-jackknife-and-bootstrap'),
('Resampling methods: Jackknife',
2,
None,
'resampling-methods-jackknife'),
('Jackknife code example', 2, None, 'jackknife-code-example'),
('Resampling methods: Bootstrap',
2,
None,
'resampling-methods-bootstrap'),
('Resampling methods: Bootstrap background',
2,
None,
'resampling-methods-bootstrap-background'),
('Resampling methods: More Bootstrap background',
2,
None,
'___sec17'),
('Resampling methods: Bootstrap approach', 2, None, '___sec18'),
('Resampling methods: Bootstrap steps', 2, None, '___sec19'),
('Code example for the Bootstrap method', 2, None, '___sec20'),
('Various steps in cross-validation', 2, None, '___sec21'),
'resampling-methods-more-bootstrap-background'),
('Resampling methods: Bootstrap approach',
2,
None,
'resampling-methods-bootstrap-approach'),
('Resampling methods: Bootstrap steps',
2,
None,
'resampling-methods-bootstrap-steps'),
('Code example for the Bootstrap method',
2,
None,
'code-example-for-the-bootstrap-method'),
('Various steps in cross-validation',
2,
None,
'various-steps-in-cross-validation'),
('How to set up the cross-validation for Ridge and/or Lasso',
2,
None,
'___sec22'),
('Cross-validation in brief', 2, None, '___sec23'),
'how-to-set-up-the-cross-validation-for-ridge-and-or-lasso'),
('Cross-validation in brief',
2,
None,
'cross-validation-in-brief'),
('Code Example for Cross-validation and $k$-fold '
'Cross-validation',
2,
None,
'___sec24'),
('The bias-variance tradeoff', 2, None, '___sec25'),
('Example code for Bias-Variance tradeoff', 2, None, '___sec26'),
('Understanding what happens', 2, None, '___sec27'),
('Summing up', 2, None, '___sec28'),
'code-example-for-cross-validation-and-k-fold-cross-validation'),
('The bias-variance tradeoff',
2,
None,
'the-bias-variance-tradeoff'),
('Example code for Bias-Variance tradeoff',
2,
None,
'example-code-for-bias-variance-tradeoff'),
('Understanding what happens',
2,
None,
'understanding-what-happens'),
('Summing up', 2, None, 'summing-up'),
("Another Example from Scikit-Learn's Repository",
2,
None,
'___sec29'),
'another-example-from-scikit-learn-s-repository'),
('More examples on bootstrap and cross-validation and errors',
2,
None,
'___sec30'),
'more-examples-on-bootstrap-and-cross-validation-and-errors'),
('The same example but now with cross-validation',
2,
None,
'___sec31'),
('Cross-validation with Ridge', 2, None, '___sec32'),
('Friday September 4', 2, None, '___sec33')]}
'the-same-example-but-now-with-cross-validation'),
('Cross-validation with Ridge',
2,
None,
'cross-validation-with-ridge')]}
end of tocinfo -->
<body>
@@ -157,7 +196,7 @@ MathJax.Hub.Config({
<center><h1>Week 36: Resampling techniques and Ordinary Least Square</h1></center> <!-- document title -->
<center><h1>Week 36: Statistical interpretation of Linear Regression and Resampling techniques</h1></center> <!-- document title -->
<p>
<!-- author(s): Morten Hjorth-Jensen -->
@@ -173,149 +212,26 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 16, 2020</h4></center> <!-- date -->
<center><h4>Sep 6, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec0">Plans for week 36 </h2>
<h2 id="plans-for-week-36">Plans for week 36 </h2>
<ul>
<li> Thursday: Statistics, probability theory and resampling methods</li>
<li> Friday: Resampling methods and motivation for Ridge Regression</li>
<li> Thursday: Statistics, probability theory and linear regression</li>
<li> Friday: Resampling methods</li>
</ul>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec1">Thursday September 3 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember3.pdf" target="_blank">handwritten notes</a>
<p>
More material will be added here, see handwritten notes also.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec2">Why resampling methods </h2>
<p>
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
<ol>
<li> look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff</li>
<li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec3">Resampling methods </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
<p>
Two resampling methods are often used in Machine Learning analyses,
<ol>
<li> The <b>bootstrap method</b></li>
<li> and <b>Cross-Validation</b></li>
</ol>
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec4">Resampling approaches can be computationally expensive </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model&#8217;s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec5">Why resampling methods ? </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Statistical analysis.</b>
<p>
<ul>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec6">Statistical analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<ul>
<li> As in other experiments, many numerical experiments have two classes of errors:</li>
<ul>
<li> Statistical errors</li>
<li> Systematical errors</li>
</ul>
<li> Statistical errors can be estimated using standard tools from statistics</li>
<li> Systematical errors are method specific and must be treated differently from case to case.</li>
</ul>
</div>
<h2 id="thursday-september-9">Thursday September 9 </h2>
<p>
<!-- !split -->
<h2 id="___sec7">Linking the regression analysis with a statistical interpretation </h2>
<h2 id="linking-the-regression-analysis-with-a-statistical-interpretation">Linking the regression analysis with a statistical interpretation </h2>
<p>
The
@@ -352,7 +268,7 @@ row number \( i \) and perform a sum over all values \( p \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec8">Assumptions made </h2>
<h2 id="assumptions-made">Assumptions made </h2>
<p>
The assumption we have made here can be summarized as (and this is going to be useful when we discuss the bias-variance trade off)
@@ -372,7 +288,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec9">Expectation value and variance </h2>
<h2 id="expectation-value-and-variance">Expectation value and variance </h2>
<p>
We can calculate the expectation value of \( \boldsymbol{y} \) for a given element \( i \)
@@ -407,7 +323,7 @@ mean value \( \boldsymbol{X}\boldsymbol{\beta} \) and variance \( \sigma^2 \) (n
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec10">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
<h2 id="expectation-value-and-variance-for-boldsymbol-beta">Expectation value and variance for \( \boldsymbol{\beta} \) </h2>
<p>
With the OLS expressions for the parameters \( \boldsymbol{\beta} \) we can evaluate the expectation value
@@ -492,7 +408,129 @@ This means the variance we obtain with the standard OLS will always for \( \lamb
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec11">Resampling methods </h2>
<h2 id="friday-september-10">Friday September 10 </h2>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="why-resampling-methods">Why resampling methods </h2>
<p>
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
<ol>
<li> look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff</li>
<li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="resampling-methods">Resampling methods </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
<p>
Two resampling methods are often used in Machine Learning analyses,
<ol>
<li> The <b>bootstrap method</b></li>
<li> and <b>Cross-Validation</b></li>
</ol>
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="resampling-approaches-can-be-computationally-expensive">Resampling approaches can be computationally expensive </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<p>
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model&#8217;s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="why-resampling-methods">Why resampling methods ? </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b>Statistical analysis</b>
<p>
<ul>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="statistical-analysis">Statistical analysis </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
<ul>
<li> As in other experiments, many numerical experiments have two classes of errors:</li>
<ul>
<li> Statistical errors</li>
<li> Systematical errors</li>
</ul>
<li> Statistical errors can be estimated using standard tools from statistics</li>
<li> Systematical errors are method specific and must be treated differently from case to case.</li>
</ul>
</div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="resampling-methods">Resampling methods </h2>
<p>
With all these analytical equations for both the OLS and Ridge
@@ -521,7 +559,7 @@ training error reaches a saturation.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec12">Resampling methods: Jackknife and Bootstrap </h2>
<h2 id="resampling-methods-jackknife-and-bootstrap">Resampling methods: Jackknife and Bootstrap </h2>
<p>
Two famous
@@ -544,7 +582,7 @@ need for bootstrapping.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec13">Resampling methods: Jackknife </h2>
<h2 id="resampling-methods-jackknife">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
@@ -563,11 +601,11 @@ number \( i \) is left out. Using this notation, define
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec14">Jackknife code example </h2>
<h2 id="jackknife-code-example">Jackknife code example </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">import</span> <span style="color: #666666">*</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">import</span> <span style="color: #666666">*</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">numpy.random</span> <span style="color: #008000; font-weight: bold">import</span> randint, randn
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">time</span> <span style="color: #008000; font-weight: bold">import</span> time
@@ -578,9 +616,9 @@ number \( i \) is left out. Using this notation, define
t[i] <span style="color: #666666">=</span> stat(delete(data,i) )
<span style="color: #408080; font-style: italic"># analysis </span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Runtime: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> sec&quot;</span> <span style="color: #666666">%</span> (time()<span style="color: #666666">-</span>t0)); <span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Jackknife Statistics :&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;original bias std. error&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;</span><span style="color: #BB6688; font-weight: bold">%8g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%14g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%15g</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> (stat(data),(n<span style="color: #666666">-1</span>)<span style="color: #666666">*</span>mean(t)<span style="color: #666666">/</span>n, (n<span style="color: #666666">*</span>var(t))<span style="color: #666666">**.5</span>))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Runtime: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> sec&quot;</span> <span style="color: #666666">%</span> (time()<span style="color: #666666">-</span>t0)); <span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Jackknife Statistics :&quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;original bias std. error&quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;</span><span style="color: #BB6688; font-weight: bold">%8g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%14g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%15g</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> (stat(data),(n<span style="color: #666666">-1</span>)<span style="color: #666666">*</span>mean(t)<span style="color: #666666">/</span>n, (n<span style="color: #666666">*</span>var(t))<span style="color: #666666">**.5</span>))
<span style="color: #008000; font-weight: bold">return</span> t
@@ -599,7 +637,7 @@ t <span style="color: #666666">=</span> jackknife(x, stat)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec15">Resampling methods: Bootstrap </h2>
<h2 id="resampling-methods-bootstrap">Resampling methods: Bootstrap </h2>
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
@@ -620,7 +658,7 @@ advantages:
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec16">Resampling methods: Bootstrap background </h2>
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
@@ -636,7 +674,7 @@ estimators.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec17">Resampling methods: More Bootstrap background </h2>
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\theta} \) has
@@ -658,7 +696,7 @@ idea is to use the relative frequency of \( \widehat{\theta}^* \)
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec18">Resampling methods: Bootstrap approach </h2>
<h2 id="resampling-methods-bootstrap-approach">Resampling methods: Bootstrap approach </h2>
<p>
But
@@ -679,7 +717,7 @@ frequency of the observation \( X_i \), just draw the values
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec19">Resampling methods: Bootstrap steps </h2>
<h2 id="resampling-methods-bootstrap-steps">Resampling methods: Bootstrap steps </h2>
<p>
The independent bootstrap works like this:
@@ -704,7 +742,7 @@ example, if you are interested in estimating the variance of \( \widehat
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec20">Code example for the Bootstrap method </h2>
<h2 id="code-example-for-the-bootstrap-method">Code example for the Bootstrap method </h2>
<p>
The following code starts with a Gaussian distribution with mean value
@@ -723,7 +761,7 @@ theorem.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">import</span> <span style="color: #666666">*</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">import</span> <span style="color: #666666">*</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">numpy.random</span> <span style="color: #008000; font-weight: bold">import</span> randint, randn
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">time</span> <span style="color: #008000; font-weight: bold">import</span> time
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.mlab</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">mlab</span>
@@ -741,9 +779,9 @@ theorem.
t[i] <span style="color: #666666">=</span> statistic(data[randint(<span style="color: #666666">0</span>,n,n)])
<span style="color: #408080; font-style: italic"># analysis </span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Runtime: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> sec&quot;</span> <span style="color: #666666">%</span> (time()<span style="color: #666666">-</span>t0)); <span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Bootstrap Statistics :&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;original bias std. error&quot;</span>)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;</span><span style="color: #BB6688; font-weight: bold">%8g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%8g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%14g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%15g</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> (statistic(data), std(data),mean(t),std(t)))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Runtime: </span><span style="color: #BB6688; font-weight: bold">%g</span><span style="color: #BA2121"> sec&quot;</span> <span style="color: #666666">%</span> (time()<span style="color: #666666">-</span>t0)); <span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Bootstrap Statistics :&quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;original bias std. error&quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;</span><span style="color: #BB6688; font-weight: bold">%8g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%8g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%14g</span><span style="color: #BA2121"> </span><span style="color: #BB6688; font-weight: bold">%15g</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> (statistic(data), std(data),mean(t),std(t)))
<span style="color: #008000; font-weight: bold">return</span> t
@@ -761,14 +799,14 @@ lt <span style="color: #666666">=</span> plt<span style="color: #666666">.</span
plt<span style="color: #666666">.</span>xlabel(<span style="color: #BA2121">&#39;Smarts&#39;</span>)
plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&#39;Probability&#39;</span>)
plt<span style="color: #666666">.</span>axis([<span style="color: #666666">99.5</span>, <span style="color: #666666">100.6</span>, <span style="color: #666666">0</span>, <span style="color: #666666">3.0</span>])
plt<span style="color: #666666">.</span>grid(<span style="color: #008000">True</span>)
plt<span style="color: #666666">.</span>grid(<span style="color: #008000; font-weight: bold">True</span>)
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split -->
<h2 id="___sec21">Various steps in cross-validation </h2>
<h2 id="various-steps-in-cross-validation">Various steps in cross-validation </h2>
<p>
When the repetitive splitting of the data set is done randomly,
@@ -789,7 +827,7 @@ cross-validation (LOOCV).
<p>
<!-- !split -->
<h2 id="___sec22">How to set up the cross-validation for Ridge and/or Lasso </h2>
<h2 id="how-to-set-up-the-cross-validation-for-ridge-and-or-lasso">How to set up the cross-validation for Ridge and/or Lasso </h2>
<ul>
<li> Define a range of interest for the penalty parameter.</li>
@@ -821,7 +859,7 @@ $$
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec23">Cross-validation in brief </h2>
<h2 id="cross-validation-in-brief">Cross-validation in brief </h2>
<p>
For the various values of \( k \)
@@ -843,14 +881,14 @@ For the various values of \( k \)
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec24">Code Example for Cross-validation and \( k \)-fold Cross-validation </h2>
<h2 id="code-example-for-cross-validation-and-k-fold-cross-validation">Code Example for Cross-validation and \( k \)-fold Cross-validation </h2>
<p>
The code here uses Ridge regression with cross-validation (CV) resampling and \( k \)-fold CV in order to fit a specific polynomial.
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> KFold
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> Ridge
@@ -943,7 +981,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec25">The bias-variance tradeoff </h2>
<h2 id="the-bias-variance-tradeoff">The bias-variance tradeoff </h2>
<p>
We will discuss the bias-variance tradeoff in the context of
@@ -1009,11 +1047,11 @@ that is the rewriting in terms of the so-called bias, the variance of the model
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec26">Example code for Bias-Variance tradeoff </h2>
<h2 id="example-code-for-bias-variance-tradeoff">Example code for Bias-Variance tradeoff </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression, Ridge, Lasso
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
@@ -1037,7 +1075,7 @@ x_train, x_test, y_train, y_test <span style="color: #666666">=</span> train_tes
<span style="color: #408080; font-style: italic"># Combine x transformation and model into one operation.</span>
<span style="color: #408080; font-style: italic"># Not neccesary, but convenient.</span>
model <span style="color: #666666">=</span> make_pipeline(PolynomialFeatures(degree<span style="color: #666666">=</span>degree), LinearRegression(fit_intercept<span style="color: #666666">=</span><span style="color: #008000">False</span>))
model <span style="color: #666666">=</span> make_pipeline(PolynomialFeatures(degree<span style="color: #666666">=</span>degree), LinearRegression(fit_intercept<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>))
<span style="color: #408080; font-style: italic"># The following (m x n_bootstraps) matrix holds the column vectors y_pred</span>
<span style="color: #408080; font-style: italic"># for each bootstrap iteration.</span>
@@ -1054,13 +1092,13 @@ y_pred <span style="color: #666666">=</span> np<span style="color: #666666">.</s
<span style="color: #408080; font-style: italic"># calculated per data point in the test set.</span>
<span style="color: #408080; font-style: italic"># Note 2: The use of keepdims=True is important in the calculation of bias as this </span>
<span style="color: #408080; font-style: italic"># maintains the column vector form. Dropping this yields very unexpected results.</span>
error <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>mean((y_test <span style="color: #666666">-</span> y_pred)<span style="color: #666666">**2</span>, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000">True</span>) )
bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( (y_test <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y_pred, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000">True</span>))<span style="color: #666666">**2</span> )
variance <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>var(y_pred, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000">True</span>) )
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Error:&#39;</span>, error)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Bias^2:&#39;</span>, bias)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Var:&#39;</span>, variance)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;{} &gt;= {} + {} = {}&#39;</span><span style="color: #666666">.</span>format(error, bias, variance, bias<span style="color: #666666">+</span>variance))
error <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>mean((y_test <span style="color: #666666">-</span> y_pred)<span style="color: #666666">**2</span>, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>) )
bias <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( (y_test <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y_pred, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>))<span style="color: #666666">**2</span> )
variance <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>var(y_pred, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>) )
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Error:&#39;</span>, error)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Bias^2:&#39;</span>, bias)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Var:&#39;</span>, variance)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> &gt;= </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> + </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&#39;</span><span style="color: #666666">.</span>format(error, bias, variance, bias<span style="color: #666666">+</span>variance))
plt<span style="color: #666666">.</span>plot(x[::<span style="color: #666666">5</span>, :], y[::<span style="color: #666666">5</span>, :], label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;f(x)&#39;</span>)
plt<span style="color: #666666">.</span>scatter(x_test, y_test, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Data points&#39;</span>)
@@ -1071,11 +1109,11 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec27">Understanding what happens </h2>
<h2 id="understanding-what-happens">Understanding what happens </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression, Ridge, Lasso
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
@@ -1100,21 +1138,21 @@ polydegree <span style="color: #666666">=</span> np<span style="color: #666666">
x_train, x_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(x, y, test_size<span style="color: #666666">=0.2</span>)
<span style="color: #008000; font-weight: bold">for</span> degree <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(maxdegree):
model <span style="color: #666666">=</span> make_pipeline(PolynomialFeatures(degree<span style="color: #666666">=</span>degree), LinearRegression(fit_intercept<span style="color: #666666">=</span><span style="color: #008000">False</span>))
model <span style="color: #666666">=</span> make_pipeline(PolynomialFeatures(degree<span style="color: #666666">=</span>degree), LinearRegression(fit_intercept<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>))
y_pred <span style="color: #666666">=</span> np<span style="color: #666666">.</span>empty((y_test<span style="color: #666666">.</span>shape[<span style="color: #666666">0</span>], n_boostraps))
<span style="color: #008000; font-weight: bold">for</span> i <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(n_boostraps):
x_, y_ <span style="color: #666666">=</span> resample(x_train, y_train)
y_pred[:, i] <span style="color: #666666">=</span> model<span style="color: #666666">.</span>fit(x_, y_)<span style="color: #666666">.</span>predict(x_test)<span style="color: #666666">.</span>ravel()
polydegree[degree] <span style="color: #666666">=</span> degree
error[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>mean((y_test <span style="color: #666666">-</span> y_pred)<span style="color: #666666">**2</span>, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000">True</span>) )
bias[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( (y_test <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y_pred, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000">True</span>))<span style="color: #666666">**2</span> )
variance[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>var(y_pred, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000">True</span>) )
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Polynomial degree:&#39;</span>, degree)
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Error:&#39;</span>, error[degree])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&#39;{} &gt;= {} + {} = {}&#39;</span><span style="color: #666666">.</span>format(error[degree], bias[degree], variance[degree], bias[degree]<span style="color: #666666">+</span>variance[degree]))
error[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>mean((y_test <span style="color: #666666">-</span> y_pred)<span style="color: #666666">**2</span>, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>) )
bias[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( (y_test <span style="color: #666666">-</span> np<span style="color: #666666">.</span>mean(y_pred, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>))<span style="color: #666666">**2</span> )
variance[degree] <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean( np<span style="color: #666666">.</span>var(y_pred, axis<span style="color: #666666">=1</span>, keepdims<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>) )
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Polynomial degree:&#39;</span>, degree)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Error:&#39;</span>, error[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Bias^2:&#39;</span>, bias[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;Var:&#39;</span>, variance[degree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&#39;</span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> &gt;= </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> + </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121"> = </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BA2121">&#39;</span><span style="color: #666666">.</span>format(error[degree], bias[degree], variance[degree], bias[degree]<span style="color: #666666">+</span>variance[degree]))
plt<span style="color: #666666">.</span>plot(polydegree, error, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Error&#39;</span>)
plt<span style="color: #666666">.</span>plot(polydegree, bias, label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;bias&#39;</span>)
@@ -1125,7 +1163,7 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec28">Summing up </h2>
<h2 id="summing-up">Summing up </h2>
<p>
The bias-variance tradeoff summarizes the fundamental tension in
@@ -1162,11 +1200,11 @@ You may also find this recent <a href="https://www.pnas.org/content/116/32/15849
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec29">Another Example from Scikit-Learn's Repository </h2>
<h2 id="another-example-from-scikit-learn-s-repository">Another Example from Scikit-Learn's Repository </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #BA2121; font-style: italic">============================</span>
<span style="color: #BA2121; font-style: italic">Underfitting vs. Overfitting</span>
<span style="color: #BA2121; font-style: italic">============================</span>
@@ -1188,7 +1226,7 @@ You may also find this recent <a href="https://www.pnas.org/content/116/32/15849
<span style="color: #BA2121; font-style: italic">training data.</span>
<span style="color: #BA2121; font-style: italic">&quot;&quot;&quot;</span>
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #19177C">__doc__</span>)
<span style="color: #008000">print</span>(<span style="color: #19177C">__doc__</span>)
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
@@ -1215,7 +1253,7 @@ plt<span style="color: #666666">.</span>figure(figsize<span style="color: #66666
plt<span style="color: #666666">.</span>setp(ax, xticks<span style="color: #666666">=</span>(), yticks<span style="color: #666666">=</span>())
polynomial_features <span style="color: #666666">=</span> PolynomialFeatures(degree<span style="color: #666666">=</span>degrees[i],
include_bias<span style="color: #666666">=</span><span style="color: #008000">False</span>)
include_bias<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>)
linear_regression <span style="color: #666666">=</span> LinearRegression()
pipeline <span style="color: #666666">=</span> Pipeline([(<span style="color: #BA2121">&quot;polynomial_features&quot;</span>, polynomial_features),
(<span style="color: #BA2121">&quot;linear_regression&quot;</span>, linear_regression)])
@@ -1234,19 +1272,19 @@ plt<span style="color: #666666">.</span>figure(figsize<span style="color: #66666
plt<span style="color: #666666">.</span>xlim((<span style="color: #666666">0</span>, <span style="color: #666666">1</span>))
plt<span style="color: #666666">.</span>ylim((<span style="color: #666666">-2</span>, <span style="color: #666666">2</span>))
plt<span style="color: #666666">.</span>legend(loc<span style="color: #666666">=</span><span style="color: #BA2121">&quot;best&quot;</span>)
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Degree {}</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">MSE = {:.2e}(+/- {:.2e})&quot;</span><span style="color: #666666">.</span>format(
plt<span style="color: #666666">.</span>title(<span style="color: #BA2121">&quot;Degree </span><span style="color: #BB6688; font-weight: bold">{}</span><span style="color: #BB6622; font-weight: bold">\n</span><span style="color: #BA2121">MSE = </span><span style="color: #BB6688; font-weight: bold">{:.2e}</span><span style="color: #BA2121">(+/- </span><span style="color: #BB6688; font-weight: bold">{:.2e}</span><span style="color: #BA2121">)&quot;</span><span style="color: #666666">.</span>format(
degrees[i], <span style="color: #666666">-</span>scores<span style="color: #666666">.</span>mean(), scores<span style="color: #666666">.</span>std()))
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec30">More examples on bootstrap and cross-validation and errors </h2>
<h2 id="more-examples-on-bootstrap-and-cross-validation-and-errors">More examples on bootstrap and cross-validation and errors </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Common imports</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># Common imports</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">os</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
@@ -1276,7 +1314,7 @@ DATA_ID <span style="color: #666666">=</span> <span style="color: #BA2121">&quot
<span style="color: #008000; font-weight: bold">return</span> os<span style="color: #666666">.</span>path<span style="color: #666666">.</span>join(DATA_ID, dat_id)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">save_fig</span>(fig_id):
plt<span style="color: #666666">.</span>savefig(image_path(fig_id) <span style="color: #666666">+</span> <span style="color: #BA2121">&quot;.png&quot;</span>, format<span style="color: #666666">=</span><span style="color: #BA2121">&#39;png&#39;</span>)
plt<span style="color: #666666">.</span>savefig(image_path(fig_id) <span style="color: #666666">+</span> <span style="color: #BA2121">&quot;.png&quot;</span>, <span style="color: #008000">format</span><span style="color: #666666">=</span><span style="color: #BA2121">&#39;png&#39;</span>)
infile <span style="color: #666666">=</span> <span style="color: #008000">open</span>(data_path(<span style="color: #BA2121">&quot;EoS.csv&quot;</span>),<span style="color: #BA2121">&#39;r&#39;</span>)
@@ -1306,7 +1344,7 @@ trials <span style="color: #666666">=</span> <span style="color: #666666">100</s
trainingerror[polydegree] <span style="color: #666666">=</span> <span style="color: #666666">0.0</span>
<span style="color: #008000; font-weight: bold">for</span> samples <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(trials):
x_train, x_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(X, Energies, test_size<span style="color: #666666">=0.2</span>)
model <span style="color: #666666">=</span> LinearRegression(fit_intercept<span style="color: #666666">=</span><span style="color: #008000">True</span>)<span style="color: #666666">.</span>fit(x_train, y_train)
model <span style="color: #666666">=</span> LinearRegression(fit_intercept<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">True</span>)<span style="color: #666666">.</span>fit(x_train, y_train)
ypred <span style="color: #666666">=</span> model<span style="color: #666666">.</span>predict(x_train)
ytilde <span style="color: #666666">=</span> model<span style="color: #666666">.</span>predict(x_test)
testerror[polydegree] <span style="color: #666666">+=</span> mean_squared_error(y_test, ytilde)
@@ -1314,9 +1352,9 @@ trials <span style="color: #666666">=</span> <span style="color: #666666">100</s
testerror[polydegree] <span style="color: #666666">/=</span> trials
trainingerror[polydegree] <span style="color: #666666">/=</span> trials
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Degree of polynomial: </span><span style="color: #BB6688; font-weight: bold">%3d</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span> polynomial[polydegree])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Mean squared error on training data: </span><span style="color: #BB6688; font-weight: bold">%.8f</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> trainingerror[polydegree])
<span style="color: #008000; font-weight: bold">print</span>(<span style="color: #BA2121">&quot;Mean squared error on test data: </span><span style="color: #BB6688; font-weight: bold">%.8f</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> testerror[polydegree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Degree of polynomial: </span><span style="color: #BB6688; font-weight: bold">%3d</span><span style="color: #BA2121">&quot;</span><span style="color: #666666">%</span> polynomial[polydegree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Mean squared error on training data: </span><span style="color: #BB6688; font-weight: bold">%.8f</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> trainingerror[polydegree])
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;Mean squared error on test data: </span><span style="color: #BB6688; font-weight: bold">%.8f</span><span style="color: #BA2121">&quot;</span> <span style="color: #666666">%</span> testerror[polydegree])
plt<span style="color: #666666">.</span>plot(polynomial, np<span style="color: #666666">.</span>log10(trainingerror), label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Training Error&#39;</span>)
plt<span style="color: #666666">.</span>plot(polynomial, np<span style="color: #666666">.</span>log10(testerror), label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Test Error&#39;</span>)
@@ -1328,12 +1366,12 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split -->
<h2 id="___sec31">The same example but now with cross-validation </h2>
<h2 id="the-same-example-but-now-with-cross-validation">The same example but now with cross-validation </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #408080; font-style: italic"># Common imports</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #408080; font-style: italic"># Common imports</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">os</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">pandas</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">pd</span>
@@ -1365,7 +1403,7 @@ DATA_ID <span style="color: #666666">=</span> <span style="color: #BA2121">&quot
<span style="color: #008000; font-weight: bold">return</span> os<span style="color: #666666">.</span>path<span style="color: #666666">.</span>join(DATA_ID, dat_id)
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">save_fig</span>(fig_id):
plt<span style="color: #666666">.</span>savefig(image_path(fig_id) <span style="color: #666666">+</span> <span style="color: #BA2121">&quot;.png&quot;</span>, format<span style="color: #666666">=</span><span style="color: #BA2121">&#39;png&#39;</span>)
plt<span style="color: #666666">.</span>savefig(image_path(fig_id) <span style="color: #666666">+</span> <span style="color: #BA2121">&quot;.png&quot;</span>, <span style="color: #008000">format</span><span style="color: #666666">=</span><span style="color: #BA2121">&#39;png&#39;</span>)
infile <span style="color: #666666">=</span> <span style="color: #008000">open</span>(data_path(<span style="color: #BA2121">&quot;EoS.csv&quot;</span>),<span style="color: #BA2121">&#39;r&#39;</span>)
@@ -1404,11 +1442,11 @@ plt<span style="color: #666666">.</span>show()
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec32">Cross-validation with Ridge </h2>
<h2 id="cross-validation-with-ridge">Cross-validation with Ridge </h2>
<p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<div class="highlight" style="background: #f8f8f8"><pre style="line-height: 125%;"><span></span><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> KFold
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> Ridge
@@ -1444,24 +1482,13 @@ plt<span style="color: #666666">.</span>ylabel(<span style="color: #BA2121">&#39
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre></div>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="___sec33">Friday September 4 </h2>
<p>
<a href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember4.mp4?vrtx=view-as-webpage" target="_blank">Video of Lecture</a> and <a href="https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember4.pdf" target="_blank">handwritten notes</a>
<p>
More material will be added here, see handwritten notes also.
<p>
<!-- ------------------- end of main content --------------- -->
<center style="font-size:80%">
<!-- copyright --> &copy; 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
<!-- copyright --> &copy; 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license
</center>
doc/pub/week36/ipynb/ipynb-week36-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week36/ipynb/week36.ipynb
+131 -129
View File
@@ -4,118 +4,26 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<!-- dom:TITLE: Week 36: Resampling techniques and Ordinary Least Square -->\n",
"# Week 36: Resampling techniques and Ordinary Least Square\n",
"<!-- dom:TITLE: Week 36: Statistical interpretation of Linear Regression and Resampling techniques -->\n",
"# Week 36: Statistical interpretation of Linear Regression and Resampling techniques\n",
"<!-- dom:AUTHOR: Morten Hjorth-Jensen at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University -->\n",
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 16, 2020**\n",
"Date: **Sep 6, 2021**\n",
"\n",
"Copyright 1999-2020, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
"\n",
"\n",
"\n",
"## Plans for week 36\n",
"\n",
"* Thursday: Statistics, probability theory and resampling methods\n",
"\n",
"* Friday: Resampling methods and motivation for Ridge Regression\n",
"\n",
"## Thursday September 3\n",
"\n",
"[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage) and [handwritten notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember3.pdf)\n",
"\n",
"More material will be added here, see handwritten notes also.\n",
"\n",
"\n",
"## Why resampling methods\n",
"\n",
"Before we proceed, we need to rethink what we have been doing. In our\n",
"eager to fit the data, we have omitted several important elements in\n",
"our regression analysis. In what follows we will\n",
"1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff\n",
"\n",
"2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more\n",
"\n",
"This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. \n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Resampling methods\n",
"Resampling methods are an indispensable tool in modern\n",
"statistics. They involve repeatedly drawing samples from a training\n",
"set and refitting a model of interest on each sample in order to\n",
"obtain additional information about the fitted model. For example, in\n",
"order to estimate the variability of a linear regression fit, we can\n",
"repeatedly draw different samples from the training data, fit a linear\n",
"regression to each new sample, and then examine the extent to which\n",
"the resulting fits differ. Such an approach may allow us to obtain\n",
"information that would not be available from fitting the model only\n",
"once using the original training sample.\n",
"\n",
"Two resampling methods are often used in Machine Learning analyses,\n",
"1. The **bootstrap method**\n",
"\n",
"2. and **Cross-Validation**\n",
"\n",
"In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular\n",
"cross-validation and the bootstrap method.\n",
"\n",
"\n",
"\n",
"\n",
"## Resampling approaches can be computationally expensive\n",
"\n",
"Resampling approaches can be computationally expensive, because they\n",
"involve fitting the same statistical method multiple times using\n",
"different subsets of the training data. However, due to recent\n",
"advances in computing power, the computational requirements of\n",
"resampling methods generally are not prohibitive. In this chapter, we\n",
"discuss two of the most commonly used resampling methods,\n",
"cross-validation and the bootstrap. Both methods are important tools\n",
"in the practical application of many statistical learning\n",
"procedures. For example, cross-validation can be used to estimate the\n",
"test error associated with a given statistical learning method in\n",
"order to evaluate its performance, or to select the appropriate level\n",
"of flexibility. The process of evaluating a model’s performance is\n",
"known as model assessment, whereas the process of selecting the proper\n",
"level of flexibility for a model is known as model selection. The\n",
"bootstrap is widely used.\n",
"\n",
"\n",
"\n",
"## Why resampling methods ?\n",
"**Statistical analysis.**\n",
"\n",
"\n",
"* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods\n",
"\n",
"* The results can be analysed with the same statistical tools as we would use analysing experimental data.\n",
"\n",
"* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.\n",
"\n",
" \n",
"\n",
"## Statistical analysis\n",
"\n",
"* As in other experiments, many numerical experiments have two classes of errors:\n",
"\n",
" * Statistical errors\n",
"\n",
" * Systematical errors\n",
"\n",
"\n",
"* Statistical errors can be estimated using standard tools from statistics\n",
"\n",
"* Systematical errors are method specific and must be treated differently from case to case.\n",
"\n",
" \n",
"* Thursday: Statistics, probability theory and linear regression\n",
"\n",
"* Friday: Resampling methods\n",
"\n",
"## Thursday September 9\n",
"\n",
"\n",
"<!-- !split -->\n",
@@ -383,6 +291,102 @@
"This means the variance we obtain with the standard OLS will always for $\\lambda > 0$ be larger than the variance of $\\boldsymbol{\\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below. \n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Friday September 10\n",
"\n",
"\n",
"\n",
"## Why resampling methods\n",
"\n",
"Before we proceed, we need to rethink what we have been doing. In our\n",
"eager to fit the data, we have omitted several important elements in\n",
"our regression analysis. In what follows we will\n",
"1. look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff\n",
"\n",
"2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more\n",
"\n",
"This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. \n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Resampling methods\n",
"Resampling methods are an indispensable tool in modern\n",
"statistics. They involve repeatedly drawing samples from a training\n",
"set and refitting a model of interest on each sample in order to\n",
"obtain additional information about the fitted model. For example, in\n",
"order to estimate the variability of a linear regression fit, we can\n",
"repeatedly draw different samples from the training data, fit a linear\n",
"regression to each new sample, and then examine the extent to which\n",
"the resulting fits differ. Such an approach may allow us to obtain\n",
"information that would not be available from fitting the model only\n",
"once using the original training sample.\n",
"\n",
"Two resampling methods are often used in Machine Learning analyses,\n",
"1. The **bootstrap method**\n",
"\n",
"2. and **Cross-Validation**\n",
"\n",
"In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular\n",
"cross-validation and the bootstrap method.\n",
"\n",
"\n",
"\n",
"\n",
"## Resampling approaches can be computationally expensive\n",
"\n",
"Resampling approaches can be computationally expensive, because they\n",
"involve fitting the same statistical method multiple times using\n",
"different subsets of the training data. However, due to recent\n",
"advances in computing power, the computational requirements of\n",
"resampling methods generally are not prohibitive. In this chapter, we\n",
"discuss two of the most commonly used resampling methods,\n",
"cross-validation and the bootstrap. Both methods are important tools\n",
"in the practical application of many statistical learning\n",
"procedures. For example, cross-validation can be used to estimate the\n",
"test error associated with a given statistical learning method in\n",
"order to evaluate its performance, or to select the appropriate level\n",
"of flexibility. The process of evaluating a model’s performance is\n",
"known as model assessment, whereas the process of selecting the proper\n",
"level of flexibility for a model is known as model selection. The\n",
"bootstrap is widely used.\n",
"\n",
"\n",
"\n",
"## Why resampling methods ?\n",
"**Statistical analysis.**\n",
"\n",
"\n",
"* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods\n",
"\n",
"* The results can be analysed with the same statistical tools as we would use analysing experimental data.\n",
"\n",
"* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.\n",
"\n",
" \n",
"\n",
"## Statistical analysis\n",
"\n",
"* As in other experiments, many numerical experiments have two classes of errors:\n",
"\n",
" * Statistical errors\n",
"\n",
" * Systematical errors\n",
"\n",
"\n",
"* Statistical errors can be estimated using standard tools from statistics\n",
"\n",
"* Systematical errors are method specific and must be treated differently from case to case.\n",
"\n",
" \n",
"\n",
"\n",
"\n",
"\n",
"\n",
"## Resampling methods\n",
"\n",
"With all these analytical equations for both the OLS and Ridge\n",
@@ -455,9 +459,10 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -598,9 +603,10 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -748,9 +754,10 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -979,9 +986,10 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1050,9 +1058,10 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1151,9 +1160,10 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1239,9 +1249,10 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1335,9 +1346,10 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1419,9 +1431,10 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": null,
"metadata": {
"collapsed": false
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
@@ -1461,20 +1474,9 @@
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Friday September 4\n",
"\n",
"[Video of Lecture](https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember4.mp4?vrtx=view-as-webpage) and [handwritten notes](https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember4.pdf)\n",
"\n",
"More material will be added here, see handwritten notes also."
]
}
],
"metadata": {},
"nbformat": 4,
"nbformat_minor": 2
"nbformat_minor": 4
}
doc/src/week36/week36.do.txt
+100 -105
View File
@@ -1,4 +1,4 @@
TITLE: Week 36: Resampling techniques and Ordinary Least Square
TITLE: Week 36: Statistical interpretation of Linear Regression and Resampling techniques
AUTHOR: Morten Hjorth-Jensen {copyright, 1999-present|CC BY-NC} at Department of Physics, University of Oslo & Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University
DATE: today
@@ -6,103 +6,12 @@ DATE: today
!split
===== Plans for week 36 =====
* Thursday: Statistics, probability theory and resampling methods
* Friday: Resampling methods and motivation for Ridge Regression
* Thursday: Statistics, probability theory and linear regression
* Friday: Resampling methods
!split
===== Thursday September 3 =====
"Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSept3.mp4?vrtx=view-as-webpage" and "handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember3.pdf"
More material will be added here, see handwritten notes also.
!split
===== Why resampling methods =====
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
o look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
o introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
!split
===== Resampling methods =====
!bblock
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
Two resampling methods are often used in Machine Learning analyses,
o The _bootstrap method_
o and _Cross-Validation_
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
!eblock
!split
===== Resampling approaches can be computationally expensive =====
!bblock
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model’s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
!eblock
!split
===== Why resampling methods ? =====
!bblock Statistical analysis
* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods
* The results can be analysed with the same statistical tools as we would use analysing experimental data.
* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
!eblock
!split
===== Statistical analysis =====
!bblock
* As in other experiments, many numerical experiments have two classes of errors:
* Statistical errors
* Systematical errors
* Statistical errors can be estimated using standard tools from statistics
* Systematical errors are method specific and must be treated differently from case to case.
!eblock
===== Thursday September 9 =====
!split
@@ -270,6 +179,102 @@ matrix product is non-negative definite.
This means the variance we obtain with the standard OLS will always for $\lambda > 0$ be larger than the variance of $\bm{\beta}$ obtained with the Ridge estimator. This has interesting consequences when we discuss the so-called bias-variance trade-off below.
!split
===== Friday September 10 =====
!split
===== Why resampling methods =====
Before we proceed, we need to rethink what we have been doing. In our
eager to fit the data, we have omitted several important elements in
our regression analysis. In what follows we will
o look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
o introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
!split
===== Resampling methods =====
!bblock
Resampling methods are an indispensable tool in modern
statistics. They involve repeatedly drawing samples from a training
set and refitting a model of interest on each sample in order to
obtain additional information about the fitted model. For example, in
order to estimate the variability of a linear regression fit, we can
repeatedly draw different samples from the training data, fit a linear
regression to each new sample, and then examine the extent to which
the resulting fits differ. Such an approach may allow us to obtain
information that would not be available from fitting the model only
once using the original training sample.
Two resampling methods are often used in Machine Learning analyses,
o The _bootstrap method_
o and _Cross-Validation_
In addition there are several other methods such as the Jackknife and the Blocking methods. We will discuss in particular
cross-validation and the bootstrap method.
!eblock
!split
===== Resampling approaches can be computationally expensive =====
!bblock
Resampling approaches can be computationally expensive, because they
involve fitting the same statistical method multiple times using
different subsets of the training data. However, due to recent
advances in computing power, the computational requirements of
resampling methods generally are not prohibitive. In this chapter, we
discuss two of the most commonly used resampling methods,
cross-validation and the bootstrap. Both methods are important tools
in the practical application of many statistical learning
procedures. For example, cross-validation can be used to estimate the
test error associated with a given statistical learning method in
order to evaluate its performance, or to select the appropriate level
of flexibility. The process of evaluating a model’s performance is
known as model assessment, whereas the process of selecting the proper
level of flexibility for a model is known as model selection. The
bootstrap is widely used.
!eblock
!split
===== Why resampling methods ? =====
!bblock Statistical analysis
* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods
* The results can be analysed with the same statistical tools as we would use analysing experimental data.
* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
!eblock
!split
===== Statistical analysis =====
!bblock
* As in other experiments, many numerical experiments have two classes of errors:
* Statistical errors
* Systematical errors
* Statistical errors can be estimated using standard tools from statistics
* Systematical errors are method specific and must be treated differently from case to case.
!eblock
!split
===== Resampling methods =====
@@ -1174,13 +1179,3 @@ plt.show()
!ec
!split
===== Friday September 4 =====
"Video of Lecture":"https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h20/forelesningsvideoer/LectureSeptember4.mp4?vrtx=view-as-webpage" and "handwritten notes":"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/NotesSeptember4.pdf"
More material will be added here, see handwritten notes also.