lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

small update with added note to weekly slides

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-09-14 06:51:54 +02:00
parent c1725a8de2
commit 10e1a87b1f
57 changed files with 2666 additions and 248 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week37/html/._week37-bs000.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs001.html
+9 -4
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
@@ -292,7 +297,7 @@ MathJax.Hub.Config({
<li> Lecture from last week on calculations of expectation values</li>
<li> Exercise for week 37</li>
<li> Work on project 1</li>
<li> See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session.</li>
<li> See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.</li>
<li> For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_self">Wessel van Wieringen's</a> article is highly recommended.</li>
</ul>
</div>
@@ -302,9 +307,9 @@ MathJax.Hub.Config({
<div class="panel-body">
<!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
<ul>
<li> Statistical interpretation of Ridge and Lasso regression</li>
<li> Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff</li>
<li> Reads and Videos:</li>
<li> Statistical interpretation of Ridge and Lasso regression</li>
<li> Readings and Videos:</li>
<ul>
<li> Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap).</li>
<li> <a href="https://www.youtube.com/watch?v=fSytzGwwBVw" target="_self">Video on cross validation</a></li>
doc/pub/week37/html/._week37-bs002.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs003.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs004.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs005.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs006.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs007.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs008.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs009.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs010.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs011.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs012.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs013.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs014.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs015.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs016.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs017.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs018.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs019.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs020.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs021.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs022.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs023.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs024.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs025.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs026.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs027.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs028.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs029.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs030.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs031.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs032.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs033.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs034.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs035.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs036.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs037.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs038.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs039.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs040.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs041.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs042.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs043.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs044.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs045.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs046.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs047.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs048.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/._week37-bs049.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/week37-bs.html
+6 -1
View File
@@ -185,7 +185,11 @@ doconce format html week37.do.txt --html_style=bootstrap --pygments_html_style=d
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -272,6 +276,7 @@ MathJax.Hub.Config({
<!-- navigation toc: --> <li><a href="._week37-bs050.html#code-example-for-cross-validation-and-k-fold-cross-validation" style="font-size: 80%;">Code Example for Cross-validation and \( k \)-fold Cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs051.html#more-examples-on-bootstrap-and-cross-validation-and-errors" style="font-size: 80%;">More examples on bootstrap and cross-validation and errors</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#the-same-example-but-now-with-cross-validation" style="font-size: 80%;">The same example but now with cross-validation</a></li>
<!-- navigation toc: --> <li><a href="._week37-bs052.html#notes-on-scaling-with-examples" style="font-size: 80%;">Notes on scaling with examples</a></li>
</ul>
</li>
doc/pub/week37/html/week37-reveal.html
+415 -4
View File
@@ -210,7 +210,7 @@ MathJax.Hub.Config({
<p><li> Work on project 1</li>
<p><li> See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session.</li>
<p><li> See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.</li>
<p><li> For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</li>
</ul>
@@ -221,11 +221,11 @@ MathJax.Hub.Config({
<p>
<ul>
<p><li> Statistical interpretation of Ridge and Lasso regression</li>
<p><li> Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff</li>
<p><li> Reads and Videos:</li>
<p><li> Statistical interpretation of Ridge and Lasso regression</li>
<p><li> Readings and Videos:</li>
<ul>
<p><li> Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap).</li>
@@ -2134,6 +2134,417 @@ plt.show()
</div>
</div>
</div>
<h2 id="notes-on-scaling-with-examples">Notes on scaling with examples </h2>
<p>The programs here use both ordinrary least squares (OLS) and Ridge
regression with one value only for the hyperparameter \( \lambda \). The
first example has no scaling and includes the intercept as well and we
are trying to fit a second-order polynomial. The second code takes out
the intercept and subtracts the mean values of each column of the
design matrix and the mean value of the outputs.
</p>
<p>The third and final code uses <b>Scikit-Learn</b> as library in order to
calculate the optimal parameters for OLS and Ridge regression. Note
that it is highly recommended to not include the intercept in Ridge
and Lasso regression, in order to avoid penalizing the optimization by
the intercept. The second and third codes do thus not include the
intercept. In the second code we do the scaling ourselves while the
last code uses the standard scaler option included in <b>Scikit-Learn</b>, known as centering (where
we subtract the mean values).
</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="font-size: 80%; line-height: 125%;"><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> train_test_split
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> StandardScaler
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">MSE</span>(y_data,y_model):
n = np.size(y_model)
<span style="color: #8B008B; font-weight: bold">return</span> np.sum((y_data-y_model)**<span style="color: #B452CD">2</span>)/n
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">OLS_fit_beta</span>(X, y):
<span style="color: #8B008B; font-weight: bold">return</span> np.linalg.pinv(X.T @ X) @ X.T @ y
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">Ridge_fit_beta</span>(X, y,L,d):
I = np.eye(d,d)
<span style="color: #8B008B; font-weight: bold">return</span> np.linalg.pinv(X.T @ X + L*I) @ X.T @ y
<span style="color: #228B22"># Same random numbers for each test.</span>
np.random.seed(<span style="color: #B452CD">2018</span>)
n = <span style="color: #B452CD">100</span>
d = <span style="color: #B452CD">3</span>
<span style="color: #228B22"># hyperparameter lambda</span>
Lambda = <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># Make data set, simple second-order polynomial</span>
x = np.linspace(-<span style="color: #B452CD">3</span>, <span style="color: #B452CD">3</span>, n)
y = <span style="color: #B452CD">2.0</span> + <span style="color: #B452CD">0.5</span>*x + <span style="color: #B452CD">5.0</span>*(x**<span style="color: #B452CD">2</span>)+ np.random.randn(n)
<span style="color: #228B22"># The design matrix X includes the intercept and no scaling is made</span>
X = np.zeros((<span style="color: #658b00">len</span>(x), d))
<span style="color: #8B008B; font-weight: bold">for</span> p <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(d):
X[:, p] = x ** (p)
<span style="color: #228B22">#Split data, no scaling is used and we include the intercept</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #228B22">#Calculate beta, own code</span>
beta_OLS = OLS_fit_beta(X_train, y_train)
beta_Ridge = Ridge_fit_beta(X_train, y_train,Lambda,d)
<span style="color: #658b00">print</span>(beta_OLS)
<span style="color: #658b00">print</span>(beta_Ridge)
<span style="color: #228B22">#predict value</span>
ytilde_test_OLS = X_test @ beta_OLS
ytilde_test_Ridge = X_test @ beta_Ridge
<span style="color: #228B22">#Calculate MSE</span>
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of OLS:&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label=<span style="color: #CD5555">&#39;Data&#39;</span>)
plt.plot(x, X @ beta_OLS,<span style="color: #CD5555">&#39;*&#39;</span>, label=<span style="color: #CD5555">&quot;OLS_Fit&quot;</span>)
plt.plot(x, X @ beta_Ridge, label=<span style="color: #CD5555">&quot;Ridge_Fit&quot;</span>)
plt.grid()
plt.legend()
plt.show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#more-on-rescaling-data" target="_blank">lecture material</a>.
see also the weekly slides <a href="https://compphysics.github.io/MachineLearning/doc/pub/week36/html/._week36-bs029.html" target="_blank">for week 36</a>.
It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.
</p>
<p>Before we discuss the code, we repeat some of the basic math from the slides of week 36.</p>
<p>Let us try to understand what this may imply mathematically when we
subtract the mean values, also known as <em>zero centering</em> or simply <em>centering</em>. For
simplicity, we will focus on ordinary regression, as done in the above example.
</p>
<p>The cost/loss function for regression is</p>
<p>&nbsp;<br>
$$
C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
$$
<p>&nbsp;<br>
<p>Recall also that we use the squared value. This expression can lead to an
increased penalty for higher differences between predicted and
output/target values.
</p>
<p>What we have done is to single out the \( \beta_0 \) term in the
definition of the mean squared error (MSE). The design matrix \( X \)
does in this case not contain any intercept column. When we take the
derivative with respect to \( \beta_0 \), we want the derivative to obey
</p>
<p>&nbsp;<br>
$$
\frac{\partial C}{\partial \beta_j} = 0,
$$
<p>&nbsp;<br>
<p>for all \( j \). For \( \beta_0 \) we have</p>
<p>&nbsp;<br>
$$
\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right).
$$
<p>&nbsp;<br>
<p>Multiplying away the constant \( 2/n \), we obtain</p>
<p>&nbsp;<br>
$$
\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
$$
<p>&nbsp;<br>
<p>Let us specialize first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \).
Our result for \( \beta_0 \) simplifies then to
</p>
<p>&nbsp;<br>
$$
n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
$$
<p>&nbsp;<br>
<p>We obtain then</p>
<p>&nbsp;<br>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
$$
<p>&nbsp;<br>
<p>If we define</p>
<p>&nbsp;<br>
$$
\mu_{\boldsymbol{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
$$
<p>&nbsp;<br>
<p>and the mean value of the outputs as</p>
<p>&nbsp;<br>
$$
\mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
$$
<p>&nbsp;<br>
<p>we have</p>
<p>&nbsp;<br>
$$
\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}.
$$
<p>&nbsp;<br>
<p>In the general case with more parameters than \( \beta_0 \) and \( \beta_1 \), we have</p>
<p>&nbsp;<br>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
$$
<p>&nbsp;<br>
<p>We can rewrite the latter equation as</p>
<p>&nbsp;<br>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j,
$$
<p>&nbsp;<br>
<p>where we have defined</p>
<p>&nbsp;<br>
$$
\mu_{\boldsymbol{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
$$
<p>&nbsp;<br>
<p>the mean value for all elements of the column vector \( \boldsymbol{x}_j \).</p>
<p>Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)</p>
<p>&nbsp;<br>
$$
C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}).
$$
<p>&nbsp;<br>
<p>If we minimize with respect to \( \boldsymbol{\beta} \) we have then</p>
<p>&nbsp;<br>
$$
\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
$$
<p>&nbsp;<br>
<p>where \( \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} \)
and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \).
</p>
<p>For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then</p>
<p>&nbsp;<br>
$$
\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
$$
<p>&nbsp;<br>
<p>Now we try to implement this.</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="font-size: 80%; line-height: 125%;">np.random.seed(<span style="color: #B452CD">2018</span>)
n = <span style="color: #B452CD">100</span>
<span style="color: #228B22"># we do not include the intercept</span>
d = <span style="color: #B452CD">2</span>
Lambda = <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># Make data set.</span>
x = np.linspace(-<span style="color: #B452CD">3</span>, <span style="color: #B452CD">3</span>, n)
y = <span style="color: #B452CD">2.0</span> + <span style="color: #B452CD">0.5</span>*x + <span style="color: #B452CD">5.0</span>*(x**<span style="color: #B452CD">2</span>)+ np.random.randn(n)
<span style="color: #228B22">#Design matrix X does not include the intercept. </span>
X = np.zeros((<span style="color: #658b00">len</span>(x), d))
<span style="color: #8B008B; font-weight: bold">for</span> p <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(d):
X[:, p] = x ** (p+<span style="color: #B452CD">1</span>)
<span style="color: #228B22">#Split data in train and test</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #228B22"># Scale data by subtracting mean value,own implementation</span>
<span style="color: #228B22">#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable</span>
X_train_mean = np.mean(X_train,axis=<span style="color: #B452CD">0</span>)
<span style="color: #228B22">#Center by removing mean from each feature</span>
X_train_scaled = X_train - X_train_mean
X_test_scaled = X_test - X_train_mean
<span style="color: #228B22">#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)</span>
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
<span style="color: #228B22">#Calculate beta</span>
beta_OLS = OLS_fit_beta(X_train_scaled, y_train_scaled)
beta_Ridge = Ridge_fit_beta(X_train_scaled, y_train_scaled,Lambda,d)
<span style="color: #658b00">print</span>(beta_OLS)
<span style="color: #658b00">print</span>(beta_Ridge)
<span style="color: #228B22"># calculate intercepts and print them</span>
interceptOLS = y_scaler - X_train_mean @ beta_OLS
interceptRidge = y_scaler - X_train_mean @ beta_Ridge
<span style="color: #658b00">print</span>(interceptOLS)
<span style="color: #658b00">print</span>(interceptRidge)
<span style="color: #228B22">#predict value with intercept</span>
ytilde_test_OLS = X_test_scaled @ beta_OLS+y_scaler
ytilde_test_Ridge = X_test_scaled @ beta_Ridge+y_scaler
<span style="color: #228B22">#Calculate MSE</span>
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of OLS:&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label=<span style="color: #CD5555">&#39;Data&#39;</span>)
plt.plot(x, X @ beta_OLS+interceptOLS,<span style="color: #CD5555">&#39;*&#39;</span>, label=<span style="color: #CD5555">&quot;OLS_Fit&quot;</span>)
plt.plot(x, X @ beta_Ridge+interceptRidge, label=<span style="color: #CD5555">&quot;Ridge_Fit&quot;</span>)
plt.grid()
plt.legend()
plt.show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Finally, instead of using our own function we repeat the same example
using the <b>standardscaler</b> functionality of the library
<b>Scikit-Learn</b>. Here we limit ourselves to Ridge regression only.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="font-size: 80%; line-height: 125%;"><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> linear_model
np.random.seed(<span style="color: #B452CD">2018</span>)
n = <span style="color: #B452CD">10</span>
d = <span style="color: #B452CD">2</span>
Lambda = <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># Make data set.</span>
x = np.linspace(-<span style="color: #B452CD">3</span>, <span style="color: #B452CD">3</span>, n)
y = <span style="color: #B452CD">2.0</span> + <span style="color: #B452CD">0.5</span>*x + <span style="color: #B452CD">5.0</span>*(x**<span style="color: #B452CD">2</span>)+ np.random.randn(n)
<span style="color: #228B22"># Design matrix X does not include the intercept. </span>
X = np.zeros((n, d))
<span style="color: #8B008B; font-weight: bold">for</span> p <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(d):
X[:, p] = x ** (p+<span style="color: #B452CD">1</span>)
<span style="color: #228B22">#Split data in train and test</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #228B22"># Scale data by subtracting mean value of the input using scikit-learn</span>
scaler = StandardScaler(with_std=<span style="color: #8B008B; font-weight: bold">False</span>)
scaler.fit(X_train)
X_train_mean = np.mean(X_train,axis=<span style="color: #B452CD">0</span>)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
<span style="color: #228B22"># We scale also the output, here by our own code</span>
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
y_test_scaled = y_test- y_scaler
<span style="color: #228B22">#Calculate beta</span>
OLS = LinearRegression()
betaOLS=OLS.fit(X_train_scaled,y_train_scaled)
ypredictOLS = OLS.predict(X_test_scaled)
linear_model.Ridge(Lambda)
RegRidge.fit(X_train_scaled,y_train_scaled)
ypredictRidge = RegRidge.predict(X_test_scaled)
betaOLS = OLS.coef_
betaRidge = RegRidge.coef_
<span style="color: #658b00">print</span>(betaOLS)
<span style="color: #658b00">print</span>(betaRidge)
interceptOLS = np.mean(y_train) - X_train_mean @ betaOLS
interceptRidge = y_scaler - X_train_mean @ betaRidge
<span style="color: #658b00">print</span>(interceptOLS)
<span style="color: #658b00">print</span>(interceptRidge)
<span style="color: #228B22">#predict value </span>
ytilde_test_Ridge = X_test_scaled @ betaRidge+y_scaler
ytilde_test_OLS = X_test_scaled @ betaOLS+y_scaler
<span style="color: #228B22">#Calculate MSE</span>
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of OLS&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label=<span style="color: #CD5555">&#39;Data&#39;</span>)
plt.plot(x, X @ RegRidge.coef_ + RegRidge.intercept_ , label=<span style="color: #CD5555">&quot;Ridge_Fit&quot;</span>)
plt.grid()
plt.legend()
plt.show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
</section>
doc/pub/week37/html/week37-solarized.html
+389 -4
View File
@@ -212,7 +212,11 @@ div.toc p,a {
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -266,7 +270,7 @@ MathJax.Hub.Config({
<li> Lecture from last week on calculations of expectation values</li>
<li> Exercise for week 37</li>
<li> Work on project 1</li>
<li> See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session.</li>
<li> See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.</li>
<li> For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</li>
</ul>
</div>
@@ -275,9 +279,9 @@ MathJax.Hub.Config({
<b>Material for the lecture on Thursday September 7</b>
<p>
<ul>
<li> Statistical interpretation of Ridge and Lasso regression</li>
<li> Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff</li>
<li> Reads and Videos:</li>
<li> Statistical interpretation of Ridge and Lasso regression</li>
<li> Readings and Videos:</li>
<ul>
<li> Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap).</li>
<li> <a href="https://www.youtube.com/watch?v=fSytzGwwBVw" target="_blank">Video on cross validation</a></li>
@@ -2019,6 +2023,387 @@ plt.show()
</div>
</div>
</div>
<h2 id="notes-on-scaling-with-examples">Notes on scaling with examples </h2>
<p>The programs here use both ordinrary least squares (OLS) and Ridge
regression with one value only for the hyperparameter \( \lambda \). The
first example has no scaling and includes the intercept as well and we
are trying to fit a second-order polynomial. The second code takes out
the intercept and subtracts the mean values of each column of the
design matrix and the mean value of the outputs.
</p>
<p>The third and final code uses <b>Scikit-Learn</b> as library in order to
calculate the optimal parameters for OLS and Ridge regression. Note
that it is highly recommended to not include the intercept in Ridge
and Lasso regression, in order to avoid penalizing the optimization by
the intercept. The second and third codes do thus not include the
intercept. In the second code we do the scaling ourselves while the
last code uses the standard scaler option included in <b>Scikit-Learn</b>, known as centering (where
we subtract the mean values).
</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">matplotlib.pyplot</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">plt</span>
<span style="color: #8B008B; font-weight: bold">import</span> <span style="color: #008b45; text-decoration: underline">numpy</span> <span style="color: #8B008B; font-weight: bold">as</span> <span style="color: #008b45; text-decoration: underline">np</span>
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.linear_model</span> <span style="color: #8B008B; font-weight: bold">import</span> LinearRegression
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.model_selection</span> <span style="color: #8B008B; font-weight: bold">import</span> train_test_split
<span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn.preprocessing</span> <span style="color: #8B008B; font-weight: bold">import</span> StandardScaler
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">MSE</span>(y_data,y_model):
n = np.size(y_model)
<span style="color: #8B008B; font-weight: bold">return</span> np.sum((y_data-y_model)**<span style="color: #B452CD">2</span>)/n
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">OLS_fit_beta</span>(X, y):
<span style="color: #8B008B; font-weight: bold">return</span> np.linalg.pinv(X.T @ X) @ X.T @ y
<span style="color: #8B008B; font-weight: bold">def</span> <span style="color: #008b45">Ridge_fit_beta</span>(X, y,L,d):
I = np.eye(d,d)
<span style="color: #8B008B; font-weight: bold">return</span> np.linalg.pinv(X.T @ X + L*I) @ X.T @ y
<span style="color: #228B22"># Same random numbers for each test.</span>
np.random.seed(<span style="color: #B452CD">2018</span>)
n = <span style="color: #B452CD">100</span>
d = <span style="color: #B452CD">3</span>
<span style="color: #228B22"># hyperparameter lambda</span>
Lambda = <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># Make data set, simple second-order polynomial</span>
x = np.linspace(-<span style="color: #B452CD">3</span>, <span style="color: #B452CD">3</span>, n)
y = <span style="color: #B452CD">2.0</span> + <span style="color: #B452CD">0.5</span>*x + <span style="color: #B452CD">5.0</span>*(x**<span style="color: #B452CD">2</span>)+ np.random.randn(n)
<span style="color: #228B22"># The design matrix X includes the intercept and no scaling is made</span>
X = np.zeros((<span style="color: #658b00">len</span>(x), d))
<span style="color: #8B008B; font-weight: bold">for</span> p <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(d):
X[:, p] = x ** (p)
<span style="color: #228B22">#Split data, no scaling is used and we include the intercept</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #228B22">#Calculate beta, own code</span>
beta_OLS = OLS_fit_beta(X_train, y_train)
beta_Ridge = Ridge_fit_beta(X_train, y_train,Lambda,d)
<span style="color: #658b00">print</span>(beta_OLS)
<span style="color: #658b00">print</span>(beta_Ridge)
<span style="color: #228B22">#predict value</span>
ytilde_test_OLS = X_test @ beta_OLS
ytilde_test_Ridge = X_test @ beta_Ridge
<span style="color: #228B22">#Calculate MSE</span>
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of OLS:&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label=<span style="color: #CD5555">&#39;Data&#39;</span>)
plt.plot(x, X @ beta_OLS,<span style="color: #CD5555">&#39;*&#39;</span>, label=<span style="color: #CD5555">&quot;OLS_Fit&quot;</span>)
plt.plot(x, X @ beta_Ridge, label=<span style="color: #CD5555">&quot;Ridge_Fit&quot;</span>)
plt.grid()
plt.legend()
plt.show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#more-on-rescaling-data" target="_blank">lecture material</a>.
see also the weekly slides <a href="https://compphysics.github.io/MachineLearning/doc/pub/week36/html/._week36-bs029.html" target="_blank">for week 36</a>.
It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.
</p>
<p>Before we discuss the code, we repeat some of the basic math from the slides of week 36.</p>
<p>Let us try to understand what this may imply mathematically when we
subtract the mean values, also known as <em>zero centering</em> or simply <em>centering</em>. For
simplicity, we will focus on ordinary regression, as done in the above example.
</p>
<p>The cost/loss function for regression is</p>
$$
C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
$$
<p>Recall also that we use the squared value. This expression can lead to an
increased penalty for higher differences between predicted and
output/target values.
</p>
<p>What we have done is to single out the \( \beta_0 \) term in the
definition of the mean squared error (MSE). The design matrix \( X \)
does in this case not contain any intercept column. When we take the
derivative with respect to \( \beta_0 \), we want the derivative to obey
</p>
$$
\frac{\partial C}{\partial \beta_j} = 0,
$$
<p>for all \( j \). For \( \beta_0 \) we have</p>
$$
\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right).
$$
<p>Multiplying away the constant \( 2/n \), we obtain</p>
$$
\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
$$
<p>Let us specialize first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \).
Our result for \( \beta_0 \) simplifies then to
</p>
$$
n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
$$
<p>We obtain then</p>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
$$
<p>If we define</p>
$$
\mu_{\boldsymbol{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
$$
<p>and the mean value of the outputs as</p>
$$
\mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
$$
<p>we have</p>
$$
\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}.
$$
<p>In the general case with more parameters than \( \beta_0 \) and \( \beta_1 \), we have</p>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
$$
<p>We can rewrite the latter equation as</p>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j,
$$
<p>where we have defined</p>
$$
\mu_{\boldsymbol{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
$$
<p>the mean value for all elements of the column vector \( \boldsymbol{x}_j \).</p>
<p>Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)</p>
$$
C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}).
$$
<p>If we minimize with respect to \( \boldsymbol{\beta} \) we have then</p>
$$
\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
$$
<p>where \( \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} \)
and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \).
</p>
<p>For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then</p>
$$
\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
$$
<p>Now we try to implement this.</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;">np.random.seed(<span style="color: #B452CD">2018</span>)
n = <span style="color: #B452CD">100</span>
<span style="color: #228B22"># we do not include the intercept</span>
d = <span style="color: #B452CD">2</span>
Lambda = <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># Make data set.</span>
x = np.linspace(-<span style="color: #B452CD">3</span>, <span style="color: #B452CD">3</span>, n)
y = <span style="color: #B452CD">2.0</span> + <span style="color: #B452CD">0.5</span>*x + <span style="color: #B452CD">5.0</span>*(x**<span style="color: #B452CD">2</span>)+ np.random.randn(n)
<span style="color: #228B22">#Design matrix X does not include the intercept. </span>
X = np.zeros((<span style="color: #658b00">len</span>(x), d))
<span style="color: #8B008B; font-weight: bold">for</span> p <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(d):
X[:, p] = x ** (p+<span style="color: #B452CD">1</span>)
<span style="color: #228B22">#Split data in train and test</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #228B22"># Scale data by subtracting mean value,own implementation</span>
<span style="color: #228B22">#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable</span>
X_train_mean = np.mean(X_train,axis=<span style="color: #B452CD">0</span>)
<span style="color: #228B22">#Center by removing mean from each feature</span>
X_train_scaled = X_train - X_train_mean
X_test_scaled = X_test - X_train_mean
<span style="color: #228B22">#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)</span>
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
<span style="color: #228B22">#Calculate beta</span>
beta_OLS = OLS_fit_beta(X_train_scaled, y_train_scaled)
beta_Ridge = Ridge_fit_beta(X_train_scaled, y_train_scaled,Lambda,d)
<span style="color: #658b00">print</span>(beta_OLS)
<span style="color: #658b00">print</span>(beta_Ridge)
<span style="color: #228B22"># calculate intercepts and print them</span>
interceptOLS = y_scaler - X_train_mean @ beta_OLS
interceptRidge = y_scaler - X_train_mean @ beta_Ridge
<span style="color: #658b00">print</span>(interceptOLS)
<span style="color: #658b00">print</span>(interceptRidge)
<span style="color: #228B22">#predict value with intercept</span>
ytilde_test_OLS = X_test_scaled @ beta_OLS+y_scaler
ytilde_test_Ridge = X_test_scaled @ beta_Ridge+y_scaler
<span style="color: #228B22">#Calculate MSE</span>
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of OLS:&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label=<span style="color: #CD5555">&#39;Data&#39;</span>)
plt.plot(x, X @ beta_OLS+interceptOLS,<span style="color: #CD5555">&#39;*&#39;</span>, label=<span style="color: #CD5555">&quot;OLS_Fit&quot;</span>)
plt.plot(x, X @ beta_Ridge+interceptRidge, label=<span style="color: #CD5555">&quot;Ridge_Fit&quot;</span>)
plt.grid()
plt.legend()
plt.show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Finally, instead of using our own function we repeat the same example
using the <b>standardscaler</b> functionality of the library
<b>Scikit-Learn</b>. Here we limit ourselves to Ridge regression only.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "perldoc" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #eeeedd">
<pre style="line-height: 125%;"><span style="color: #8B008B; font-weight: bold">from</span> <span style="color: #008b45; text-decoration: underline">sklearn</span> <span style="color: #8B008B; font-weight: bold">import</span> linear_model
np.random.seed(<span style="color: #B452CD">2018</span>)
n = <span style="color: #B452CD">10</span>
d = <span style="color: #B452CD">2</span>
Lambda = <span style="color: #B452CD">0.01</span>
<span style="color: #228B22"># Make data set.</span>
x = np.linspace(-<span style="color: #B452CD">3</span>, <span style="color: #B452CD">3</span>, n)
y = <span style="color: #B452CD">2.0</span> + <span style="color: #B452CD">0.5</span>*x + <span style="color: #B452CD">5.0</span>*(x**<span style="color: #B452CD">2</span>)+ np.random.randn(n)
<span style="color: #228B22"># Design matrix X does not include the intercept. </span>
X = np.zeros((n, d))
<span style="color: #8B008B; font-weight: bold">for</span> p <span style="color: #8B008B">in</span> <span style="color: #658b00">range</span>(d):
X[:, p] = x ** (p+<span style="color: #B452CD">1</span>)
<span style="color: #228B22">#Split data in train and test</span>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=<span style="color: #B452CD">0.2</span>)
<span style="color: #228B22"># Scale data by subtracting mean value of the input using scikit-learn</span>
scaler = StandardScaler(with_std=<span style="color: #8B008B; font-weight: bold">False</span>)
scaler.fit(X_train)
X_train_mean = np.mean(X_train,axis=<span style="color: #B452CD">0</span>)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
<span style="color: #228B22"># We scale also the output, here by our own code</span>
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
y_test_scaled = y_test- y_scaler
<span style="color: #228B22">#Calculate beta</span>
OLS = LinearRegression()
betaOLS=OLS.fit(X_train_scaled,y_train_scaled)
ypredictOLS = OLS.predict(X_test_scaled)
linear_model.Ridge(Lambda)
RegRidge.fit(X_train_scaled,y_train_scaled)
ypredictRidge = RegRidge.predict(X_test_scaled)
betaOLS = OLS.coef_
betaRidge = RegRidge.coef_
<span style="color: #658b00">print</span>(betaOLS)
<span style="color: #658b00">print</span>(betaRidge)
interceptOLS = np.mean(y_train) - X_train_mean @ betaOLS
interceptRidge = y_scaler - X_train_mean @ betaRidge
<span style="color: #658b00">print</span>(interceptOLS)
<span style="color: #658b00">print</span>(interceptRidge)
<span style="color: #228B22">#predict value </span>
ytilde_test_Ridge = X_test_scaled @ betaRidge+y_scaler
ytilde_test_OLS = X_test_scaled @ betaOLS+y_scaler
<span style="color: #228B22">#Calculate MSE</span>
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of OLS&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot; &quot;</span>)
<span style="color: #658b00">print</span>(<span style="color: #CD5555">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #658b00">print</span>(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label=<span style="color: #CD5555">&#39;Data&#39;</span>)
plt.plot(x, X @ RegRidge.coef_ + RegRidge.intercept_ , label=<span style="color: #CD5555">&quot;Ridge_Fit&quot;</span>)
plt.grid()
plt.legend()
plt.show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- ------------------- end of main content --------------- -->
doc/pub/week37/html/week37.html
+389 -4
View File
@@ -289,7 +289,11 @@ div.toc p,a {
('The same example but now with cross-validation',
2,
None,
'the-same-example-but-now-with-cross-validation')]}
'the-same-example-but-now-with-cross-validation'),
('Notes on scaling with examples',
2,
None,
'notes-on-scaling-with-examples')]}
end of tocinfo -->
<body>
@@ -343,7 +347,7 @@ MathJax.Hub.Config({
<li> Lecture from last week on calculations of expectation values</li>
<li> Exercise for week 37</li>
<li> Work on project 1</li>
<li> See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session.</li>
<li> See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.</li>
<li> For more discussions of Ridge regression and calculation of averages, <a href="https://arxiv.org/abs/1509.09169" target="_blank">Wessel van Wieringen's</a> article is highly recommended.</li>
</ul>
</div>
@@ -352,9 +356,9 @@ MathJax.Hub.Config({
<b>Material for the lecture on Thursday September 7</b>
<p>
<ul>
<li> Statistical interpretation of Ridge and Lasso regression</li>
<li> Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff</li>
<li> Reads and Videos:</li>
<li> Statistical interpretation of Ridge and Lasso regression</li>
<li> Readings and Videos:</li>
<ul>
<li> Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap).</li>
<li> <a href="https://www.youtube.com/watch?v=fSytzGwwBVw" target="_blank">Video on cross validation</a></li>
@@ -2096,6 +2100,387 @@ plt<span style="color: #666666">.</span>show()
</div>
</div>
</div>
<h2 id="notes-on-scaling-with-examples">Notes on scaling with examples </h2>
<p>The programs here use both ordinrary least squares (OLS) and Ridge
regression with one value only for the hyperparameter \( \lambda \). The
first example has no scaling and includes the intercept as well and we
are trying to fit a second-order polynomial. The second code takes out
the intercept and subtracts the mean values of each column of the
design matrix and the mean value of the outputs.
</p>
<p>The third and final code uses <b>Scikit-Learn</b> as library in order to
calculate the optimal parameters for OLS and Ridge regression. Note
that it is highly recommended to not include the intercept in Ridge
and Lasso regression, in order to avoid penalizing the optimization by
the intercept. The second and third codes do thus not include the
intercept. In the second code we do the scaling ourselves while the
last code uses the standard scaler option included in <b>Scikit-Learn</b>, known as centering (where
we subtract the mean values).
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">matplotlib.pyplot</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">plt</span>
<span style="color: #008000; font-weight: bold">import</span> <span style="color: #0000FF; font-weight: bold">numpy</span> <span style="color: #008000; font-weight: bold">as</span> <span style="color: #0000FF; font-weight: bold">np</span>
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.linear_model</span> <span style="color: #008000; font-weight: bold">import</span> LinearRegression
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> PolynomialFeatures
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.model_selection</span> <span style="color: #008000; font-weight: bold">import</span> train_test_split
<span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn.preprocessing</span> <span style="color: #008000; font-weight: bold">import</span> StandardScaler
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">MSE</span>(y_data,y_model):
n <span style="color: #666666">=</span> np<span style="color: #666666">.</span>size(y_model)
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>sum((y_data<span style="color: #666666">-</span>y_model)<span style="color: #666666">**2</span>)<span style="color: #666666">/</span>n
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">OLS_fit_beta</span>(X, y):
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X) <span style="color: #666666">@</span> X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y
<span style="color: #008000; font-weight: bold">def</span> <span style="color: #0000FF">Ridge_fit_beta</span>(X, y,L,d):
I <span style="color: #666666">=</span> np<span style="color: #666666">.</span>eye(d,d)
<span style="color: #008000; font-weight: bold">return</span> np<span style="color: #666666">.</span>linalg<span style="color: #666666">.</span>pinv(X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> X <span style="color: #666666">+</span> L<span style="color: #666666">*</span>I) <span style="color: #666666">@</span> X<span style="color: #666666">.</span>T <span style="color: #666666">@</span> y
<span style="color: #408080; font-style: italic"># Same random numbers for each test.</span>
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">2018</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
d <span style="color: #666666">=</span> <span style="color: #666666">3</span>
<span style="color: #408080; font-style: italic"># hyperparameter lambda</span>
Lambda <span style="color: #666666">=</span> <span style="color: #666666">0.01</span>
<span style="color: #408080; font-style: italic"># Make data set, simple second-order polynomial</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-3</span>, <span style="color: #666666">3</span>, n)
y <span style="color: #666666">=</span> <span style="color: #666666">2.0</span> <span style="color: #666666">+</span> <span style="color: #666666">0.5*</span>x <span style="color: #666666">+</span> <span style="color: #666666">5.0*</span>(x<span style="color: #666666">**2</span>)<span style="color: #666666">+</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(n)
<span style="color: #408080; font-style: italic"># The design matrix X includes the intercept and no scaling is made</span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(x), d))
<span style="color: #008000; font-weight: bold">for</span> p <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(d):
X[:, p] <span style="color: #666666">=</span> x <span style="color: #666666">**</span> (p)
<span style="color: #408080; font-style: italic">#Split data, no scaling is used and we include the intercept</span>
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(X, y, test_size<span style="color: #666666">=0.2</span>)
<span style="color: #408080; font-style: italic">#Calculate beta, own code</span>
beta_OLS <span style="color: #666666">=</span> OLS_fit_beta(X_train, y_train)
beta_Ridge <span style="color: #666666">=</span> Ridge_fit_beta(X_train, y_train,Lambda,d)
<span style="color: #008000">print</span>(beta_OLS)
<span style="color: #008000">print</span>(beta_Ridge)
<span style="color: #408080; font-style: italic">#predict value</span>
ytilde_test_OLS <span style="color: #666666">=</span> X_test <span style="color: #666666">@</span> beta_OLS
ytilde_test_Ridge <span style="color: #666666">=</span> X_test <span style="color: #666666">@</span> beta_Ridge
<span style="color: #408080; font-style: italic">#Calculate MSE</span>
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot; &quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;test MSE of OLS:&quot;</span>)
<span style="color: #008000">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot; &quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #008000">print</span>(MSE(y_test,ytilde_test_Ridge))
plt<span style="color: #666666">.</span>scatter(x,y,label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Data&#39;</span>)
plt<span style="color: #666666">.</span>plot(x, X <span style="color: #666666">@</span> beta_OLS,<span style="color: #BA2121">&#39;*&#39;</span>, label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;OLS_Fit&quot;</span>)
plt<span style="color: #666666">.</span>plot(x, X <span style="color: #666666">@</span> beta_Ridge, label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;Ridge_Fit&quot;</span>)
plt<span style="color: #666666">.</span>grid()
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the <a href="https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#more-on-rescaling-data" target="_blank">lecture material</a>.
see also the weekly slides <a href="https://compphysics.github.io/MachineLearning/doc/pub/week36/html/._week36-bs029.html" target="_blank">for week 36</a>.
It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.
</p>
<p>Before we discuss the code, we repeat some of the basic math from the slides of week 36.</p>
<p>Let us try to understand what this may imply mathematically when we
subtract the mean values, also known as <em>zero centering</em> or simply <em>centering</em>. For
simplicity, we will focus on ordinary regression, as done in the above example.
</p>
<p>The cost/loss function for regression is</p>
$$
C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
$$
<p>Recall also that we use the squared value. This expression can lead to an
increased penalty for higher differences between predicted and
output/target values.
</p>
<p>What we have done is to single out the \( \beta_0 \) term in the
definition of the mean squared error (MSE). The design matrix \( X \)
does in this case not contain any intercept column. When we take the
derivative with respect to \( \beta_0 \), we want the derivative to obey
</p>
$$
\frac{\partial C}{\partial \beta_j} = 0,
$$
<p>for all \( j \). For \( \beta_0 \) we have</p>
$$
\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right).
$$
<p>Multiplying away the constant \( 2/n \), we obtain</p>
$$
\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
$$
<p>Let us specialize first to the case where we have only two parameters \( \beta_0 \) and \( \beta_1 \).
Our result for \( \beta_0 \) simplifies then to
</p>
$$
n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
$$
<p>We obtain then</p>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
$$
<p>If we define</p>
$$
\mu_{\boldsymbol{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
$$
<p>and the mean value of the outputs as</p>
$$
\mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
$$
<p>we have</p>
$$
\beta_0 = \mu_y - \beta_1\mu_{\boldsymbol{x}_1}.
$$
<p>In the general case with more parameters than \( \beta_0 \) and \( \beta_1 \), we have</p>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
$$
<p>We can rewrite the latter equation as</p>
$$
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\boldsymbol{x}_j}\beta_j,
$$
<p>where we have defined</p>
$$
\mu_{\boldsymbol{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
$$
<p>the mean value for all elements of the column vector \( \boldsymbol{x}_j \).</p>
<p>Replacing \( y_i \) with \( y_i - y_i - \overline{\boldsymbol{y}} \) and centering also our design matrix results in a cost function (in vector-matrix disguise)</p>
$$
C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}).
$$
<p>If we minimize with respect to \( \boldsymbol{\beta} \) we have then</p>
$$
\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
$$
<p>where \( \boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\boldsymbol{y}} \)
and \( \tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj} \).
</p>
<p>For Ridge regression we need to add \( \lambda \boldsymbol{\beta}^T\boldsymbol{\beta} \) to the cost function and get then</p>
$$
\hat{\boldsymbol{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
$$
<p>Now we try to implement this.</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;">np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">2018</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">100</span>
<span style="color: #408080; font-style: italic"># we do not include the intercept</span>
d <span style="color: #666666">=</span> <span style="color: #666666">2</span>
Lambda <span style="color: #666666">=</span> <span style="color: #666666">0.01</span>
<span style="color: #408080; font-style: italic"># Make data set.</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-3</span>, <span style="color: #666666">3</span>, n)
y <span style="color: #666666">=</span> <span style="color: #666666">2.0</span> <span style="color: #666666">+</span> <span style="color: #666666">0.5*</span>x <span style="color: #666666">+</span> <span style="color: #666666">5.0*</span>(x<span style="color: #666666">**2</span>)<span style="color: #666666">+</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(n)
<span style="color: #408080; font-style: italic">#Design matrix X does not include the intercept. </span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((<span style="color: #008000">len</span>(x), d))
<span style="color: #008000; font-weight: bold">for</span> p <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(d):
X[:, p] <span style="color: #666666">=</span> x <span style="color: #666666">**</span> (p<span style="color: #666666">+1</span>)
<span style="color: #408080; font-style: italic">#Split data in train and test</span>
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(X, y, test_size<span style="color: #666666">=0.2</span>)
<span style="color: #408080; font-style: italic"># Scale data by subtracting mean value,own implementation</span>
<span style="color: #408080; font-style: italic">#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable</span>
X_train_mean <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(X_train,axis<span style="color: #666666">=0</span>)
<span style="color: #408080; font-style: italic">#Center by removing mean from each feature</span>
X_train_scaled <span style="color: #666666">=</span> X_train <span style="color: #666666">-</span> X_train_mean
X_test_scaled <span style="color: #666666">=</span> X_test <span style="color: #666666">-</span> X_train_mean
<span style="color: #408080; font-style: italic">#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)</span>
y_scaler <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(y_train)
y_train_scaled <span style="color: #666666">=</span> y_train <span style="color: #666666">-</span> y_scaler
<span style="color: #408080; font-style: italic">#Calculate beta</span>
beta_OLS <span style="color: #666666">=</span> OLS_fit_beta(X_train_scaled, y_train_scaled)
beta_Ridge <span style="color: #666666">=</span> Ridge_fit_beta(X_train_scaled, y_train_scaled,Lambda,d)
<span style="color: #008000">print</span>(beta_OLS)
<span style="color: #008000">print</span>(beta_Ridge)
<span style="color: #408080; font-style: italic"># calculate intercepts and print them</span>
interceptOLS <span style="color: #666666">=</span> y_scaler <span style="color: #666666">-</span> X_train_mean <span style="color: #666666">@</span> beta_OLS
interceptRidge <span style="color: #666666">=</span> y_scaler <span style="color: #666666">-</span> X_train_mean <span style="color: #666666">@</span> beta_Ridge
<span style="color: #008000">print</span>(interceptOLS)
<span style="color: #008000">print</span>(interceptRidge)
<span style="color: #408080; font-style: italic">#predict value with intercept</span>
ytilde_test_OLS <span style="color: #666666">=</span> X_test_scaled <span style="color: #666666">@</span> beta_OLS<span style="color: #666666">+</span>y_scaler
ytilde_test_Ridge <span style="color: #666666">=</span> X_test_scaled <span style="color: #666666">@</span> beta_Ridge<span style="color: #666666">+</span>y_scaler
<span style="color: #408080; font-style: italic">#Calculate MSE</span>
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot; &quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;test MSE of OLS:&quot;</span>)
<span style="color: #008000">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot; &quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #008000">print</span>(MSE(y_test,ytilde_test_Ridge))
plt<span style="color: #666666">.</span>scatter(x,y,label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Data&#39;</span>)
plt<span style="color: #666666">.</span>plot(x, X <span style="color: #666666">@</span> beta_OLS<span style="color: #666666">+</span>interceptOLS,<span style="color: #BA2121">&#39;*&#39;</span>, label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;OLS_Fit&quot;</span>)
plt<span style="color: #666666">.</span>plot(x, X <span style="color: #666666">@</span> beta_Ridge<span style="color: #666666">+</span>interceptRidge, label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;Ridge_Fit&quot;</span>)
plt<span style="color: #666666">.</span>grid()
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<p>Finally, instead of using our own function we repeat the same example
using the <b>standardscaler</b> functionality of the library
<b>Scikit-Learn</b>. Here we limit ourselves to Ridge regression only.
</p>
<!-- code=python (!bc pycod) typeset with pygments style "default" -->
<div class="cell border-box-sizing code_cell rendered">
<div class="input">
<div class="inner_cell">
<div class="input_area">
<div class="highlight" style="background: #f8f8f8">
<pre style="line-height: 125%;"><span style="color: #008000; font-weight: bold">from</span> <span style="color: #0000FF; font-weight: bold">sklearn</span> <span style="color: #008000; font-weight: bold">import</span> linear_model
np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>seed(<span style="color: #666666">2018</span>)
n <span style="color: #666666">=</span> <span style="color: #666666">10</span>
d <span style="color: #666666">=</span> <span style="color: #666666">2</span>
Lambda <span style="color: #666666">=</span> <span style="color: #666666">0.01</span>
<span style="color: #408080; font-style: italic"># Make data set.</span>
x <span style="color: #666666">=</span> np<span style="color: #666666">.</span>linspace(<span style="color: #666666">-3</span>, <span style="color: #666666">3</span>, n)
y <span style="color: #666666">=</span> <span style="color: #666666">2.0</span> <span style="color: #666666">+</span> <span style="color: #666666">0.5*</span>x <span style="color: #666666">+</span> <span style="color: #666666">5.0*</span>(x<span style="color: #666666">**2</span>)<span style="color: #666666">+</span> np<span style="color: #666666">.</span>random<span style="color: #666666">.</span>randn(n)
<span style="color: #408080; font-style: italic"># Design matrix X does not include the intercept. </span>
X <span style="color: #666666">=</span> np<span style="color: #666666">.</span>zeros((n, d))
<span style="color: #008000; font-weight: bold">for</span> p <span style="color: #AA22FF; font-weight: bold">in</span> <span style="color: #008000">range</span>(d):
X[:, p] <span style="color: #666666">=</span> x <span style="color: #666666">**</span> (p<span style="color: #666666">+1</span>)
<span style="color: #408080; font-style: italic">#Split data in train and test</span>
X_train, X_test, y_train, y_test <span style="color: #666666">=</span> train_test_split(X, y, test_size<span style="color: #666666">=0.2</span>)
<span style="color: #408080; font-style: italic"># Scale data by subtracting mean value of the input using scikit-learn</span>
scaler <span style="color: #666666">=</span> StandardScaler(with_std<span style="color: #666666">=</span><span style="color: #008000; font-weight: bold">False</span>)
scaler<span style="color: #666666">.</span>fit(X_train)
X_train_mean <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(X_train,axis<span style="color: #666666">=0</span>)
X_train_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_train)
X_test_scaled <span style="color: #666666">=</span> scaler<span style="color: #666666">.</span>transform(X_test)
<span style="color: #408080; font-style: italic"># We scale also the output, here by our own code</span>
y_scaler <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(y_train)
y_train_scaled <span style="color: #666666">=</span> y_train <span style="color: #666666">-</span> y_scaler
y_test_scaled <span style="color: #666666">=</span> y_test<span style="color: #666666">-</span> y_scaler
<span style="color: #408080; font-style: italic">#Calculate beta</span>
OLS <span style="color: #666666">=</span> LinearRegression()
betaOLS<span style="color: #666666">=</span>OLS<span style="color: #666666">.</span>fit(X_train_scaled,y_train_scaled)
ypredictOLS <span style="color: #666666">=</span> OLS<span style="color: #666666">.</span>predict(X_test_scaled)
linear_model<span style="color: #666666">.</span>Ridge(Lambda)
RegRidge<span style="color: #666666">.</span>fit(X_train_scaled,y_train_scaled)
ypredictRidge <span style="color: #666666">=</span> RegRidge<span style="color: #666666">.</span>predict(X_test_scaled)
betaOLS <span style="color: #666666">=</span> OLS<span style="color: #666666">.</span>coef_
betaRidge <span style="color: #666666">=</span> RegRidge<span style="color: #666666">.</span>coef_
<span style="color: #008000">print</span>(betaOLS)
<span style="color: #008000">print</span>(betaRidge)
interceptOLS <span style="color: #666666">=</span> np<span style="color: #666666">.</span>mean(y_train) <span style="color: #666666">-</span> X_train_mean <span style="color: #666666">@</span> betaOLS
interceptRidge <span style="color: #666666">=</span> y_scaler <span style="color: #666666">-</span> X_train_mean <span style="color: #666666">@</span> betaRidge
<span style="color: #008000">print</span>(interceptOLS)
<span style="color: #008000">print</span>(interceptRidge)
<span style="color: #408080; font-style: italic">#predict value </span>
ytilde_test_Ridge <span style="color: #666666">=</span> X_test_scaled <span style="color: #666666">@</span> betaRidge<span style="color: #666666">+</span>y_scaler
ytilde_test_OLS <span style="color: #666666">=</span> X_test_scaled <span style="color: #666666">@</span> betaOLS<span style="color: #666666">+</span>y_scaler
<span style="color: #408080; font-style: italic">#Calculate MSE</span>
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot; &quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;test MSE of OLS&quot;</span>)
<span style="color: #008000">print</span>(MSE(y_test,ytilde_test_OLS))
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot; &quot;</span>)
<span style="color: #008000">print</span>(<span style="color: #BA2121">&quot;test MSE of Ridge&quot;</span>)
<span style="color: #008000">print</span>(MSE(y_test,ytilde_test_Ridge))
plt<span style="color: #666666">.</span>scatter(x,y,label<span style="color: #666666">=</span><span style="color: #BA2121">&#39;Data&#39;</span>)
plt<span style="color: #666666">.</span>plot(x, X <span style="color: #666666">@</span> RegRidge<span style="color: #666666">.</span>coef_ <span style="color: #666666">+</span> RegRidge<span style="color: #666666">.</span>intercept_ , label<span style="color: #666666">=</span><span style="color: #BA2121">&quot;Ridge_Fit&quot;</span>)
plt<span style="color: #666666">.</span>grid()
plt<span style="color: #666666">.</span>legend()
plt<span style="color: #666666">.</span>show()
</pre>
</div>
</div>
</div>
</div>
<div class="output_wrapper">
<div class="output">
<div class="output_area">
<div class="output_subarea output_stream output_stdout output_text">
</div>
</div>
</div>
</div>
</div>
<!-- ------------------- end of main content --------------- -->
doc/pub/week37/ipynb/ipynb-week37-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week37/ipynb/week37.ipynb
+799 -179
View File
File diff suppressed because it is too large Load Diff
doc/src/week37/week37.do.txt
+365 -3
View File
@@ -12,13 +12,13 @@ DATE: today
* Lecture from last week on calculations of expectation values
* Exercise for week 37
* Work on project 1
* See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session.
* See also additional note on scaling (jupyter-notebook) sent separately. This will be discussed during the first hour of each session. This note is added at the end of these slides.
* For more discussions of Ridge regression and calculation of averages, "Wessel van Wieringen's":"https://arxiv.org/abs/1509.09169" article is highly recommended.
!eblock
!bblock Material for the lecture on Thursday September 7
* Statistical interpretation of Ridge and Lasso regression
* Resampling techniques, Bootstrap and cross validation and bias-variance tradeoff
* Reads and Videos:
* Statistical interpretation of Ridge and Lasso regression
* Readings and Videos:
* Hastie et al Chapter 7, here we recommend 7.1-7.5 and 7.10 (cross-validation) and 7.11 (bootstrap).
* "Video on cross validation":"https://www.youtube.com/watch?v=fSytzGwwBVw"
* "Video on Bootstrapping":"https://www.youtube.com/watch?v=Xz0x-8-cgaQ"
@@ -1645,3 +1645,365 @@ plt.show()
===== Notes on scaling with examples =====
The programs here use both ordinrary least squares (OLS) and Ridge
regression with one value only for the hyperparameter $\lambda$. The
first example has no scaling and includes the intercept as well and we
are trying to fit a second-order polynomial. The second code takes out
the intercept and subtracts the mean values of each column of the
design matrix and the mean value of the outputs.
The third and final code uses _Scikit-Learn_ as library in order to
calculate the optimal parameters for OLS and Ridge regression. Note
that it is highly recommended to not include the intercept in Ridge
and Lasso regression, in order to avoid penalizing the optimization by
the intercept. The second and third codes do thus not include the
intercept. In the second code we do the scaling ourselves while the
last code uses the standard scaler option included in _Scikit-Learn_, known as centering (where
we subtract the mean values).
!bc pycod
import matplotlib.pyplot as plt
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
def MSE(y_data,y_model):
n = np.size(y_model)
return np.sum((y_data-y_model)**2)/n
def OLS_fit_beta(X, y):
return np.linalg.pinv(X.T @ X) @ X.T @ y
def Ridge_fit_beta(X, y,L,d):
I = np.eye(d,d)
return np.linalg.pinv(X.T @ X + L*I) @ X.T @ y
# Same random numbers for each test.
np.random.seed(2018)
n = 100
d = 3
# hyperparameter lambda
Lambda = 0.01
# Make data set, simple second-order polynomial
x = np.linspace(-3, 3, n)
y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)
# The design matrix X includes the intercept and no scaling is made
X = np.zeros((len(x), d))
for p in range(d):
X[:, p] = x ** (p)
#Split data, no scaling is used and we include the intercept
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
#Calculate beta, own code
beta_OLS = OLS_fit_beta(X_train, y_train)
beta_Ridge = Ridge_fit_beta(X_train, y_train,Lambda,d)
print(beta_OLS)
print(beta_Ridge)
#predict value
ytilde_test_OLS = X_test @ beta_OLS
ytilde_test_Ridge = X_test @ beta_Ridge
#Calculate MSE
print(" ")
print("test MSE of OLS:")
print(MSE(y_test,ytilde_test_OLS))
print(" ")
print("test MSE of Ridge")
print(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label='Data')
plt.plot(x, X @ beta_OLS,'*', label="OLS_Fit")
plt.plot(x, X @ beta_Ridge, label="Ridge_Fit")
plt.grid()
plt.legend()
plt.show()
!ec
In this example we do not include the intercept and we scale the data by subtracting the mean values. This follows the discussion in the "lecture material":"https://compphysics.github.io/MachineLearning/doc/LectureNotes/_build/html/chapter3.html#more-on-rescaling-data".
see also the weekly slides "for week 36":"https://compphysics.github.io/MachineLearning/doc/pub/week36/html/._week36-bs029.html".
It is recommended whrn we use Ridge and Lasso regression to not include the intercept in the optimization process.
Before we discuss the code, we repeat some of the basic math from the slides of week 36.
Let us try to understand what this may imply mathematically when we
subtract the mean values, also known as *zero centering* or simply *centering*. For
simplicity, we will focus on ordinary regression, as done in the above example.
The cost/loss function for regression is
!bt
\[
C(\beta_0, \beta_1, ... , \beta_{p-1}) = \frac{1}{n}\sum_{i=0}^{n} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij}\beta_j\right)^2,.
\]
!et
Recall also that we use the squared value. This expression can lead to an
increased penalty for higher differences between predicted and
output/target values.
What we have done is to single out the $\beta_0$ term in the
definition of the mean squared error (MSE). The design matrix $X$
does in this case not contain any intercept column. When we take the
derivative with respect to $\beta_0$, we want the derivative to obey
!bt
\[
\frac{\partial C}{\partial \beta_j} = 0,
\]
!et
for all $j$. For $\beta_0$ we have
!bt
\[
\frac{\partial C}{\partial \beta_0} = -\frac{2}{n}\sum_{i=0}^{n-1} \left(y_i - \beta_0 - \sum_{j=1}^{p-1} X_{ij} \beta_j\right).
\]
!et
Multiplying away the constant $2/n$, we obtain
!bt
\[
\sum_{i=0}^{n-1} \beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} \sum_{j=1}^{p-1} X_{ij} \beta_j.
\]
!et
Let us specialize first to the case where we have only two parameters $\beta_0$ and $\beta_1$.
Our result for $\beta_0$ simplifies then to
!bt
\[
n\beta_0 = \sum_{i=0}^{n-1}y_i - \sum_{i=0}^{n-1} X_{i1} \beta_1.
\]
!et
We obtain then
!bt
\[
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \beta_1\frac{1}{n}\sum_{i=0}^{n-1} X_{i1}.
\]
!et
If we define
!bt
\[
\mu_{\bm{x}_1}=\frac{1}{n}\sum_{i=0}^{n-1} X_{i1},
\]
!et
and the mean value of the outputs as
!bt
\[
\mu_y=\frac{1}{n}\sum_{i=0}^{n-1}y_i,
\]
!et
we have
!bt
\[
\beta_0 = \mu_y - \beta_1\mu_{\bm{x}_1}.
\]
!et
In the general case with more parameters than $\beta_0$ and $\beta_1$, we have
!bt
\[
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \frac{1}{n}\sum_{i=0}^{n-1}\sum_{j=1}^{p-1} X_{ij}\beta_j.
\]
!et
We can rewrite the latter equation as
!bt
\[
\beta_0 = \frac{1}{n}\sum_{i=0}^{n-1}y_i - \sum_{j=1}^{p-1} \mu_{\bm{x}_j}\beta_j,
\]
!et
where we have defined
!bt
\[
\mu_{\bm{x}_j}=\frac{1}{n}\sum_{i=0}^{n-1} X_{ij},
\]
!et
the mean value for all elements of the column vector $\bm{x}_j$.
Replacing $y_i$ with $y_i - y_i - \overline{\bm{y}}$ and centering also our design matrix results in a cost function (in vector-matrix disguise)
!bt
\[
C(\boldsymbol{\beta}) = (\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta})^T(\boldsymbol{\tilde{y}} - \tilde{X}\boldsymbol{\beta}).
\]
!et
If we minimize with respect to $\bm{\beta}$ we have then
!bt
\[
\hat{\bm{\beta}} = (\tilde{X}^T\tilde{X})^{-1}\tilde{X}^T\boldsymbol{\tilde{y}},
\]
!et
where $\boldsymbol{\tilde{y}} = \boldsymbol{y} - \overline{\bm{y}}$
and $\tilde{X}_{ij} = X_{ij} - \frac{1}{n}\sum_{k=0}^{n-1}X_{kj}$.
For Ridge regression we need to add $\lambda \boldsymbol{\beta}^T\boldsymbol{\beta}$ to the cost function and get then
!bt
\[
\hat{\bm{\beta}} = (\tilde{X}^T\tilde{X} + \lambda I)^{-1}\tilde{X}^T\boldsymbol{\tilde{y}}.
\]
!et
Now we try to implement this.
!bc pycod
np.random.seed(2018)
n = 100
# we do not include the intercept
d = 2
Lambda = 0.01
# Make data set.
x = np.linspace(-3, 3, n)
y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)
#Design matrix X does not include the intercept.
X = np.zeros((len(x), d))
for p in range(d):
X[:, p] = x ** (p+1)
#Split data in train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# Scale data by subtracting mean value,own implementation
#For our own implementation, we will need to deal with the intercept by centering the design matrix and the target variable
X_train_mean = np.mean(X_train,axis=0)
#Center by removing mean from each feature
X_train_scaled = X_train - X_train_mean
X_test_scaled = X_test - X_train_mean
#The model intercept (called y_scaler) is given by the mean of the target variable (IF X is centered, note)
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
#Calculate beta
beta_OLS = OLS_fit_beta(X_train_scaled, y_train_scaled)
beta_Ridge = Ridge_fit_beta(X_train_scaled, y_train_scaled,Lambda,d)
print(beta_OLS)
print(beta_Ridge)
# calculate intercepts and print them
interceptOLS = y_scaler - X_train_mean @ beta_OLS
interceptRidge = y_scaler - X_train_mean @ beta_Ridge
print(interceptOLS)
print(interceptRidge)
#predict value with intercept
ytilde_test_OLS = X_test_scaled @ beta_OLS+y_scaler
ytilde_test_Ridge = X_test_scaled @ beta_Ridge+y_scaler
#Calculate MSE
print(" ")
print("test MSE of OLS:")
print(MSE(y_test,ytilde_test_OLS))
print(" ")
print("test MSE of Ridge")
print(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label='Data')
plt.plot(x, X @ beta_OLS+interceptOLS,'*', label="OLS_Fit")
plt.plot(x, X @ beta_Ridge+interceptRidge, label="Ridge_Fit")
plt.grid()
plt.legend()
plt.show()
!ec
Finally, instead of using our own function we repeat the same example
using the _standardscaler_ functionality of the library
_Scikit-Learn_. Here we limit ourselves to Ridge regression only.
!bc pycod
from sklearn import linear_model
np.random.seed(2018)
n = 10
d = 2
Lambda = 0.01
# Make data set.
x = np.linspace(-3, 3, n)
y = 2.0 + 0.5*x + 5.0*(x**2)+ np.random.randn(n)
# Design matrix X does not include the intercept.
X = np.zeros((n, d))
for p in range(d):
X[:, p] = x ** (p+1)
#Split data in train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
# Scale data by subtracting mean value of the input using scikit-learn
scaler = StandardScaler(with_std=False)
scaler.fit(X_train)
X_train_mean = np.mean(X_train,axis=0)
X_train_scaled = scaler.transform(X_train)
X_test_scaled = scaler.transform(X_test)
# We scale also the output, here by our own code
y_scaler = np.mean(y_train)
y_train_scaled = y_train - y_scaler
y_test_scaled = y_test- y_scaler
#Calculate beta
OLS = LinearRegression()
betaOLS=OLS.fit(X_train_scaled,y_train_scaled)
ypredictOLS = OLS.predict(X_test_scaled)
linear_model.Ridge(Lambda)
RegRidge.fit(X_train_scaled,y_train_scaled)
ypredictRidge = RegRidge.predict(X_test_scaled)
betaOLS = OLS.coef_
betaRidge = RegRidge.coef_
print(betaOLS)
print(betaRidge)
interceptOLS = np.mean(y_train) - X_train_mean @ betaOLS
interceptRidge = y_scaler - X_train_mean @ betaRidge
print(interceptOLS)
print(interceptRidge)
#predict value
ytilde_test_Ridge = X_test_scaled @ betaRidge+y_scaler
ytilde_test_OLS = X_test_scaled @ betaOLS+y_scaler
#Calculate MSE
print(" ")
print("test MSE of OLS")
print(MSE(y_test,ytilde_test_OLS))
print(" ")
print("test MSE of Ridge")
print(MSE(y_test,ytilde_test_Ridge))
plt.scatter(x,y,label='Data')
plt.plot(x, X @ RegRidge.coef_ + RegRidge.intercept_ , label="Ridge_Fit")
plt.grid()
plt.legend()
plt.show()
!ec