lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updating book

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-08-26 10:38:31 +02:00
parent e23be69563
commit c20f947816
66 changed files with 13205 additions and 445 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter1.html
+458 -73
View File
@@ -781,13 +781,13 @@ example of the functionality of <strong>Scikit-Learn</strong>.</p>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The intercept alpha:
[2.18954529]
[1.98452685]
Coefficient beta :
[[4.57051369]]
Mean squared error: 0.23
Variance score: 0.88
[[5.00109273]]
Mean squared error: 0.19
Variance score: 0.92
Mean squared log error: 0.01
Mean absolute error: 0.39
Mean absolute error: 0.35
</pre></div>
</div>
<img alt="_images/chapter1_13_1.png" src="_images/chapter1_13_1.png" />
@@ -888,7 +888,7 @@ a linear <span class="math notranslate nohighlight">\(x\)</span>-dependence we s
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter1_27_0.png" src="_images/chapter1_27_0.png" />
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.004999999999999997
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.004999999999999996
</pre></div>
</div>
</div>
@@ -997,18 +997,6 @@ After having downloaded this file to our own computer, we are now ready to read
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">FileNotFoundError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="o">&lt;</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">5</span><span class="o">-</span><span class="mi">3</span><span class="n">cd19a0768e1</span><span class="o">&gt;</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span>
<span class="g g-Whitespace"> </span><span class="mi">31</span> <span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot;.png&quot;</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">&#39;png&#39;</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">32</span>
<span class="ne">---&gt; </span><span class="mi">33</span> <span class="n">infile</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="n">data_path</span><span class="p">(</span><span class="s2">&quot;MassEval2016.dat&quot;</span><span class="p">),</span><span class="s1">&#39;r&#39;</span><span class="p">)</span>
<span class="ne">FileNotFoundError</span>: [Errno 2] No such file or directory: &#39;DataFiles/MassEval2016.dat&#39;
</pre></div>
</div>
</div>
</div>
<p>Before we proceed, we define also a function for making our plots. You can obviously avoid this and simply set up various <strong>matplotlib</strong> commands every time you need them. You may however find it convenient to collect all such commands in one function and simply call this function.</p>
<div class="cell docutils container">
@@ -1049,6 +1037,11 @@ data) to actually open the file and simply take a look at it!</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&#39; \nThis is taken from the data file of the mass 2016 evaluation. \nAll files are 3436 lines long with 124 character per line. \n Headers are 39 lines long. \n col 1 : Fortran character control: 1 = page feed 0 = line feed \n format : a1,i3,i5,i5,i5,1x,a3,a4,1x,f13.5,f11.5,f11.3,f9.3,1x,a2,f11.3,f9.3,1x,i3,1x,f12.5,f11.5 \n These formats are reflected in the pandas widths variable below, see the statement \n widths=(1,3,5,5,5,1,3,4,1,13,11,11,9,1,2,11,9,1,3,1,12,11,1), \n Pandas has also a variable header, with length 39 in this case. \n&#39;
</pre></div>
</div>
</div>
</div>
<p>The data we are interested in are in columns 2, 3, 4 and 11, giving us
the number of neutrons, protons, mass numbers and binding energies,
@@ -1098,6 +1091,25 @@ the number of nucleons <span class="math notranslate nohighlight">\(A\)</span>,
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> N Z A Element Ebinding
A
1 0 0 1 1 H 0.000000
2 1 1 1 2 H 1.112283
3 2 2 1 3 H 2.827265
4 6 2 2 4 He 7.073915
5 9 3 2 5 He 5.512132
... ... ... ... ... ...
264 3304 156 108 264 Hs 7.298375
265 3310 157 108 265 Hs 7.296247
266 3317 158 108 266 Hs 7.298273
269 3338 159 110 269 Ds 7.250154
270 3344 160 110 270 Ds 7.253775
[267 rows x 5 columns]
</pre></div>
</div>
</div>
</div>
<p>The next step, and we will define this mathematically later, is to set up the so-called <strong>design matrix</strong>. We will throughout call this matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>.
It has dimensionality <span class="math notranslate nohighlight">\(p\times n\)</span>, where <span class="math notranslate nohighlight">\(n\)</span> is the number of data points and <span class="math notranslate nohighlight">\(p\)</span> are the so-called predictors. In our case here they are given by the number of polynomials in <span class="math notranslate nohighlight">\(A\)</span> we wish to include in the fit.</p>
@@ -1150,6 +1162,16 @@ Now we can print measures of how our fit is doing, the coefficients from the fit
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Mean squared error: 0.04
Variance score: 0.95
Mean absolute error: 0.05
[ 0.00000000e+00 7.06492086e-03 -1.73091052e-01 -1.66020213e+01
1.17385778e+00] 15.212327334149492
</pre></div>
</div>
<img alt="_images/chapter1_57_1.png" src="_images/chapter1_57_1.png" />
</div>
</div>
<p>As a teaser, let us now see how we can do this with decision trees using <strong>scikit-learn</strong>. Later we will switch to so-called <strong>random forests</strong>!</p>
<div class="cell docutils container">
@@ -1186,6 +1208,27 @@ Now we can print measures of how our fit is doing, the coefficients from the fit
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter1_59_0.png" src="_images/chapter1_59_0.png" />
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> N Z A Element Ebinding Eapprox
A
1 0 0 1 1 H 0.000000 0.000000
2 1 1 1 2 H 1.112283 1.112283
3 2 2 1 3 H 2.827265 2.827265
4 6 2 2 4 He 7.073915 7.073915
5 9 3 2 5 He 5.512132 5.512132
... ... ... ... ... ... ...
264 3304 156 108 264 Hs 7.298375 7.298375
265 3310 157 108 265 Hs 7.296247 7.297260
266 3317 158 108 266 Hs 7.298273 7.297260
269 3338 159 110 269 Ds 7.250154 7.250154
270 3344 160 110 270 Ds 7.253775 7.253775
[267 rows x 6 columns]
0.009883615646716184
</pre></div>
</div>
</div>
</div>
<p>The <strong>seaborn</strong> package allows us to visualize data in an efficient way. Note that we use <strong>scikit-learn</strong>’s multi-layer perceptron (or feed forward neural network)
functionality.</p>
@@ -1223,6 +1266,91 @@ functionality.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/sklearn/neural_network/_multilayer_perceptron.py:582: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn&#39;t converged yet.
warnings.warn(
</pre></div>
</div>
<img alt="_images/chapter1_61_9.png" src="_images/chapter1_61_9.png" />
</div>
</div>
</div>
</div>
@@ -1459,6 +1587,133 @@ our matrix as <span class="math notranslate nohighlight">\(\boldsymbol{X}\in {\m
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_html"><div>
<style scoped>
.dataframe tbody tr th:only-of-type {
vertical-align: middle;
}
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
</style>
<table border="1" class="dataframe">
<thead>
<tr style="text-align: right;">
<th></th>
<th>1</th>
<th>A</th>
<th>A^(2/3)</th>
<th>A^(-1/3)</th>
<th>1/A</th>
</tr>
<tr>
<th>A</th>
<th></th>
<th></th>
<th></th>
<th></th>
<th></th>
</tr>
</thead>
<tbody>
<tr>
<th>1</th>
<td>1.0</td>
<td>1.0</td>
<td>1.000000</td>
<td>1.000000</td>
<td>1.000000</td>
</tr>
<tr>
<th>2</th>
<td>1.0</td>
<td>2.0</td>
<td>1.587401</td>
<td>0.793701</td>
<td>0.500000</td>
</tr>
<tr>
<th>3</th>
<td>1.0</td>
<td>3.0</td>
<td>2.080084</td>
<td>0.693361</td>
<td>0.333333</td>
</tr>
<tr>
<th>4</th>
<td>1.0</td>
<td>4.0</td>
<td>2.519842</td>
<td>0.629961</td>
<td>0.250000</td>
</tr>
<tr>
<th>5</th>
<td>1.0</td>
<td>5.0</td>
<td>2.924018</td>
<td>0.584804</td>
<td>0.200000</td>
</tr>
<tr>
<th>...</th>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
<td>...</td>
</tr>
<tr>
<th>264</th>
<td>1.0</td>
<td>264.0</td>
<td>41.153106</td>
<td>0.155883</td>
<td>0.003788</td>
</tr>
<tr>
<th>265</th>
<td>1.0</td>
<td>265.0</td>
<td>41.256962</td>
<td>0.155687</td>
<td>0.003774</td>
</tr>
<tr>
<th>266</th>
<td>1.0</td>
<td>266.0</td>
<td>41.360688</td>
<td>0.155491</td>
<td>0.003759</td>
</tr>
<tr>
<th>269</th>
<td>1.0</td>
<td>269.0</td>
<td>41.671089</td>
<td>0.154911</td>
<td>0.003717</td>
</tr>
<tr>
<th>270</th>
<td>1.0</td>
<td>270.0</td>
<td>41.774300</td>
<td>0.154720</td>
<td>0.003704</td>
</tr>
</tbody>
</table>
<p>267 rows × 5 columns</p>
</div></div></div>
</div>
<p>With <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\in {\mathbb{R}}^{p\times 1}\)</span>, it means that we will hereafter write our equations for the approximation as</p>
<div class="math notranslate nohighlight">
@@ -1610,7 +1865,7 @@ C
K</p>
<div class="math notranslate nohighlight">
\[
\frac{\partial \log{\vert\boldsymbol{A}\vert}}{\partial \boldsymbol{A}} = (\boldsymbol{A}^{-1})^T.
\frac{\partial\log{\vert\boldsymbol{A}\vert}}{\partial \boldsymbol{A}}=(\boldsymbol{A}^{-1})^T.
\]</div>
<p>The residuals <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> are in turn given by</p>
<div class="math notranslate nohighlight">
@@ -1668,6 +1923,9 @@ write</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter1_132_0.png" src="_images/chapter1_132_0.png" />
</div>
</div>
<p>We can easily test our fit by computing the <span class="math notranslate nohighlight">\(R2\)</span> score that we discussed in connection with the functionality of <strong>Scikit-Learn</strong> in the introductory slides.
Since we are not using <strong>Scikit-Learn</strong> here we can define our own <span class="math notranslate nohighlight">\(R2\)</span> function as</p>
@@ -1686,6 +1944,11 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.9547578478889096
</pre></div>
</div>
</div>
</div>
<p>We can easily add our <strong>MSE</strong> score as</p>
<div class="cell docutils container">
@@ -1698,6 +1961,11 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>0.03787596148305236
</pre></div>
</div>
</div>
</div>
<p>and finally the relative error as</p>
<div class="cell docutils container">
@@ -1708,6 +1976,23 @@ Since we are not using <strong>Scikit-Learn</strong> here we can define our own
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>A
1 0 inf
2 1 1.123190
3 2 0.327631
4 6 0.344172
5 9 0.044402
...
264 3304 0.009911
265 3310 0.009154
266 3317 0.007824
269 3338 0.011347
270 3344 0.009790
Name: Ebinding, Length: 267, dtype: float64
</pre></div>
</div>
</div>
</div>
<div class="section" id="the-chi-2-function">
<h3><span class="section-number">3.4.1. </span>The <span class="math notranslate nohighlight">\(\chi^2\)</span> function<a class="headerlink" href="#the-chi-2-function" title="Permalink to this headline"></a></h3>
@@ -1935,6 +2220,19 @@ hyperparameter <span class="math notranslate nohighlight">\(\lambda\)</span>, al
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Mean squared error: 12.36
Variance score: 1.00
Mean absolute error: 2.83
[ 0. 618.32047562 -861.13519106 1404.91549644] -11.057088709963637
Mean squared error: 197.93
Variance score: 1.00
Mean absolute error: 11.69
[ 0. 28.18220995 282.79902342 842.30879705] 12.946893955211749
</pre></div>
</div>
<img alt="_images/chapter1_179_1.png" src="_images/chapter1_179_1.png" />
</div>
</div>
<p>The above simple polynomial in density <span class="math notranslate nohighlight">\(\rho\)</span> gives an excellent fit
to the data.</p>
@@ -2024,6 +2322,18 @@ but now splitting the data into a training set and a test set.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Training R2
0.9999864543345858
Training MSE
6.180092462880674
Test R2
0.9999822527140678
Test MSE
7.205466494327873
</pre></div>
</div>
</div>
</div>
</div>
<div class="section" id="the-boston-housing-data-example">
@@ -2078,6 +2388,11 @@ the house using the features (predictors) listed here.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>dict_keys([&#39;data&#39;, &#39;target&#39;, &#39;feature_names&#39;, &#39;DESCR&#39;, &#39;filename&#39;])
</pre></div>
</div>
</div>
</div>
<p>Then we invoke Pandas</p>
<div class="cell docutils container">
@@ -2097,6 +2412,25 @@ the house using the features (predictors) listed here.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>CRIM 0
ZN 0
INDUS 0
CHAS 0
NOX 0
RM 0
AGE 0
DIS 0
RAD 0
TAX 0
PTRATIO 0
B 0
LSTAT 0
MEDV 0
dtype: int64
</pre></div>
</div>
</div>
</div>
<p>We can then visualize the data</p>
<div class="cell docutils container">
@@ -2110,6 +2444,13 @@ the house using the features (predictors) listed here.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/opt/anaconda3/lib/python3.8/site-packages/seaborn/distributions.py:2551: FutureWarning: `distplot` is a deprecated function and will be removed in a future version. Please adapt your code to use either `displot` (a figure-level function with similar flexibility) or `histplot` (an axes-level function for histograms).
warnings.warn(msg, FutureWarning)
</pre></div>
</div>
<img alt="_images/chapter1_191_1.png" src="_images/chapter1_191_1.png" />
</div>
</div>
<p>It is now useful to look at the correlation matrix</p>
<div class="cell docutils container">
@@ -2122,6 +2463,12 @@ the house using the features (predictors) listed here.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output text_plain highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;AxesSubplot:&gt;
</pre></div>
</div>
<img alt="_images/chapter1_193_1.png" src="_images/chapter1_193_1.png" />
</div>
</div>
<p>From the above coorelation plot we can see that <strong>MEDV</strong> is strongly correlated to <strong>LSTAT</strong> and <strong>RM</strong>. We see also that <strong>RAD</strong> and <strong>TAX</strong> are stronly correlated, but we don’t include this in our features together to avoid multi-colinearity</p>
<div class="cell docutils container">
@@ -2142,6 +2489,9 @@ the house using the features (predictors) listed here.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter1_195_0.png" src="_images/chapter1_195_0.png" />
</div>
</div>
<p>Now we start training our model</p>
<div class="cell docutils container">
@@ -2167,6 +2517,14 @@ the house using the features (predictors) listed here.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>(404, 2)
(102, 2)
(404,)
(102,)
</pre></div>
</div>
</div>
</div>
<p>Then we use the linear regression functionality from <strong>Scikit-Learn</strong></p>
<div class="cell docutils container">
@@ -2205,6 +2563,20 @@ the house using the features (predictors) listed here.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>The model performance for training set
--------------------------------------
RMSE is 5.637129335071195
R2 score is 0.6300745149331701
The model performance for testing set
--------------------------------------
RMSE is 5.137400784702912
R2 score is 0.6628996975186952
</pre></div>
</div>
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -2215,6 +2587,9 @@ the house using the features (predictors) listed here.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter1_202_0.png" src="_images/chapter1_202_0.png" />
</div>
</div>
</div>
<div class="section" id="reducing-the-number-of-degrees-of-freedom-overarching-view">
@@ -2364,6 +2739,36 @@ techniques.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>MSE before scaling: 0.00
R2 score before scaling 0.99
Feature min values before scaling:
[1.00000000e+00 1.10094646e-03 9.51523276e-04 1.21208310e-06
1.04757618e-06 9.05396545e-07 1.33443859e-09 1.15332528e-09
9.96793117e-10 8.61505887e-10 1.46914544e-12 1.26974938e-12
1.09741585e-12 9.48471852e-13 8.19742904e-13 1.61745046e-15
1.39792608e-15 1.20819609e-15 1.04421672e-15 9.02493044e-16
7.80004454e-16]
Feature max values before scaling:
[1. 0.99825997 0.99883879 0.99652296 0.99710078 0.99767893
0.99478898 0.9953658 0.99594294 0.99652042 0.99305802 0.99363383
0.99420997 0.99478645 0.99536326 0.99133007 0.99190487 0.99248001
0.99305549 0.99363129 0.99420743]
Feature min values after scaling:
[ 0. -1.61869821 -1.66880047 -1.06209126 -1.08170444 -1.10131725
-0.84087101 -0.85396354 -0.86692943 -0.87972591 -0.71351486 -0.7240496
-0.73453972 -0.74495014 -0.75524378 -0.62783293 -0.63680118 -0.64580686
-0.65482578 -0.66383151 -0.67279536]
Feature max values after scaling:
[0. 1.78944806 1.68342382 2.34172919 2.25457052 2.16553696
2.7899453 2.71281409 2.63374631 2.55280484 3.17117385 3.10307631
3.03303359 2.96104648 2.88712946 3.50394742 3.44395541 3.38216436
3.31853484 3.25303483 3.1856411 ]
MSE after scaling: 0.00
R2 score for scaled data: 0.99
</pre></div>
</div>
</div>
</div>
</div>
</div>
@@ -2460,55 +2865,6 @@ R^2(\hat{y}, \tilde{\hat{y}}) = 1 - \frac{\sum_{i=0}^{n - 1} (y_i - \tilde{y}_i)
\]</div>
<p>You can use the functionality included in scikit-learn. If you feel for it, you can use your own program and define functions which compute the above two functions.
Discuss the meaning of these results. Try also to vary the coefficient in front of the added stochastic noise term and discuss the quality of the fits.</p>
<p>!bsol
The code here is an example of where we define our own design matrix and fit parameters <span class="math notranslate nohighlight">\(\beta\)</span>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">import</span> <span class="nn">pandas</span> <span class="k">as</span> <span class="nn">pd</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">from</span> <span class="nn">sklearn.model_selection</span> <span class="kn">import</span> <span class="n">train_test_split</span>
<span class="k">def</span> <span class="nf">save_fig</span><span class="p">(</span><span class="n">fig_id</span><span class="p">):</span>
<span class="n">plt</span><span class="o">.</span><span class="n">savefig</span><span class="p">(</span><span class="n">image_path</span><span class="p">(</span><span class="n">fig_id</span><span class="p">)</span> <span class="o">+</span> <span class="s2">&quot;.png&quot;</span><span class="p">,</span> <span class="nb">format</span><span class="o">=</span><span class="s1">&#39;png&#39;</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">R2</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span> <span class="n">y_model</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">1</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">y_model</span><span class="p">)</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span> <span class="o">/</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span> <span class="o">-</span> <span class="n">np</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="n">y_data</span><span class="p">))</span> <span class="o">**</span> <span class="mi">2</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">MSE</span><span class="p">(</span><span class="n">y_data</span><span class="p">,</span><span class="n">y_model</span><span class="p">):</span>
<span class="n">n</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">size</span><span class="p">(</span><span class="n">y_model</span><span class="p">)</span>
<span class="k">return</span> <span class="n">np</span><span class="o">.</span><span class="n">sum</span><span class="p">((</span><span class="n">y_data</span><span class="o">-</span><span class="n">y_model</span><span class="p">)</span><span class="o">**</span><span class="mi">2</span><span class="p">)</span><span class="o">/</span><span class="n">n</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="n">y</span> <span class="o">=</span> <span class="mf">2.0</span><span class="o">+</span><span class="mi">5</span><span class="o">*</span><span class="n">x</span><span class="o">*</span><span class="n">x</span><span class="o">+</span><span class="mf">0.1</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span>
<span class="c1"># The design matrix now as function of a given polynomial</span>
<span class="n">X</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">zeros</span><span class="p">((</span><span class="nb">len</span><span class="p">(</span><span class="n">x</span><span class="p">),</span><span class="mi">3</span><span class="p">))</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="mf">1.0</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">1</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span>
<span class="n">X</span><span class="p">[:,</span><span class="mi">2</span><span class="p">]</span> <span class="o">=</span> <span class="n">x</span><span class="o">**</span><span class="mi">2</span>
<span class="c1"># We split the data in test and training data</span>
<span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="c1"># matrix inversion to find beta</span>
<span class="n">beta</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">linalg</span><span class="o">.</span><span class="n">inv</span><span class="p">(</span><span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">X_train</span><span class="p">)</span> <span class="o">@</span> <span class="n">X_train</span><span class="o">.</span><span class="n">T</span> <span class="o">@</span> <span class="n">y_train</span>
<span class="nb">print</span><span class="p">(</span><span class="n">beta</span><span class="p">)</span>
<span class="c1"># and then make the prediction</span>
<span class="n">ytilde</span> <span class="o">=</span> <span class="n">X_train</span> <span class="o">@</span> <span class="n">beta</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Training R2&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Training MSE&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_train</span><span class="p">,</span><span class="n">ytilde</span><span class="p">))</span>
<span class="n">ypredict</span> <span class="o">=</span> <span class="n">X_test</span> <span class="o">@</span> <span class="n">beta</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test R2&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">R2</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Test MSE&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">MSE</span><span class="p">(</span><span class="n">y_test</span><span class="p">,</span><span class="n">ypredict</span><span class="p">))</span>
</pre></div>
</div>
</div>
</div>
<p>!esol</p>
</div>
<div class="section" id="exercise-normalizing-our-data">
<h3><span class="section-number">3.9.3. </span>Exercise: Normalizing our data<a class="headerlink" href="#exercise-normalizing-our-data" title="Permalink to this headline"></a></h3>
@@ -2549,6 +2905,38 @@ for testing. This can be done as follows with our design matrix <span class="mat
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">ValueError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="o">&lt;</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">38</span><span class="o">-</span><span class="mi">9</span><span class="n">b9cf4fa1a95</span><span class="o">&gt;</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span>
<span class="g g-Whitespace"> </span><span class="mi">1</span> <span class="c1"># split in training and test data</span>
<span class="ne">----&gt; </span><span class="mi">2</span> <span class="n">X_train</span><span class="p">,</span> <span class="n">X_test</span><span class="p">,</span> <span class="n">y_train</span><span class="p">,</span> <span class="n">y_test</span> <span class="o">=</span> <span class="n">train_test_split</span><span class="p">(</span><span class="n">X</span><span class="p">,</span><span class="n">y</span><span class="p">,</span><span class="n">test_size</span><span class="o">=</span><span class="mf">0.2</span><span class="p">)</span>
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/sklearn/model_selection/_split.py</span> in <span class="ni">train_test_split</span><span class="nt">(*arrays, **options)</span>
<span class="g g-Whitespace"> </span><span class="mi">2125</span> <span class="k">raise</span> <span class="ne">TypeError</span><span class="p">(</span><span class="s2">&quot;Invalid parameters passed: </span><span class="si">%s</span><span class="s2">&quot;</span> <span class="o">%</span> <span class="nb">str</span><span class="p">(</span><span class="n">options</span><span class="p">))</span>
<span class="g g-Whitespace"> </span><span class="mi">2126</span>
<span class="ne">-&gt; </span><span class="mi">2127</span> <span class="n">arrays</span> <span class="o">=</span> <span class="n">indexable</span><span class="p">(</span><span class="o">*</span><span class="n">arrays</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">2128</span>
<span class="g g-Whitespace"> </span><span class="mi">2129</span> <span class="n">n_samples</span> <span class="o">=</span> <span class="n">_num_samples</span><span class="p">(</span><span class="n">arrays</span><span class="p">[</span><span class="mi">0</span><span class="p">])</span>
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py</span> in <span class="ni">indexable</span><span class="nt">(*iterables)</span>
<span class="g g-Whitespace"> </span><span class="mi">290</span> <span class="s2">&quot;&quot;&quot;</span>
<span class="g g-Whitespace"> </span><span class="mi">291</span><span class="s2"> result = [_make_indexable(X) for X in iterables]</span>
<span class="ne">--&gt; </span><span class="mi">292</span><span class="s2"> check_consistent_length(*result)</span>
<span class="g g-Whitespace"> </span><span class="mi">293</span><span class="s2"> return result</span>
<span class="g g-Whitespace"> </span><span class="mi">294</span><span class="s2"> </span>
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/sklearn/utils/validation.py</span> in <span class="ni">check_consistent_length</span><span class="nt">(*arrays)</span>
<span class="g g-Whitespace"> </span><span class="mi">253</span><span class="s2"> uniques = np.unique(lengths)</span>
<span class="g g-Whitespace"> </span><span class="mi">254</span><span class="s2"> if len(uniques) &gt; 1:</span>
<span class="ne">--&gt; </span><span class="mi">255</span><span class="s2"> raise ValueError(&quot;Found input variables with inconsistent numbers of&quot;</span>
<span class="g g-Whitespace"> </span><span class="mi">256</span><span class="s2"> &quot; samples: </span><span class="si">%r</span><span class="s2">&quot; % [int(l) for l in lengths])</span>
<span class="g g-Whitespace"> </span><span class="mi">257</span><span class="s2"> </span>
<span class="ne">ValueError</span>: Found input variables with inconsistent numbers of samples: [1000, 100]
</pre></div>
</div>
</div>
</div>
<p>Then we can use the standard scaler to scale our data as</p>
<div class="cell docutils container">
@@ -2587,11 +2975,8 @@ Write a first code which sets up a design matrix <span class="math notranslate n
Perform an ordinary least squares and compute the means squared error and the <span class="math notranslate nohighlight">\(R2\)</span> factor for the training data and the test data, with and without scaling.
!esubex</p>
<p>!bsubex
Add now a model which allows you to make polynomials up to degree <span class="math notranslate nohighlight">\(15\)</span>. Perform a standard OLS fitting of the training data and compute the MSE and <span class="math notranslate nohighlight">\(R2\)</span> for the training and test data and plot both test and training data MSE and <span class="math notranslate nohighlight">\(R2\)</span> as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?
!bsol
Here you simply need to change the degree of the polynomial in the above code to <span class="math notranslate nohighlight">\(n=15\)</span>.
!esol
!esubex</p>
Add now a model which allows you to make polynomials up to degree <span class="math notranslate nohighlight">\(15\)</span>. Perform a standard OLS fitting of the training data and compute the MSE and <span class="math notranslate nohighlight">\(R2\)</span> for the training and test data and plot both test and training data MSE and <span class="math notranslate nohighlight">\(R2\)</span> as functions of the polynomial degree. Compare what you see with Figure 2.11 of Hastie et al. Comment your results. For which polynomial degree do you find an optimal MSE (smallest value)?</p>
<p>!esubex</p>
</div>
</div>
</div>