lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2023-10-15 21:55:49 +02:00
parent cb5ab447ad
commit edf33b439c
171 changed files with 18722 additions and 1691 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter3.html
+154 -26
View File
@@ -313,6 +313,26 @@ const thebe_selector_output = ".output, .cell_output"
Week 40: Gradient descent methods (continued) and start Neural networks
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek41.html">
Exercises week 41
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week41.html">
Week 41 Neural networks and constructing a neural network code
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="exercisesweek42.html">
Exercises week 42
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="week42.html">
Week 42 Constructing a Neural Network code with introduction to Tensor flow
</a>
</li>
</ul>
<p aria-level="2" class="caption" role="heading">
<span class="caption-text">
@@ -325,6 +345,11 @@ const thebe_selector_output = ".output, .cell_output"
Project 1 on Machine Learning, deadline October 9 (midnight), 2023
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="project2.html">
Project 2 on Machine Learning, deadline November 13 (Midnight)
</a>
</li>
</ul>
</div>
@@ -844,10 +869,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.179404 sec
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.136236 sec
Jackknife Statistics :
original bias std. error
100.039 100.029 0.150726
99.979 99.969 0.14845
</pre></div>
</div>
</div>
@@ -1066,7 +1091,7 @@ theorem.</p>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Bootstrap Statistics :
original bias std. error
100.092 14.9578 100.093 0.149299
99.8342 14.8306 99.8351 0.14857
</pre></div>
</div>
</div>
@@ -1288,14 +1313,14 @@ Error: 0.06844519414009445
Bias^2: 0.06453579006728322
Var: 0.003909404072811221
0.06844519414009445 &gt;= 0.06453579006728322 + 0.003909404072811221 = 0.06844519414009444
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 5
Polynomial degree: 5
Error: 0.05227921801205679
Bias^2: 0.04818727730430286
Var: 0.004091940707753925
0.05227921801205679 &gt;= 0.04818727730430286 + 0.004091940707753925 = 0.05227921801205679
Polynomial degree: 6
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 6
Error: 0.03781367141738902
Bias^2: 0.03365768507152769
Var: 0.0041559863458613296
@@ -1569,12 +1594,12 @@ Mean squared error on test data: 5.98822371
Degree of polynomial: 6
Mean squared error on training data: 3.66204648
Mean squared error on test data: 8.14812206
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 7
Degree of polynomial: 7
Mean squared error on training data: 0.47075725
Mean squared error on test data: 2.00607783
Degree of polynomial: 8
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 8
Mean squared error on training data: 0.04912436
Mean squared error on test data: 0.21596432
Degree of polynomial: 9
@@ -1589,15 +1614,15 @@ Mean squared error on test data: 1.35533773
Degree of polynomial: 12
Mean squared error on training data: 0.00813803
Mean squared error on test data: 0.17446471
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 13
Degree of polynomial: 13
Mean squared error on training data: 0.00759119
Mean squared error on test data: 1.08131003
Degree of polynomial: 14
Mean squared error on training data: 0.00472199
Mean squared error on test data: 0.81333804
Degree of polynomial: 15
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 15
Mean squared error on training data: 0.00410478
Mean squared error on test data: 92.09172409
Degree of polynomial: 16
@@ -1609,9 +1634,7 @@ Mean squared error on test data: 1271.35771826
Degree of polynomial: 18
Mean squared error on training data: 0.00228742
Mean squared error on test data: 108.27092910
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 19
Degree of polynomial: 19
Mean squared error on training data: 0.00156376
Mean squared error on test data: 1371.99051150
Degree of polynomial: 20
@@ -1620,7 +1643,9 @@ Mean squared error on test data: 1887.86252988
Degree of polynomial: 21
Mean squared error on training data: 0.00118508
Mean squared error on test data: 14859.69908626
Degree of polynomial: 22
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 22
Mean squared error on training data: 0.00092647
Mean squared error on test data: 876.51191552
Degree of polynomial: 23
@@ -1629,9 +1654,7 @@ Mean squared error on test data: 5594.60815105
Degree of polynomial: 24
Mean squared error on training data: 0.00084705
Mean squared error on test data: 1277.61702282
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 25
Degree of polynomial: 25
Mean squared error on training data: 0.00079129
Mean squared error on test data: 128664.31650694
Degree of polynomial: 26
@@ -1640,7 +1663,9 @@ Mean squared error on test data: 19003.94822514
Degree of polynomial: 27
Mean squared error on training data: 0.00068946
Mean squared error on test data: 2379.66219404
Degree of polynomial: 28
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 28
Mean squared error on training data: 0.00062595
Mean squared error on test data: 4082.19983530
Degree of polynomial: 29
@@ -1648,9 +1673,9 @@ Mean squared error on training data: 0.00060705
Mean squared error on test data: 3250.17647619
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19176/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(trainingerror), label=&#39;Training Error&#39;)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19176/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(testerror), label=&#39;Test Error&#39;)
</pre></div>
</div>
@@ -1884,7 +1909,7 @@ cross-validation (LOOCV).</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_19176/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(estimated_mse_sklearn), label=&#39;Test Error&#39;)
</pre></div>
</div>
@@ -2772,6 +2797,15 @@ linear system as an equation would reduce this down to
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/4162706317.py:6: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
cb = fig.colorbar(im)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/4162706317.py:7: UserWarning: FixedFormatter should only be used together with FixedLocator
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
</pre></div>
</div>
<img alt="_images/chapter3_154_1.png" src="_images/chapter3_154_1.png" />
</div>
</div>
<p>It is interesting to note that OLS
considers both <span class="math notranslate nohighlight">\(J_{j, j + 1} = -0.5\)</span> and <span class="math notranslate nohighlight">\(J_{j, j - 1} = -0.5\)</span> as
@@ -2909,6 +2943,15 @@ with the form utilized in linear regression, viz.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/3777801602.py:6: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
cb = fig.colorbar(im)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/3777801602.py:7: UserWarning: FixedFormatter should only be used together with FixedLocator
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
</pre></div>
</div>
<img alt="_images/chapter3_172_1.png" src="_images/chapter3_172_1.png" />
</div>
</div>
<p>The results agree perfectly with our previous discussion where we used our own code.</p>
<p>Having explored the ordinary least squares we move on to ridge
@@ -2942,6 +2985,15 @@ cost function is given by</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/438060758.py:9: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
cb = fig.colorbar(im)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/438060758.py:10: UserWarning: FixedFormatter should only be used together with FixedLocator
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
</pre></div>
</div>
<img alt="_images/chapter3_175_1.png" src="_images/chapter3_175_1.png" />
</div>
</div>
<p>In the <strong>Least Absolute Shrinkage and Selection Operator</strong> (LASSO)-method we get a third cost function.</p>
<!-- Equation labels as ordinary links -->
@@ -2970,6 +3022,15 @@ cost function is given by</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/3544313922.py:8: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
cb = fig.colorbar(im)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/3544313922.py:9: UserWarning: FixedFormatter should only be used together with FixedLocator
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
</pre></div>
</div>
<img alt="_images/chapter3_179_1.png" src="_images/chapter3_179_1.png" />
</div>
</div>
<p>It is quite striking how LASSO breaks the symmetry of the coupling
constant as opposed to ridge and OLS. We get a sparse solution with
@@ -3016,6 +3077,51 @@ constant as opposed to ridge and OLS. We get a sparse solution with
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0%| | 0/10 [00:00&lt;?, ?it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_coordinate_descent.py:647: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.924e+00, tolerance: 1.797e+00
model = cd_fast.enet_coordinate_descent(
10%|███████████▍ | 1/10 [00:00&lt;00:08, 1.06it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 20%|██████████████████████▊ | 2/10 [00:01&lt;00:07, 1.04it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 30%|██████████████████████████████████▏ | 3/10 [00:02&lt;00:05, 1.25it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 40%|█████████████████████████████████████████████▌ | 4/10 [00:03&lt;00:04, 1.36it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 50%|█████████████████████████████████████████████████████████ | 5/10 [00:03&lt;00:03, 1.43it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 60%|████████████████████████████████████████████████████████████████████▍ | 6/10 [00:04&lt;00:02, 1.50it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 70%|███████████████████████████████████████████████████████████████████████████████▊ | 7/10 [00:05&lt;00:01, 1.55it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 80%|███████████████████████████████████████████████████████████████████████████████████████████▏ | 8/10 [00:05&lt;00:01, 1.60it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 9/10 [00:06&lt;00:00, 1.61it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:06&lt;00:00, 1.62it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:06&lt;00:00, 1.47it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
</pre></div>
</div>
<img alt="_images/chapter3_181_13.png" src="_images/chapter3_181_13.png" />
</div>
</div>
<p>We see that LASSO reaches a good solution for low
values of <span class="math notranslate nohighlight">\(\lambda\)</span>, but will “wither” when we increase <span class="math notranslate nohighlight">\(\lambda\)</span> too
@@ -3062,6 +3168,9 @@ testing set that is close to the accuracy of the training set.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter3_183_0.png" src="_images/chapter3_183_0.png" />
</div>
</div>
<p>From the above figure we can see that LASSO with <span class="math notranslate nohighlight">\(\lambda = 10^{-2}\)</span>
achieves a very good accuracy on the test set. This by far surpasses the
@@ -3151,6 +3260,15 @@ which polynomial fits the data best.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/3980313467.py:9: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
ax = fig.gca(projection=&#39;3d&#39;)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_31624/3980313467.py:37: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
fig.colorbar(surf, shrink=0.5, aspect=5)
</pre></div>
</div>
<img alt="_images/chapter3_188_1.png" src="_images/chapter3_188_1.png" />
</div>
</div>
<div class="section" id="exercise-ordinary-least-square-ols-on-the-franke-function">
<h3><span class="section-number">5.8.1. </span>Exercise: Ordinary Least Square (OLS) on the Franke function<a class="headerlink" href="#exercise-ordinary-least-square-ols-on-the-franke-function" title="Permalink to this headline"></a></h3>
@@ -3303,6 +3421,16 @@ Python program using</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">NameError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="nn">Input In [31],</span> in <span class="ni">&lt;cell line: 1&gt;</span><span class="nt">()</span>
<span class="ne">----&gt; </span><span class="mi">1</span> <span class="n">scipy</span><span class="o">.</span><span class="n">misc</span><span class="o">.</span><span class="n">imread</span>
<span class="ne">NameError</span>: name &#39;scipy&#39; is not defined
</pre></div>
</div>
</div>
</div>
<p>Here is a simple part of a Python code which reads and plots the data
from such files</p>