lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update of notes

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2022-12-11 22:07:05 +01:00
parent 8051880d95
commit 5473efe28f
70 changed files with 3777 additions and 1484 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter3.html
+80 -88
View File
@@ -499,7 +499,8 @@ const thebe_selector_output = ".output, .cell_output"
<div>
<div class="tex2jax_ignore mathjax_ignore section" id="resampling-methods">
<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)
doconce format html chapter3.do.txt --><div class="tex2jax_ignore mathjax_ignore section" id="resampling-methods">
<h1><span class="section-number">5. </span>Resampling Methods<a class="headerlink" href="#resampling-methods" title="Permalink to this headline"></a></h1>
<div class="section" id="introduction">
<h2><span class="section-number">5.1. </span>Introduction<a class="headerlink" href="#introduction" title="Permalink to this headline"></a></h2>
@@ -625,9 +626,8 @@ mean value <span class="math notranslate nohighlight">\(\boldsymbol{X}\boldsymbo
\[
\mathbb{E}(\boldsymbol{\beta}) = \mathbb{E}[ (\mathbf{X}^{\top} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1}\mathbf{X}^{T} \mathbb{E}[ \mathbf{Y}]=(\mathbf{X}^{T} \mathbf{X})^{-1} \mathbf{X}^{T}\mathbf{X}\boldsymbol{\beta}=\boldsymbol{\beta}.
\]</div>
<p>This means that the estimator of the regression parameters is unbiased.
v
We can also calculate the variance</p>
<p>This means that the estimator of the regression parameters is unbiased.</p>
<p>We can also calculate the variance</p>
<p>The variance of <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span> is</p>
<div class="math notranslate nohighlight">
\[\begin{split}
@@ -700,7 +700,7 @@ we discuss the</p>
<li><p>prediction error or simply the <strong>test error</strong> <span class="math notranslate nohighlight">\(\mathrm{Err_{Test}}\)</span>, where we have a fixed training set and the test error is the MSE arising from the data reserved for testing. We discuss also the</p></li>
<li><p>training error <span class="math notranslate nohighlight">\(\mathrm{Err_{Train}}\)</span>, which is the average loss over the training data.</p></li>
</ol>
<p>As our model becomes more and more complex, more of the training data tends to used. The training may thence adapt to more complicated structures in the data. This may lead to a decrease in the bias (see below for code example) and a slight increase of the variance for the test error.
<p>As our model becomes more and more complex, more of the training data tends to be used. The training may thence adapt to more complicated structures in the data. This may lead to a decrease in the bias (see below for code example) and a slight increase of the variance for the test error.
For a certain level of complexity the test error will reach minimum, before starting to increase again. The
training error reaches a saturation.</p>
<p>Two famous
@@ -760,10 +760,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.0903549 sec
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.0893679 sec
Jackknife Statistics :
original bias std. error
100.107 100.097 0.150184
99.9524 99.9424 0.148854
</pre></div>
</div>
</div>
@@ -828,7 +828,7 @@ estimate any statistics thereof. In principle you never draw the
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>. Instead
you use the estimators corresponding to the statistic of interest. For
example, if you are interested in estimating the variance of <span class="math notranslate nohighlight">\(\widehat
\beta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
\beta\)</span>, apply the estimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
<span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>.</p>
<p>Before we proceed however, we need to remind ourselves about a central
theorem in statistics, namely the so-called <strong>central limit theorem</strong>.
@@ -982,7 +982,7 @@ theorem.</p>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Bootstrap Statistics :
original bias std. error
99.9033 14.9678 99.904 0.151348
100.188 15.1133 100.19 0.149655
</pre></div>
</div>
</div>
@@ -1004,7 +1004,7 @@ original bias std. error
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter3_47_0.png" src="_images/chapter3_47_0.png" />
<img alt="_images/chapter3_51_0.png" src="_images/chapter3_51_0.png" />
</div>
</div>
</div>
@@ -1036,12 +1036,12 @@ C(\boldsymbol{X},\boldsymbol{\beta}) =\frac{1}{n}\sum_{i=0}^{n-1}(y_i-\tilde{y}_
\[
\mathbb{E}\left[(\boldsymbol{y}-\boldsymbol{\tilde{y}})^2\right]=\frac{1}{n}\sum_i(f_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\frac{1}{n}\sum_i(\tilde{y}_i-\mathbb{E}\left[\boldsymbol{\tilde{y}}\right])^2+\sigma^2.
\]</div>
<p>The three terms represent the square of the bias of the learning
<p>The first term represents the square of the bias of the learning
method, which can be thought of as the error caused by the simplifying
assumptions built into the method. The second term represents the
variance of the chosen model and finally the last terms is variance of
the error <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span>.</p>
<p>To derive this equation, we need to recall that the variance of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> are both equal to <span class="math notranslate nohighlight">\(\sigma^2\)</span>. The mean value of <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> is by definition equal to zero. Furthermore, the function <span class="math notranslate nohighlight">\(f\)</span> is not a stochastics variable, idem for <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span>.
<p>To derive this equation, we need to recall that the variance of <span class="math notranslate nohighlight">\(\boldsymbol{y}\)</span> and <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> are both equal to <span class="math notranslate nohighlight">\(\sigma^2\)</span>. The mean value of <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span> is by definition equal to zero. Furthermore, the function <span class="math notranslate nohighlight">\(f\)</span> is not a stochastic variable, idem for <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span>.
We use a more compact notation in terms of the expectation value</p>
<div class="math notranslate nohighlight">
\[
@@ -1124,7 +1124,7 @@ Var: 0.0010479245926411787
0.013121574062587286 &gt;= 0.012073649469946107 + 0.0010479245926411787 = 0.013121574062587286
</pre></div>
</div>
<img alt="_images/chapter3_61_1.png" src="_images/chapter3_61_1.png" />
<img alt="_images/chapter3_65_1.png" src="_images/chapter3_65_1.png" />
</div>
</div>
<div class="cell docutils container">
@@ -1216,7 +1216,9 @@ Error: 0.03781367141738902
Bias^2: 0.03365768507152769
Var: 0.0041559863458613296
0.03781367141738902 &gt;= 0.03365768507152769 + 0.0041559863458613296 = 0.03781367141738902
Polynomial degree: 7
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 7
Error: 0.027609773491022394
Bias^2: 0.022999498260366198
Var: 0.004610275230656182
@@ -1253,7 +1255,7 @@ Var: 0.20867052175003364
0.2284246870217162 &gt;= 0.01975416527168255 + 0.20867052175003364 = 0.2284246870217162
</pre></div>
</div>
<img alt="_images/chapter3_62_2.png" src="_images/chapter3_62_2.png" />
<img alt="_images/chapter3_66_3.png" src="_images/chapter3_66_3.png" />
</div>
</div>
<p>The bias-variance tradeoff summarizes the fundamental tension in
@@ -1378,7 +1380,7 @@ set, the higher, the less likely the model generalizes correctly from the
training data.
</pre></div>
</div>
<img alt="_images/chapter3_64_1.png" src="_images/chapter3_64_1.png" />
<img alt="_images/chapter3_68_1.png" src="_images/chapter3_68_1.png" />
</div>
</div>
<div class="cell docutils container">
@@ -1466,9 +1468,7 @@ training data.
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 1
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Mean squared error on training data: 439230.69504801
Mean squared error on training data: 439230.69504801
Mean squared error on test data: 481979.17861098
Degree of polynomial: 2
Mean squared error on training data: 115822.95008046
@@ -1508,12 +1508,12 @@ Mean squared error on test data: 0.17446471
Degree of polynomial: 13
Mean squared error on training data: 0.00759119
Mean squared error on test data: 1.08131003
Degree of polynomial: 14
Mean squared error on training data: 0.00472199
Mean squared error on test data: 0.81333804
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 15
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 14
Mean squared error on training data: 0.00472199
Mean squared error on test data: 0.81333804
Degree of polynomial: 15
Mean squared error on training data: 0.00410478
Mean squared error on test data: 92.09172409
Degree of polynomial: 16
@@ -1528,12 +1528,12 @@ Mean squared error on test data: 108.27092910
Degree of polynomial: 19
Mean squared error on training data: 0.00156376
Mean squared error on test data: 1371.99051150
Degree of polynomial: 20
Mean squared error on training data: 0.00137818
Mean squared error on test data: 1887.86252988
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 21
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 20
Mean squared error on training data: 0.00137818
Mean squared error on test data: 1887.86252988
Degree of polynomial: 21
Mean squared error on training data: 0.00118508
Mean squared error on test data: 14859.69908626
Degree of polynomial: 22
@@ -1548,12 +1548,12 @@ Mean squared error on test data: 1277.61702282
Degree of polynomial: 25
Mean squared error on training data: 0.00079129
Mean squared error on test data: 128664.31650694
Degree of polynomial: 26
Mean squared error on training data: 0.00076905
Mean squared error on test data: 19003.94822514
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 27
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 26
Mean squared error on training data: 0.00076905
Mean squared error on test data: 19003.94822514
Degree of polynomial: 27
Mean squared error on training data: 0.00068946
Mean squared error on test data: 2379.66219404
Degree of polynomial: 28
@@ -1564,13 +1564,13 @@ Mean squared error on training data: 0.00060705
Mean squared error on test data: 3250.17647619
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/626635268.py:73: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(trainingerror), label=&#39;Training Error&#39;)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/626635268.py:74: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(testerror), label=&#39;Test Error&#39;)
</pre></div>
</div>
<img alt="_images/chapter3_65_7.png" src="_images/chapter3_65_7.png" />
<img alt="_images/chapter3_69_6.png" src="_images/chapter3_69_6.png" />
</div>
</div>
</div>
@@ -1723,7 +1723,7 @@ cross-validation (LOOCV).</p>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter3_71_0.png" src="_images/chapter3_71_0.png" />
<img alt="_images/chapter3_75_0.png" src="_images/chapter3_75_0.png" />
</div>
</div>
<p>More examples of the application of cross-validation follow here.</p>
@@ -1800,14 +1800,14 @@ cross-validation (LOOCV).</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/3817475779.py:63: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(estimated_mse_sklearn), label=&#39;Test Error&#39;)
</pre></div>
</div>
<img alt="_images/chapter3_73_1.png" src="_images/chapter3_73_1.png" />
<img alt="_images/chapter3_77_1.png" src="_images/chapter3_77_1.png" />
</div>
</div>
<p>Note that we have kept the intercept in the first column of design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. When we call the corresponding <strong>Scikit-Learn</strong> function we need thus to set the intercept to <strong>False</strong>. Libraries like <strong>Scikit-Learn</strong> normally scale the design matrix and does not fit intercept. See the discussions below.</p>
<p>Note that we have kept the intercept in the first column of design matrix <span class="math notranslate nohighlight">\(\boldsymbol{X}\)</span>. When we call the corresponding <strong>Scikit-Learn</strong> function we need thus to set the intercept to <strong>False</strong>. Libraries like <strong>Scikit-Learn</strong> normally scale the design matrix and do not fit intercept. See the discussions below.</p>
</div>
<div class="section" id="more-on-rescaling-data">
<h2><span class="section-number">5.6. </span>More on Rescaling data<a class="headerlink" href="#more-on-rescaling-data" title="Permalink to this headline"></a></h2>
@@ -2077,7 +2077,7 @@ MSE with Sklearn intercept
0.004113634617443131
</pre></div>
</div>
<img alt="_images/chapter3_107_1.png" src="_images/chapter3_107_1.png" />
<img alt="_images/chapter3_112_1.png" src="_images/chapter3_112_1.png" />
</div>
</div>
<p>The intercept is the value of our output/target variable
@@ -2088,7 +2088,7 @@ the way we treat the intercept may give a larger or smaller MSE,
meaning that the MSE can be penalized by the value of the
intercept. Not including the intercept in the fit, means that the
regularization term does not include <span class="math notranslate nohighlight">\(\beta_0\)</span>. For different values
of <span class="math notranslate nohighlight">\(\lambda\)</span>, this may lead to differeing MSE values.</p>
of <span class="math notranslate nohighlight">\(\lambda\)</span>, this may lead to different MSE values.</p>
<p>To remind the reader, the regularization term, with the intercept in Ridge regression, is given by</p>
<div class="math notranslate nohighlight">
\[
@@ -2272,7 +2272,7 @@ MSE values for Scikit-Learn Ridge implementation
0.26409315307910025
</pre></div>
</div>
<img alt="_images/chapter3_115_1.png" src="_images/chapter3_115_1.png" />
<img alt="_images/chapter3_120_1.png" src="_images/chapter3_120_1.png" />
</div>
</div>
<p>The results here agree when we force <strong>Scikit-Learn</strong>’s Ridge function to include the first column in our design matrix.
@@ -2475,7 +2475,7 @@ MSE values for Scikit-Learn Ridge implementation
0.002381316302584886
</pre></div>
</div>
<img alt="_images/chapter3_117_1.png" src="_images/chapter3_117_1.png" />
<img alt="_images/chapter3_122_1.png" src="_images/chapter3_122_1.png" />
</div>
</div>
<p>We see here, when compared to the code which includes explicitely the
@@ -2689,13 +2689,13 @@ linear system as an equation would reduce this down to
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/4162706317.py:6: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/4162706317.py:6: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
cb = fig.colorbar(im)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/4162706317.py:7: UserWarning: FixedFormatter should only be used together with FixedLocator
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/4162706317.py:7: UserWarning: FixedFormatter should only be used together with FixedLocator
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
</pre></div>
</div>
<img alt="_images/chapter3_148_1.png" src="_images/chapter3_148_1.png" />
<img alt="_images/chapter3_154_1.png" src="_images/chapter3_154_1.png" />
</div>
</div>
<p>It is interesting to note that OLS
@@ -2835,13 +2835,13 @@ with the form utilized in linear regression, viz.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/3777801602.py:6: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/3777801602.py:6: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
cb = fig.colorbar(im)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/3777801602.py:7: UserWarning: FixedFormatter should only be used together with FixedLocator
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/3777801602.py:7: UserWarning: FixedFormatter should only be used together with FixedLocator
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
</pre></div>
</div>
<img alt="_images/chapter3_166_1.png" src="_images/chapter3_166_1.png" />
<img alt="_images/chapter3_172_1.png" src="_images/chapter3_172_1.png" />
</div>
</div>
<p>The results agree perfectly with our previous discussion where we used our own code.</p>
@@ -2850,23 +2850,15 @@ regression. In ridge regression we include a <strong>regularizer</strong>. This
involves a new cost function which leads to a new estimate for the
weights <span class="math notranslate nohighlight">\(\boldsymbol{\beta}\)</span>. This results in a penalized regression problem. The
cost function is given by</p>
<p>6
0</p>
<p>&lt;
&lt;
&lt;
!
!
M
A
T
H
_
B
L
O
C
K</p>
<!-- Equation labels as ordinary links -->
<div id="_auto11"></div>
<div class="math notranslate nohighlight">
\[
\begin{equation}
C(\boldsymbol{X}, \boldsymbol{\beta}; \lambda) = (\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y})^T(\boldsymbol{X}\boldsymbol{\beta} - \boldsymbol{y}) + \lambda \boldsymbol{\beta}^T\boldsymbol{\beta}.
\label{_auto11} \tag{11}
\end{equation}
\]</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="n">_lambda</span> <span class="o">=</span> <span class="mf">0.1</span>
@@ -2885,13 +2877,13 @@ K</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/438060758.py:9: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/438060758.py:9: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
cb = fig.colorbar(im)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/438060758.py:10: UserWarning: FixedFormatter should only be used together with FixedLocator
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/438060758.py:10: UserWarning: FixedFormatter should only be used together with FixedLocator
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
</pre></div>
</div>
<img alt="_images/chapter3_169_1.png" src="_images/chapter3_169_1.png" />
<img alt="_images/chapter3_175_1.png" src="_images/chapter3_175_1.png" />
</div>
</div>
<p>In the <strong>Least Absolute Shrinkage and Selection Operator</strong> (LASSO)-method we get a third cost function.</p>
@@ -2922,13 +2914,13 @@ K</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/3544313922.py:8: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/3544313922.py:8: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
cb = fig.colorbar(im)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/3544313922.py:9: UserWarning: FixedFormatter should only be used together with FixedLocator
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/3544313922.py:9: UserWarning: FixedFormatter should only be used together with FixedLocator
cb.ax.set_yticklabels(cb.ax.get_yticklabels(), fontsize=18)
</pre></div>
</div>
<img alt="_images/chapter3_173_1.png" src="_images/chapter3_173_1.png" />
<img alt="_images/chapter3_179_1.png" src="_images/chapter3_179_1.png" />
</div>
</div>
<p>It is quite striking how LASSO breaks the symmetry of the coupling
@@ -2977,49 +2969,49 @@ constant as opposed to ridge and OLS. We get a sparse solution with
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0%| | 0/10 [00:00&lt;?, ?it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 0%| | 0/10 [00:00&lt;?, ?it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/linear_model/_coordinate_descent.py:647: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.924e+00, tolerance: 1.797e+00
model = cd_fast.enet_coordinate_descent(
10%|███████████▊ | 1/10 [00:00&lt;00:06, 1.32it/s]
10%|███████████▉ | 1/10 [00:00&lt;00:06, 1.50it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 20%|███████████████████████▌ | 2/10 [00:01&lt;00:05, 1.39it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 20%|███████████████████████▊ | 2/10 [00:01&lt;00:04, 1.83it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 30%|███████████████████████████████████▍ | 3/10 [00:01&lt;00:03, 1.83it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 30%|███████████████████████████████████▋ | 3/10 [00:01&lt;00:02, 2.70it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 40%|███████████████████████████████████████████████▏ | 4/10 [00:01&lt;00:02, 2.44it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 40%|███████████████████████████████████████████████▌ | 4/10 [00:01&lt;00:01, 3.15it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 50%|███████████████████████████████████████████████████████████ | 5/10 [00:02&lt;00:01, 3.02it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 50%|███████████████████████████████████████████████████████████▌ | 5/10 [00:01&lt;00:01, 3.60it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 60%|██████████████████████████████████████████████████████████████████████▊ | 6/10 [00:02&lt;00:01, 3.70it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 60%|███████████████████████████████████████████████████████████████████████▍ | 6/10 [00:01&lt;00:01, 3.91it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 70%|██████████████████████████████████████████████████████████████████████████████████▌ | 7/10 [00:02&lt;00:00, 4.31it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 70%|███████████████████████████████████████████████████████████████████████████████████▎ | 7/10 [00:02&lt;00:00, 4.65it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 80%|██████████████████████████████████████████████████████████████████████████████████████████████▍ | 8/10 [00:02&lt;00:00, 4.48it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 80%|███████████████████████████████████████████████████████████████████████████████████████████████▏ | 8/10 [00:02&lt;00:00, 5.13it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 90%|██████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 9/10 [00:02&lt;00:00, 4.92it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span> 90%|███████████████████████████████████████████████████████████████████████████████████████████████████████████ | 9/10 [00:02&lt;00:00, 5.60it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02&lt;00:00, 5.65it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02&lt;00:00, 5.96it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02&lt;00:00, 3.37it/s]
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:02&lt;00:00, 3.97it/s]
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>
</pre></div>
</div>
<img alt="_images/chapter3_175_13.png" src="_images/chapter3_175_13.png" />
<img alt="_images/chapter3_181_13.png" src="_images/chapter3_181_13.png" />
</div>
</div>
<p>We see that LASSO reaches a good solution for low
@@ -3068,7 +3060,7 @@ testing set that is close to the accuracy of the training set.</p>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter3_177_0.png" src="_images/chapter3_177_0.png" />
<img alt="_images/chapter3_183_0.png" src="_images/chapter3_183_0.png" />
</div>
</div>
<p>From the above figure we can see that LASSO with <span class="math notranslate nohighlight">\(\lambda = 10^{-2}\)</span>
@@ -3160,13 +3152,13 @@ which polynomial fits the data best.</p>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/3980313467.py:9: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/3980313467.py:9: MatplotlibDeprecationWarning: Calling gca() with keyword arguments was deprecated in Matplotlib 3.4. Starting two minor releases later, gca() will take no keyword arguments. The gca() function should only be used to get the current axes, or if no axes exist, create new axes with default keyword arguments. To create a new axes with non-default arguments, use plt.axes() or plt.subplot().
ax = fig.gca(projection=&#39;3d&#39;)
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_94529/3980313467.py:37: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
/var/folders/td/3yk470mj5p931p9dtkk0y6jw0000gn/T/ipykernel_41811/3980313467.py:37: MatplotlibDeprecationWarning: Auto-removal of grids by pcolor() and pcolormesh() is deprecated since 3.5 and will be removed two minor releases later; please call grid(False) first.
fig.colorbar(surf, shrink=0.5, aspect=5)
</pre></div>
</div>
<img alt="_images/chapter3_181_1.png" src="_images/chapter3_181_1.png" />
<img alt="_images/chapter3_188_1.png" src="_images/chapter3_188_1.png" />
</div>
</div>
<div class="section" id="exercise-ordinary-least-square-ols-on-the-franke-function">