lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on notes

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 15:21:30 +02:00
parent dd5df56326
commit bbb241c88f
60 changed files with 15229 additions and 14601 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/_build/html/chapter3.html
+272 -128
View File
@@ -157,6 +157,11 @@
6. Logistic Regression
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapteroptimization.html">
7. Optimization, the central part of any Machine Learning algortithm
</a>
</li>
<li class="toctree-l1">
<a class="reference internal" href="chapter5.html">
8. Support Vector Machines, overarching aims
@@ -542,7 +547,7 @@ satisfied, the methods will fail. Yet, it should be said that if the data are
independent, identically distributed, and we only want to estimate the
variance of <span class="math notranslate nohighlight">\(\overline{X}\)</span> (which often is the case), then there is no
need for bootstrapping.</p>
<p>The Jackknife works by making many replicas of the estimator <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span>.
<p>The Jackknife works by making many replicas of the estimator <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span>.
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values <span class="math notranslate nohighlight">\(\boldsymbol{x} = (x_1,x_2,\cdots,X_n)\)</span>.
Let <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> denote the vector</p>
<div class="math notranslate nohighlight">
@@ -551,8 +556,8 @@ Let <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> denot
\]</div>
<p>which equals the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> with the exception that observation
number <span class="math notranslate nohighlight">\(i\)</span> is left out. Using this notation, define
<span class="math notranslate nohighlight">\(\widehat{\theta}_i\)</span> to be the estimator
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> computed using <span class="math notranslate nohighlight">\(\vec{X}_i\)</span>.</p>
<span class="math notranslate nohighlight">\(\widehat{\beta}_i\)</span> to be the estimator
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> computed using <span class="math notranslate nohighlight">\(\vec{X}_i\)</span>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="o">*</span>
@@ -587,10 +592,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.138472 sec
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.1375 sec
Jackknife Statistics :
original bias std. error
100.094 100.084 0.150306
100.029 100.019 0.150581
</pre></div>
</div>
</div>
@@ -607,27 +612,27 @@ advantages:</p>
<li><p>It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.</p></li>
<li><p>It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).</p></li>
</ol>
<p>Since <span class="math notranslate nohighlight">\(\widehat{\theta} = \widehat{\theta}(\boldsymbol{X})\)</span> is a function of random variables,
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> itself must be a random variable. Thus it has
<p>Since <span class="math notranslate nohighlight">\(\widehat{\beta} = \widehat{\beta}(\boldsymbol{X})\)</span> is a function of random variables,
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> itself must be a random variable. Thus it has
a pdf, call this function <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. The aim of the bootstrap is to
estimate <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> by the relative frequency of
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span>. You can think of this as using a histogram
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span>. You can think of this as using a histogram
in the place of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. If the relative frequency closely
resembles <span class="math notranslate nohighlight">\(p(\vec{t})\)</span>, then using numerics, it is straight forward to
estimate all the interesting parameters of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> using point
estimators.</p>
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> has
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> has
more than one component, and the components are independent, we use the
same estimator on each component separately. If the probability
density function of <span class="math notranslate nohighlight">\(X_i\)</span>, <span class="math notranslate nohighlight">\(p(x)\)</span>, had been known, then it would have
been straight forward to do this by:</p>
<ol class="simple">
<li><p>Drawing lots of numbers from <span class="math notranslate nohighlight">\(p(x)\)</span>, suppose we call one such set of numbers <span class="math notranslate nohighlight">\((X_1^*, X_2^*, \cdots, X_n^*)\)</span>.</p></li>
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>.</p></li>
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>.</p></li>
</ol>
<p>By repeated use of (1) and (2), many
estimates of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> could have been obtained. The
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>
estimates of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> could have been obtained. The
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>
(think of a histogram) as an estimate of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>.</p>
<p>But
unless there is enough information available about the process that
@@ -645,128 +650,18 @@ frequency of the observation <span class="math notranslate nohighlight">\(X_i\)<
<ol class="simple">
<li><p>Draw with replacement <span class="math notranslate nohighlight">\(n\)</span> numbers for the observed variables <span class="math notranslate nohighlight">\(\boldsymbol{x} = (x_1,x_2,\cdots,x_n)\)</span>.</p></li>
<li><p>Define a vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> containing the values which were drawn from <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>.</p></li>
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \theta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \beta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
<li><p>Repeat this process <span class="math notranslate nohighlight">\(k\)</span> times.</p></li>
</ol>
<p>When you are done, you can draw a histogram of the relative frequency
of <span class="math notranslate nohighlight">\(\widehat \theta^*\)</span>. This is your estimate of the probability
of <span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>. This is your estimate of the probability
distribution <span class="math notranslate nohighlight">\(p(t)\)</span>. Using this probability distribution you can
estimate any statistics thereof. In principle you never draw the
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>. Instead
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>. Instead
you use the estimators corresponding to the statistic of interest. For
example, if you are interested in estimating the variance of <span class="math notranslate nohighlight">\(\widehat
\theta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
<span class="math notranslate nohighlight">\(\widehat \theta ^*\)</span>.</p>
<p>The following code starts with a Gaussian distribution with mean value
<span class="math notranslate nohighlight">\(\mu =100\)</span> and variance <span class="math notranslate nohighlight">\(\sigma=15\)</span>. We use this to generate the data
used in the bootstrap analysis. The bootstrap analysis returns a data
set after a given number of bootstrap operations (as many as we have
data points). This data set consists of estimated mean values for each
bootstrap operation. The histogram generated by the bootstrap method
shows that the distribution for these mean values is also a Gaussian,
centered around the mean value <span class="math notranslate nohighlight">\(\mu=100\)</span> but with standard deviation
<span class="math notranslate nohighlight">\(\sigma/\sqrt{n}\)</span>, where <span class="math notranslate nohighlight">\(n\)</span> is the number of bootstrap samples (in
this case the same as the number of original data points). The value
of the standard deviation is what we expect from the central limit
theorem.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
<span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="o">*</span>
<span class="kn">from</span> <span class="nn">numpy.random</span> <span class="kn">import</span> <span class="n">randint</span><span class="p">,</span> <span class="n">randn</span>
<span class="kn">from</span> <span class="nn">time</span> <span class="kn">import</span> <span class="n">time</span>
<span class="kn">import</span> <span class="nn">matplotlib.mlab</span> <span class="k">as</span> <span class="nn">mlab</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="c1"># Returns mean of bootstrap samples </span>
<span class="k">def</span> <span class="nf">stat</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="k">return</span> <span class="n">mean</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
<span class="c1"># Bootstrap algorithm</span>
<span class="k">def</span> <span class="nf">bootstrap</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">statistic</span><span class="p">,</span> <span class="n">R</span><span class="p">):</span>
<span class="n">t</span> <span class="o">=</span> <span class="n">zeros</span><span class="p">(</span><span class="n">R</span><span class="p">);</span> <span class="n">n</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">);</span> <span class="n">inds</span> <span class="o">=</span> <span class="n">arange</span><span class="p">(</span><span class="n">n</span><span class="p">);</span> <span class="n">t0</span> <span class="o">=</span> <span class="n">time</span><span class="p">()</span>
<span class="c1"># non-parametric bootstrap </span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">R</span><span class="p">):</span>
<span class="n">t</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">statistic</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="n">n</span><span class="p">,</span><span class="n">n</span><span class="p">)])</span>
<span class="c1"># analysis </span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Runtime: </span><span class="si">%g</span><span class="s2"> sec&quot;</span> <span class="o">%</span> <span class="p">(</span><span class="n">time</span><span class="p">()</span><span class="o">-</span><span class="n">t0</span><span class="p">));</span> <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Bootstrap Statistics :&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;original bias std. error&quot;</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="s2">&quot;</span><span class="si">%8g</span><span class="s2"> </span><span class="si">%8g</span><span class="s2"> </span><span class="si">%14g</span><span class="s2"> </span><span class="si">%15g</span><span class="s2">&quot;</span> <span class="o">%</span> <span class="p">(</span><span class="n">statistic</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="n">std</span><span class="p">(</span><span class="n">data</span><span class="p">),</span><span class="n">mean</span><span class="p">(</span><span class="n">t</span><span class="p">),</span><span class="n">std</span><span class="p">(</span><span class="n">t</span><span class="p">)))</span>
<span class="k">return</span> <span class="n">t</span>
<span class="n">mu</span><span class="p">,</span> <span class="n">sigma</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">15</span>
<span class="n">datapoints</span> <span class="o">=</span> <span class="mi">10000</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">mu</span> <span class="o">+</span> <span class="n">sigma</span><span class="o">*</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">datapoints</span><span class="p">)</span>
<span class="c1"># bootstrap returns the data sample </span>
<span class="n">t</span> <span class="o">=</span> <span class="n">bootstrap</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">stat</span><span class="p">,</span> <span class="n">datapoints</span><span class="p">)</span>
<span class="c1"># the histogram of the bootstrapped data </span>
<span class="n">n</span><span class="p">,</span> <span class="n">binsboot</span><span class="p">,</span> <span class="n">patches</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="n">normed</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">facecolor</span><span class="o">=</span><span class="s1">&#39;red&#39;</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.75</span><span class="p">)</span>
<span class="c1"># add a &#39;best fit&#39; line </span>
<span class="n">y</span> <span class="o">=</span> <span class="n">mlab</span><span class="o">.</span><span class="n">normpdf</span><span class="p">(</span> <span class="n">binsboot</span><span class="p">,</span> <span class="n">mean</span><span class="p">(</span><span class="n">t</span><span class="p">),</span> <span class="n">std</span><span class="p">(</span><span class="n">t</span><span class="p">))</span>
<span class="n">lt</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">binsboot</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="s1">&#39;r--&#39;</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">&#39;Smarts&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">&#39;Probability&#39;</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">([</span><span class="mf">99.5</span><span class="p">,</span> <span class="mf">100.6</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">])</span>
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="kc">True</span><span class="p">)</span>
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 2.06834 sec
Bootstrap Statistics :
original bias std. error
99.9933 15.0354 99.9941 0.149739
</pre></div>
</div>
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">AttributeError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
<span class="o">&lt;</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">2</span><span class="o">-</span><span class="mi">772</span><span class="n">b904ae9cb</span><span class="o">&gt;</span> <span class="ow">in</span> <span class="o">&lt;</span><span class="n">module</span><span class="o">&gt;</span>
<span class="g g-Whitespace"> </span><span class="mi">31</span> <span class="n">t</span> <span class="o">=</span> <span class="n">bootstrap</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">stat</span><span class="p">,</span> <span class="n">datapoints</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">32</span> <span class="c1"># the histogram of the bootstrapped data</span>
<span class="ne">---&gt; </span><span class="mi">33</span> <span class="n">n</span><span class="p">,</span> <span class="n">binsboot</span><span class="p">,</span> <span class="n">patches</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="n">normed</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">facecolor</span><span class="o">=</span><span class="s1">&#39;red&#39;</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.75</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">34</span>
<span class="g g-Whitespace"> </span><span class="mi">35</span> <span class="c1"># add a &#39;best fit&#39; line</span>
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/matplotlib/pyplot.py</span> in <span class="ni">hist</span><span class="nt">(x, bins, range, density, weights, cumulative, bottom, histtype, align, orientation, rwidth, log, color, label, stacked, data, **kwargs)</span>
<span class="g g-Whitespace"> </span><span class="mi">2683</span> <span class="n">orientation</span><span class="o">=</span><span class="s1">&#39;vertical&#39;</span><span class="p">,</span> <span class="n">rwidth</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">log</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">2684</span> <span class="n">label</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">stacked</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="ne">-&gt; </span><span class="mi">2685</span> <span class="k">return</span> <span class="n">gca</span><span class="p">()</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span>
<span class="g g-Whitespace"> </span><span class="mi">2686</span> <span class="n">x</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="n">bins</span><span class="p">,</span> <span class="nb">range</span><span class="o">=</span><span class="nb">range</span><span class="p">,</span> <span class="n">density</span><span class="o">=</span><span class="n">density</span><span class="p">,</span> <span class="n">weights</span><span class="o">=</span><span class="n">weights</span><span class="p">,</span>
<span class="g g-Whitespace"> </span><span class="mi">2687</span> <span class="n">cumulative</span><span class="o">=</span><span class="n">cumulative</span><span class="p">,</span> <span class="n">bottom</span><span class="o">=</span><span class="n">bottom</span><span class="p">,</span> <span class="n">histtype</span><span class="o">=</span><span class="n">histtype</span><span class="p">,</span>
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/matplotlib/__init__.py</span> in <span class="ni">inner</span><span class="nt">(ax, data, *args, **kwargs)</span>
<span class="g g-Whitespace"> </span><span class="mi">1445</span> <span class="k">def</span> <span class="nf">inner</span><span class="p">(</span><span class="n">ax</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="g g-Whitespace"> </span><span class="mi">1446</span> <span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
<span class="ne">-&gt; </span><span class="mi">1447</span> <span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="n">ax</span><span class="p">,</span> <span class="o">*</span><span class="nb">map</span><span class="p">(</span><span class="n">sanitize_sequence</span><span class="p">,</span> <span class="n">args</span><span class="p">),</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">1448</span>
<span class="g g-Whitespace"> </span><span class="mi">1449</span> <span class="n">bound</span> <span class="o">=</span> <span class="n">new_sig</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">ax</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/matplotlib/axes/_axes.py</span> in <span class="ni">hist</span><span class="nt">(self, x, bins, range, density, weights, cumulative, bottom, histtype, align, orientation, rwidth, log, color, label, stacked, **kwargs)</span>
<span class="g g-Whitespace"> </span><span class="mi">6813</span> <span class="k">if</span> <span class="n">patch</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">6814</span> <span class="n">p</span> <span class="o">=</span> <span class="n">patch</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="ne">-&gt; </span><span class="mi">6815</span> <span class="n">p</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">kwargs</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">6816</span> <span class="k">if</span> <span class="n">lbl</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
<span class="g g-Whitespace"> </span><span class="mi">6817</span> <span class="n">p</span><span class="o">.</span><span class="n">set_label</span><span class="p">(</span><span class="n">lbl</span><span class="p">)</span>
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/matplotlib/artist.py</span> in <span class="ni">update</span><span class="nt">(self, props)</span>
<span class="g g-Whitespace"> </span><span class="mi">994</span> <span class="n">func</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;set_</span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">995</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">callable</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
<span class="ne">--&gt; </span><span class="mi">996</span> <span class="k">raise</span> <span class="ne">AttributeError</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="si">!r}</span><span class="s2"> object &quot;</span>
<span class="g g-Whitespace"> </span><span class="mi">997</span> <span class="sa">f</span><span class="s2">&quot;has no property </span><span class="si">{</span><span class="n">k</span><span class="si">!r}</span><span class="s2">&quot;</span><span class="p">)</span>
<span class="g g-Whitespace"> </span><span class="mi">998</span> <span class="n">ret</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">func</span><span class="p">(</span><span class="n">v</span><span class="p">))</span>
<span class="ne">AttributeError</span>: &#39;Rectangle&#39; object has no property &#39;normed&#39;
</pre></div>
</div>
<img alt="_images/chapter3_25_2.png" src="_images/chapter3_25_2.png" />
</div>
</div>
\beta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
<span class="math notranslate nohighlight">\(\widehat \beta ^*\)</span>.</p>
</div>
</div>
<div class="section" id="the-bias-variance-tradeoff">
@@ -820,7 +715,9 @@ We use a more compact notation in terms of the expectation value</p>
<p>that is the rewriting in terms of the so-called bias, the variance of the model <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> and the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span>.</p>
<div class="cell docutils container">
<div class="cell_input docutils container">
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LinearRegression</span><span class="p">,</span> <span class="n">Ridge</span><span class="p">,</span> <span class="n">Lasso</span>
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">PolynomialFeatures</span>
@@ -877,6 +774,15 @@ We use a more compact notation in terms of the expectation value</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Error: 0.01312157412031145
Bias^2: 0.012073649480472317
Var: 0.0010479246398391328
0.01312157412031145 &gt;= 0.012073649480472317 + 0.0010479246398391328 = 0.01312157412031145
</pre></div>
</div>
<img alt="_images/chapter3_37_1.png" src="_images/chapter3_37_1.png" />
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -929,6 +835,91 @@ We use a more compact notation in terms of the expectation value</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 0
Error: 0.32149601703519126
Bias^2: 0.3123314713548606
Var: 0.009164545680330616
0.32149601703519126 &gt;= 0.3123314713548606 + 0.009164545680330616 = 0.3214960170351912
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 1
Error: 0.08426840630693411
Bias^2: 0.07968918676726028
Var: 0.004579219539673833
0.08426840630693411 &gt;= 0.07968918676726028 + 0.004579219539673833 = 0.08426840630693411
Polynomial degree: 2
Error: 0.10398646080125035
Bias^2: 0.10077114273548986
Var: 0.0032153180657605086
0.10398646080125035 &gt;= 0.10077114273548986 + 0.0032153180657605086 = 0.10398646080125036
Polynomial degree: 3
Error: 0.06547790180152352
Bias^2: 0.062082386342319454
Var: 0.0033955154592040936
0.06547790180152352 &gt;= 0.062082386342319454 + 0.0033955154592040936 = 0.06547790180152355
Polynomial degree: 4
Error: 0.06844519414009442
Bias^2: 0.06453579006728317
Var: 0.003909404072811237
0.06844519414009442 &gt;= 0.06453579006728317 + 0.003909404072811237 = 0.06844519414009441
Polynomial degree: 5
Error: 0.05227921801205707
Bias^2: 0.048187277304303125
Var: 0.004091940707753964
0.05227921801205707 &gt;= 0.048187277304303125 + 0.004091940707753964 = 0.05227921801205709
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 6
Error: 0.03781367141738898
Bias^2: 0.03365768507152761
Var: 0.004155986345861379
0.03781367141738898 &gt;= 0.03365768507152761 + 0.004155986345861379 = 0.03781367141738899
Polynomial degree: 7
Error: 0.027609773491022498
Bias^2: 0.02299949826036597
Var: 0.004610275230656537
0.027609773491022498 &gt;= 0.02299949826036597 + 0.004610275230656537 = 0.027609773491022505
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 8
Error: 0.017355848195591973
Bias^2: 0.010331721306655588
Var: 0.007024126888936384
0.017355848195591973 &gt;= 0.010331721306655588 + 0.007024126888936384 = 0.017355848195591973
Polynomial degree: 9
Error: 0.026605727637189085
Bias^2: 0.010018312644140933
Var: 0.016587414993048166
0.026605727637189085 &gt;= 0.010018312644140933 + 0.016587414993048166 = 0.0266057276371891
Polynomial degree: 10
Error: 0.021592704588043153
Bias^2: 0.010516485576652981
Var: 0.011076219011390184
0.021592704588043153 &gt;= 0.010516485576652981 + 0.011076219011390184 = 0.021592704588043167
Polynomial degree: 11
Error: 0.07160048164228314
Bias^2: 0.01443680008897583
Var: 0.0571636815533073
0.07160048164228314 &gt;= 0.01443680008897583 + 0.0571636815533073 = 0.07160048164228312
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 12
Error: 0.1154777721897675
Bias^2: 0.01628578269590588
Var: 0.09919198949386163
0.1154777721897675 &gt;= 0.01628578269590588 + 0.09919198949386163 = 0.11547777218976751
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 13
Error: 0.22842468702166951
Bias^2: 0.01975416527163567
Var: 0.20867052175003387
0.22842468702166951 &gt;= 0.01975416527163567 + 0.20867052175003387 = 0.22842468702166954
</pre></div>
</div>
<img alt="_images/chapter3_38_6.png" src="_images/chapter3_38_6.png" />
</div>
</div>
<p>The bias-variance tradeoff summarizes the fundamental tension in
machine learning, particularly supervised learning, between the
@@ -1030,6 +1021,30 @@ flexible statistical methods have higher variance.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>============================
Underfitting vs. Overfitting
============================
This example demonstrates the problems of underfitting and overfitting and
how we can use linear regression with polynomial features to approximate
nonlinear functions. The plot shows the function that we want to approximate,
which is a part of the cosine function. In addition, the samples from the
real function and the approximations of different models are displayed. The
models have polynomial features of different degrees. We can see that a
linear function (polynomial with degree 1) is not sufficient to fit the
training samples. This is called **underfitting**. A polynomial of degree 4
approximates the true function almost perfectly. However, for higher degrees
the model will **overfit** the training data, i.e. it learns the noise of the
training data.
We evaluate quantitatively **overfitting** / **underfitting** by using
cross-validation. We calculate the mean squared error (MSE) on the validation
set, the higher, the less likely the model generalizes correctly from the
training data.
</pre></div>
</div>
<img alt="_images/chapter3_40_1.png" src="_images/chapter3_40_1.png" />
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -1114,6 +1129,122 @@ flexible statistical methods have higher variance.</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 1
Mean squared error on training data: 439230.69504801
Mean squared error on test data: 481979.17861098
Degree of polynomial: 2
Mean squared error on training data: 115822.95008046
Mean squared error on test data: 123711.53703498
Degree of polynomial: 3
Mean squared error on training data: 9011.85263220
Mean squared error on test data: 10913.84780262
Degree of polynomial: 4
Mean squared error on training data: 303.47610036
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Mean squared error on test data: 426.30787294
Degree of polynomial: 5
Mean squared error on training data: 3.80354994
Mean squared error on test data: 5.98822371
Degree of polynomial: 6
Mean squared error on training data: 3.66204648
Mean squared error on test data: 8.14812206
Degree of polynomial: 7
Mean squared error on training data: 0.47075725
Mean squared error on test data: 2.00607783
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 8
Mean squared error on training data: 0.04912436
Mean squared error on test data: 0.21596432
Degree of polynomial: 9
Mean squared error on training data: 0.02522069
Mean squared error on test data: 0.08576932
Degree of polynomial: 10
Mean squared error on training data: 0.02511518
Mean squared error on test data: 1.20015436
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 11
Mean squared error on training data: 0.01640891
Mean squared error on test data: 1.35533773
Degree of polynomial: 12
Mean squared error on training data: 0.00813803
Mean squared error on test data: 0.17446471
Degree of polynomial: 13
Mean squared error on training data: 0.00759119
Mean squared error on test data: 1.08131003
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 14
Mean squared error on training data: 0.00472199
Mean squared error on test data: 0.81333793
Degree of polynomial: 15
Mean squared error on training data: 0.00410478
Mean squared error on test data: 92.09145189
Degree of polynomial: 16
Mean squared error on training data: 0.00315593
Mean squared error on test data: 234.39716546
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 17
Mean squared error on training data: 0.00242998
Mean squared error on test data: 1271.05295709
Degree of polynomial: 18
Mean squared error on training data: 0.00228740
Mean squared error on test data: 108.42208194
Degree of polynomial: 19
Mean squared error on training data: 0.00156372
Mean squared error on test data: 1388.41078073
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 20
Mean squared error on training data: 0.00137982
Mean squared error on test data: 1761.43341615
Degree of polynomial: 21
Mean squared error on training data: 0.00118170
Mean squared error on test data: 15061.31603087
Degree of polynomial: 22
Mean squared error on training data: 0.00092354
Mean squared error on test data: 890.63488525
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 23
Mean squared error on training data: 0.00085887
Mean squared error on test data: 5483.16796929
Degree of polynomial: 24
Mean squared error on training data: 0.00084589
Mean squared error on test data: 1695.57143061
Degree of polynomial: 25
Mean squared error on training data: 0.00078806
Mean squared error on test data: 131343.30655001
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 26
Mean squared error on training data: 0.00076916
Mean squared error on test data: 17709.14370264
Degree of polynomial: 27
Mean squared error on training data: 0.00068970
Mean squared error on test data: 2975.38903780
Degree of polynomial: 28
Mean squared error on training data: 0.00062588
Mean squared error on test data: 3848.64522721
</pre></div>
</div>
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 29
Mean squared error on training data: 0.00060728
Mean squared error on test data: 2988.64001211
</pre></div>
</div>
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;ipython-input-5-8dc29df57a8c&gt;:73: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(trainingerror), label=&#39;Training Error&#39;)
&lt;ipython-input-5-8dc29df57a8c&gt;:74: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(testerror), label=&#39;Test Error&#39;)
</pre></div>
</div>
<img alt="_images/chapter3_41_11.png" src="_images/chapter3_41_11.png" />
</div>
</div>
</div>
<div class="section" id="cross-validation">
@@ -1264,6 +1395,9 @@ cross-validation (LOOCV).</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter3_47_0.png" src="_images/chapter3_47_0.png" />
</div>
</div>
<p>More examples of the application of cross-validation follow here.</p>
<div class="cell docutils container">
@@ -1338,6 +1472,13 @@ cross-validation (LOOCV).</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>&lt;ipython-input-7-49b0ef2e51e2&gt;:63: RuntimeWarning: divide by zero encountered in log10
plt.plot(polynomial, np.log10(estimated_mse_sklearn), label=&#39;Test Error&#39;)
</pre></div>
</div>
<img alt="_images/chapter3_49_1.png" src="_images/chapter3_49_1.png" />
</div>
</div>
<div class="cell docutils container">
<div class="cell_input docutils container">
@@ -1379,6 +1520,9 @@ cross-validation (LOOCV).</p>
</pre></div>
</div>
</div>
<div class="cell_output docutils container">
<img alt="_images/chapter3_50_0.png" src="_images/chapter3_50_0.png" />
</div>
</div>
</div>
</div>