lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on week 37

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-15 09:31:26 +02:00
parent 5921cb0a5b
commit c556620351
18 changed files with 198 additions and 182 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/pub/week37/html/._week37-bs000.html
+1 -1
View File
@@ -270,7 +270,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 14, 2021</h4></center> <!-- date -->
<center><h4>Sep 15, 2021</h4></center> <!-- date -->
<br>
<p>
doc/pub/week37/html/._week37-bs013.html
+1 -1
View File
@@ -263,7 +263,7 @@ our regression analysis. In what follows we will
<li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
and discuss how to select a given model (one of the difficult parts in machine learning).
<p>
<p>
doc/pub/week37/html/._week37-bs016.html
+2 -2
View File
@@ -257,8 +257,8 @@ MathJax.Hub.Config({
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
<ul>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses.</li>
<li> The results can be analysed with the same statistical tools as we would use when analysing experimental data.</li>
<li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
doc/pub/week37/html/._week37-bs018.html
+1 -1
View File
@@ -256,7 +256,7 @@ MathJax.Hub.Config({
<p>
With all these analytical equations for both the OLS and Ridge
regression, we will now outline how to assess a given model. This will
lead us to a discussion of the so-called bias-variance tradeoff (see
lead to a discussion of the so-called bias-variance tradeoff (see
below) and so-called resampling methods.
<p>
doc/pub/week37/html/._week37-bs019.html
+1 -1
View File
@@ -260,7 +260,7 @@ resampling methods are the <b>independent bootstrap</b> and <b>the jackknife</b>
<p>
The jackknife is a special case of the independent bootstrap. Still, the jackknife was made
popular prior to the independent bootstrap. And as the popularity of
the independent bootstrap soared, new variants, such as <b>the dependent bootstrap</b>.
the independent bootstrap soared, new variants, such as <b>the dependent bootstrap</b> have also been developed..
<p>
The Jackknife and independent bootstrap work for
doc/pub/week37/html/._week37-bs020.html
+3 -3
View File
@@ -254,7 +254,7 @@ MathJax.Hub.Config({
<h2 id="resampling-methods-jackknife" class="anchor">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
The Jackknife works by making many replicas of the estimator \( \widehat{\beta} \).
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \).
Let \( \boldsymbol{x}_i \) denote the vector
$$
@@ -264,8 +264,8 @@ $$
<p>
which equals the vector \( \boldsymbol{x} \) with the exception that observation
number \( i \) is left out. Using this notation, define
\( \widehat{\theta}_i \) to be the estimator
\( \widehat{\theta} \) computed using \( \vec{X}_i \).
\( \widehat{\beta}_i \) to be the estimator
\( \widehat{\beta} \) computed using \( \vec{X}_i \).
<p>
<p>
doc/pub/week37/html/._week37-bs022.html
+7 -1
View File
@@ -255,7 +255,7 @@ MathJax.Hub.Config({
<div class="panel panel-default">
<div class="panel-body">
<p> <!-- subsequent paragraphs come in larger fonts, so start with a paragraph -->
Bootstrapping is a nonparametric approach to statistical inference
Bootstrapping is a non-parametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. Bootstrapping offers a number of
advantages:
@@ -270,6 +270,12 @@ advantages:
</div>
<p>
The textbook by <a href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" target="_self">Davison on the Bootstrap Methods and their Applications</a> provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by <a href="https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317" target="_self">Efron and Tibshirani</a>.
<p>
Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called <b>central limit theorem</b>.
<p>
<p>
<!-- navigation buttons at the bottom of the page -->
doc/pub/week37/html/._week37-bs023.html
+3 -3
View File
@@ -254,11 +254,11 @@ MathJax.Hub.Config({
<h2 id="resampling-methods-bootstrap-background" class="anchor">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
\( \widehat{\theta} \) itself must be a random variable. Thus it has
Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables,
\( \widehat{\beta} \) itself must be a random variable. Thus it has
a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to
estimate \( p(\boldsymbol{t}) \) by the relative frequency of
\( \widehat{\theta} \). You can think of this as using a histogram
\( \widehat{\beta} \). You can think of this as using a histogram
in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point
doc/pub/week37/html/._week37-bs024.html
+6 -6
View File
@@ -254,20 +254,20 @@ MathJax.Hub.Config({
<h2 id="resampling-methods-more-bootstrap-background" class="anchor">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\theta} \) has
In the case that \( \widehat{\beta} \) has
more than one component, and the components are independent, we use the
same estimator on each component separately. If the probability
density function of \( X_i \), \( p(x) \), had been known, then it would have
been straight forward to do this by:
been straightforward to do this by:
<ol>
<li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
<li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
<li> Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).</li>
</ol>
By repeated use of (1) and (2), many
estimates of \( \widehat{\theta} \) could have been obtained. The
idea is to use the relative frequency of \( \widehat{\theta}^* \)
By repeated use of the above two points, many
estimates of \( \widehat{\beta} \) can be obtained. The
idea is to use the relative frequency of \( \widehat{\beta}^* \)
(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).
<p>
doc/pub/week37/html/._week37-bs025.html
+4 -7
View File
@@ -259,15 +259,12 @@ unless there is enough information available about the process that
generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general
unknown. Therefore, <a href="https://projecteuclid.org/euclid.aos/1176344552" target="_self">Efron in 1979</a> asked the
question: What if we replace \( p(x) \) by the relative frequency
of the observation \( X_i \); if we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
of the observation \( X_i \)?
<p>
Instead of generating the histogram for the relative
frequency of the observation \( X_i \), just draw the values
\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector
\( \boldsymbol{X} \).
If we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
<p>
<p>
doc/pub/week37/html/._week37-bs026.html
+5 -5
View File
@@ -259,19 +259,19 @@ The independent bootstrap works like this:
<ol>
<li> Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).</li>
<li> Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).</li>
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).</li>
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).</li>
<li> Repeat this process \( k \) times.</li>
</ol>
When you are done, you can draw a histogram of the relative frequency
of \( \widehat \theta^* \). This is your estimate of the probability
of \( \widehat \beta^* \). This is your estimate of the probability
distribution \( p(t) \). Using this probability distribution you can
estimate any statistics thereof. In principle you never draw the
histogram of the relative frequency of \( \widehat{\theta}^* \). Instead
histogram of the relative frequency of \( \widehat{\beta}^* \). Instead
you use the estimators corresponding to the statistic of interest. For
example, if you are interested in estimating the variance of \( \widehat
\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values
\( \widehat \theta ^* \).
\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values
\( \widehat \beta^* \).
<p>
<p>
doc/pub/week37/html/week37-bs.html
+1 -1
View File
@@ -270,7 +270,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 14, 2021</h4></center> <!-- date -->
<center><h4>Sep 15, 2021</h4></center> <!-- date -->
<br>
<p>
doc/pub/week37/html/week37-reveal.html
+34 -31
View File
@@ -148,7 +148,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>&nbsp;<br>
<center><h4>Sep 14, 2021</h4></center> <!-- date -->
<center><h4>Sep 15, 2021</h4></center> <!-- date -->
<br>
<p>
@@ -591,7 +591,7 @@ our regression analysis. In what follows we will
</ol>
<p>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
and discuss how to select a given model (one of the difficult parts in machine learning).
</section>
@@ -659,8 +659,8 @@ bootstrap is widely used.
<div class="alert alert-block alert-block alert-text-normal">
<b>Statistical analysis</b>
<ul>
<p><li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<p><li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<p><li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses.</li>
<p><li> The results can be analysed with the same statistical tools as we would use when analysing experimental data.</li>
<p><li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
@@ -693,7 +693,7 @@ bootstrap is widely used.
<p>
With all these analytical equations for both the OLS and Ridge
regression, we will now outline how to assess a given model. This will
lead us to a discussion of the so-called bias-variance tradeoff (see
lead to a discussion of the so-called bias-variance tradeoff (see
below) and so-called resampling methods.
<p>
@@ -727,7 +727,7 @@ resampling methods are the <b>independent bootstrap</b> and <b>the jackknife</b>
<p>
The jackknife is a special case of the independent bootstrap. Still, the jackknife was made
popular prior to the independent bootstrap. And as the popularity of
the independent bootstrap soared, new variants, such as <b>the dependent bootstrap</b>.
the independent bootstrap soared, new variants, such as <b>the dependent bootstrap</b> have also been developed..
<p>
The Jackknife and independent bootstrap work for
@@ -744,7 +744,7 @@ need for bootstrapping.
<h2 id="resampling-methods-jackknife">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
The Jackknife works by making many replicas of the estimator \( \widehat{\beta} \).
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \).
Let \( \boldsymbol{x}_i \) denote the vector
<p>&nbsp;<br>
@@ -756,8 +756,8 @@ $$
<p>
which equals the vector \( \boldsymbol{x} \) with the exception that observation
number \( i \) is left out. Using this notation, define
\( \widehat{\theta}_i \) to be the estimator
\( \widehat{\theta} \) computed using \( \vec{X}_i \).
\( \widehat{\beta}_i \) to be the estimator
\( \widehat{\beta} \) computed using \( \vec{X}_i \).
</section>
@@ -803,7 +803,7 @@ t = jackknife(x, stat)
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Bootstrapping is a nonparametric approach to statistical inference
Bootstrapping is a non-parametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. Bootstrapping offers a number of
advantages:
@@ -817,6 +817,12 @@ advantages:
<p><li> It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).</li>
</ol>
</div>
<p>
The textbook by <a href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" target="_blank">Davison on the Bootstrap Methods and their Applications</a> provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by <a href="https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317" target="_blank">Efron and Tibshirani</a>.
<p>
Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called <b>central limit theorem</b>.
</section>
@@ -824,11 +830,11 @@ advantages:
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
\( \widehat{\theta} \) itself must be a random variable. Thus it has
Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables,
\( \widehat{\beta} \) itself must be a random variable. Thus it has
a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to
estimate \( p(\boldsymbol{t}) \) by the relative frequency of
\( \widehat{\theta} \). You can think of this as using a histogram
\( \widehat{\beta} \). You can think of this as using a histogram
in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point
@@ -840,21 +846,21 @@ estimators.
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\theta} \) has
In the case that \( \widehat{\beta} \) has
more than one component, and the components are independent, we use the
same estimator on each component separately. If the probability
density function of \( X_i \), \( p(x) \), had been known, then it would have
been straight forward to do this by:
been straightforward to do this by:
<ol>
<p><li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
<p><li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
<p><li> Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).</li>
</ol>
<p>
By repeated use of (1) and (2), many
estimates of \( \widehat{\theta} \) could have been obtained. The
idea is to use the relative frequency of \( \widehat{\theta}^* \)
By repeated use of the above two points, many
estimates of \( \widehat{\beta} \) can be obtained. The
idea is to use the relative frequency of \( \widehat{\beta}^* \)
(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).
</section>
@@ -868,15 +874,12 @@ unless there is enough information available about the process that
generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general
unknown. Therefore, <a href="https://projecteuclid.org/euclid.aos/1176344552" target="_blank">Efron in 1979</a> asked the
question: What if we replace \( p(x) \) by the relative frequency
of the observation \( X_i \); if we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
of the observation \( X_i \)?
<p>
Instead of generating the histogram for the relative
frequency of the observation \( X_i \), just draw the values
\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector
\( \boldsymbol{X} \).
If we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
</section>
@@ -889,20 +892,20 @@ The independent bootstrap works like this:
<ol>
<p><li> Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).</li>
<p><li> Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).</li>
<p><li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).</li>
<p><li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).</li>
<p><li> Repeat this process \( k \) times.</li>
</ol>
<p>
When you are done, you can draw a histogram of the relative frequency
of \( \widehat \theta^* \). This is your estimate of the probability
of \( \widehat \beta^* \). This is your estimate of the probability
distribution \( p(t) \). Using this probability distribution you can
estimate any statistics thereof. In principle you never draw the
histogram of the relative frequency of \( \widehat{\theta}^* \). Instead
histogram of the relative frequency of \( \widehat{\beta}^* \). Instead
you use the estimators corresponding to the statistic of interest. For
example, if you are interested in estimating the variance of \( \widehat
\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values
\( \widehat \theta ^* \).
\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values
\( \widehat \beta^* \).
</section>
doc/pub/week37/html/week37-solarized.html
+34 -31
View File
@@ -221,7 +221,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 14, 2021</h4></center> <!-- date -->
<center><h4>Sep 15, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -609,7 +609,7 @@ our regression analysis. In what follows we will
<li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
and discuss how to select a given model (one of the difficult parts in machine learning).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -682,8 +682,8 @@ bootstrap is widely used.
<p>
<ul>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses.</li>
<li> The results can be analysed with the same statistical tools as we would use when analysing experimental data.</li>
<li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
@@ -719,7 +719,7 @@ bootstrap is widely used.
<p>
With all these analytical equations for both the OLS and Ridge
regression, we will now outline how to assess a given model. This will
lead us to a discussion of the so-called bias-variance tradeoff (see
lead to a discussion of the so-called bias-variance tradeoff (see
below) and so-called resampling methods.
<p>
@@ -752,7 +752,7 @@ resampling methods are the <b>independent bootstrap</b> and <b>the jackknife</b>
<p>
The jackknife is a special case of the independent bootstrap. Still, the jackknife was made
popular prior to the independent bootstrap. And as the popularity of
the independent bootstrap soared, new variants, such as <b>the dependent bootstrap</b>.
the independent bootstrap soared, new variants, such as <b>the dependent bootstrap</b> have also been developed..
<p>
The Jackknife and independent bootstrap work for
@@ -769,7 +769,7 @@ need for bootstrapping.
<h2 id="resampling-methods-jackknife">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
The Jackknife works by making many replicas of the estimator \( \widehat{\beta} \).
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \).
Let \( \boldsymbol{x}_i \) denote the vector
$$
@@ -779,8 +779,8 @@ $$
<p>
which equals the vector \( \boldsymbol{x} \) with the exception that observation
number \( i \) is left out. Using this notation, define
\( \widehat{\theta}_i \) to be the estimator
\( \widehat{\theta} \) computed using \( \vec{X}_i \).
\( \widehat{\beta}_i \) to be the estimator
\( \widehat{\beta} \) computed using \( \vec{X}_i \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -825,7 +825,7 @@ t = jackknife(x, stat)
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Bootstrapping is a nonparametric approach to statistical inference
Bootstrapping is a non-parametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. Bootstrapping offers a number of
advantages:
@@ -839,17 +839,23 @@ advantages:
</div>
<p>
The textbook by <a href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" target="_blank">Davison on the Bootstrap Methods and their Applications</a> provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by <a href="https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317" target="_blank">Efron and Tibshirani</a>.
<p>
Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called <b>central limit theorem</b>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
\( \widehat{\theta} \) itself must be a random variable. Thus it has
Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables,
\( \widehat{\beta} \) itself must be a random variable. Thus it has
a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to
estimate \( p(\boldsymbol{t}) \) by the relative frequency of
\( \widehat{\theta} \). You can think of this as using a histogram
\( \widehat{\beta} \). You can think of this as using a histogram
in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point
@@ -861,20 +867,20 @@ estimators.
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\theta} \) has
In the case that \( \widehat{\beta} \) has
more than one component, and the components are independent, we use the
same estimator on each component separately. If the probability
density function of \( X_i \), \( p(x) \), had been known, then it would have
been straight forward to do this by:
been straightforward to do this by:
<ol>
<li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
<li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
<li> Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).</li>
</ol>
By repeated use of (1) and (2), many
estimates of \( \widehat{\theta} \) could have been obtained. The
idea is to use the relative frequency of \( \widehat{\theta}^* \)
By repeated use of the above two points, many
estimates of \( \widehat{\beta} \) can be obtained. The
idea is to use the relative frequency of \( \widehat{\beta}^* \)
(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).
<p>
@@ -888,15 +894,12 @@ unless there is enough information available about the process that
generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general
unknown. Therefore, <a href="https://projecteuclid.org/euclid.aos/1176344552" target="_blank">Efron in 1979</a> asked the
question: What if we replace \( p(x) \) by the relative frequency
of the observation \( X_i \); if we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
of the observation \( X_i \)?
<p>
Instead of generating the histogram for the relative
frequency of the observation \( X_i \), just draw the values
\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector
\( \boldsymbol{X} \).
If we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -909,19 +912,19 @@ The independent bootstrap works like this:
<ol>
<li> Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).</li>
<li> Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).</li>
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).</li>
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).</li>
<li> Repeat this process \( k \) times.</li>
</ol>
When you are done, you can draw a histogram of the relative frequency
of \( \widehat \theta^* \). This is your estimate of the probability
of \( \widehat \beta^* \). This is your estimate of the probability
distribution \( p(t) \). Using this probability distribution you can
estimate any statistics thereof. In principle you never draw the
histogram of the relative frequency of \( \widehat{\theta}^* \). Instead
histogram of the relative frequency of \( \widehat{\beta}^* \). Instead
you use the estimators corresponding to the statistic of interest. For
example, if you are interested in estimating the variance of \( \widehat
\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values
\( \widehat \theta ^* \).
\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values
\( \widehat \beta^* \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week37/html/week37.html
+34 -31
View File
@@ -226,7 +226,7 @@ MathJax.Hub.Config({
<center>[2] <b>Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University</b></center>
<br>
<p>
<center><h4>Sep 14, 2021</h4></center> <!-- date -->
<center><h4>Sep 15, 2021</h4></center> <!-- date -->
<br>
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -614,7 +614,7 @@ our regression analysis. In what follows we will
<li> introduce resampling techniques like cross-validation, bootstrapping and jackknife and more</li>
</ol>
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
and discuss how to select a given model (one of the difficult parts in machine learning).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -687,8 +687,8 @@ bootstrap is widely used.
<p>
<ul>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods</li>
<li> The results can be analysed with the same statistical tools as we would use analysing experimental data.</li>
<li> Our simulations can be treated as <em>computer experiments</em>. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses.</li>
<li> The results can be analysed with the same statistical tools as we would use when analysing experimental data.</li>
<li> As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.</li>
</ul>
</div>
@@ -724,7 +724,7 @@ bootstrap is widely used.
<p>
With all these analytical equations for both the OLS and Ridge
regression, we will now outline how to assess a given model. This will
lead us to a discussion of the so-called bias-variance tradeoff (see
lead to a discussion of the so-called bias-variance tradeoff (see
below) and so-called resampling methods.
<p>
@@ -757,7 +757,7 @@ resampling methods are the <b>independent bootstrap</b> and <b>the jackknife</b>
<p>
The jackknife is a special case of the independent bootstrap. Still, the jackknife was made
popular prior to the independent bootstrap. And as the popularity of
the independent bootstrap soared, new variants, such as <b>the dependent bootstrap</b>.
the independent bootstrap soared, new variants, such as <b>the dependent bootstrap</b> have also been developed..
<p>
The Jackknife and independent bootstrap work for
@@ -774,7 +774,7 @@ need for bootstrapping.
<h2 id="resampling-methods-jackknife">Resampling methods: Jackknife </h2>
<p>
The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \).
The Jackknife works by making many replicas of the estimator \( \widehat{\beta} \).
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \).
Let \( \boldsymbol{x}_i \) denote the vector
$$
@@ -784,8 +784,8 @@ $$
<p>
which equals the vector \( \boldsymbol{x} \) with the exception that observation
number \( i \) is left out. Using this notation, define
\( \widehat{\theta}_i \) to be the estimator
\( \widehat{\theta} \) computed using \( \vec{X}_i \).
\( \widehat{\beta}_i \) to be the estimator
\( \widehat{\beta} \) computed using \( \vec{X}_i \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -830,7 +830,7 @@ t <span style="color: #666666">=</span> jackknife(x, stat)
<div class="alert alert-block alert-block alert-text-normal">
<b></b>
<p>
Bootstrapping is a nonparametric approach to statistical inference
Bootstrapping is a non-parametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. Bootstrapping offers a number of
advantages:
@@ -844,17 +844,23 @@ advantages:
</div>
<p>
The textbook by <a href="https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" target="_blank">Davison on the Bootstrap Methods and their Applications</a> provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by <a href="https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317" target="_blank">Efron and Tibshirani</a>.
<p>
Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called <b>central limit theorem</b>.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
<h2 id="resampling-methods-bootstrap-background">Resampling methods: Bootstrap background </h2>
<p>
Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables,
\( \widehat{\theta} \) itself must be a random variable. Thus it has
Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables,
\( \widehat{\beta} \) itself must be a random variable. Thus it has
a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to
estimate \( p(\boldsymbol{t}) \) by the relative frequency of
\( \widehat{\theta} \). You can think of this as using a histogram
\( \widehat{\beta} \). You can think of this as using a histogram
in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely
resembles \( p(\vec{t}) \), then using numerics, it is straight forward to
estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point
@@ -866,20 +872,20 @@ estimators.
<h2 id="resampling-methods-more-bootstrap-background">Resampling methods: More Bootstrap background </h2>
<p>
In the case that \( \widehat{\theta} \) has
In the case that \( \widehat{\beta} \) has
more than one component, and the components are independent, we use the
same estimator on each component separately. If the probability
density function of \( X_i \), \( p(x) \), had been known, then it would have
been straight forward to do this by:
been straightforward to do this by:
<ol>
<li> Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).</li>
<li> Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).</li>
<li> Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).</li>
</ol>
By repeated use of (1) and (2), many
estimates of \( \widehat{\theta} \) could have been obtained. The
idea is to use the relative frequency of \( \widehat{\theta}^* \)
By repeated use of the above two points, many
estimates of \( \widehat{\beta} \) can be obtained. The
idea is to use the relative frequency of \( \widehat{\beta}^* \)
(think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).
<p>
@@ -893,15 +899,12 @@ unless there is enough information available about the process that
generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general
unknown. Therefore, <a href="https://projecteuclid.org/euclid.aos/1176344552" target="_blank">Efron in 1979</a> asked the
question: What if we replace \( p(x) \) by the relative frequency
of the observation \( X_i \); if we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
of the observation \( X_i \)?
<p>
Instead of generating the histogram for the relative
frequency of the observation \( X_i \), just draw the values
\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector
\( \boldsymbol{X} \).
If we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
@@ -914,19 +917,19 @@ The independent bootstrap works like this:
<ol>
<li> Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).</li>
<li> Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).</li>
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).</li>
<li> Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).</li>
<li> Repeat this process \( k \) times.</li>
</ol>
When you are done, you can draw a histogram of the relative frequency
of \( \widehat \theta^* \). This is your estimate of the probability
of \( \widehat \beta^* \). This is your estimate of the probability
distribution \( p(t) \). Using this probability distribution you can
estimate any statistics thereof. In principle you never draw the
histogram of the relative frequency of \( \widehat{\theta}^* \). Instead
histogram of the relative frequency of \( \widehat{\beta}^* \). Instead
you use the estimators corresponding to the statistic of interest. For
example, if you are interested in estimating the variance of \( \widehat
\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values
\( \widehat \theta ^* \).
\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values
\( \widehat \beta^* \).
<p>
<!-- !split --><br><br><br><br><br><br><br><br><br><br>
doc/pub/week37/ipynb/ipynb-week37-src.tar.gz
BIN
View File
Binary file not shown.
doc/pub/week37/ipynb/week37.ipynb
+31 -29
View File
@@ -10,7 +10,7 @@
"<!-- Author: --> \n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n",
"\n",
"Date: **Sep 14, 2021**\n",
"Date: **Sep 15, 2021**\n",
"\n",
"Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n",
"\n",
@@ -593,7 +593,7 @@
"\n",
"2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more\n",
"\n",
"This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. \n",
"and discuss how to select a given model (one of the difficult parts in machine learning).\n",
"\n",
"\n",
"\n",
@@ -646,9 +646,9 @@
"**Statistical analysis.**\n",
"\n",
"\n",
"* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods\n",
"* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses.\n",
"\n",
"* The results can be analysed with the same statistical tools as we would use analysing experimental data.\n",
"* The results can be analysed with the same statistical tools as we would use when analysing experimental data.\n",
"\n",
"* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.\n",
"\n",
@@ -677,7 +677,7 @@
"\n",
"With all these analytical equations for both the OLS and Ridge\n",
"regression, we will now outline how to assess a given model. This will\n",
"lead us to a discussion of the so-called bias-variance tradeoff (see\n",
"lead to a discussion of the so-called bias-variance tradeoff (see\n",
"below) and so-called resampling methods.\n",
"\n",
"One of the quantities we have discussed as a way to measure errors is\n",
@@ -704,7 +704,7 @@
"\n",
"The jackknife is a special case of the independent bootstrap. Still, the jackknife was made\n",
"popular prior to the independent bootstrap. And as the popularity of\n",
"the independent bootstrap soared, new variants, such as **the dependent bootstrap**.\n",
"the independent bootstrap soared, new variants, such as **the dependent bootstrap** have also been developed..\n",
"\n",
"The Jackknife and independent bootstrap work for\n",
"independent, identically distributed random variables.\n",
@@ -716,7 +716,7 @@
"\n",
"## Resampling methods: Jackknife\n",
"\n",
"The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n",
"The Jackknife works by making many replicas of the estimator $\\widehat{\\beta}$. \n",
"The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n",
"Let $\\boldsymbol{x}_i$ denote the vector"
]
@@ -736,8 +736,8 @@
"source": [
"which equals the vector $\\boldsymbol{x}$ with the exception that observation\n",
"number $i$ is left out. Using this notation, define\n",
"$\\widehat{\\theta}_i$ to be the estimator\n",
"$\\widehat{\\theta}$ computed using $\\vec{X}_i$. \n",
"$\\widehat{\\beta}_i$ to be the estimator\n",
"$\\widehat{\\beta}$ computed using $\\vec{X}_i$. \n",
"\n",
"\n",
"## Jackknife code example"
@@ -787,7 +787,7 @@
"metadata": {},
"source": [
"## Resampling methods: Bootstrap\n",
"Bootstrapping is a nonparametric approach to statistical inference\n",
"Bootstrapping is a non-parametric approach to statistical inference\n",
"that substitutes computation for more traditional distributional\n",
"assumptions and asymptotic results. Bootstrapping offers a number of\n",
"advantages: \n",
@@ -801,14 +801,18 @@
"\n",
"\n",
"\n",
"The textbook by [Davison on the Bootstrap Methods and their Applications](https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A) provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by [Efron and Tibshirani](https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317).\n",
"\n",
"\n",
"Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called **central limit theorem**.\n",
"\n",
"## Resampling methods: Bootstrap background\n",
"\n",
"Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n",
"$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n",
"Since $\\widehat{\\beta} = \\widehat{\\beta}(\\boldsymbol{X})$ is a function of random variables,\n",
"$\\widehat{\\beta}$ itself must be a random variable. Thus it has\n",
"a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n",
"estimate $p(\\boldsymbol{t})$ by the relative frequency of\n",
"$\\widehat{\\theta}$. You can think of this as using a histogram\n",
"$\\widehat{\\beta}$. You can think of this as using a histogram\n",
"in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n",
"resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n",
"estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n",
@@ -817,18 +821,18 @@
"\n",
"## Resampling methods: More Bootstrap background\n",
"\n",
"In the case that $\\widehat{\\theta}$ has\n",
"In the case that $\\widehat{\\beta}$ has\n",
"more than one component, and the components are independent, we use the\n",
"same estimator on each component separately. If the probability\n",
"density function of $X_i$, $p(x)$, had been known, then it would have\n",
"been straight forward to do this by: \n",
"been straightforward to do this by: \n",
"1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n",
"\n",
"2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n",
"2. Then using these numbers, we could compute a replica of $\\widehat{\\beta}$ called $\\widehat{\\beta}^*$. \n",
"\n",
"By repeated use of (1) and (2), many\n",
"estimates of $\\widehat{\\theta}$ could have been obtained. The\n",
"idea is to use the relative frequency of $\\widehat{\\theta}^*$\n",
"By repeated use of the above two points, many\n",
"estimates of $\\widehat{\\beta}$ can be obtained. The\n",
"idea is to use the relative frequency of $\\widehat{\\beta}^*$\n",
"(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n",
"\n",
"## Resampling methods: Bootstrap approach\n",
@@ -838,15 +842,13 @@
"generated $X_1,X_2,\\cdots,X_n$, $p(x)$ is in general\n",
"unknown. Therefore, [Efron in 1979](https://projecteuclid.org/euclid.aos/1176344552) asked the\n",
"question: What if we replace $p(x)$ by the relative frequency\n",
"of the observation $X_i$; if we draw observations in accordance with\n",
"of the observation $X_i$?\n",
"\n",
"If we draw observations in accordance with\n",
"the relative frequency of the observations, will we obtain the same\n",
"result in some asymptotic sense? The answer is yes.\n",
"\n",
"\n",
"Instead of generating the histogram for the relative\n",
"frequency of the observation $X_i$, just draw the values\n",
"$(X_1^*,X_2^*,\\cdots,X_n^*)$ with replacement from the vector\n",
"$\\boldsymbol{X}$. \n",
"\n",
"## Resampling methods: Bootstrap steps\n",
"\n",
@@ -856,19 +858,19 @@
"\n",
"2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n",
"\n",
"3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n",
"3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\beta}^*$ by evaluating $\\widehat \\beta$ under the observations $\\boldsymbol{x}^*$. \n",
"\n",
"4. Repeat this process $k$ times. \n",
"\n",
"When you are done, you can draw a histogram of the relative frequency\n",
"of $\\widehat \\theta^*$. This is your estimate of the probability\n",
"of $\\widehat \\beta^*$. This is your estimate of the probability\n",
"distribution $p(t)$. Using this probability distribution you can\n",
"estimate any statistics thereof. In principle you never draw the\n",
"histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n",
"histogram of the relative frequency of $\\widehat{\\beta}^*$. Instead\n",
"you use the estimators corresponding to the statistic of interest. For\n",
"example, if you are interested in estimating the variance of $\\widehat\n",
"\\theta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n",
"$\\widehat \\theta ^*$.\n",
"\\beta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n",
"$\\widehat \\beta^*$.\n",
"\n",
"\n",
"## Code example for the Bootstrap method\n",
doc/src/week37/week37.do.txt
+30 -28
View File
@@ -377,7 +377,7 @@ our regression analysis. In what follows we will
o look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff
o introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods.
and discuss how to select a given model (one of the difficult parts in machine learning).
@@ -434,8 +434,8 @@ bootstrap is widely used.
===== Why resampling methods ? =====
!bblock Statistical analysis
* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods
* The results can be analysed with the same statistical tools as we would use analysing experimental data.
* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses.
* The results can be analysed with the same statistical tools as we would use when analysing experimental data.
* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.
@@ -461,7 +461,7 @@ bootstrap is widely used.
With all these analytical equations for both the OLS and Ridge
regression, we will now outline how to assess a given model. This will
lead us to a discussion of the so-called bias-variance tradeoff (see
lead to a discussion of the so-called bias-variance tradeoff (see
below) and so-called resampling methods.
One of the quantities we have discussed as a way to measure errors is
@@ -488,7 +488,7 @@ resampling methods are the _independent bootstrap_ and _the jackknife_.
The jackknife is a special case of the independent bootstrap. Still, the jackknife was made
popular prior to the independent bootstrap. And as the popularity of
the independent bootstrap soared, new variants, such as _the dependent bootstrap_.
the independent bootstrap soared, new variants, such as _the dependent bootstrap_ have also been developed..
The Jackknife and independent bootstrap work for
independent, identically distributed random variables.
@@ -501,7 +501,7 @@ need for bootstrapping.
!split
===== Resampling methods: Jackknife =====
The Jackknife works by making many replicas of the estimator $\widehat{\theta}$.
The Jackknife works by making many replicas of the estimator $\widehat{\beta}$.
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\bm{x} = (x_1,x_2,\cdots,X_n)$.
Let $\bm{x}_i$ denote the vector
!bt
@@ -512,8 +512,8 @@ Let $\bm{x}_i$ denote the vector
which equals the vector $\bm{x}$ with the exception that observation
number $i$ is left out. Using this notation, define
$\widehat{\theta}_i$ to be the estimator
$\widehat{\theta}$ computed using $\vec{X}_i$.
$\widehat{\beta}_i$ to be the estimator
$\widehat{\beta}$ computed using $\vec{X}_i$.
!split
@@ -554,7 +554,7 @@ t = jackknife(x, stat)
!split
===== Resampling methods: Bootstrap =====
!bblock
Bootstrapping is a nonparametric approach to statistical inference
Bootstrapping is a non-parametric approach to statistical inference
that substitutes computation for more traditional distributional
assumptions and asymptotic results. Bootstrapping offers a number of
advantages:
@@ -564,15 +564,19 @@ o It is possible to apply the bootstrap to statistics with sampling distribution
o It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
!eblock
The textbook by "Davison on the Bootstrap Methods and their Applications":"https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by "Efron and Tibshirani":"https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317".
Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called _central limit theorem_.
!split
===== Resampling methods: Bootstrap background =====
Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables,
$\widehat{\theta}$ itself must be a random variable. Thus it has
Since $\widehat{\beta} = \widehat{\beta}(\bm{X})$ is a function of random variables,
$\widehat{\beta}$ itself must be a random variable. Thus it has
a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to
estimate $p(\bm{t})$ by the relative frequency of
$\widehat{\theta}$. You can think of this as using a histogram
$\widehat{\beta}$. You can think of this as using a histogram
in the place of $p(\bm{t})$. If the relative frequency closely
resembles $p(\vec{t})$, then using numerics, it is straight forward to
estimate all the interesting parameters of $p(\bm{t})$ using point
@@ -582,17 +586,17 @@ estimators.
!split
===== Resampling methods: More Bootstrap background =====
In the case that $\widehat{\theta}$ has
In the case that $\widehat{\beta}$ has
more than one component, and the components are independent, we use the
same estimator on each component separately. If the probability
density function of $X_i$, $p(x)$, had been known, then it would have
been straight forward to do this by:
been straightforward to do this by:
o Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$.
o Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$.
o Then using these numbers, we could compute a replica of $\widehat{\beta}$ called $\widehat{\beta}^*$.
By repeated use of (1) and (2), many
estimates of $\widehat{\theta}$ could have been obtained. The
idea is to use the relative frequency of $\widehat{\theta}^*$
By repeated use of the above two points, many
estimates of $\widehat{\beta}$ can be obtained. The
idea is to use the relative frequency of $\widehat{\beta}^*$
(think of a histogram) as an estimate of $p(\bm{t})$.
!split
@@ -603,15 +607,13 @@ unless there is enough information available about the process that
generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general
unknown. Therefore, "Efron in 1979":"https://projecteuclid.org/euclid.aos/1176344552" asked the
question: What if we replace $p(x)$ by the relative frequency
of the observation $X_i$; if we draw observations in accordance with
of the observation $X_i$?
If we draw observations in accordance with
the relative frequency of the observations, will we obtain the same
result in some asymptotic sense? The answer is yes.
Instead of generating the histogram for the relative
frequency of the observation $X_i$, just draw the values
$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector
$\bm{X}$.
!split
===== Resampling methods: Bootstrap steps =====
@@ -620,18 +622,18 @@ The independent bootstrap works like this:
o Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$.
o Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$.
o Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$.
o Using the vector $\bm{x}^*$ compute $\widehat{\beta}^*$ by evaluating $\widehat \beta$ under the observations $\bm{x}^*$.
o Repeat this process $k$ times.
When you are done, you can draw a histogram of the relative frequency
of $\widehat \theta^*$. This is your estimate of the probability
of $\widehat \beta^*$. This is your estimate of the probability
distribution $p(t)$. Using this probability distribution you can
estimate any statistics thereof. In principle you never draw the
histogram of the relative frequency of $\widehat{\theta}^*$. Instead
histogram of the relative frequency of $\widehat{\beta}^*$. Instead
you use the estimators corresponding to the statistic of interest. For
example, if you are interested in estimating the variance of $\widehat
\theta$, apply the etsimator $\widehat \sigma^2$ to the values
$\widehat \theta ^*$.
\beta$, apply the etsimator $\widehat \sigma^2$ to the values
$\widehat \beta^*$.
!split