diff --git a/doc/pub/week37/html/._week37-bs000.html b/doc/pub/week37/html/._week37-bs000.html index 6dc865aec..a60ece31a 100644 --- a/doc/pub/week37/html/._week37-bs000.html +++ b/doc/pub/week37/html/._week37-bs000.html @@ -270,7 +270,7 @@ MathJax.Hub.Config({
[2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

-

Sep 14, 2021

+

Sep 15, 2021


diff --git a/doc/pub/week37/html/._week37-bs013.html b/doc/pub/week37/html/._week37-bs013.html index 21a5ab5ff..32ee0175e 100644 --- a/doc/pub/week37/html/._week37-bs013.html +++ b/doc/pub/week37/html/._week37-bs013.html @@ -263,7 +263,7 @@ our regression analysis. In what follows we will

  • introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
    • -This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. +and discuss how to select a given model (one of the difficult parts in machine learning).

    diff --git a/doc/pub/week37/html/._week37-bs016.html b/doc/pub/week37/html/._week37-bs016.html index 9e8cae8bf..d4d915059 100644 --- a/doc/pub/week37/html/._week37-bs016.html +++ b/doc/pub/week37/html/._week37-bs016.html @@ -257,8 +257,8 @@ MathJax.Hub.Config({

    diff --git a/doc/pub/week37/html/._week37-bs018.html b/doc/pub/week37/html/._week37-bs018.html index 80662ce5f..cf010edb8 100644 --- a/doc/pub/week37/html/._week37-bs018.html +++ b/doc/pub/week37/html/._week37-bs018.html @@ -256,7 +256,7 @@ MathJax.Hub.Config({

    With all these analytical equations for both the OLS and Ridge regression, we will now outline how to assess a given model. This will -lead us to a discussion of the so-called bias-variance tradeoff (see +lead to a discussion of the so-called bias-variance tradeoff (see below) and so-called resampling methods.

    diff --git a/doc/pub/week37/html/._week37-bs019.html b/doc/pub/week37/html/._week37-bs019.html index cde89f6d3..b965e2612 100644 --- a/doc/pub/week37/html/._week37-bs019.html +++ b/doc/pub/week37/html/._week37-bs019.html @@ -260,7 +260,7 @@ resampling methods are the independent bootstrap and the jackknife

    The jackknife is a special case of the independent bootstrap. Still, the jackknife was made popular prior to the independent bootstrap. And as the popularity of -the independent bootstrap soared, new variants, such as the dependent bootstrap. +the independent bootstrap soared, new variants, such as the dependent bootstrap have also been developed..

    The Jackknife and independent bootstrap work for diff --git a/doc/pub/week37/html/._week37-bs020.html b/doc/pub/week37/html/._week37-bs020.html index 6f6fe8ced..c5abe310a 100644 --- a/doc/pub/week37/html/._week37-bs020.html +++ b/doc/pub/week37/html/._week37-bs020.html @@ -254,7 +254,7 @@ MathJax.Hub.Config({

    Resampling methods: Jackknife

    -The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \). +The Jackknife works by making many replicas of the estimator \( \widehat{\beta} \). The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \). Let \( \boldsymbol{x}_i \) denote the vector $$ @@ -264,8 +264,8 @@ $$

    which equals the vector \( \boldsymbol{x} \) with the exception that observation number \( i \) is left out. Using this notation, define -\( \widehat{\theta}_i \) to be the estimator -\( \widehat{\theta} \) computed using \( \vec{X}_i \). +\( \widehat{\beta}_i \) to be the estimator +\( \widehat{\beta} \) computed using \( \vec{X}_i \).

    diff --git a/doc/pub/week37/html/._week37-bs022.html b/doc/pub/week37/html/._week37-bs022.html index 4776ce29f..11fb837fd 100644 --- a/doc/pub/week37/html/._week37-bs022.html +++ b/doc/pub/week37/html/._week37-bs022.html @@ -255,7 +255,7 @@ MathJax.Hub.Config({

    -Bootstrapping is a nonparametric approach to statistical inference +Bootstrapping is a non-parametric approach to statistical inference that substitutes computation for more traditional distributional assumptions and asymptotic results. Bootstrapping offers a number of advantages: @@ -270,6 +270,12 @@ advantages:

    +

    +The textbook by Davison on the Bootstrap Methods and their Applications provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by Efron and Tibshirani. + +

    +Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called central limit theorem. +

    diff --git a/doc/pub/week37/html/._week37-bs023.html b/doc/pub/week37/html/._week37-bs023.html index 1bf506740..06ccacb99 100644 --- a/doc/pub/week37/html/._week37-bs023.html +++ b/doc/pub/week37/html/._week37-bs023.html @@ -254,11 +254,11 @@ MathJax.Hub.Config({

    Resampling methods: Bootstrap background

    -Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables, -\( \widehat{\theta} \) itself must be a random variable. Thus it has +Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables, +\( \widehat{\beta} \) itself must be a random variable. Thus it has a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to estimate \( p(\boldsymbol{t}) \) by the relative frequency of -\( \widehat{\theta} \). You can think of this as using a histogram +\( \widehat{\beta} \). You can think of this as using a histogram in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely resembles \( p(\vec{t}) \), then using numerics, it is straight forward to estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point diff --git a/doc/pub/week37/html/._week37-bs024.html b/doc/pub/week37/html/._week37-bs024.html index de5da0b60..d505dbe57 100644 --- a/doc/pub/week37/html/._week37-bs024.html +++ b/doc/pub/week37/html/._week37-bs024.html @@ -254,20 +254,20 @@ MathJax.Hub.Config({

    Resampling methods: More Bootstrap background

    -In the case that \( \widehat{\theta} \) has +In the case that \( \widehat{\beta} \) has more than one component, and the components are independent, we use the same estimator on each component separately. If the probability density function of \( X_i \), \( p(x) \), had been known, then it would have -been straight forward to do this by: +been straightforward to do this by:

    1. Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).
      2. -
    2. Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).
      3. +
    3. Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).
    -By repeated use of (1) and (2), many -estimates of \( \widehat{\theta} \) could have been obtained. The -idea is to use the relative frequency of \( \widehat{\theta}^* \) +By repeated use of the above two points, many +estimates of \( \widehat{\beta} \) can be obtained. The +idea is to use the relative frequency of \( \widehat{\beta}^* \) (think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).

    diff --git a/doc/pub/week37/html/._week37-bs025.html b/doc/pub/week37/html/._week37-bs025.html index ce192a72b..c773534d4 100644 --- a/doc/pub/week37/html/._week37-bs025.html +++ b/doc/pub/week37/html/._week37-bs025.html @@ -259,15 +259,12 @@ unless there is enough information available about the process that generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general unknown. Therefore, Efron in 1979 asked the question: What if we replace \( p(x) \) by the relative frequency -of the observation \( X_i \); if we draw observations in accordance with -the relative frequency of the observations, will we obtain the same -result in some asymptotic sense? The answer is yes. +of the observation \( X_i \)?

    -Instead of generating the histogram for the relative -frequency of the observation \( X_i \), just draw the values -\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector -\( \boldsymbol{X} \). +If we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes.

    diff --git a/doc/pub/week37/html/._week37-bs026.html b/doc/pub/week37/html/._week37-bs026.html index 7df37e7d1..9d05e820e 100644 --- a/doc/pub/week37/html/._week37-bs026.html +++ b/doc/pub/week37/html/._week37-bs026.html @@ -259,19 +259,19 @@ The independent bootstrap works like this:

    1. Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).
    2. Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).
      3. -
    3. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).
      4. +
    4. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).
    5. Repeat this process \( k \) times.
    When you are done, you can draw a histogram of the relative frequency -of \( \widehat \theta^* \). This is your estimate of the probability +of \( \widehat \beta^* \). This is your estimate of the probability distribution \( p(t) \). Using this probability distribution you can estimate any statistics thereof. In principle you never draw the -histogram of the relative frequency of \( \widehat{\theta}^* \). Instead +histogram of the relative frequency of \( \widehat{\beta}^* \). Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of \( \widehat -\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values -\( \widehat \theta ^* \). +\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values +\( \widehat \beta^* \).

    diff --git a/doc/pub/week37/html/week37-bs.html b/doc/pub/week37/html/week37-bs.html index 6dc865aec..a60ece31a 100644 --- a/doc/pub/week37/html/week37-bs.html +++ b/doc/pub/week37/html/week37-bs.html @@ -270,7 +270,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 14, 2021

    +

    Sep 15, 2021


    diff --git a/doc/pub/week37/html/week37-reveal.html b/doc/pub/week37/html/week37-reveal.html index 4cfdaf06f..ed627abc0 100644 --- a/doc/pub/week37/html/week37-reveal.html +++ b/doc/pub/week37/html/week37-reveal.html @@ -148,7 +148,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

     
    -

    Sep 14, 2021

    +

    Sep 15, 2021


    @@ -591,7 +591,7 @@ our regression analysis. In what follows we will

    -This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. +and discuss how to select a given model (one of the difficult parts in machine learning). @@ -659,8 +659,8 @@ bootstrap is widely used.

    Statistical analysis
    @@ -693,7 +693,7 @@ bootstrap is widely used.

    With all these analytical equations for both the OLS and Ridge regression, we will now outline how to assess a given model. This will -lead us to a discussion of the so-called bias-variance tradeoff (see +lead to a discussion of the so-called bias-variance tradeoff (see below) and so-called resampling methods.

    @@ -727,7 +727,7 @@ resampling methods are the independent bootstrap and the jackknife

    The jackknife is a special case of the independent bootstrap. Still, the jackknife was made popular prior to the independent bootstrap. And as the popularity of -the independent bootstrap soared, new variants, such as the dependent bootstrap. +the independent bootstrap soared, new variants, such as the dependent bootstrap have also been developed..

    The Jackknife and independent bootstrap work for @@ -744,7 +744,7 @@ need for bootstrapping.

    Resampling methods: Jackknife

    -The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \). +The Jackknife works by making many replicas of the estimator \( \widehat{\beta} \). The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \). Let \( \boldsymbol{x}_i \) denote the vector

     
    @@ -756,8 +756,8 @@ $$

    which equals the vector \( \boldsymbol{x} \) with the exception that observation number \( i \) is left out. Using this notation, define -\( \widehat{\theta}_i \) to be the estimator -\( \widehat{\theta} \) computed using \( \vec{X}_i \). +\( \widehat{\beta}_i \) to be the estimator +\( \widehat{\beta} \) computed using \( \vec{X}_i \). @@ -803,7 +803,7 @@ t = jackknife(x, stat)

    -Bootstrapping is a nonparametric approach to statistical inference +Bootstrapping is a non-parametric approach to statistical inference that substitutes computation for more traditional distributional assumptions and asymptotic results. Bootstrapping offers a number of advantages: @@ -817,6 +817,12 @@ advantages:

  • It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
    • + +

    +The textbook by Davison on the Bootstrap Methods and their Applications provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by Efron and Tibshirani. + +

    +Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called central limit theorem. @@ -824,11 +830,11 @@ advantages:

    Resampling methods: Bootstrap background

    -Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables, -\( \widehat{\theta} \) itself must be a random variable. Thus it has +Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables, +\( \widehat{\beta} \) itself must be a random variable. Thus it has a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to estimate \( p(\boldsymbol{t}) \) by the relative frequency of -\( \widehat{\theta} \). You can think of this as using a histogram +\( \widehat{\beta} \). You can think of this as using a histogram in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely resembles \( p(\vec{t}) \), then using numerics, it is straight forward to estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point @@ -840,21 +846,21 @@ estimators.

    Resampling methods: More Bootstrap background

    -In the case that \( \widehat{\theta} \) has +In the case that \( \widehat{\beta} \) has more than one component, and the components are independent, we use the same estimator on each component separately. If the probability density function of \( X_i \), \( p(x) \), had been known, then it would have -been straight forward to do this by: +been straightforward to do this by:

    1. Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).
      2. -

    2. Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).
      3. +

    3. Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).

    -By repeated use of (1) and (2), many -estimates of \( \widehat{\theta} \) could have been obtained. The -idea is to use the relative frequency of \( \widehat{\theta}^* \) +By repeated use of the above two points, many +estimates of \( \widehat{\beta} \) can be obtained. The +idea is to use the relative frequency of \( \widehat{\beta}^* \) (think of a histogram) as an estimate of \( p(\boldsymbol{t}) \). @@ -868,15 +874,12 @@ unless there is enough information available about the process that generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general unknown. Therefore, Efron in 1979 asked the question: What if we replace \( p(x) \) by the relative frequency -of the observation \( X_i \); if we draw observations in accordance with -the relative frequency of the observations, will we obtain the same -result in some asymptotic sense? The answer is yes. +of the observation \( X_i \)?

    -Instead of generating the histogram for the relative -frequency of the observation \( X_i \), just draw the values -\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector -\( \boldsymbol{X} \). +If we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes. @@ -889,20 +892,20 @@ The independent bootstrap works like this:

    1. Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).

    2. Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).
      3. -

    3. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).
      4. +

    4. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).

    5. Repeat this process \( k \) times.

    When you are done, you can draw a histogram of the relative frequency -of \( \widehat \theta^* \). This is your estimate of the probability +of \( \widehat \beta^* \). This is your estimate of the probability distribution \( p(t) \). Using this probability distribution you can estimate any statistics thereof. In principle you never draw the -histogram of the relative frequency of \( \widehat{\theta}^* \). Instead +histogram of the relative frequency of \( \widehat{\beta}^* \). Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of \( \widehat -\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values -\( \widehat \theta ^* \). +\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values +\( \widehat \beta^* \). diff --git a/doc/pub/week37/html/week37-solarized.html b/doc/pub/week37/html/week37-solarized.html index 16be56986..b54409fe1 100644 --- a/doc/pub/week37/html/week37-solarized.html +++ b/doc/pub/week37/html/week37-solarized.html @@ -221,7 +221,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 14, 2021

    +

    Sep 15, 2021












    @@ -609,7 +609,7 @@ our regression analysis. In what follows we will

  • introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
    • -This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. +and discuss how to select a given model (one of the difficult parts in machine learning).











    @@ -682,8 +682,8 @@ bootstrap is widely used.

    @@ -719,7 +719,7 @@ bootstrap is widely used.

    With all these analytical equations for both the OLS and Ridge regression, we will now outline how to assess a given model. This will -lead us to a discussion of the so-called bias-variance tradeoff (see +lead to a discussion of the so-called bias-variance tradeoff (see below) and so-called resampling methods.

    @@ -752,7 +752,7 @@ resampling methods are the independent bootstrap and the jackknife

    The jackknife is a special case of the independent bootstrap. Still, the jackknife was made popular prior to the independent bootstrap. And as the popularity of -the independent bootstrap soared, new variants, such as the dependent bootstrap. +the independent bootstrap soared, new variants, such as the dependent bootstrap have also been developed..

    The Jackknife and independent bootstrap work for @@ -769,7 +769,7 @@ need for bootstrapping.

    Resampling methods: Jackknife

    -The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \). +The Jackknife works by making many replicas of the estimator \( \widehat{\beta} \). The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \). Let \( \boldsymbol{x}_i \) denote the vector $$ @@ -779,8 +779,8 @@ $$

    which equals the vector \( \boldsymbol{x} \) with the exception that observation number \( i \) is left out. Using this notation, define -\( \widehat{\theta}_i \) to be the estimator -\( \widehat{\theta} \) computed using \( \vec{X}_i \). +\( \widehat{\beta}_i \) to be the estimator +\( \widehat{\beta} \) computed using \( \vec{X}_i \).











    @@ -825,7 +825,7 @@ t = jackknife(x, stat)

    -Bootstrapping is a nonparametric approach to statistical inference +Bootstrapping is a non-parametric approach to statistical inference that substitutes computation for more traditional distributional assumptions and asymptotic results. Bootstrapping offers a number of advantages: @@ -839,17 +839,23 @@ advantages:

    +

    +The textbook by Davison on the Bootstrap Methods and their Applications provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by Efron and Tibshirani. + +

    +Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called central limit theorem. +











    Resampling methods: Bootstrap background

    -Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables, -\( \widehat{\theta} \) itself must be a random variable. Thus it has +Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables, +\( \widehat{\beta} \) itself must be a random variable. Thus it has a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to estimate \( p(\boldsymbol{t}) \) by the relative frequency of -\( \widehat{\theta} \). You can think of this as using a histogram +\( \widehat{\beta} \). You can think of this as using a histogram in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely resembles \( p(\vec{t}) \), then using numerics, it is straight forward to estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point @@ -861,20 +867,20 @@ estimators.

    Resampling methods: More Bootstrap background

    -In the case that \( \widehat{\theta} \) has +In the case that \( \widehat{\beta} \) has more than one component, and the components are independent, we use the same estimator on each component separately. If the probability density function of \( X_i \), \( p(x) \), had been known, then it would have -been straight forward to do this by: +been straightforward to do this by:

    1. Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).
      2. -
    2. Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).
      3. +
    3. Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).
    -By repeated use of (1) and (2), many -estimates of \( \widehat{\theta} \) could have been obtained. The -idea is to use the relative frequency of \( \widehat{\theta}^* \) +By repeated use of the above two points, many +estimates of \( \widehat{\beta} \) can be obtained. The +idea is to use the relative frequency of \( \widehat{\beta}^* \) (think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).

    @@ -888,15 +894,12 @@ unless there is enough information available about the process that generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general unknown. Therefore, Efron in 1979 asked the question: What if we replace \( p(x) \) by the relative frequency -of the observation \( X_i \); if we draw observations in accordance with -the relative frequency of the observations, will we obtain the same -result in some asymptotic sense? The answer is yes. +of the observation \( X_i \)?

    -Instead of generating the histogram for the relative -frequency of the observation \( X_i \), just draw the values -\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector -\( \boldsymbol{X} \). +If we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes.











    @@ -909,19 +912,19 @@ The independent bootstrap works like this:

    1. Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).
    2. Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).
      3. -
    3. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).
      4. +
    4. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).
    5. Repeat this process \( k \) times.
    When you are done, you can draw a histogram of the relative frequency -of \( \widehat \theta^* \). This is your estimate of the probability +of \( \widehat \beta^* \). This is your estimate of the probability distribution \( p(t) \). Using this probability distribution you can estimate any statistics thereof. In principle you never draw the -histogram of the relative frequency of \( \widehat{\theta}^* \). Instead +histogram of the relative frequency of \( \widehat{\beta}^* \). Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of \( \widehat -\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values -\( \widehat \theta ^* \). +\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values +\( \widehat \beta^* \).











    diff --git a/doc/pub/week37/html/week37.html b/doc/pub/week37/html/week37.html index 4227ad31a..de5b07941 100644 --- a/doc/pub/week37/html/week37.html +++ b/doc/pub/week37/html/week37.html @@ -226,7 +226,7 @@ MathJax.Hub.Config({

    [2] Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

    -

    Sep 14, 2021

    +

    Sep 15, 2021












    @@ -614,7 +614,7 @@ our regression analysis. In what follows we will

  • introduce resampling techniques like cross-validation, bootstrapping and jackknife and more
    • -This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. +and discuss how to select a given model (one of the difficult parts in machine learning).











    @@ -687,8 +687,8 @@ bootstrap is widely used.

    @@ -724,7 +724,7 @@ bootstrap is widely used.

    With all these analytical equations for both the OLS and Ridge regression, we will now outline how to assess a given model. This will -lead us to a discussion of the so-called bias-variance tradeoff (see +lead to a discussion of the so-called bias-variance tradeoff (see below) and so-called resampling methods.

    @@ -757,7 +757,7 @@ resampling methods are the independent bootstrap and the jackknife

    The jackknife is a special case of the independent bootstrap. Still, the jackknife was made popular prior to the independent bootstrap. And as the popularity of -the independent bootstrap soared, new variants, such as the dependent bootstrap. +the independent bootstrap soared, new variants, such as the dependent bootstrap have also been developed..

    The Jackknife and independent bootstrap work for @@ -774,7 +774,7 @@ need for bootstrapping.

    Resampling methods: Jackknife

    -The Jackknife works by making many replicas of the estimator \( \widehat{\theta} \). +The Jackknife works by making many replicas of the estimator \( \widehat{\beta} \). The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values \( \boldsymbol{x} = (x_1,x_2,\cdots,X_n) \). Let \( \boldsymbol{x}_i \) denote the vector $$ @@ -784,8 +784,8 @@ $$

    which equals the vector \( \boldsymbol{x} \) with the exception that observation number \( i \) is left out. Using this notation, define -\( \widehat{\theta}_i \) to be the estimator -\( \widehat{\theta} \) computed using \( \vec{X}_i \). +\( \widehat{\beta}_i \) to be the estimator +\( \widehat{\beta} \) computed using \( \vec{X}_i \).











    @@ -830,7 +830,7 @@ t = jackknife(x, stat)

    -Bootstrapping is a nonparametric approach to statistical inference +Bootstrapping is a non-parametric approach to statistical inference that substitutes computation for more traditional distributional assumptions and asymptotic results. Bootstrapping offers a number of advantages: @@ -844,17 +844,23 @@ advantages:

    +

    +The textbook by Davison on the Bootstrap Methods and their Applications provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by Efron and Tibshirani. + +

    +Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called central limit theorem. +











    Resampling methods: Bootstrap background

    -Since \( \widehat{\theta} = \widehat{\theta}(\boldsymbol{X}) \) is a function of random variables, -\( \widehat{\theta} \) itself must be a random variable. Thus it has +Since \( \widehat{\beta} = \widehat{\beta}(\boldsymbol{X}) \) is a function of random variables, +\( \widehat{\beta} \) itself must be a random variable. Thus it has a pdf, call this function \( p(\boldsymbol{t}) \). The aim of the bootstrap is to estimate \( p(\boldsymbol{t}) \) by the relative frequency of -\( \widehat{\theta} \). You can think of this as using a histogram +\( \widehat{\beta} \). You can think of this as using a histogram in the place of \( p(\boldsymbol{t}) \). If the relative frequency closely resembles \( p(\vec{t}) \), then using numerics, it is straight forward to estimate all the interesting parameters of \( p(\boldsymbol{t}) \) using point @@ -866,20 +872,20 @@ estimators.

    Resampling methods: More Bootstrap background

    -In the case that \( \widehat{\theta} \) has +In the case that \( \widehat{\beta} \) has more than one component, and the components are independent, we use the same estimator on each component separately. If the probability density function of \( X_i \), \( p(x) \), had been known, then it would have -been straight forward to do this by: +been straightforward to do this by:

    1. Drawing lots of numbers from \( p(x) \), suppose we call one such set of numbers \( (X_1^*, X_2^*, \cdots, X_n^*) \).
      2. -
    2. Then using these numbers, we could compute a replica of \( \widehat{\theta} \) called \( \widehat{\theta}^* \).
      3. +
    3. Then using these numbers, we could compute a replica of \( \widehat{\beta} \) called \( \widehat{\beta}^* \).
    -By repeated use of (1) and (2), many -estimates of \( \widehat{\theta} \) could have been obtained. The -idea is to use the relative frequency of \( \widehat{\theta}^* \) +By repeated use of the above two points, many +estimates of \( \widehat{\beta} \) can be obtained. The +idea is to use the relative frequency of \( \widehat{\beta}^* \) (think of a histogram) as an estimate of \( p(\boldsymbol{t}) \).

    @@ -893,15 +899,12 @@ unless there is enough information available about the process that generated \( X_1,X_2,\cdots,X_n \), \( p(x) \) is in general unknown. Therefore, Efron in 1979 asked the question: What if we replace \( p(x) \) by the relative frequency -of the observation \( X_i \); if we draw observations in accordance with -the relative frequency of the observations, will we obtain the same -result in some asymptotic sense? The answer is yes. +of the observation \( X_i \)?

    -Instead of generating the histogram for the relative -frequency of the observation \( X_i \), just draw the values -\( (X_1^*,X_2^*,\cdots,X_n^*) \) with replacement from the vector -\( \boldsymbol{X} \). +If we draw observations in accordance with +the relative frequency of the observations, will we obtain the same +result in some asymptotic sense? The answer is yes.











    @@ -914,19 +917,19 @@ The independent bootstrap works like this:

    1. Draw with replacement \( n \) numbers for the observed variables \( \boldsymbol{x} = (x_1,x_2,\cdots,x_n) \).
    2. Define a vector \( \boldsymbol{x}^* \) containing the values which were drawn from \( \boldsymbol{x} \).
      3. -
    3. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\theta}^* \) by evaluating \( \widehat \theta \) under the observations \( \boldsymbol{x}^* \).
      4. +
    4. Using the vector \( \boldsymbol{x}^* \) compute \( \widehat{\beta}^* \) by evaluating \( \widehat \beta \) under the observations \( \boldsymbol{x}^* \).
    5. Repeat this process \( k \) times.
    When you are done, you can draw a histogram of the relative frequency -of \( \widehat \theta^* \). This is your estimate of the probability +of \( \widehat \beta^* \). This is your estimate of the probability distribution \( p(t) \). Using this probability distribution you can estimate any statistics thereof. In principle you never draw the -histogram of the relative frequency of \( \widehat{\theta}^* \). Instead +histogram of the relative frequency of \( \widehat{\beta}^* \). Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of \( \widehat -\theta \), apply the etsimator \( \widehat \sigma^2 \) to the values -\( \widehat \theta ^* \). +\beta \), apply the etsimator \( \widehat \sigma^2 \) to the values +\( \widehat \beta^* \).











    diff --git a/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz b/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz index 0c191f258..5edd4061c 100644 Binary files a/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz and b/doc/pub/week37/ipynb/ipynb-week37-src.tar.gz differ diff --git a/doc/pub/week37/ipynb/week37.ipynb b/doc/pub/week37/ipynb/week37.ipynb index 971467289..c075114df 100644 --- a/doc/pub/week37/ipynb/week37.ipynb +++ b/doc/pub/week37/ipynb/week37.ipynb @@ -10,7 +10,7 @@ " \n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", "\n", - "Date: **Sep 14, 2021**\n", + "Date: **Sep 15, 2021**\n", "\n", "Copyright 1999-2021, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license\n", "\n", @@ -593,7 +593,7 @@ "\n", "2. introduce resampling techniques like cross-validation, bootstrapping and jackknife and more\n", "\n", - "This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. \n", + "and discuss how to select a given model (one of the difficult parts in machine learning).\n", "\n", "\n", "\n", @@ -646,9 +646,9 @@ "**Statistical analysis.**\n", "\n", "\n", - "* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods\n", + "* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses.\n", "\n", - "* The results can be analysed with the same statistical tools as we would use analysing experimental data.\n", + "* The results can be analysed with the same statistical tools as we would use when analysing experimental data.\n", "\n", "* As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors.\n", "\n", @@ -677,7 +677,7 @@ "\n", "With all these analytical equations for both the OLS and Ridge\n", "regression, we will now outline how to assess a given model. This will\n", - "lead us to a discussion of the so-called bias-variance tradeoff (see\n", + "lead to a discussion of the so-called bias-variance tradeoff (see\n", "below) and so-called resampling methods.\n", "\n", "One of the quantities we have discussed as a way to measure errors is\n", @@ -704,7 +704,7 @@ "\n", "The jackknife is a special case of the independent bootstrap. Still, the jackknife was made\n", "popular prior to the independent bootstrap. And as the popularity of\n", - "the independent bootstrap soared, new variants, such as **the dependent bootstrap**.\n", + "the independent bootstrap soared, new variants, such as **the dependent bootstrap** have also been developed..\n", "\n", "The Jackknife and independent bootstrap work for\n", "independent, identically distributed random variables.\n", @@ -716,7 +716,7 @@ "\n", "## Resampling methods: Jackknife\n", "\n", - "The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n", + "The Jackknife works by making many replicas of the estimator $\\widehat{\\beta}$. \n", "The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n", "Let $\\boldsymbol{x}_i$ denote the vector" ] @@ -736,8 +736,8 @@ "source": [ "which equals the vector $\\boldsymbol{x}$ with the exception that observation\n", "number $i$ is left out. Using this notation, define\n", - "$\\widehat{\\theta}_i$ to be the estimator\n", - "$\\widehat{\\theta}$ computed using $\\vec{X}_i$. \n", + "$\\widehat{\\beta}_i$ to be the estimator\n", + "$\\widehat{\\beta}$ computed using $\\vec{X}_i$. \n", "\n", "\n", "## Jackknife code example" @@ -787,7 +787,7 @@ "metadata": {}, "source": [ "## Resampling methods: Bootstrap\n", - "Bootstrapping is a nonparametric approach to statistical inference\n", + "Bootstrapping is a non-parametric approach to statistical inference\n", "that substitutes computation for more traditional distributional\n", "assumptions and asymptotic results. Bootstrapping offers a number of\n", "advantages: \n", @@ -801,14 +801,18 @@ "\n", "\n", "\n", + "The textbook by [Davison on the Bootstrap Methods and their Applications](https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A) provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by [Efron and Tibshirani](https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317).\n", + "\n", + "\n", + "Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called **central limit theorem**.\n", "\n", "## Resampling methods: Bootstrap background\n", "\n", - "Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n", - "$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n", + "Since $\\widehat{\\beta} = \\widehat{\\beta}(\\boldsymbol{X})$ is a function of random variables,\n", + "$\\widehat{\\beta}$ itself must be a random variable. Thus it has\n", "a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n", "estimate $p(\\boldsymbol{t})$ by the relative frequency of\n", - "$\\widehat{\\theta}$. You can think of this as using a histogram\n", + "$\\widehat{\\beta}$. You can think of this as using a histogram\n", "in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n", "resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n", "estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n", @@ -817,18 +821,18 @@ "\n", "## Resampling methods: More Bootstrap background\n", "\n", - "In the case that $\\widehat{\\theta}$ has\n", + "In the case that $\\widehat{\\beta}$ has\n", "more than one component, and the components are independent, we use the\n", "same estimator on each component separately. If the probability\n", "density function of $X_i$, $p(x)$, had been known, then it would have\n", - "been straight forward to do this by: \n", + "been straightforward to do this by: \n", "1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n", "\n", - "2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n", + "2. Then using these numbers, we could compute a replica of $\\widehat{\\beta}$ called $\\widehat{\\beta}^*$. \n", "\n", - "By repeated use of (1) and (2), many\n", - "estimates of $\\widehat{\\theta}$ could have been obtained. The\n", - "idea is to use the relative frequency of $\\widehat{\\theta}^*$\n", + "By repeated use of the above two points, many\n", + "estimates of $\\widehat{\\beta}$ can be obtained. The\n", + "idea is to use the relative frequency of $\\widehat{\\beta}^*$\n", "(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n", "\n", "## Resampling methods: Bootstrap approach\n", @@ -838,15 +842,13 @@ "generated $X_1,X_2,\\cdots,X_n$, $p(x)$ is in general\n", "unknown. Therefore, [Efron in 1979](https://projecteuclid.org/euclid.aos/1176344552) asked the\n", "question: What if we replace $p(x)$ by the relative frequency\n", - "of the observation $X_i$; if we draw observations in accordance with\n", + "of the observation $X_i$?\n", + "\n", + "If we draw observations in accordance with\n", "the relative frequency of the observations, will we obtain the same\n", "result in some asymptotic sense? The answer is yes.\n", "\n", "\n", - "Instead of generating the histogram for the relative\n", - "frequency of the observation $X_i$, just draw the values\n", - "$(X_1^*,X_2^*,\\cdots,X_n^*)$ with replacement from the vector\n", - "$\\boldsymbol{X}$. \n", "\n", "## Resampling methods: Bootstrap steps\n", "\n", @@ -856,19 +858,19 @@ "\n", "2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n", "\n", - "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n", + "3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\beta}^*$ by evaluating $\\widehat \\beta$ under the observations $\\boldsymbol{x}^*$. \n", "\n", "4. Repeat this process $k$ times. \n", "\n", "When you are done, you can draw a histogram of the relative frequency\n", - "of $\\widehat \\theta^*$. This is your estimate of the probability\n", + "of $\\widehat \\beta^*$. This is your estimate of the probability\n", "distribution $p(t)$. Using this probability distribution you can\n", "estimate any statistics thereof. In principle you never draw the\n", - "histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n", + "histogram of the relative frequency of $\\widehat{\\beta}^*$. Instead\n", "you use the estimators corresponding to the statistic of interest. For\n", "example, if you are interested in estimating the variance of $\\widehat\n", - "\\theta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n", - "$\\widehat \\theta ^*$.\n", + "\\beta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n", + "$\\widehat \\beta^*$.\n", "\n", "\n", "## Code example for the Bootstrap method\n", diff --git a/doc/src/week37/week37.do.txt b/doc/src/week37/week37.do.txt index e139db917..c7c6fa37c 100644 --- a/doc/src/week37/week37.do.txt +++ b/doc/src/week37/week37.do.txt @@ -377,7 +377,7 @@ our regression analysis. In what follows we will o look at statistical properties, including a discussion of mean values, variance and the so-called bias-variance tradeoff o introduce resampling techniques like cross-validation, bootstrapping and jackknife and more -This will allow us to link the standard linear algebra methods we have discussed above to a statistical interpretation of the methods. +and discuss how to select a given model (one of the difficult parts in machine learning). @@ -434,8 +434,8 @@ bootstrap is widely used. ===== Why resampling methods ? ===== !bblock Statistical analysis -* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods -* The results can be analysed with the same statistical tools as we would use analysing experimental data. +* Our simulations can be treated as *computer experiments*. This is particularly the case for Monte Carlo methods which are widely used in statistical analyses. +* The results can be analysed with the same statistical tools as we would use when analysing experimental data. * As in all experiments, we are looking for expectation values and an estimate of how accurate they are, i.e., possible sources for errors. @@ -461,7 +461,7 @@ bootstrap is widely used. With all these analytical equations for both the OLS and Ridge regression, we will now outline how to assess a given model. This will -lead us to a discussion of the so-called bias-variance tradeoff (see +lead to a discussion of the so-called bias-variance tradeoff (see below) and so-called resampling methods. One of the quantities we have discussed as a way to measure errors is @@ -488,7 +488,7 @@ resampling methods are the _independent bootstrap_ and _the jackknife_. The jackknife is a special case of the independent bootstrap. Still, the jackknife was made popular prior to the independent bootstrap. And as the popularity of -the independent bootstrap soared, new variants, such as _the dependent bootstrap_. +the independent bootstrap soared, new variants, such as _the dependent bootstrap_ have also been developed.. The Jackknife and independent bootstrap work for independent, identically distributed random variables. @@ -501,7 +501,7 @@ need for bootstrapping. !split ===== Resampling methods: Jackknife ===== -The Jackknife works by making many replicas of the estimator $\widehat{\theta}$. +The Jackknife works by making many replicas of the estimator $\widehat{\beta}$. The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\bm{x} = (x_1,x_2,\cdots,X_n)$. Let $\bm{x}_i$ denote the vector !bt @@ -512,8 +512,8 @@ Let $\bm{x}_i$ denote the vector which equals the vector $\bm{x}$ with the exception that observation number $i$ is left out. Using this notation, define -$\widehat{\theta}_i$ to be the estimator -$\widehat{\theta}$ computed using $\vec{X}_i$. +$\widehat{\beta}_i$ to be the estimator +$\widehat{\beta}$ computed using $\vec{X}_i$. !split @@ -554,7 +554,7 @@ t = jackknife(x, stat) !split ===== Resampling methods: Bootstrap ===== !bblock -Bootstrapping is a nonparametric approach to statistical inference +Bootstrapping is a non-parametric approach to statistical inference that substitutes computation for more traditional distributional assumptions and asymptotic results. Bootstrapping offers a number of advantages: @@ -564,15 +564,19 @@ o It is possible to apply the bootstrap to statistics with sampling distribution o It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples). !eblock +The textbook by "Davison on the Bootstrap Methods and their Applications":"https://www.cambridge.org/core/books/bootstrap-methods-and-their-application/ED2FD043579F27952363566DC09CBD6A" provides many more insights and proofs. In this course we will take a more practical approach and use the results and theorems provided in the literature. For those interested in reading more about the bootstrap methods, we recommend the above text and the one by "Efron and Tibshirani":"https://www.routledge.com/An-Introduction-to-the-Bootstrap/Efron-Tibshirani/p/book/9780412042317". + + +Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called _central limit theorem_. !split ===== Resampling methods: Bootstrap background ===== -Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables, -$\widehat{\theta}$ itself must be a random variable. Thus it has +Since $\widehat{\beta} = \widehat{\beta}(\bm{X})$ is a function of random variables, +$\widehat{\beta}$ itself must be a random variable. Thus it has a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to estimate $p(\bm{t})$ by the relative frequency of -$\widehat{\theta}$. You can think of this as using a histogram +$\widehat{\beta}$. You can think of this as using a histogram in the place of $p(\bm{t})$. If the relative frequency closely resembles $p(\vec{t})$, then using numerics, it is straight forward to estimate all the interesting parameters of $p(\bm{t})$ using point @@ -582,17 +586,17 @@ estimators. !split ===== Resampling methods: More Bootstrap background ===== -In the case that $\widehat{\theta}$ has +In the case that $\widehat{\beta}$ has more than one component, and the components are independent, we use the same estimator on each component separately. If the probability density function of $X_i$, $p(x)$, had been known, then it would have -been straight forward to do this by: +been straightforward to do this by: o Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$. -o Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$. +o Then using these numbers, we could compute a replica of $\widehat{\beta}$ called $\widehat{\beta}^*$. -By repeated use of (1) and (2), many -estimates of $\widehat{\theta}$ could have been obtained. The -idea is to use the relative frequency of $\widehat{\theta}^*$ +By repeated use of the above two points, many +estimates of $\widehat{\beta}$ can be obtained. The +idea is to use the relative frequency of $\widehat{\beta}^*$ (think of a histogram) as an estimate of $p(\bm{t})$. !split @@ -603,15 +607,13 @@ unless there is enough information available about the process that generated $X_1,X_2,\cdots,X_n$, $p(x)$ is in general unknown. Therefore, "Efron in 1979":"https://projecteuclid.org/euclid.aos/1176344552" asked the question: What if we replace $p(x)$ by the relative frequency -of the observation $X_i$; if we draw observations in accordance with +of the observation $X_i$? + +If we draw observations in accordance with the relative frequency of the observations, will we obtain the same result in some asymptotic sense? The answer is yes. -Instead of generating the histogram for the relative -frequency of the observation $X_i$, just draw the values -$(X_1^*,X_2^*,\cdots,X_n^*)$ with replacement from the vector -$\bm{X}$. !split ===== Resampling methods: Bootstrap steps ===== @@ -620,18 +622,18 @@ The independent bootstrap works like this: o Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$. o Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$. -o Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$. +o Using the vector $\bm{x}^*$ compute $\widehat{\beta}^*$ by evaluating $\widehat \beta$ under the observations $\bm{x}^*$. o Repeat this process $k$ times. When you are done, you can draw a histogram of the relative frequency -of $\widehat \theta^*$. This is your estimate of the probability +of $\widehat \beta^*$. This is your estimate of the probability distribution $p(t)$. Using this probability distribution you can estimate any statistics thereof. In principle you never draw the -histogram of the relative frequency of $\widehat{\theta}^*$. Instead +histogram of the relative frequency of $\widehat{\beta}^*$. Instead you use the estimators corresponding to the statistic of interest. For example, if you are interested in estimating the variance of $\widehat -\theta$, apply the etsimator $\widehat \sigma^2$ to the values -$\widehat \theta ^*$. +\beta$, apply the etsimator $\widehat \sigma^2$ to the values +$\widehat \beta^*$. !split