lars/FYS-STK4155
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update on notes

Browse Source
This commit is contained in:
Morten Hjorth-Jensen
2021-09-13 15:21:30 +02:00
parent dd5df56326
commit bbb241c88f
60 changed files with 15229 additions and 14601 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/LectureNotes/chapter3.ipynb
+17 -86
View File
@@ -361,7 +361,7 @@
"need for bootstrapping. \n",
"\n",
"\n",
"The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n",
"The Jackknife works by making many replicas of the estimator $\\widehat{\\beta}$. \n",
"The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n",
"Let $\\boldsymbol{x}_i$ denote the vector"
]
@@ -381,8 +381,8 @@
"source": [
"which equals the vector $\\boldsymbol{x}$ with the exception that observation\n",
"number $i$ is left out. Using this notation, define\n",
"$\\widehat{\\theta}_i$ to be the estimator\n",
"$\\widehat{\\theta}$ computed using $\\vec{X}_i$."
"$\\widehat{\\beta}_i$ to be the estimator\n",
"$\\widehat{\\beta}$ computed using $\\vec{X}_i$."
]
},
{
@@ -442,11 +442,11 @@
"\n",
"4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).\n",
"\n",
"Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n",
"$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n",
"Since $\\widehat{\\beta} = \\widehat{\\beta}(\\boldsymbol{X})$ is a function of random variables,\n",
"$\\widehat{\\beta}$ itself must be a random variable. Thus it has\n",
"a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n",
"estimate $p(\\boldsymbol{t})$ by the relative frequency of\n",
"$\\widehat{\\theta}$. You can think of this as using a histogram\n",
"$\\widehat{\\beta}$. You can think of this as using a histogram\n",
"in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n",
"resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n",
"estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n",
@@ -454,18 +454,18 @@
"\n",
"\n",
"\n",
"In the case that $\\widehat{\\theta}$ has\n",
"In the case that $\\widehat{\\beta}$ has\n",
"more than one component, and the components are independent, we use the\n",
"same estimator on each component separately. If the probability\n",
"density function of $X_i$, $p(x)$, had been known, then it would have\n",
"been straight forward to do this by: \n",
"1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n",
"\n",
"2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n",
"2. Then using these numbers, we could compute a replica of $\\widehat{\\beta}$ called $\\widehat{\\beta}^*$. \n",
"\n",
"By repeated use of (1) and (2), many\n",
"estimates of $\\widehat{\\theta}$ could have been obtained. The\n",
"idea is to use the relative frequency of $\\widehat{\\theta}^*$\n",
"estimates of $\\widehat{\\beta}$ could have been obtained. The\n",
"idea is to use the relative frequency of $\\widehat{\\beta}^*$\n",
"(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n",
"\n",
"\n",
@@ -491,94 +491,23 @@
"\n",
"2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n",
"\n",
"3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n",
"3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\beta}^*$ by evaluating $\\widehat \\beta$ under the observations $\\boldsymbol{x}^*$. \n",
"\n",
"4. Repeat this process $k$ times. \n",
"\n",
"When you are done, you can draw a histogram of the relative frequency\n",
"of $\\widehat \\theta^*$. This is your estimate of the probability\n",
"of $\\widehat \\beta^*$. This is your estimate of the probability\n",
"distribution $p(t)$. Using this probability distribution you can\n",
"estimate any statistics thereof. In principle you never draw the\n",
"histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n",
"histogram of the relative frequency of $\\widehat{\\beta}^*$. Instead\n",
"you use the estimators corresponding to the statistic of interest. For\n",
"example, if you are interested in estimating the variance of $\\widehat\n",
"\\theta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n",
"$\\widehat \\theta ^*$.\n",
"\\beta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n",
"$\\widehat \\beta ^*$.\n",
"\n",
"\n",
"\n",
"The following code starts with a Gaussian distribution with mean value\n",
"$\\mu =100$ and variance $\\sigma=15$. We use this to generate the data\n",
"used in the bootstrap analysis. The bootstrap analysis returns a data\n",
"set after a given number of bootstrap operations (as many as we have\n",
"data points). This data set consists of estimated mean values for each\n",
"bootstrap operation. The histogram generated by the bootstrap method\n",
"shows that the distribution for these mean values is also a Gaussian,\n",
"centered around the mean value $\\mu=100$ but with standard deviation\n",
"$\\sigma/\\sqrt{n}$, where $n$ is the number of bootstrap samples (in\n",
"this case the same as the number of original data points). The value\n",
"of the standard deviation is what we expect from the central limit\n",
"theorem."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false,
"editable": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"from numpy import *\n",
"from numpy.random import randint, randn\n",
"from time import time\n",
"import matplotlib.mlab as mlab\n",
"import matplotlib.pyplot as plt\n",
"\n",
"# Returns mean of bootstrap samples \n",
"def stat(data):\n",
" return mean(data)\n",
"\n",
"# Bootstrap algorithm\n",
"def bootstrap(data, statistic, R):\n",
" t = zeros(R); n = len(data); inds = arange(n); t0 = time()\n",
" # non-parametric bootstrap \n",
" for i in range(R):\n",
" t[i] = statistic(data[randint(0,n,n)])\n",
"\n",
" # analysis \n",
" print(\"Runtime: %g sec\" % (time()-t0)); print(\"Bootstrap Statistics :\")\n",
" print(\"original bias std. error\")\n",
" print(\"%8g %8g %14g %15g\" % (statistic(data), std(data),mean(t),std(t)))\n",
" return t\n",
"\n",
"\n",
"mu, sigma = 100, 15\n",
"datapoints = 10000\n",
"x = mu + sigma*random.randn(datapoints)\n",
"# bootstrap returns the data sample \n",
"t = bootstrap(x, stat, datapoints)\n",
"# the histogram of the bootstrapped data \n",
"n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)\n",
"\n",
"# add a 'best fit' line \n",
"y = mlab.normpdf( binsboot, mean(t), std(t))\n",
"lt = plt.plot(binsboot, y, 'r--', linewidth=1)\n",
"plt.xlabel('Smarts')\n",
"plt.ylabel('Probability')\n",
"plt.axis([99.5, 100.6, 0, 3.0])\n",
"plt.grid(True)\n",
"\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The bias-variance tradeoff\n",
"\n",
"\n",
@@ -710,6 +639,8 @@
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",