update on notes
This commit is contained in:
Morten Hjorth-Jensen
@@ -244,7 +244,7 @@ variance of $\overline{X}$ (which often is the case), then there is no
|
||||
need for bootstrapping.
|
||||
|
||||
|
||||
The Jackknife works by making many replicas of the estimator $\widehat{\theta}$.
|
||||
The Jackknife works by making many replicas of the estimator $\widehat{\beta}$.
|
||||
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\bm{x} = (x_1,x_2,\cdots,X_n)$.
|
||||
Let $\bm{x}_i$ denote the vector
|
||||
!bt
|
||||
@@ -255,8 +255,8 @@ Let $\bm{x}_i$ denote the vector
|
||||
|
||||
which equals the vector $\bm{x}$ with the exception that observation
|
||||
number $i$ is left out. Using this notation, define
|
||||
$\widehat{\theta}_i$ to be the estimator
|
||||
$\widehat{\theta}$ computed using $\vec{X}_i$.
|
||||
$\widehat{\beta}_i$ to be the estimator
|
||||
$\widehat{\beta}$ computed using $\vec{X}_i$.
|
||||
|
||||
|
||||
|
||||
@@ -306,11 +306,11 @@ o It is relatively simple to apply the bootstrap to complex data-collection plan
|
||||
|
||||
|
||||
|
||||
Since $\widehat{\theta} = \widehat{\theta}(\bm{X})$ is a function of random variables,
|
||||
$\widehat{\theta}$ itself must be a random variable. Thus it has
|
||||
Since $\widehat{\beta} = \widehat{\beta}(\bm{X})$ is a function of random variables,
|
||||
$\widehat{\beta}$ itself must be a random variable. Thus it has
|
||||
a pdf, call this function $p(\bm{t})$. The aim of the bootstrap is to
|
||||
estimate $p(\bm{t})$ by the relative frequency of
|
||||
$\widehat{\theta}$. You can think of this as using a histogram
|
||||
$\widehat{\beta}$. You can think of this as using a histogram
|
||||
in the place of $p(\bm{t})$. If the relative frequency closely
|
||||
resembles $p(\vec{t})$, then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of $p(\bm{t})$ using point
|
||||
@@ -318,17 +318,17 @@ estimators.
|
||||
|
||||
|
||||
|
||||
In the case that $\widehat{\theta}$ has
|
||||
In the case that $\widehat{\beta}$ has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of $X_i$, $p(x)$, had been known, then it would have
|
||||
been straight forward to do this by:
|
||||
o Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$.
|
||||
o Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$.
|
||||
o Then using these numbers, we could compute a replica of $\widehat{\beta}$ called $\widehat{\beta}^*$.
|
||||
|
||||
By repeated use of (1) and (2), many
|
||||
estimates of $\widehat{\theta}$ could have been obtained. The
|
||||
idea is to use the relative frequency of $\widehat{\theta}^*$
|
||||
estimates of $\widehat{\beta}$ could have been obtained. The
|
||||
idea is to use the relative frequency of $\widehat{\beta}^*$
|
||||
(think of a histogram) as an estimate of $p(\bm{t})$.
|
||||
|
||||
|
||||
@@ -352,80 +352,21 @@ The independent bootstrap works like this:
|
||||
|
||||
o Draw with replacement $n$ numbers for the observed variables $\bm{x} = (x_1,x_2,\cdots,x_n)$.
|
||||
o Define a vector $\bm{x}^*$ containing the values which were drawn from $\bm{x}$.
|
||||
o Using the vector $\bm{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\bm{x}^*$.
|
||||
o Using the vector $\bm{x}^*$ compute $\widehat{\beta}^*$ by evaluating $\widehat \beta$ under the observations $\bm{x}^*$.
|
||||
o Repeat this process $k$ times.
|
||||
|
||||
When you are done, you can draw a histogram of the relative frequency
|
||||
of $\widehat \theta^*$. This is your estimate of the probability
|
||||
of $\widehat \beta^*$. This is your estimate of the probability
|
||||
distribution $p(t)$. Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of $\widehat{\theta}^*$. Instead
|
||||
histogram of the relative frequency of $\widehat{\beta}^*$. Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of $\widehat
|
||||
\theta$, apply the etsimator $\widehat \sigma^2$ to the values
|
||||
$\widehat \theta ^*$.
|
||||
\beta$, apply the etsimator $\widehat \sigma^2$ to the values
|
||||
$\widehat \beta ^*$.
|
||||
|
||||
|
||||
|
||||
The following code starts with a Gaussian distribution with mean value
|
||||
$\mu =100$ and variance $\sigma=15$. We use this to generate the data
|
||||
used in the bootstrap analysis. The bootstrap analysis returns a data
|
||||
set after a given number of bootstrap operations (as many as we have
|
||||
data points). This data set consists of estimated mean values for each
|
||||
bootstrap operation. The histogram generated by the bootstrap method
|
||||
shows that the distribution for these mean values is also a Gaussian,
|
||||
centered around the mean value $\mu=100$ but with standard deviation
|
||||
$\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in
|
||||
this case the same as the number of original data points). The value
|
||||
of the standard deviation is what we expect from the central limit
|
||||
theorem.
|
||||
|
||||
|
||||
!bc pycod
|
||||
from numpy import *
|
||||
from numpy.random import randint, randn
|
||||
from time import time
|
||||
import matplotlib.mlab as mlab
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
# Returns mean of bootstrap samples
|
||||
def stat(data):
|
||||
return mean(data)
|
||||
|
||||
# Bootstrap algorithm
|
||||
def bootstrap(data, statistic, R):
|
||||
t = zeros(R); n = len(data); inds = arange(n); t0 = time()
|
||||
# non-parametric bootstrap
|
||||
for i in range(R):
|
||||
t[i] = statistic(data[randint(0,n,n)])
|
||||
|
||||
# analysis
|
||||
print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
|
||||
print("original bias std. error")
|
||||
print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
|
||||
return t
|
||||
|
||||
|
||||
mu, sigma = 100, 15
|
||||
datapoints = 10000
|
||||
x = mu + sigma*random.randn(datapoints)
|
||||
# bootstrap returns the data sample
|
||||
t = bootstrap(x, stat, datapoints)
|
||||
# the histogram of the bootstrapped data
|
||||
n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)
|
||||
|
||||
# add a 'best fit' line
|
||||
y = mlab.normpdf( binsboot, mean(t), std(t))
|
||||
lt = plt.plot(binsboot, y, 'r--', linewidth=1)
|
||||
plt.xlabel('Smarts')
|
||||
plt.ylabel('Probability')
|
||||
plt.axis([99.5, 100.6, 0, 3.0])
|
||||
plt.grid(True)
|
||||
|
||||
plt.show()
|
||||
|
||||
!ec
|
||||
|
||||
|
||||
===== The bias-variance tradeoff =====
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
@@ -361,7 +361,7 @@
|
||||
"need for bootstrapping. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n",
|
||||
"The Jackknife works by making many replicas of the estimator $\\widehat{\\beta}$. \n",
|
||||
"The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n",
|
||||
"Let $\\boldsymbol{x}_i$ denote the vector"
|
||||
]
|
||||
@@ -381,8 +381,8 @@
|
||||
"source": [
|
||||
"which equals the vector $\\boldsymbol{x}$ with the exception that observation\n",
|
||||
"number $i$ is left out. Using this notation, define\n",
|
||||
"$\\widehat{\\theta}_i$ to be the estimator\n",
|
||||
"$\\widehat{\\theta}$ computed using $\\vec{X}_i$."
|
||||
"$\\widehat{\\beta}_i$ to be the estimator\n",
|
||||
"$\\widehat{\\beta}$ computed using $\\vec{X}_i$."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -442,11 +442,11 @@
|
||||
"\n",
|
||||
"4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).\n",
|
||||
"\n",
|
||||
"Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n",
|
||||
"$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n",
|
||||
"Since $\\widehat{\\beta} = \\widehat{\\beta}(\\boldsymbol{X})$ is a function of random variables,\n",
|
||||
"$\\widehat{\\beta}$ itself must be a random variable. Thus it has\n",
|
||||
"a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n",
|
||||
"estimate $p(\\boldsymbol{t})$ by the relative frequency of\n",
|
||||
"$\\widehat{\\theta}$. You can think of this as using a histogram\n",
|
||||
"$\\widehat{\\beta}$. You can think of this as using a histogram\n",
|
||||
"in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n",
|
||||
"resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n",
|
||||
"estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n",
|
||||
@@ -454,18 +454,18 @@
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"In the case that $\\widehat{\\theta}$ has\n",
|
||||
"In the case that $\\widehat{\\beta}$ has\n",
|
||||
"more than one component, and the components are independent, we use the\n",
|
||||
"same estimator on each component separately. If the probability\n",
|
||||
"density function of $X_i$, $p(x)$, had been known, then it would have\n",
|
||||
"been straight forward to do this by: \n",
|
||||
"1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n",
|
||||
"\n",
|
||||
"2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n",
|
||||
"2. Then using these numbers, we could compute a replica of $\\widehat{\\beta}$ called $\\widehat{\\beta}^*$. \n",
|
||||
"\n",
|
||||
"By repeated use of (1) and (2), many\n",
|
||||
"estimates of $\\widehat{\\theta}$ could have been obtained. The\n",
|
||||
"idea is to use the relative frequency of $\\widehat{\\theta}^*$\n",
|
||||
"estimates of $\\widehat{\\beta}$ could have been obtained. The\n",
|
||||
"idea is to use the relative frequency of $\\widehat{\\beta}^*$\n",
|
||||
"(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -491,94 +491,23 @@
|
||||
"\n",
|
||||
"2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n",
|
||||
"\n",
|
||||
"3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n",
|
||||
"3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\beta}^*$ by evaluating $\\widehat \\beta$ under the observations $\\boldsymbol{x}^*$. \n",
|
||||
"\n",
|
||||
"4. Repeat this process $k$ times. \n",
|
||||
"\n",
|
||||
"When you are done, you can draw a histogram of the relative frequency\n",
|
||||
"of $\\widehat \\theta^*$. This is your estimate of the probability\n",
|
||||
"of $\\widehat \\beta^*$. This is your estimate of the probability\n",
|
||||
"distribution $p(t)$. Using this probability distribution you can\n",
|
||||
"estimate any statistics thereof. In principle you never draw the\n",
|
||||
"histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n",
|
||||
"histogram of the relative frequency of $\\widehat{\\beta}^*$. Instead\n",
|
||||
"you use the estimators corresponding to the statistic of interest. For\n",
|
||||
"example, if you are interested in estimating the variance of $\\widehat\n",
|
||||
"\\theta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n",
|
||||
"$\\widehat \\theta ^*$.\n",
|
||||
"\\beta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n",
|
||||
"$\\widehat \\beta ^*$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The following code starts with a Gaussian distribution with mean value\n",
|
||||
"$\\mu =100$ and variance $\\sigma=15$. We use this to generate the data\n",
|
||||
"used in the bootstrap analysis. The bootstrap analysis returns a data\n",
|
||||
"set after a given number of bootstrap operations (as many as we have\n",
|
||||
"data points). This data set consists of estimated mean values for each\n",
|
||||
"bootstrap operation. The histogram generated by the bootstrap method\n",
|
||||
"shows that the distribution for these mean values is also a Gaussian,\n",
|
||||
"centered around the mean value $\\mu=100$ but with standard deviation\n",
|
||||
"$\\sigma/\\sqrt{n}$, where $n$ is the number of bootstrap samples (in\n",
|
||||
"this case the same as the number of original data points). The value\n",
|
||||
"of the standard deviation is what we expect from the central limit\n",
|
||||
"theorem."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%matplotlib inline\n",
|
||||
"\n",
|
||||
"from numpy import *\n",
|
||||
"from numpy.random import randint, randn\n",
|
||||
"from time import time\n",
|
||||
"import matplotlib.mlab as mlab\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"# Returns mean of bootstrap samples \n",
|
||||
"def stat(data):\n",
|
||||
" return mean(data)\n",
|
||||
"\n",
|
||||
"# Bootstrap algorithm\n",
|
||||
"def bootstrap(data, statistic, R):\n",
|
||||
" t = zeros(R); n = len(data); inds = arange(n); t0 = time()\n",
|
||||
" # non-parametric bootstrap \n",
|
||||
" for i in range(R):\n",
|
||||
" t[i] = statistic(data[randint(0,n,n)])\n",
|
||||
"\n",
|
||||
" # analysis \n",
|
||||
" print(\"Runtime: %g sec\" % (time()-t0)); print(\"Bootstrap Statistics :\")\n",
|
||||
" print(\"original bias std. error\")\n",
|
||||
" print(\"%8g %8g %14g %15g\" % (statistic(data), std(data),mean(t),std(t)))\n",
|
||||
" return t\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"mu, sigma = 100, 15\n",
|
||||
"datapoints = 10000\n",
|
||||
"x = mu + sigma*random.randn(datapoints)\n",
|
||||
"# bootstrap returns the data sample \n",
|
||||
"t = bootstrap(x, stat, datapoints)\n",
|
||||
"# the histogram of the bootstrapped data \n",
|
||||
"n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)\n",
|
||||
"\n",
|
||||
"# add a 'best fit' line \n",
|
||||
"y = mlab.normpdf( binsboot, mean(t), std(t))\n",
|
||||
"lt = plt.plot(binsboot, y, 'r--', linewidth=1)\n",
|
||||
"plt.xlabel('Smarts')\n",
|
||||
"plt.ylabel('Probability')\n",
|
||||
"plt.axis([99.5, 100.6, 0, 3.0])\n",
|
||||
"plt.grid(True)\n",
|
||||
"\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## The bias-variance tradeoff\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -710,6 +639,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%matplotlib inline\n",
|
||||
"\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import numpy as np\n",
|
||||
"from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -22,91 +22,99 @@ For the reading assignments we use the following abbreviations:
|
||||
- HTF: Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning
|
||||
- AG: Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow
|
||||
|
||||
### Week 35 August 23-27
|
||||
### Week 34 August 23-27
|
||||
- Lab Wednesday: Introduction to software and repetition of Python Programming
|
||||
- Lecture Thursday: Introduction to the course, what is Machine Learning and introduction to Linear Regression
|
||||
- Lecture Thursday: Introduction to the course, what is Machine Learning and introduction to Linear Regression.
|
||||
- Video of Lecture August 26, 2021 at https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureThursdayAugust26.mp4?vrtx=view-as-webpage
|
||||
- Lecture Friday: Basics of Linear Regression
|
||||
- Reading recommendations: Refresh linear algebra, GBC chapters 1 and 2. CMB sections 1.1 and 3.1. HTF chapters 2 and 3. Install scikit-learn. See lecture notes for week 35 at https://compphysics.github.io/MachineLearning/doc/web/course.html
|
||||
- Video of Lecture August 27, 2021 at https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureThursdayAugust27.mp4?vrtx=view-as-webpage
|
||||
- Reading recommendations: Refresh linear algebra, GBC chapters 1 and 2. CMB sections 1.1 and 3.1. HTF chapters 2 and 3. Install scikit-learn. See lecture notes for week 34 at https://compphysics.github.io/MachineLearning/doc/web/course.html
|
||||
|
||||
### Week 36 August 30-September 3
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Linear Regression, from ordinary linear regression to Ridge and Lasso regression, linear algebra analysis, examples and discussions of codes
|
||||
- Lecture Friday: Linear Regression, Linear algebra and Ridge and Lasso Regression, linear algebra analysis, examples and discussions of codes
|
||||
- Reading recommendations: See lecture notes for week 36 at https://compphysics.github.io/MachineLearning/doc/web/course.html. HTF chapter 3. GBC chapters 1 and and sections 3.1-3.11 and 5.1 and CMB sections 1.1 and 3.1
|
||||
### Week 35 August 30-September 3
|
||||
- Lab Wednesday: Work on exercises 1-3 for week 35
|
||||
- Thursday: Review of ordinary Least Squares with applications and discussion of Ridge Regression and Singular Value Decomposition
|
||||
- Video of lecture Thursday at https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember2.mp4?vrtx=view-as-webpage".
|
||||
- Friday: Analysis of Ridge and Lasso Regression and links with Singular Value Decomposition
|
||||
- Video of lecture Friday at https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember3.mp4?vrtx=view-as-webpage"
|
||||
- Reading recommendations: See lecture notes for week 35 at https://compphysics.github.io/MachineLearning/doc/web/course.html. HTF chapter 3. GBC chapters 1 and and sections 3.1-3.11 and 5.1 and CMB sections 1.1 and 3.1
|
||||
|
||||
### Week 37 September 6-10
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Statistical interpretation of Linear Regression
|
||||
- Lecture Friday: Bias-Variance tradeoff
|
||||
- Reading recommendations: See lecture notes for week 37 at https://compphysics.github.io/MachineLearning/doc/web/course.html. GBC sections 5.2-5.5, CMB section 3.2
|
||||
- Chapter
|
||||
### Week 38 September 13-17
|
||||
|
||||
### Week 36 September 6-10
|
||||
- Lab Wednesday: Exercises 1 and 2 from week 36
|
||||
- Lecture Thursday: Summary from last week on SVD, Statistics, probability theory and linear regression
|
||||
- Video of Lecture https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember9.mp4?vrtx=view-as-webpage".
|
||||
- Friday: Linear Regression and links with Statistics, Resampling methods and presentation of first project.
|
||||
- Video of Lecture at https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureSeptember10.mp4?vrtx=view-as-webpage"
|
||||
|
||||
- Recommended Reading: Lectures on Regression, Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1, Hastie et al chapter 3
|
||||
|
||||
### Week 37 September 13-17
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Resampling methods, cross-validation and Bootstrap
|
||||
- Lecture Friday: More on Resampling methods and summary of linear regression
|
||||
- Reading recommendations: See lecture notes for week 38 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
|
||||
- Reading recommendations: See lecture notes for week 37 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
|
||||
- Chapter
|
||||
### Week 39 September 20-24
|
||||
### Week 38 September 20-24
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Classification problems and Logistic Regression, from binary cases to several categories
|
||||
- Lecture Friday: Logistic Regression and gradient optimization
|
||||
- Reading recommendations: See lecture notes for week 39 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
|
||||
- Reading recommendations: See lecture notes for week 38 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
|
||||
- Chapter
|
||||
### Week 40 September 27- October 1
|
||||
### Week 39 September 27- October 1
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Gradient Optimization methods
|
||||
- Lecture Friday: Deep Learning and Neural Networks
|
||||
- Reading recommendations: See lecture notes for week 40 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
|
||||
- Reading recommendations: See lecture notes for week 39 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
|
||||
- Chapter
|
||||
### Week 41 October 4-8
|
||||
### Week 40 October 4-8
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Writing a feed-forward Neural Network code for regression and classification
|
||||
- Lecture Friday: Deep Learning and TensorFlow and Keras
|
||||
- Reading recommendations:
|
||||
- Chapter
|
||||
### Week 42 October 11-15
|
||||
### Week 41 October 11-15
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Deep learning and Neural Networks
|
||||
- Lecture Friday: Convolutional Neural Networks, basic elements
|
||||
- Reading recommendations:
|
||||
- GoodFellow et al, Chapter 9
|
||||
### Week 43 October 18-22
|
||||
### Week 42 October 18-22
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Convolutional Neural Networks and classification problems
|
||||
- Lecture Friday: Convolutional Neural Networks and classification problems
|
||||
- Reading recommendations:
|
||||
- Chapter
|
||||
### Week 44 October 25-29
|
||||
### Week 43 October 25-29
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Recurrent Neural Networks
|
||||
- Lecture Friday: Recurrent Neural Networks and time series
|
||||
- Reading recommendations:
|
||||
- Chapter
|
||||
### Week 45 November 1-5
|
||||
### Week 44 November 1-5
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Decision trees, classification and regression
|
||||
- Lecture Friday: Decision trees, basic algorithms
|
||||
- Reading recommendations:
|
||||
- Chapter
|
||||
### Week 46 November 8-12
|
||||
### Week 45 November 8-12
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Ensemble methods, bagging and random forests
|
||||
- Lecture Friday: Boosting and gradient boosting
|
||||
- Reading recommendations:
|
||||
- Chapter
|
||||
### Week 47 November 15-19
|
||||
### Week 46 November 15-19
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday:
|
||||
- Lecture Friday: Unsupervised Learning, k-means
|
||||
- Reading recommendations:
|
||||
- Chapter
|
||||
### Week 48 November 22-26
|
||||
### Week 47 November 22-26
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Unsupervised Learning, Principal Component Analysis (PCA)
|
||||
- Lecture Friday: Unsupervised Learning and PCA and Clustering
|
||||
- Reading recommendations:
|
||||
- Chapter
|
||||
### Week 49 November 29- December 2
|
||||
### Week 48 November 29- December 2
|
||||
- Lab Wednesday:
|
||||
- Lecture Thursday: Unsupervised Learning and Clustering
|
||||
- Lecture Friday: Summary of course
|
||||
|
||||
@@ -157,6 +157,11 @@
|
||||
6. Logistic Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapteroptimization.html">
|
||||
7. Optimization, the central part of any Machine Learning algortithm
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter5.html">
|
||||
8. Support Vector Machines, overarching aims
|
||||
@@ -542,7 +547,7 @@ satisfied, the methods will fail. Yet, it should be said that if the data are
|
||||
independent, identically distributed, and we only want to estimate the
|
||||
variance of <span class="math notranslate nohighlight">\(\overline{X}\)</span> (which often is the case), then there is no
|
||||
need for bootstrapping.</p>
|
||||
<p>The Jackknife works by making many replicas of the estimator <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span>.
|
||||
<p>The Jackknife works by making many replicas of the estimator <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span>.
|
||||
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values <span class="math notranslate nohighlight">\(\boldsymbol{x} = (x_1,x_2,\cdots,X_n)\)</span>.
|
||||
Let <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> denote the vector</p>
|
||||
<div class="math notranslate nohighlight">
|
||||
@@ -551,8 +556,8 @@ Let <span class="math notranslate nohighlight">\(\boldsymbol{x}_i\)</span> denot
|
||||
\]</div>
|
||||
<p>which equals the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span> with the exception that observation
|
||||
number <span class="math notranslate nohighlight">\(i\)</span> is left out. Using this notation, define
|
||||
<span class="math notranslate nohighlight">\(\widehat{\theta}_i\)</span> to be the estimator
|
||||
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> computed using <span class="math notranslate nohighlight">\(\vec{X}_i\)</span>.</p>
|
||||
<span class="math notranslate nohighlight">\(\widehat{\beta}_i\)</span> to be the estimator
|
||||
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> computed using <span class="math notranslate nohighlight">\(\vec{X}_i\)</span>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="o">*</span>
|
||||
@@ -587,10 +592,10 @@ number <span class="math notranslate nohighlight">\(i\)</span> is left out. Usin
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.138472 sec
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 0.1375 sec
|
||||
Jackknife Statistics :
|
||||
original bias std. error
|
||||
100.094 100.084 0.150306
|
||||
100.029 100.019 0.150581
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -607,27 +612,27 @@ advantages:</p>
|
||||
<li><p>It is possible to apply the bootstrap to statistics with sampling distributions that are difficult to derive, even asymptotically.</p></li>
|
||||
<li><p>It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).</p></li>
|
||||
</ol>
|
||||
<p>Since <span class="math notranslate nohighlight">\(\widehat{\theta} = \widehat{\theta}(\boldsymbol{X})\)</span> is a function of random variables,
|
||||
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> itself must be a random variable. Thus it has
|
||||
<p>Since <span class="math notranslate nohighlight">\(\widehat{\beta} = \widehat{\beta}(\boldsymbol{X})\)</span> is a function of random variables,
|
||||
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> itself must be a random variable. Thus it has
|
||||
a pdf, call this function <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. The aim of the bootstrap is to
|
||||
estimate <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> by the relative frequency of
|
||||
<span class="math notranslate nohighlight">\(\widehat{\theta}\)</span>. You can think of this as using a histogram
|
||||
<span class="math notranslate nohighlight">\(\widehat{\beta}\)</span>. You can think of this as using a histogram
|
||||
in the place of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>. If the relative frequency closely
|
||||
resembles <span class="math notranslate nohighlight">\(p(\vec{t})\)</span>, then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span> using point
|
||||
estimators.</p>
|
||||
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> has
|
||||
<p>In the case that <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of <span class="math notranslate nohighlight">\(X_i\)</span>, <span class="math notranslate nohighlight">\(p(x)\)</span>, had been known, then it would have
|
||||
been straight forward to do this by:</p>
|
||||
<ol class="simple">
|
||||
<li><p>Drawing lots of numbers from <span class="math notranslate nohighlight">\(p(x)\)</span>, suppose we call one such set of numbers <span class="math notranslate nohighlight">\((X_1^*, X_2^*, \cdots, X_n^*)\)</span>.</p></li>
|
||||
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>.</p></li>
|
||||
<li><p>Then using these numbers, we could compute a replica of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> called <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>.</p></li>
|
||||
</ol>
|
||||
<p>By repeated use of (1) and (2), many
|
||||
estimates of <span class="math notranslate nohighlight">\(\widehat{\theta}\)</span> could have been obtained. The
|
||||
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>
|
||||
estimates of <span class="math notranslate nohighlight">\(\widehat{\beta}\)</span> could have been obtained. The
|
||||
idea is to use the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>
|
||||
(think of a histogram) as an estimate of <span class="math notranslate nohighlight">\(p(\boldsymbol{t})\)</span>.</p>
|
||||
<p>But
|
||||
unless there is enough information available about the process that
|
||||
@@ -645,128 +650,18 @@ frequency of the observation <span class="math notranslate nohighlight">\(X_i\)<
|
||||
<ol class="simple">
|
||||
<li><p>Draw with replacement <span class="math notranslate nohighlight">\(n\)</span> numbers for the observed variables <span class="math notranslate nohighlight">\(\boldsymbol{x} = (x_1,x_2,\cdots,x_n)\)</span>.</p></li>
|
||||
<li><p>Define a vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> containing the values which were drawn from <span class="math notranslate nohighlight">\(\boldsymbol{x}\)</span>.</p></li>
|
||||
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \theta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
|
||||
<li><p>Using the vector <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span> compute <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span> by evaluating <span class="math notranslate nohighlight">\(\widehat \beta\)</span> under the observations <span class="math notranslate nohighlight">\(\boldsymbol{x}^*\)</span>.</p></li>
|
||||
<li><p>Repeat this process <span class="math notranslate nohighlight">\(k\)</span> times.</p></li>
|
||||
</ol>
|
||||
<p>When you are done, you can draw a histogram of the relative frequency
|
||||
of <span class="math notranslate nohighlight">\(\widehat \theta^*\)</span>. This is your estimate of the probability
|
||||
of <span class="math notranslate nohighlight">\(\widehat \beta^*\)</span>. This is your estimate of the probability
|
||||
distribution <span class="math notranslate nohighlight">\(p(t)\)</span>. Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\theta}^*\)</span>. Instead
|
||||
histogram of the relative frequency of <span class="math notranslate nohighlight">\(\widehat{\beta}^*\)</span>. Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of <span class="math notranslate nohighlight">\(\widehat
|
||||
\theta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
|
||||
<span class="math notranslate nohighlight">\(\widehat \theta ^*\)</span>.</p>
|
||||
<p>The following code starts with a Gaussian distribution with mean value
|
||||
<span class="math notranslate nohighlight">\(\mu =100\)</span> and variance <span class="math notranslate nohighlight">\(\sigma=15\)</span>. We use this to generate the data
|
||||
used in the bootstrap analysis. The bootstrap analysis returns a data
|
||||
set after a given number of bootstrap operations (as many as we have
|
||||
data points). This data set consists of estimated mean values for each
|
||||
bootstrap operation. The histogram generated by the bootstrap method
|
||||
shows that the distribution for these mean values is also a Gaussian,
|
||||
centered around the mean value <span class="math notranslate nohighlight">\(\mu=100\)</span> but with standard deviation
|
||||
<span class="math notranslate nohighlight">\(\sigma/\sqrt{n}\)</span>, where <span class="math notranslate nohighlight">\(n\)</span> is the number of bootstrap samples (in
|
||||
this case the same as the number of original data points). The value
|
||||
of the standard deviation is what we expect from the central limit
|
||||
theorem.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
|
||||
|
||||
<span class="kn">from</span> <span class="nn">numpy</span> <span class="kn">import</span> <span class="o">*</span>
|
||||
<span class="kn">from</span> <span class="nn">numpy.random</span> <span class="kn">import</span> <span class="n">randint</span><span class="p">,</span> <span class="n">randn</span>
|
||||
<span class="kn">from</span> <span class="nn">time</span> <span class="kn">import</span> <span class="n">time</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.mlab</span> <span class="k">as</span> <span class="nn">mlab</span>
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
|
||||
<span class="c1"># Returns mean of bootstrap samples </span>
|
||||
<span class="k">def</span> <span class="nf">stat</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
|
||||
<span class="k">return</span> <span class="n">mean</span><span class="p">(</span><span class="n">data</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># Bootstrap algorithm</span>
|
||||
<span class="k">def</span> <span class="nf">bootstrap</span><span class="p">(</span><span class="n">data</span><span class="p">,</span> <span class="n">statistic</span><span class="p">,</span> <span class="n">R</span><span class="p">):</span>
|
||||
<span class="n">t</span> <span class="o">=</span> <span class="n">zeros</span><span class="p">(</span><span class="n">R</span><span class="p">);</span> <span class="n">n</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">data</span><span class="p">);</span> <span class="n">inds</span> <span class="o">=</span> <span class="n">arange</span><span class="p">(</span><span class="n">n</span><span class="p">);</span> <span class="n">t0</span> <span class="o">=</span> <span class="n">time</span><span class="p">()</span>
|
||||
<span class="c1"># non-parametric bootstrap </span>
|
||||
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">R</span><span class="p">):</span>
|
||||
<span class="n">t</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="n">statistic</span><span class="p">(</span><span class="n">data</span><span class="p">[</span><span class="n">randint</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span><span class="n">n</span><span class="p">,</span><span class="n">n</span><span class="p">)])</span>
|
||||
|
||||
<span class="c1"># analysis </span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"Runtime: </span><span class="si">%g</span><span class="s2"> sec"</span> <span class="o">%</span> <span class="p">(</span><span class="n">time</span><span class="p">()</span><span class="o">-</span><span class="n">t0</span><span class="p">));</span> <span class="nb">print</span><span class="p">(</span><span class="s2">"Bootstrap Statistics :"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"original bias std. error"</span><span class="p">)</span>
|
||||
<span class="nb">print</span><span class="p">(</span><span class="s2">"</span><span class="si">%8g</span><span class="s2"> </span><span class="si">%8g</span><span class="s2"> </span><span class="si">%14g</span><span class="s2"> </span><span class="si">%15g</span><span class="s2">"</span> <span class="o">%</span> <span class="p">(</span><span class="n">statistic</span><span class="p">(</span><span class="n">data</span><span class="p">),</span> <span class="n">std</span><span class="p">(</span><span class="n">data</span><span class="p">),</span><span class="n">mean</span><span class="p">(</span><span class="n">t</span><span class="p">),</span><span class="n">std</span><span class="p">(</span><span class="n">t</span><span class="p">)))</span>
|
||||
<span class="k">return</span> <span class="n">t</span>
|
||||
|
||||
|
||||
<span class="n">mu</span><span class="p">,</span> <span class="n">sigma</span> <span class="o">=</span> <span class="mi">100</span><span class="p">,</span> <span class="mi">15</span>
|
||||
<span class="n">datapoints</span> <span class="o">=</span> <span class="mi">10000</span>
|
||||
<span class="n">x</span> <span class="o">=</span> <span class="n">mu</span> <span class="o">+</span> <span class="n">sigma</span><span class="o">*</span><span class="n">random</span><span class="o">.</span><span class="n">randn</span><span class="p">(</span><span class="n">datapoints</span><span class="p">)</span>
|
||||
<span class="c1"># bootstrap returns the data sample </span>
|
||||
<span class="n">t</span> <span class="o">=</span> <span class="n">bootstrap</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">stat</span><span class="p">,</span> <span class="n">datapoints</span><span class="p">)</span>
|
||||
<span class="c1"># the histogram of the bootstrapped data </span>
|
||||
<span class="n">n</span><span class="p">,</span> <span class="n">binsboot</span><span class="p">,</span> <span class="n">patches</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="n">normed</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">facecolor</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.75</span><span class="p">)</span>
|
||||
|
||||
<span class="c1"># add a 'best fit' line </span>
|
||||
<span class="n">y</span> <span class="o">=</span> <span class="n">mlab</span><span class="o">.</span><span class="n">normpdf</span><span class="p">(</span> <span class="n">binsboot</span><span class="p">,</span> <span class="n">mean</span><span class="p">(</span><span class="n">t</span><span class="p">),</span> <span class="n">std</span><span class="p">(</span><span class="n">t</span><span class="p">))</span>
|
||||
<span class="n">lt</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">plot</span><span class="p">(</span><span class="n">binsboot</span><span class="p">,</span> <span class="n">y</span><span class="p">,</span> <span class="s1">'r--'</span><span class="p">,</span> <span class="n">linewidth</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">xlabel</span><span class="p">(</span><span class="s1">'Smarts'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">ylabel</span><span class="p">(</span><span class="s1">'Probability'</span><span class="p">)</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">axis</span><span class="p">([</span><span class="mf">99.5</span><span class="p">,</span> <span class="mf">100.6</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="mf">3.0</span><span class="p">])</span>
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">grid</span><span class="p">(</span><span class="kc">True</span><span class="p">)</span>
|
||||
|
||||
<span class="n">plt</span><span class="o">.</span><span class="n">show</span><span class="p">()</span>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Runtime: 2.06834 sec
|
||||
Bootstrap Statistics :
|
||||
original bias std. error
|
||||
99.9933 15.0354 99.9941 0.149739
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output traceback highlight-ipythontb notranslate"><div class="highlight"><pre><span></span><span class="gt">---------------------------------------------------------------------------</span>
|
||||
<span class="ne">AttributeError</span><span class="g g-Whitespace"> </span>Traceback (most recent call last)
|
||||
<span class="o"><</span><span class="n">ipython</span><span class="o">-</span><span class="nb">input</span><span class="o">-</span><span class="mi">2</span><span class="o">-</span><span class="mi">772</span><span class="n">b904ae9cb</span><span class="o">></span> <span class="ow">in</span> <span class="o"><</span><span class="n">module</span><span class="o">></span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">31</span> <span class="n">t</span> <span class="o">=</span> <span class="n">bootstrap</span><span class="p">(</span><span class="n">x</span><span class="p">,</span> <span class="n">stat</span><span class="p">,</span> <span class="n">datapoints</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">32</span> <span class="c1"># the histogram of the bootstrapped data</span>
|
||||
<span class="ne">---> </span><span class="mi">33</span> <span class="n">n</span><span class="p">,</span> <span class="n">binsboot</span><span class="p">,</span> <span class="n">patches</span> <span class="o">=</span> <span class="n">plt</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span><span class="n">t</span><span class="p">,</span> <span class="mi">50</span><span class="p">,</span> <span class="n">normed</span><span class="o">=</span><span class="mi">1</span><span class="p">,</span> <span class="n">facecolor</span><span class="o">=</span><span class="s1">'red'</span><span class="p">,</span> <span class="n">alpha</span><span class="o">=</span><span class="mf">0.75</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">34</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">35</span> <span class="c1"># add a 'best fit' line</span>
|
||||
|
||||
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/matplotlib/pyplot.py</span> in <span class="ni">hist</span><span class="nt">(x, bins, range, density, weights, cumulative, bottom, histtype, align, orientation, rwidth, log, color, label, stacked, data, **kwargs)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">2683</span> <span class="n">orientation</span><span class="o">=</span><span class="s1">'vertical'</span><span class="p">,</span> <span class="n">rwidth</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">log</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">color</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">2684</span> <span class="n">label</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">stacked</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="o">*</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
|
||||
<span class="ne">-> </span><span class="mi">2685</span> <span class="k">return</span> <span class="n">gca</span><span class="p">()</span><span class="o">.</span><span class="n">hist</span><span class="p">(</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">2686</span> <span class="n">x</span><span class="p">,</span> <span class="n">bins</span><span class="o">=</span><span class="n">bins</span><span class="p">,</span> <span class="nb">range</span><span class="o">=</span><span class="nb">range</span><span class="p">,</span> <span class="n">density</span><span class="o">=</span><span class="n">density</span><span class="p">,</span> <span class="n">weights</span><span class="o">=</span><span class="n">weights</span><span class="p">,</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">2687</span> <span class="n">cumulative</span><span class="o">=</span><span class="n">cumulative</span><span class="p">,</span> <span class="n">bottom</span><span class="o">=</span><span class="n">bottom</span><span class="p">,</span> <span class="n">histtype</span><span class="o">=</span><span class="n">histtype</span><span class="p">,</span>
|
||||
|
||||
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/matplotlib/__init__.py</span> in <span class="ni">inner</span><span class="nt">(ax, data, *args, **kwargs)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">1445</span> <span class="k">def</span> <span class="nf">inner</span><span class="p">(</span><span class="n">ax</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">data</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">1446</span> <span class="k">if</span> <span class="n">data</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
|
||||
<span class="ne">-> </span><span class="mi">1447</span> <span class="k">return</span> <span class="n">func</span><span class="p">(</span><span class="n">ax</span><span class="p">,</span> <span class="o">*</span><span class="nb">map</span><span class="p">(</span><span class="n">sanitize_sequence</span><span class="p">,</span> <span class="n">args</span><span class="p">),</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">1448</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">1449</span> <span class="n">bound</span> <span class="o">=</span> <span class="n">new_sig</span><span class="o">.</span><span class="n">bind</span><span class="p">(</span><span class="n">ax</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
|
||||
|
||||
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/matplotlib/axes/_axes.py</span> in <span class="ni">hist</span><span class="nt">(self, x, bins, range, density, weights, cumulative, bottom, histtype, align, orientation, rwidth, log, color, label, stacked, **kwargs)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">6813</span> <span class="k">if</span> <span class="n">patch</span><span class="p">:</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">6814</span> <span class="n">p</span> <span class="o">=</span> <span class="n">patch</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
|
||||
<span class="ne">-> </span><span class="mi">6815</span> <span class="n">p</span><span class="o">.</span><span class="n">update</span><span class="p">(</span><span class="n">kwargs</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">6816</span> <span class="k">if</span> <span class="n">lbl</span> <span class="ow">is</span> <span class="ow">not</span> <span class="kc">None</span><span class="p">:</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">6817</span> <span class="n">p</span><span class="o">.</span><span class="n">set_label</span><span class="p">(</span><span class="n">lbl</span><span class="p">)</span>
|
||||
|
||||
<span class="nn">~/opt/anaconda3/lib/python3.8/site-packages/matplotlib/artist.py</span> in <span class="ni">update</span><span class="nt">(self, props)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">994</span> <span class="n">func</span> <span class="o">=</span> <span class="nb">getattr</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="sa">f</span><span class="s2">"set_</span><span class="si">{</span><span class="n">k</span><span class="si">}</span><span class="s2">"</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">995</span> <span class="k">if</span> <span class="ow">not</span> <span class="n">callable</span><span class="p">(</span><span class="n">func</span><span class="p">):</span>
|
||||
<span class="ne">--> </span><span class="mi">996</span> <span class="k">raise</span> <span class="ne">AttributeError</span><span class="p">(</span><span class="sa">f</span><span class="s2">"</span><span class="si">{</span><span class="nb">type</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span><span class="o">.</span><span class="vm">__name__</span><span class="si">!r}</span><span class="s2"> object "</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">997</span> <span class="sa">f</span><span class="s2">"has no property </span><span class="si">{</span><span class="n">k</span><span class="si">!r}</span><span class="s2">"</span><span class="p">)</span>
|
||||
<span class="g g-Whitespace"> </span><span class="mi">998</span> <span class="n">ret</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">func</span><span class="p">(</span><span class="n">v</span><span class="p">))</span>
|
||||
|
||||
<span class="ne">AttributeError</span>: 'Rectangle' object has no property 'normed'
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter3_25_2.png" src="_images/chapter3_25_2.png" />
|
||||
</div>
|
||||
</div>
|
||||
\beta\)</span>, apply the etsimator <span class="math notranslate nohighlight">\(\widehat \sigma^2\)</span> to the values
|
||||
<span class="math notranslate nohighlight">\(\widehat \beta ^*\)</span>.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="the-bias-variance-tradeoff">
|
||||
@@ -820,7 +715,9 @@ We use a more compact notation in terms of the expectation value</p>
|
||||
<p>that is the rewriting in terms of the so-called bias, the variance of the model <span class="math notranslate nohighlight">\(\boldsymbol{\tilde{y}}\)</span> and the variance of <span class="math notranslate nohighlight">\(\boldsymbol{\epsilon}\)</span>.</p>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<div class="highlight-ipython3 notranslate"><div class="highlight"><pre><span></span><span class="o">%</span><span class="k">matplotlib</span> inline
|
||||
|
||||
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="nn">plt</span>
|
||||
<span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="nn">np</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.linear_model</span> <span class="kn">import</span> <span class="n">LinearRegression</span><span class="p">,</span> <span class="n">Ridge</span><span class="p">,</span> <span class="n">Lasso</span>
|
||||
<span class="kn">from</span> <span class="nn">sklearn.preprocessing</span> <span class="kn">import</span> <span class="n">PolynomialFeatures</span>
|
||||
@@ -877,6 +774,15 @@ We use a more compact notation in terms of the expectation value</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Error: 0.01312157412031145
|
||||
Bias^2: 0.012073649480472317
|
||||
Var: 0.0010479246398391328
|
||||
0.01312157412031145 >= 0.012073649480472317 + 0.0010479246398391328 = 0.01312157412031145
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter3_37_1.png" src="_images/chapter3_37_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -929,6 +835,91 @@ We use a more compact notation in terms of the expectation value</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 0
|
||||
Error: 0.32149601703519126
|
||||
Bias^2: 0.3123314713548606
|
||||
Var: 0.009164545680330616
|
||||
0.32149601703519126 >= 0.3123314713548606 + 0.009164545680330616 = 0.3214960170351912
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 1
|
||||
Error: 0.08426840630693411
|
||||
Bias^2: 0.07968918676726028
|
||||
Var: 0.004579219539673833
|
||||
0.08426840630693411 >= 0.07968918676726028 + 0.004579219539673833 = 0.08426840630693411
|
||||
Polynomial degree: 2
|
||||
Error: 0.10398646080125035
|
||||
Bias^2: 0.10077114273548986
|
||||
Var: 0.0032153180657605086
|
||||
0.10398646080125035 >= 0.10077114273548986 + 0.0032153180657605086 = 0.10398646080125036
|
||||
Polynomial degree: 3
|
||||
Error: 0.06547790180152352
|
||||
Bias^2: 0.062082386342319454
|
||||
Var: 0.0033955154592040936
|
||||
0.06547790180152352 >= 0.062082386342319454 + 0.0033955154592040936 = 0.06547790180152355
|
||||
Polynomial degree: 4
|
||||
Error: 0.06844519414009442
|
||||
Bias^2: 0.06453579006728317
|
||||
Var: 0.003909404072811237
|
||||
0.06844519414009442 >= 0.06453579006728317 + 0.003909404072811237 = 0.06844519414009441
|
||||
Polynomial degree: 5
|
||||
Error: 0.05227921801205707
|
||||
Bias^2: 0.048187277304303125
|
||||
Var: 0.004091940707753964
|
||||
0.05227921801205707 >= 0.048187277304303125 + 0.004091940707753964 = 0.05227921801205709
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 6
|
||||
Error: 0.03781367141738898
|
||||
Bias^2: 0.03365768507152761
|
||||
Var: 0.004155986345861379
|
||||
0.03781367141738898 >= 0.03365768507152761 + 0.004155986345861379 = 0.03781367141738899
|
||||
Polynomial degree: 7
|
||||
Error: 0.027609773491022498
|
||||
Bias^2: 0.02299949826036597
|
||||
Var: 0.004610275230656537
|
||||
0.027609773491022498 >= 0.02299949826036597 + 0.004610275230656537 = 0.027609773491022505
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 8
|
||||
Error: 0.017355848195591973
|
||||
Bias^2: 0.010331721306655588
|
||||
Var: 0.007024126888936384
|
||||
0.017355848195591973 >= 0.010331721306655588 + 0.007024126888936384 = 0.017355848195591973
|
||||
Polynomial degree: 9
|
||||
Error: 0.026605727637189085
|
||||
Bias^2: 0.010018312644140933
|
||||
Var: 0.016587414993048166
|
||||
0.026605727637189085 >= 0.010018312644140933 + 0.016587414993048166 = 0.0266057276371891
|
||||
Polynomial degree: 10
|
||||
Error: 0.021592704588043153
|
||||
Bias^2: 0.010516485576652981
|
||||
Var: 0.011076219011390184
|
||||
0.021592704588043153 >= 0.010516485576652981 + 0.011076219011390184 = 0.021592704588043167
|
||||
Polynomial degree: 11
|
||||
Error: 0.07160048164228314
|
||||
Bias^2: 0.01443680008897583
|
||||
Var: 0.0571636815533073
|
||||
0.07160048164228314 >= 0.01443680008897583 + 0.0571636815533073 = 0.07160048164228312
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 12
|
||||
Error: 0.1154777721897675
|
||||
Bias^2: 0.01628578269590588
|
||||
Var: 0.09919198949386163
|
||||
0.1154777721897675 >= 0.01628578269590588 + 0.09919198949386163 = 0.11547777218976751
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Polynomial degree: 13
|
||||
Error: 0.22842468702166951
|
||||
Bias^2: 0.01975416527163567
|
||||
Var: 0.20867052175003387
|
||||
0.22842468702166951 >= 0.01975416527163567 + 0.20867052175003387 = 0.22842468702166954
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter3_38_6.png" src="_images/chapter3_38_6.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>The bias-variance tradeoff summarizes the fundamental tension in
|
||||
machine learning, particularly supervised learning, between the
|
||||
@@ -1030,6 +1021,30 @@ flexible statistical methods have higher variance.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>============================
|
||||
Underfitting vs. Overfitting
|
||||
============================
|
||||
|
||||
This example demonstrates the problems of underfitting and overfitting and
|
||||
how we can use linear regression with polynomial features to approximate
|
||||
nonlinear functions. The plot shows the function that we want to approximate,
|
||||
which is a part of the cosine function. In addition, the samples from the
|
||||
real function and the approximations of different models are displayed. The
|
||||
models have polynomial features of different degrees. We can see that a
|
||||
linear function (polynomial with degree 1) is not sufficient to fit the
|
||||
training samples. This is called **underfitting**. A polynomial of degree 4
|
||||
approximates the true function almost perfectly. However, for higher degrees
|
||||
the model will **overfit** the training data, i.e. it learns the noise of the
|
||||
training data.
|
||||
We evaluate quantitatively **overfitting** / **underfitting** by using
|
||||
cross-validation. We calculate the mean squared error (MSE) on the validation
|
||||
set, the higher, the less likely the model generalizes correctly from the
|
||||
training data.
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter3_40_1.png" src="_images/chapter3_40_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -1114,6 +1129,122 @@ flexible statistical methods have higher variance.</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 1
|
||||
Mean squared error on training data: 439230.69504801
|
||||
Mean squared error on test data: 481979.17861098
|
||||
Degree of polynomial: 2
|
||||
Mean squared error on training data: 115822.95008046
|
||||
Mean squared error on test data: 123711.53703498
|
||||
Degree of polynomial: 3
|
||||
Mean squared error on training data: 9011.85263220
|
||||
Mean squared error on test data: 10913.84780262
|
||||
Degree of polynomial: 4
|
||||
Mean squared error on training data: 303.47610036
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Mean squared error on test data: 426.30787294
|
||||
Degree of polynomial: 5
|
||||
Mean squared error on training data: 3.80354994
|
||||
Mean squared error on test data: 5.98822371
|
||||
Degree of polynomial: 6
|
||||
Mean squared error on training data: 3.66204648
|
||||
Mean squared error on test data: 8.14812206
|
||||
Degree of polynomial: 7
|
||||
Mean squared error on training data: 0.47075725
|
||||
Mean squared error on test data: 2.00607783
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 8
|
||||
Mean squared error on training data: 0.04912436
|
||||
Mean squared error on test data: 0.21596432
|
||||
Degree of polynomial: 9
|
||||
Mean squared error on training data: 0.02522069
|
||||
Mean squared error on test data: 0.08576932
|
||||
Degree of polynomial: 10
|
||||
Mean squared error on training data: 0.02511518
|
||||
Mean squared error on test data: 1.20015436
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 11
|
||||
Mean squared error on training data: 0.01640891
|
||||
Mean squared error on test data: 1.35533773
|
||||
Degree of polynomial: 12
|
||||
Mean squared error on training data: 0.00813803
|
||||
Mean squared error on test data: 0.17446471
|
||||
Degree of polynomial: 13
|
||||
Mean squared error on training data: 0.00759119
|
||||
Mean squared error on test data: 1.08131003
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 14
|
||||
Mean squared error on training data: 0.00472199
|
||||
Mean squared error on test data: 0.81333793
|
||||
Degree of polynomial: 15
|
||||
Mean squared error on training data: 0.00410478
|
||||
Mean squared error on test data: 92.09145189
|
||||
Degree of polynomial: 16
|
||||
Mean squared error on training data: 0.00315593
|
||||
Mean squared error on test data: 234.39716546
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 17
|
||||
Mean squared error on training data: 0.00242998
|
||||
Mean squared error on test data: 1271.05295709
|
||||
Degree of polynomial: 18
|
||||
Mean squared error on training data: 0.00228740
|
||||
Mean squared error on test data: 108.42208194
|
||||
Degree of polynomial: 19
|
||||
Mean squared error on training data: 0.00156372
|
||||
Mean squared error on test data: 1388.41078073
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 20
|
||||
Mean squared error on training data: 0.00137982
|
||||
Mean squared error on test data: 1761.43341615
|
||||
Degree of polynomial: 21
|
||||
Mean squared error on training data: 0.00118170
|
||||
Mean squared error on test data: 15061.31603087
|
||||
Degree of polynomial: 22
|
||||
Mean squared error on training data: 0.00092354
|
||||
Mean squared error on test data: 890.63488525
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 23
|
||||
Mean squared error on training data: 0.00085887
|
||||
Mean squared error on test data: 5483.16796929
|
||||
Degree of polynomial: 24
|
||||
Mean squared error on training data: 0.00084589
|
||||
Mean squared error on test data: 1695.57143061
|
||||
Degree of polynomial: 25
|
||||
Mean squared error on training data: 0.00078806
|
||||
Mean squared error on test data: 131343.30655001
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 26
|
||||
Mean squared error on training data: 0.00076916
|
||||
Mean squared error on test data: 17709.14370264
|
||||
Degree of polynomial: 27
|
||||
Mean squared error on training data: 0.00068970
|
||||
Mean squared error on test data: 2975.38903780
|
||||
Degree of polynomial: 28
|
||||
Mean squared error on training data: 0.00062588
|
||||
Mean squared error on test data: 3848.64522721
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stream highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span>Degree of polynomial: 29
|
||||
Mean squared error on training data: 0.00060728
|
||||
Mean squared error on test data: 2988.64001211
|
||||
</pre></div>
|
||||
</div>
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><ipython-input-5-8dc29df57a8c>:73: RuntimeWarning: divide by zero encountered in log10
|
||||
plt.plot(polynomial, np.log10(trainingerror), label='Training Error')
|
||||
<ipython-input-5-8dc29df57a8c>:74: RuntimeWarning: divide by zero encountered in log10
|
||||
plt.plot(polynomial, np.log10(testerror), label='Test Error')
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter3_41_11.png" src="_images/chapter3_41_11.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="section" id="cross-validation">
|
||||
@@ -1264,6 +1395,9 @@ cross-validation (LOOCV).</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter3_47_0.png" src="_images/chapter3_47_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
<p>More examples of the application of cross-validation follow here.</p>
|
||||
<div class="cell docutils container">
|
||||
@@ -1338,6 +1472,13 @@ cross-validation (LOOCV).</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<div class="output stderr highlight-myst-ansi notranslate"><div class="highlight"><pre><span></span><ipython-input-7-49b0ef2e51e2>:63: RuntimeWarning: divide by zero encountered in log10
|
||||
plt.plot(polynomial, np.log10(estimated_mse_sklearn), label='Test Error')
|
||||
</pre></div>
|
||||
</div>
|
||||
<img alt="_images/chapter3_49_1.png" src="_images/chapter3_49_1.png" />
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell docutils container">
|
||||
<div class="cell_input docutils container">
|
||||
@@ -1379,6 +1520,9 @@ cross-validation (LOOCV).</p>
|
||||
</pre></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="cell_output docutils container">
|
||||
<img alt="_images/chapter3_50_0.png" src="_images/chapter3_50_0.png" />
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -155,6 +155,11 @@
|
||||
6. Logistic Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapteroptimization.html">
|
||||
7. Optimization, the central part of any Machine Learning algortithm
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter5.html">
|
||||
8. Support Vector Machines, overarching aims
|
||||
|
||||
@@ -156,6 +156,11 @@
|
||||
6. Logistic Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapteroptimization.html">
|
||||
7. Optimization, the central part of any Machine Learning algortithm
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter5.html">
|
||||
8. Support Vector Machines, overarching aims
|
||||
|
||||
Binary file not shown.
@@ -139,17 +139,17 @@
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter1.html">
|
||||
3. Linear Regression, basic Elements
|
||||
3. Linear Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter2.html">
|
||||
4. Resampling Methods
|
||||
4. Ridge and Lasso Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter3.html">
|
||||
5. Ridge and Lasso Regression
|
||||
5. Resampling Methods
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
@@ -157,9 +157,14 @@
|
||||
6. Logistic Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapteroptimization.html">
|
||||
7. Optimization, the central part of any Machine Learning algortithm
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter5.html">
|
||||
7. Support Vector Machines, overarching aims
|
||||
8. Support Vector Machines, overarching aims
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -171,12 +176,12 @@
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter6.html">
|
||||
8. Decision trees, overarching aims
|
||||
9. Decision trees, overarching aims
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter7.html">
|
||||
9. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
|
||||
10. Ensemble Methods: From a Single Tree to Many Trees and Extreme Boosting, Meet the Jungle of Methods
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -188,12 +193,12 @@
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter8.html">
|
||||
10. Basic ideas of the Principal Component Analysis (PCA)
|
||||
11. Basic ideas of the Principal Component Analysis (PCA)
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="Clustering.html">
|
||||
11. Clustering Analysis
|
||||
12. Clustering Analysis
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -205,12 +210,12 @@
|
||||
<ul class="nav bd-sidenav">
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter9.html">
|
||||
12. Neural networks
|
||||
13. Neural networks
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter10.html">
|
||||
13. Building a Feed Forward Neural Network
|
||||
14. Building a Feed Forward Neural Network
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -290,78 +295,78 @@
|
||||
</a>
|
||||
<ul class="nav section-nav flex-column">
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-35-august-23-27">
|
||||
Week 35 August 23-27
|
||||
<a class="reference internal nav-link" href="#week-34-august-23-27">
|
||||
Week 34 August 23-27
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-36-august-30-september-3">
|
||||
Week 36 August 30-September 3
|
||||
<a class="reference internal nav-link" href="#week-35-august-30-september-3">
|
||||
Week 35 August 30-September 3
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-37-september-6-10">
|
||||
Week 37 September 6-10
|
||||
<a class="reference internal nav-link" href="#week-36-september-6-10">
|
||||
Week 36 September 6-10
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-38-september-13-17">
|
||||
Week 38 September 13-17
|
||||
<a class="reference internal nav-link" href="#week-37-september-13-17">
|
||||
Week 37 September 13-17
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-39-september-20-24">
|
||||
Week 39 September 20-24
|
||||
<a class="reference internal nav-link" href="#week-38-september-20-24">
|
||||
Week 38 September 20-24
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-40-september-27-october-1">
|
||||
Week 40 September 27- October 1
|
||||
<a class="reference internal nav-link" href="#week-39-september-27-october-1">
|
||||
Week 39 September 27- October 1
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-41-october-4-8">
|
||||
Week 41 October 4-8
|
||||
<a class="reference internal nav-link" href="#week-40-october-4-8">
|
||||
Week 40 October 4-8
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-42-october-11-15">
|
||||
Week 42 October 11-15
|
||||
<a class="reference internal nav-link" href="#week-41-october-11-15">
|
||||
Week 41 October 11-15
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-43-october-18-22">
|
||||
Week 43 October 18-22
|
||||
<a class="reference internal nav-link" href="#week-42-october-18-22">
|
||||
Week 42 October 18-22
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-44-october-25-29">
|
||||
Week 44 October 25-29
|
||||
<a class="reference internal nav-link" href="#week-43-october-25-29">
|
||||
Week 43 October 25-29
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-45-november-1-5">
|
||||
Week 45 November 1-5
|
||||
<a class="reference internal nav-link" href="#week-44-november-1-5">
|
||||
Week 44 November 1-5
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-46-november-8-12">
|
||||
Week 46 November 8-12
|
||||
<a class="reference internal nav-link" href="#week-45-november-8-12">
|
||||
Week 45 November 8-12
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-47-november-15-19">
|
||||
Week 47 November 15-19
|
||||
<a class="reference internal nav-link" href="#week-46-november-15-19">
|
||||
Week 46 November 15-19
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-48-november-22-26">
|
||||
Week 48 November 22-26
|
||||
<a class="reference internal nav-link" href="#week-47-november-22-26">
|
||||
Week 47 November 22-26
|
||||
</a>
|
||||
</li>
|
||||
<li class="toc-h3 nav-item toc-entry">
|
||||
<a class="reference internal nav-link" href="#week-49-november-29-december-2">
|
||||
Week 49 November 29- December 2
|
||||
<a class="reference internal nav-link" href="#week-48-november-29-december-2">
|
||||
Week 48 November 29- December 2
|
||||
</a>
|
||||
</li>
|
||||
</ul>
|
||||
@@ -399,43 +404,58 @@
|
||||
<li><p>HTF: Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning</p></li>
|
||||
<li><p>AG: Aurelien Geron, Hands‑On Machine Learning with Scikit‑Learn and TensorFlow</p></li>
|
||||
</ul>
|
||||
<div class="section" id="week-35-august-23-27">
|
||||
<h3>Week 35 August 23-27<a class="headerlink" href="#week-35-august-23-27" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-34-august-23-27">
|
||||
<h3>Week 34 August 23-27<a class="headerlink" href="#week-34-august-23-27" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday: Introduction to software and repetition of Python Programming</p></li>
|
||||
<li><p>Lecture Thursday: Introduction to the course, what is Machine Learning and introduction to Linear Regression</p></li>
|
||||
<li><p>Lecture Thursday: Introduction to the course, what is Machine Learning and introduction to Linear Regression.</p></li>
|
||||
<li><p>Video of Lecture August 26, 2021 at <a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureThursdayAugust26.mp4?vrtx=view-as-webpage">https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureThursdayAugust26.mp4?vrtx=view-as-webpage</a></p></li>
|
||||
<li><p>Lecture Friday: Basics of Linear Regression</p></li>
|
||||
<li><p>Reading recommendations: Refresh linear algebra, GBC chapters 1 and 2. CMB sections 1.1 and 3.1. HTF chapters 2 and 3. Install scikit-learn. See lecture notes for week 35 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a></p></li>
|
||||
<li><p>Video of Lecture August 27, 2021 at <a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureThursdayAugust27.mp4?vrtx=view-as-webpage">https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureThursdayAugust27.mp4?vrtx=view-as-webpage</a></p></li>
|
||||
<li><p>Reading recommendations: Refresh linear algebra, GBC chapters 1 and 2. CMB sections 1.1 and 3.1. HTF chapters 2 and 3. Install scikit-learn. See lecture notes for week 34 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a></p></li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-36-august-30-september-3">
|
||||
<h3>Week 36 August 30-September 3<a class="headerlink" href="#week-36-august-30-september-3" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-35-august-30-september-3">
|
||||
<h3>Week 35 August 30-September 3<a class="headerlink" href="#week-35-august-30-september-3" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Linear Regression, from ordinary linear regression to Ridge and Lasso regression, linear algebra analysis, examples and discussions of codes</p></li>
|
||||
<li><p>Lecture Friday: Linear Regression, Linear algebra and Ridge and Lasso Regression, linear algebra analysis, examples and discussions of codes</p></li>
|
||||
<li><p>Reading recommendations: See lecture notes for week 36 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>. HTF chapter 3. GBC chapters 1 and and sections 3.1-3.11 and 5.1 and CMB sections 1.1 and 3.1</p></li>
|
||||
<li><p>Lab Wednesday: Work on exercises 1-3 for week 35</p></li>
|
||||
<li><p>Thursday: Review of ordinary Least Squares with applications and discussion of Ridge Regression and Singular Value Decomposition</p></li>
|
||||
<li><p>Video of lecture Thursday at <a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember2.mp4?vrtx=view-as-webpage">https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember2.mp4?vrtx=view-as-webpage</a>”.</p></li>
|
||||
<li><p>Friday: Analysis of Ridge and Lasso Regression and links with Singular Value Decomposition</p></li>
|
||||
<li><p>Video of lecture Friday at <a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember3.mp4?vrtx=view-as-webpage">https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember3.mp4?vrtx=view-as-webpage</a>”</p></li>
|
||||
<li><p>Reading recommendations: See lecture notes for week 35 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>. HTF chapter 3. GBC chapters 1 and and sections 3.1-3.11 and 5.1 and CMB sections 1.1 and 3.1</p></li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-37-september-6-10">
|
||||
<h3>Week 37 September 6-10<a class="headerlink" href="#week-37-september-6-10" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-36-september-6-10">
|
||||
<h3>Week 36 September 6-10<a class="headerlink" href="#week-36-september-6-10" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday: Exercises 1 and 2 from week 36</p></li>
|
||||
<li><p>Lecture Thursday: Summary from last week on SVD, Statistics, probability theory and linear regression</p></li>
|
||||
<li><p>Video of Lecture <a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember9.mp4?vrtx=view-as-webpage">https://www.uio.no/studier/emner/matnat/fys/FYS-STK3155/h21/forelesningsvideoer/LectureSeptember9.mp4?vrtx=view-as-webpage</a>”.</p></li>
|
||||
<li><p>Friday: Linear Regression and links with Statistics, Resampling methods and presentation of first project.</p></li>
|
||||
<li><p>Video of Lecture at <a class="reference external" href="https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureSeptember10.mp4?vrtx=view-as-webpage">https://www.uio.no/studier/emner/matnat/fys/FYS-STK4155/h21/forelesningsvideoer/LectureSeptember10.mp4?vrtx=view-as-webpage</a>”</p></li>
|
||||
<li><p>Recommended Reading: Lectures on Regression, Bishop 1.1, 1.2, 2.1, 2.2, 2.3 and 3.1, Hastie et al chapter 3</p></li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-37-september-13-17">
|
||||
<h3>Week 37 September 13-17<a class="headerlink" href="#week-37-september-13-17" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Statistical interpretation of Linear Regression</p></li>
|
||||
<li><p>Lecture Friday: Bias-Variance tradeoff</p></li>
|
||||
<li><p>Reading recommendations: See lecture notes for week 37 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>. GBC sections 5.2-5.5, CMB section 3.2</p>
|
||||
<li><p>Lecture Thursday: Resampling methods, cross-validation and Bootstrap</p></li>
|
||||
<li><p>Lecture Friday: More on Resampling methods and summary of linear regression</p></li>
|
||||
<li><p>Reading recommendations: See lecture notes for week 37 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>.</p>
|
||||
<ul>
|
||||
<li><p>Chapter</p></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-38-september-13-17">
|
||||
<h3>Week 38 September 13-17<a class="headerlink" href="#week-38-september-13-17" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-38-september-20-24">
|
||||
<h3>Week 38 September 20-24<a class="headerlink" href="#week-38-september-20-24" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Resampling methods, cross-validation and Bootstrap</p></li>
|
||||
<li><p>Lecture Friday: More on Resampling methods and summary of linear regression</p></li>
|
||||
<li><p>Lecture Thursday: Classification problems and Logistic Regression, from binary cases to several categories</p></li>
|
||||
<li><p>Lecture Friday: Logistic Regression and gradient optimization</p></li>
|
||||
<li><p>Reading recommendations: See lecture notes for week 38 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>.</p>
|
||||
<ul>
|
||||
<li><p>Chapter</p></li>
|
||||
@@ -443,12 +463,12 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-39-september-20-24">
|
||||
<h3>Week 39 September 20-24<a class="headerlink" href="#week-39-september-20-24" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-39-september-27-october-1">
|
||||
<h3>Week 39 September 27- October 1<a class="headerlink" href="#week-39-september-27-october-1" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Classification problems and Logistic Regression, from binary cases to several categories</p></li>
|
||||
<li><p>Lecture Friday: Logistic Regression and gradient optimization</p></li>
|
||||
<li><p>Lecture Thursday: Gradient Optimization methods</p></li>
|
||||
<li><p>Lecture Friday: Deep Learning and Neural Networks</p></li>
|
||||
<li><p>Reading recommendations: See lecture notes for week 39 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>.</p>
|
||||
<ul>
|
||||
<li><p>Chapter</p></li>
|
||||
@@ -456,21 +476,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-40-september-27-october-1">
|
||||
<h3>Week 40 September 27- October 1<a class="headerlink" href="#week-40-september-27-october-1" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Gradient Optimization methods</p></li>
|
||||
<li><p>Lecture Friday: Deep Learning and Neural Networks</p></li>
|
||||
<li><p>Reading recommendations: See lecture notes for week 40 at <a class="reference external" href="https://compphysics.github.io/MachineLearning/doc/web/course.html">https://compphysics.github.io/MachineLearning/doc/web/course.html</a>.</p>
|
||||
<ul>
|
||||
<li><p>Chapter</p></li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-41-october-4-8">
|
||||
<h3>Week 41 October 4-8<a class="headerlink" href="#week-41-october-4-8" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-40-october-4-8">
|
||||
<h3>Week 40 October 4-8<a class="headerlink" href="#week-40-october-4-8" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Writing a feed-forward Neural Network code for regression and classification</p></li>
|
||||
@@ -482,8 +489,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-42-october-11-15">
|
||||
<h3>Week 42 October 11-15<a class="headerlink" href="#week-42-october-11-15" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-41-october-11-15">
|
||||
<h3>Week 41 October 11-15<a class="headerlink" href="#week-41-october-11-15" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Deep learning and Neural Networks</p></li>
|
||||
@@ -495,8 +502,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-43-october-18-22">
|
||||
<h3>Week 43 October 18-22<a class="headerlink" href="#week-43-october-18-22" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-42-october-18-22">
|
||||
<h3>Week 42 October 18-22<a class="headerlink" href="#week-42-october-18-22" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Convolutional Neural Networks and classification problems</p></li>
|
||||
@@ -508,8 +515,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-44-october-25-29">
|
||||
<h3>Week 44 October 25-29<a class="headerlink" href="#week-44-october-25-29" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-43-october-25-29">
|
||||
<h3>Week 43 October 25-29<a class="headerlink" href="#week-43-october-25-29" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Recurrent Neural Networks</p></li>
|
||||
@@ -521,8 +528,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-45-november-1-5">
|
||||
<h3>Week 45 November 1-5<a class="headerlink" href="#week-45-november-1-5" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-44-november-1-5">
|
||||
<h3>Week 44 November 1-5<a class="headerlink" href="#week-44-november-1-5" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Decision trees, classification and regression</p></li>
|
||||
@@ -534,8 +541,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-46-november-8-12">
|
||||
<h3>Week 46 November 8-12<a class="headerlink" href="#week-46-november-8-12" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-45-november-8-12">
|
||||
<h3>Week 45 November 8-12<a class="headerlink" href="#week-45-november-8-12" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Ensemble methods, bagging and random forests</p></li>
|
||||
@@ -547,8 +554,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-47-november-15-19">
|
||||
<h3>Week 47 November 15-19<a class="headerlink" href="#week-47-november-15-19" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-46-november-15-19">
|
||||
<h3>Week 46 November 15-19<a class="headerlink" href="#week-46-november-15-19" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday:</p></li>
|
||||
@@ -560,8 +567,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-48-november-22-26">
|
||||
<h3>Week 48 November 22-26<a class="headerlink" href="#week-48-november-22-26" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-47-november-22-26">
|
||||
<h3>Week 47 November 22-26<a class="headerlink" href="#week-47-november-22-26" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Unsupervised Learning, Principal Component Analysis (PCA)</p></li>
|
||||
@@ -573,8 +580,8 @@
|
||||
</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="section" id="week-49-november-29-december-2">
|
||||
<h3>Week 49 November 29- December 2<a class="headerlink" href="#week-49-november-29-december-2" title="Permalink to this headline">¶</a></h3>
|
||||
<div class="section" id="week-48-november-29-december-2">
|
||||
<h3>Week 48 November 29- December 2<a class="headerlink" href="#week-48-november-29-december-2" title="Permalink to this headline">¶</a></h3>
|
||||
<ul class="simple">
|
||||
<li><p>Lab Wednesday:</p></li>
|
||||
<li><p>Lecture Thursday: Unsupervised Learning and Clustering</p></li>
|
||||
|
||||
@@ -160,6 +160,11 @@
|
||||
6. Logistic Regression
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapteroptimization.html">
|
||||
7. Optimization, the central part of any Machine Learning algortithm
|
||||
</a>
|
||||
</li>
|
||||
<li class="toctree-l1">
|
||||
<a class="reference internal" href="chapter5.html">
|
||||
8. Support Vector Machines, overarching aims
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -255,7 +255,7 @@ variance of $\overline{X}$ (which often is the case), then there is no
|
||||
need for bootstrapping.
|
||||
|
||||
|
||||
The Jackknife works by making many replicas of the estimator $\widehat{\theta}$.
|
||||
The Jackknife works by making many replicas of the estimator $\widehat{\beta}$.
|
||||
The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\boldsymbol{x} = (x_1,x_2,\cdots,X_n)$.
|
||||
Let $\boldsymbol{x}_i$ denote the vector
|
||||
|
||||
@@ -265,8 +265,8 @@ $$
|
||||
|
||||
which equals the vector $\boldsymbol{x}$ with the exception that observation
|
||||
number $i$ is left out. Using this notation, define
|
||||
$\widehat{\theta}_i$ to be the estimator
|
||||
$\widehat{\theta}$ computed using $\vec{X}_i$.
|
||||
$\widehat{\beta}_i$ to be the estimator
|
||||
$\widehat{\beta}$ computed using $\vec{X}_i$.
|
||||
|
||||
from numpy import *
|
||||
from numpy.random import randint, randn
|
||||
@@ -311,11 +311,11 @@ advantages:
|
||||
|
||||
4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).
|
||||
|
||||
Since $\widehat{\theta} = \widehat{\theta}(\boldsymbol{X})$ is a function of random variables,
|
||||
$\widehat{\theta}$ itself must be a random variable. Thus it has
|
||||
Since $\widehat{\beta} = \widehat{\beta}(\boldsymbol{X})$ is a function of random variables,
|
||||
$\widehat{\beta}$ itself must be a random variable. Thus it has
|
||||
a pdf, call this function $p(\boldsymbol{t})$. The aim of the bootstrap is to
|
||||
estimate $p(\boldsymbol{t})$ by the relative frequency of
|
||||
$\widehat{\theta}$. You can think of this as using a histogram
|
||||
$\widehat{\beta}$. You can think of this as using a histogram
|
||||
in the place of $p(\boldsymbol{t})$. If the relative frequency closely
|
||||
resembles $p(\vec{t})$, then using numerics, it is straight forward to
|
||||
estimate all the interesting parameters of $p(\boldsymbol{t})$ using point
|
||||
@@ -323,18 +323,18 @@ estimators.
|
||||
|
||||
|
||||
|
||||
In the case that $\widehat{\theta}$ has
|
||||
In the case that $\widehat{\beta}$ has
|
||||
more than one component, and the components are independent, we use the
|
||||
same estimator on each component separately. If the probability
|
||||
density function of $X_i$, $p(x)$, had been known, then it would have
|
||||
been straight forward to do this by:
|
||||
1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \cdots, X_n^*)$.
|
||||
|
||||
2. Then using these numbers, we could compute a replica of $\widehat{\theta}$ called $\widehat{\theta}^*$.
|
||||
2. Then using these numbers, we could compute a replica of $\widehat{\beta}$ called $\widehat{\beta}^*$.
|
||||
|
||||
By repeated use of (1) and (2), many
|
||||
estimates of $\widehat{\theta}$ could have been obtained. The
|
||||
idea is to use the relative frequency of $\widehat{\theta}^*$
|
||||
estimates of $\widehat{\beta}$ could have been obtained. The
|
||||
idea is to use the relative frequency of $\widehat{\beta}^*$
|
||||
(think of a histogram) as an estimate of $p(\boldsymbol{t})$.
|
||||
|
||||
|
||||
@@ -360,78 +360,22 @@ The independent bootstrap works like this:
|
||||
|
||||
2. Define a vector $\boldsymbol{x}^*$ containing the values which were drawn from $\boldsymbol{x}$.
|
||||
|
||||
3. Using the vector $\boldsymbol{x}^*$ compute $\widehat{\theta}^*$ by evaluating $\widehat \theta$ under the observations $\boldsymbol{x}^*$.
|
||||
3. Using the vector $\boldsymbol{x}^*$ compute $\widehat{\beta}^*$ by evaluating $\widehat \beta$ under the observations $\boldsymbol{x}^*$.
|
||||
|
||||
4. Repeat this process $k$ times.
|
||||
|
||||
When you are done, you can draw a histogram of the relative frequency
|
||||
of $\widehat \theta^*$. This is your estimate of the probability
|
||||
of $\widehat \beta^*$. This is your estimate of the probability
|
||||
distribution $p(t)$. Using this probability distribution you can
|
||||
estimate any statistics thereof. In principle you never draw the
|
||||
histogram of the relative frequency of $\widehat{\theta}^*$. Instead
|
||||
histogram of the relative frequency of $\widehat{\beta}^*$. Instead
|
||||
you use the estimators corresponding to the statistic of interest. For
|
||||
example, if you are interested in estimating the variance of $\widehat
|
||||
\theta$, apply the etsimator $\widehat \sigma^2$ to the values
|
||||
$\widehat \theta ^*$.
|
||||
\beta$, apply the etsimator $\widehat \sigma^2$ to the values
|
||||
$\widehat \beta ^*$.
|
||||
|
||||
|
||||
|
||||
The following code starts with a Gaussian distribution with mean value
|
||||
$\mu =100$ and variance $\sigma=15$. We use this to generate the data
|
||||
used in the bootstrap analysis. The bootstrap analysis returns a data
|
||||
set after a given number of bootstrap operations (as many as we have
|
||||
data points). This data set consists of estimated mean values for each
|
||||
bootstrap operation. The histogram generated by the bootstrap method
|
||||
shows that the distribution for these mean values is also a Gaussian,
|
||||
centered around the mean value $\mu=100$ but with standard deviation
|
||||
$\sigma/\sqrt{n}$, where $n$ is the number of bootstrap samples (in
|
||||
this case the same as the number of original data points). The value
|
||||
of the standard deviation is what we expect from the central limit
|
||||
theorem.
|
||||
|
||||
%matplotlib inline
|
||||
|
||||
from numpy import *
|
||||
from numpy.random import randint, randn
|
||||
from time import time
|
||||
import matplotlib.mlab as mlab
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
# Returns mean of bootstrap samples
|
||||
def stat(data):
|
||||
return mean(data)
|
||||
|
||||
# Bootstrap algorithm
|
||||
def bootstrap(data, statistic, R):
|
||||
t = zeros(R); n = len(data); inds = arange(n); t0 = time()
|
||||
# non-parametric bootstrap
|
||||
for i in range(R):
|
||||
t[i] = statistic(data[randint(0,n,n)])
|
||||
|
||||
# analysis
|
||||
print("Runtime: %g sec" % (time()-t0)); print("Bootstrap Statistics :")
|
||||
print("original bias std. error")
|
||||
print("%8g %8g %14g %15g" % (statistic(data), std(data),mean(t),std(t)))
|
||||
return t
|
||||
|
||||
|
||||
mu, sigma = 100, 15
|
||||
datapoints = 10000
|
||||
x = mu + sigma*random.randn(datapoints)
|
||||
# bootstrap returns the data sample
|
||||
t = bootstrap(x, stat, datapoints)
|
||||
# the histogram of the bootstrapped data
|
||||
n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)
|
||||
|
||||
# add a 'best fit' line
|
||||
y = mlab.normpdf( binsboot, mean(t), std(t))
|
||||
lt = plt.plot(binsboot, y, 'r--', linewidth=1)
|
||||
plt.xlabel('Smarts')
|
||||
plt.ylabel('Probability')
|
||||
plt.axis([99.5, 100.6, 0, 3.0])
|
||||
plt.grid(True)
|
||||
|
||||
plt.show()
|
||||
|
||||
## The bias-variance tradeoff
|
||||
|
||||
@@ -494,6 +438,8 @@ $$
|
||||
|
||||
that is the rewriting in terms of the so-called bias, the variance of the model $\boldsymbol{\tilde{y}}$ and the variance of $\boldsymbol{\epsilon}$.
|
||||
|
||||
%matplotlib inline
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
from sklearn.linear_model import LinearRegression, Ridge, Lasso
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 23 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 37 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 10 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 11 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 17 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 47 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 27 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
@@ -18,6 +18,7 @@ parts:
|
||||
- file: chapter2.ipynb
|
||||
- file: chapter3.ipynb
|
||||
- file: chapter4.ipynb
|
||||
- file: chapteroptimization.ipynb
|
||||
- file: chapter5.ipynb
|
||||
- caption: Decision Trees, Ensemble Methods and Boosting
|
||||
numbered: true
|
||||
|
||||
@@ -361,7 +361,7 @@
|
||||
"need for bootstrapping. \n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The Jackknife works by making many replicas of the estimator $\\widehat{\\theta}$. \n",
|
||||
"The Jackknife works by making many replicas of the estimator $\\widehat{\\beta}$. \n",
|
||||
"The jackknife is a resampling method where we systematically leave out one observation from the vector of observed values $\\boldsymbol{x} = (x_1,x_2,\\cdots,X_n)$. \n",
|
||||
"Let $\\boldsymbol{x}_i$ denote the vector"
|
||||
]
|
||||
@@ -381,8 +381,8 @@
|
||||
"source": [
|
||||
"which equals the vector $\\boldsymbol{x}$ with the exception that observation\n",
|
||||
"number $i$ is left out. Using this notation, define\n",
|
||||
"$\\widehat{\\theta}_i$ to be the estimator\n",
|
||||
"$\\widehat{\\theta}$ computed using $\\vec{X}_i$."
|
||||
"$\\widehat{\\beta}_i$ to be the estimator\n",
|
||||
"$\\widehat{\\beta}$ computed using $\\vec{X}_i$."
|
||||
]
|
||||
},
|
||||
{
|
||||
@@ -442,11 +442,11 @@
|
||||
"\n",
|
||||
"4. It is relatively simple to apply the bootstrap to complex data-collection plans (such as stratified and clustered samples).\n",
|
||||
"\n",
|
||||
"Since $\\widehat{\\theta} = \\widehat{\\theta}(\\boldsymbol{X})$ is a function of random variables,\n",
|
||||
"$\\widehat{\\theta}$ itself must be a random variable. Thus it has\n",
|
||||
"Since $\\widehat{\\beta} = \\widehat{\\beta}(\\boldsymbol{X})$ is a function of random variables,\n",
|
||||
"$\\widehat{\\beta}$ itself must be a random variable. Thus it has\n",
|
||||
"a pdf, call this function $p(\\boldsymbol{t})$. The aim of the bootstrap is to\n",
|
||||
"estimate $p(\\boldsymbol{t})$ by the relative frequency of\n",
|
||||
"$\\widehat{\\theta}$. You can think of this as using a histogram\n",
|
||||
"$\\widehat{\\beta}$. You can think of this as using a histogram\n",
|
||||
"in the place of $p(\\boldsymbol{t})$. If the relative frequency closely\n",
|
||||
"resembles $p(\\vec{t})$, then using numerics, it is straight forward to\n",
|
||||
"estimate all the interesting parameters of $p(\\boldsymbol{t})$ using point\n",
|
||||
@@ -454,18 +454,18 @@
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"In the case that $\\widehat{\\theta}$ has\n",
|
||||
"In the case that $\\widehat{\\beta}$ has\n",
|
||||
"more than one component, and the components are independent, we use the\n",
|
||||
"same estimator on each component separately. If the probability\n",
|
||||
"density function of $X_i$, $p(x)$, had been known, then it would have\n",
|
||||
"been straight forward to do this by: \n",
|
||||
"1. Drawing lots of numbers from $p(x)$, suppose we call one such set of numbers $(X_1^*, X_2^*, \\cdots, X_n^*)$. \n",
|
||||
"\n",
|
||||
"2. Then using these numbers, we could compute a replica of $\\widehat{\\theta}$ called $\\widehat{\\theta}^*$. \n",
|
||||
"2. Then using these numbers, we could compute a replica of $\\widehat{\\beta}$ called $\\widehat{\\beta}^*$. \n",
|
||||
"\n",
|
||||
"By repeated use of (1) and (2), many\n",
|
||||
"estimates of $\\widehat{\\theta}$ could have been obtained. The\n",
|
||||
"idea is to use the relative frequency of $\\widehat{\\theta}^*$\n",
|
||||
"estimates of $\\widehat{\\beta}$ could have been obtained. The\n",
|
||||
"idea is to use the relative frequency of $\\widehat{\\beta}^*$\n",
|
||||
"(think of a histogram) as an estimate of $p(\\boldsymbol{t})$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -491,94 +491,23 @@
|
||||
"\n",
|
||||
"2. Define a vector $\\boldsymbol{x}^*$ containing the values which were drawn from $\\boldsymbol{x}$. \n",
|
||||
"\n",
|
||||
"3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\theta}^*$ by evaluating $\\widehat \\theta$ under the observations $\\boldsymbol{x}^*$. \n",
|
||||
"3. Using the vector $\\boldsymbol{x}^*$ compute $\\widehat{\\beta}^*$ by evaluating $\\widehat \\beta$ under the observations $\\boldsymbol{x}^*$. \n",
|
||||
"\n",
|
||||
"4. Repeat this process $k$ times. \n",
|
||||
"\n",
|
||||
"When you are done, you can draw a histogram of the relative frequency\n",
|
||||
"of $\\widehat \\theta^*$. This is your estimate of the probability\n",
|
||||
"of $\\widehat \\beta^*$. This is your estimate of the probability\n",
|
||||
"distribution $p(t)$. Using this probability distribution you can\n",
|
||||
"estimate any statistics thereof. In principle you never draw the\n",
|
||||
"histogram of the relative frequency of $\\widehat{\\theta}^*$. Instead\n",
|
||||
"histogram of the relative frequency of $\\widehat{\\beta}^*$. Instead\n",
|
||||
"you use the estimators corresponding to the statistic of interest. For\n",
|
||||
"example, if you are interested in estimating the variance of $\\widehat\n",
|
||||
"\\theta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n",
|
||||
"$\\widehat \\theta ^*$.\n",
|
||||
"\\beta$, apply the etsimator $\\widehat \\sigma^2$ to the values\n",
|
||||
"$\\widehat \\beta ^*$.\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"The following code starts with a Gaussian distribution with mean value\n",
|
||||
"$\\mu =100$ and variance $\\sigma=15$. We use this to generate the data\n",
|
||||
"used in the bootstrap analysis. The bootstrap analysis returns a data\n",
|
||||
"set after a given number of bootstrap operations (as many as we have\n",
|
||||
"data points). This data set consists of estimated mean values for each\n",
|
||||
"bootstrap operation. The histogram generated by the bootstrap method\n",
|
||||
"shows that the distribution for these mean values is also a Gaussian,\n",
|
||||
"centered around the mean value $\\mu=100$ but with standard deviation\n",
|
||||
"$\\sigma/\\sqrt{n}$, where $n$ is the number of bootstrap samples (in\n",
|
||||
"this case the same as the number of original data points). The value\n",
|
||||
"of the standard deviation is what we expect from the central limit\n",
|
||||
"theorem."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {
|
||||
"collapsed": false,
|
||||
"editable": true
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%matplotlib inline\n",
|
||||
"\n",
|
||||
"from numpy import *\n",
|
||||
"from numpy.random import randint, randn\n",
|
||||
"from time import time\n",
|
||||
"import matplotlib.mlab as mlab\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"\n",
|
||||
"# Returns mean of bootstrap samples \n",
|
||||
"def stat(data):\n",
|
||||
" return mean(data)\n",
|
||||
"\n",
|
||||
"# Bootstrap algorithm\n",
|
||||
"def bootstrap(data, statistic, R):\n",
|
||||
" t = zeros(R); n = len(data); inds = arange(n); t0 = time()\n",
|
||||
" # non-parametric bootstrap \n",
|
||||
" for i in range(R):\n",
|
||||
" t[i] = statistic(data[randint(0,n,n)])\n",
|
||||
"\n",
|
||||
" # analysis \n",
|
||||
" print(\"Runtime: %g sec\" % (time()-t0)); print(\"Bootstrap Statistics :\")\n",
|
||||
" print(\"original bias std. error\")\n",
|
||||
" print(\"%8g %8g %14g %15g\" % (statistic(data), std(data),mean(t),std(t)))\n",
|
||||
" return t\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"mu, sigma = 100, 15\n",
|
||||
"datapoints = 10000\n",
|
||||
"x = mu + sigma*random.randn(datapoints)\n",
|
||||
"# bootstrap returns the data sample \n",
|
||||
"t = bootstrap(x, stat, datapoints)\n",
|
||||
"# the histogram of the bootstrapped data \n",
|
||||
"n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)\n",
|
||||
"\n",
|
||||
"# add a 'best fit' line \n",
|
||||
"y = mlab.normpdf( binsboot, mean(t), std(t))\n",
|
||||
"lt = plt.plot(binsboot, y, 'r--', linewidth=1)\n",
|
||||
"plt.xlabel('Smarts')\n",
|
||||
"plt.ylabel('Probability')\n",
|
||||
"plt.axis([99.5, 100.6, 0, 3.0])\n",
|
||||
"plt.grid(True)\n",
|
||||
"\n",
|
||||
"plt.show()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## The bias-variance tradeoff\n",
|
||||
"\n",
|
||||
"\n",
|
||||
@@ -710,6 +639,8 @@
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"%matplotlib inline\n",
|
||||
"\n",
|
||||
"import matplotlib.pyplot as plt\n",
|
||||
"import numpy as np\n",
|
||||
"from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user