updating
This commit is contained in:
Morten Hjorth-Jensen
@@ -1,38 +0,0 @@
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from IPython.display import display
|
||||
import matplotlib.pyplot as plt
|
||||
from sklearn.model_selection import train_test_split
|
||||
from sklearn import linear_model
|
||||
|
||||
# Make data set.
|
||||
n = 1000
|
||||
x = np.random.rand(n)
|
||||
y = np.exp(-x**2) + 1.5 * np.exp(-(x-2)**2)#+ np.random.randn(n)
|
||||
|
||||
Maxpolydegree = 5
|
||||
X = np.zeros((len(x),Maxpolydegree))
|
||||
X[:,0] = 1.0
|
||||
|
||||
for polydegree in range(1, Maxpolydegree):
|
||||
for degree in range(polydegree):
|
||||
X[:,degree] = x**(degree)
|
||||
|
||||
|
||||
# We split the data in test and training data
|
||||
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
|
||||
|
||||
# Decide which values of lambda to use
|
||||
nlambdas = 2
|
||||
lambdas = np.logspace(-3, -1, nlambdas)
|
||||
for i in range(nlambdas):
|
||||
lmb = lambdas[i]
|
||||
# Make the fit using Ridge only
|
||||
RegRidge = linear_model.Ridge(lmb,fit_intercept=False)
|
||||
RegRidge.fit(X_train,y_train)
|
||||
# and then make the prediction
|
||||
ypredictRidge = RegRidge.predict(X_test)
|
||||
Coeffs = np.array(RegRidge.coef_)
|
||||
BetaValues = pd.DataFrame(Coeffs)
|
||||
BetaValues.columns = ['beta']
|
||||
display(BetaValues)
|
||||
@@ -636,6 +636,145 @@ The textbook by "Davison on the Bootstrap Methods and their Applications":"https
|
||||
|
||||
Before we proceed however, we need to remind ourselves about a central theorem in statistics, namely the so-called _central limit theorem_.
|
||||
|
||||
!split
|
||||
===== The Central Limit Theorem =====
|
||||
|
||||
|
||||
Suppose we have a PDF $p(x)$ from which we generate a series $N$
|
||||
of averages $\langle x_i \rangle$. Each mean value $\langle x_i \rangle$
|
||||
is viewed as the average of a specific measurement, e.g., throwing
|
||||
dice 100 times and then taking the average value, or producing a certain
|
||||
amount of random numbers.
|
||||
For notational ease, we set $\langle x_i \rangle=x_i$ in the discussion
|
||||
which follows.
|
||||
|
||||
If we compute the mean $z$ of $m$ such mean values $x_i$
|
||||
!bt
|
||||
\[
|
||||
z=\frac{x_1+x_2+\dots+x_m}{m},
|
||||
\]
|
||||
!et
|
||||
the question we pose is which is the PDF of the new variable $z$.
|
||||
|
||||
!split
|
||||
===== Finding the Limit =====
|
||||
|
||||
The probability of obtaining an average value $z$ is the product of the
|
||||
probabilities of obtaining arbitrary individual mean values $x_i$,
|
||||
but with the constraint that the average is $z$. We can express this through
|
||||
the following expression
|
||||
!bt
|
||||
\[
|
||||
\tilde{p}(z)=\int dx_1p(x_1)\int dx_2p(x_2)\dots\int dx_mp(x_m)
|
||||
\delta(z-\frac{x_1+x_2+\dots+x_m}{m}),
|
||||
\]
|
||||
!et
|
||||
where the $\delta$-function enbodies the constraint that the mean is $z$.
|
||||
All measurements that lead to each individual $x_i$ are expected to
|
||||
be independent, which in turn means that we can express $\tilde{p}$ as the
|
||||
product of individual $p(x_i)$. The independence assumption is important in the derivation of the central limit theorem.
|
||||
|
||||
|
||||
!split
|
||||
===== Rewriting the $\delta$-function =====
|
||||
|
||||
If we use the integral expression for the $\delta$-function
|
||||
|
||||
!bt
|
||||
\[
|
||||
\delta(z-\frac{x_1+x_2+\dots+x_m}{m})=\frac{1}{2\pi}\int_{-\infty}^{\infty}
|
||||
dq\exp{\left(iq(z-\frac{x_1+x_2+\dots+x_m}{m})\right)},
|
||||
\]
|
||||
!et
|
||||
and inserting $e^{i\mu q-i\mu q}$ where $\mu$ is the mean value
|
||||
we arrive at
|
||||
!bt
|
||||
\[
|
||||
\tilde{p}(z)=\frac{1}{2\pi}\int_{-\infty}^{\infty}
|
||||
dq\exp{\left(iq(z-\mu)\right)}\left[\int_{-\infty}^{\infty}
|
||||
dxp(x)\exp{\left(iq(\mu-x)/m\right)}\right]^m,
|
||||
\]
|
||||
!et
|
||||
with the integral over $x$ resulting in
|
||||
|
||||
!bt
|
||||
\[
|
||||
\int_{-\infty}^{\infty}dxp(x)\exp{\left(iq(\mu-x)/m\right)}=
|
||||
\int_{-\infty}^{\infty}dxp(x)
|
||||
\left[1+\frac{iq(\mu-x)}{m}-\frac{q^2(\mu-x)^2}{2m^2}+\dots\right].
|
||||
\]
|
||||
!et
|
||||
|
||||
!split
|
||||
===== Identifying Terms =====
|
||||
|
||||
The second term on the rhs disappears since this is just the mean and
|
||||
employing the definition of $\sigma^2$ we have
|
||||
!bt
|
||||
\[
|
||||
\int_{-\infty}^{\infty}dxp(x)e^{\left(iq(\mu-x)/m\right)}=
|
||||
1-\frac{q^2\sigma^2}{2m^2}+\dots,
|
||||
\]
|
||||
!et
|
||||
resulting in
|
||||
|
||||
!bt
|
||||
\[
|
||||
\left[\int_{-\infty}^{\infty}dxp(x)\exp{\left(iq(\mu-x)/m\right)}\right]^m\approx
|
||||
\left[1-\frac{q^2\sigma^2}{2m^2}+\dots \right]^m,
|
||||
\]
|
||||
!et
|
||||
and in the limit $m\rightarrow \infty$ we obtain
|
||||
|
||||
!bt
|
||||
\[
|
||||
\tilde{p}(z)=\frac{1}{\sqrt{2\pi}(\sigma/\sqrt{m})}
|
||||
\exp{\left(-\frac{(z-\mu)^2}{2(\sigma/\sqrt{m})^2}\right)},
|
||||
\]
|
||||
!et
|
||||
which is the normal distribution with variance
|
||||
$\sigma^2_m=\sigma^2/m$, where $\sigma$ is the variance of the PDF $p(x)$
|
||||
and $\mu$ is also the mean of the PDF $p(x)$.
|
||||
|
||||
!split
|
||||
===== Wrapping it up =====
|
||||
|
||||
Thus, the central limit theorem states that the PDF $\tilde{p}(z)$ of
|
||||
the average of $m$ random values corresponding to a PDF $p(x)$
|
||||
is a normal distribution whose mean is the
|
||||
mean value of the PDF $p(x)$ and whose variance is the variance
|
||||
of the PDF $p(x)$ divided by $m$, the number of values used to compute $z$.
|
||||
|
||||
The theorem is satisfied by a large class of PDFs. Note however that for a
|
||||
finite $m$, it is not always possible to find a closed expression for
|
||||
$\tilde{p}(x)$.
|
||||
The central limit theorem leads then to the well-known expression for the
|
||||
standard deviation, given by
|
||||
|
||||
!bt
|
||||
\[
|
||||
\sigma_m=
|
||||
\frac{\sigma}{\sqrt{m}}.
|
||||
\]
|
||||
!et
|
||||
|
||||
The latter is true only if the average value is known exactly. This is obtained in the limit
|
||||
$m\rightarrow \infty$ only. Because the mean and the variance are measured quantities we obtain
|
||||
the familiar expression in statistics
|
||||
!bt
|
||||
\[
|
||||
\sigma_m\approx
|
||||
\frac{\sigma}{\sqrt{m-1}}.
|
||||
\]
|
||||
!et
|
||||
|
||||
In many cases however the above estimate for the standard deviation,
|
||||
in particular if correlations are strong, may be too simplistic.
|
||||
|
||||
!split
|
||||
===== Confidence Intervals =====
|
||||
|
||||
|
||||
!split
|
||||
===== Resampling methods: Bootstrap background =====
|
||||
|
||||
@@ -727,10 +866,12 @@ from time import time
|
||||
import matplotlib.mlab as mlab
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
# Returns mean of bootstrap samples
|
||||
def stat(data):
|
||||
# Returns mean of bootstrap samples # Alternatively, we can run it using Scikit-Learn's function resample # See the examples below
|
||||
|
||||
def statistics(data):
|
||||
return mean(data)
|
||||
|
||||
|
||||
# Bootstrap algorithm
|
||||
def bootstrap(data, statistic, R):
|
||||
t = zeros(R); n = len(data); inds = arange(n); t0 = time()
|
||||
@@ -749,7 +890,13 @@ mu, sigma = 100, 15
|
||||
datapoints = 10000
|
||||
x = mu + sigma*random.randn(datapoints)
|
||||
# bootstrap returns the data sample
|
||||
t = bootstrap(x, stat, datapoints)
|
||||
t = bootstrap(x, statistics, datapoints)
|
||||
!ec
|
||||
We see that our new variance and from that the standard deviation, agrees with the central limit theorem.
|
||||
|
||||
!split
|
||||
===== Plotting the Histogram =====
|
||||
!bc pycod
|
||||
# the histogram of the bootstrapped data
|
||||
n, binsboot, patches = plt.hist(t, 50, normed=1, facecolor='red', alpha=0.75)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user