update chapter 3

doc/LectureNotes/chapter3.ipynb

@@ -735,11 +735,92 @@
     "$\\tilde{p}(x)$.\n",
     "\n",
     "\n",
+    "The following code starts with a Gaussian distribution with mean value\n",
+    "$\\mu =100$ and variance $\\sigma=15$. We use this to generate the data\n",
+    "used in the bootstrap analysis. The bootstrap analysis returns a data\n",
+    "set after a given number of bootstrap operations (as many as we have\n",
+    "data points). This data set consists of estimated mean values for each\n",
+    "bootstrap operation. The histogram generated by the bootstrap method\n",
+    "shows that the distribution for these mean values is also a Gaussian,\n",
+    "centered around the mean value $\\mu=100$ but with standard deviation\n",
+    "$\\sigma/\\sqrt{n}$, where $n$ is the number of bootstrap samples (in\n",
+    "this case the same as the number of original data points). The value\n",
+    "of the standard deviation is what we expect from the central limit\n",
+    "theorem."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "%matplotlib inline\n",
     "\n",
+    "import numpy as np\n",
+    "from time import time\n",
+    "from scipy.stats import norm\n",
+    "import matplotlib.pyplot as plt\n",
     "\n",
+    "# Returns mean of bootstrap samples \n",
+    "# Bootstrap algorithm\n",
+    "def bootstrap(data, datapoints):\n",
+    "    t = np.zeros(datapoints)\n",
+    "    n = len(data)\n",
+    "    # non-parametric bootstrap         \n",
+    "    for i in range(datapoints):\n",
+    "        t[i] = np.mean(data[np.random.randint(0,n,n)])\n",
+    "    # analysis    \n",
+    "    print(\"Bootstrap Statistics :\")\n",
+    "    print(\"original           bias      std. error\")\n",
+    "    print(\"%8g %8g %14g %15g\" % (np.mean(data), np.std(data),np.mean(t),np.std(t)))\n",
+    "    return t\n",
     "\n",
+    "# We set the mean value to 100 and the standard deviation to 15\n",
+    "mu, sigma = 100, 15\n",
+    "datapoints = 10000\n",
+    "# We generate random numbers according to the normal distribution\n",
+    "x = mu + sigma*np.random.randn(datapoints)\n",
+    "# bootstrap returns the data sample                                    \n",
+    "t = bootstrap(x, datapoints)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We see that our new variance and from that the standard deviation, agrees with the central limit theorem.\n",
     "\n",
-    "\n",
+    "We plot then the histogram together with a best fit for the data set."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": false,
+    "editable": true
+   },
+   "outputs": [],
+   "source": [
+    "# the histogram of the bootstrapped data (normalized data if density = True)\n",
+    "n, binsboot, patches = plt.hist(t, 50, density=True, facecolor='red', alpha=0.75)\n",
+    "# add a 'best fit' line  \n",
+    "y = norm.pdf(binsboot, np.mean(t), np.std(t))\n",
+    "lt = plt.plot(binsboot, y, 'b', linewidth=1)\n",
+    "plt.xlabel('x')\n",
+    "plt.ylabel('Probability')\n",
+    "plt.grid(True)\n",
+    "plt.show()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
     "## The bias-variance tradeoff\n",
     "\n",
     "\n",
@@ -871,8 +952,6 @@
    },
    "outputs": [],
    "source": [
-    "%matplotlib inline\n",
-    "\n",
     "import matplotlib.pyplot as plt\n",
     "import numpy as np\n",
     "from sklearn.linear_model import LinearRegression, Ridge, Lasso\n",