From 25c1a391cedc07a918750b8d40361427b73a3135 Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Tue, 20 Sep 2022 22:33:50 +0200 Subject: [PATCH] Update week37.ipynb --- doc/pub/week37/ipynb/week37.ipynb | 776 +++++++++++++----------------- 1 file changed, 334 insertions(+), 442 deletions(-) diff --git a/doc/pub/week37/ipynb/week37.ipynb b/doc/pub/week37/ipynb/week37.ipynb index 14af4623d..7c0027279 100644 --- a/doc/pub/week37/ipynb/week37.ipynb +++ b/doc/pub/week37/ipynb/week37.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "9707d916", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "a044c354", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 37: Summary of Ridge and Lasso Regression and Resampling Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -30,9 +26,7 @@ { "cell_type": "markdown", "id": "1c862c30", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plans for week 37\n", "\n", @@ -57,9 +51,7 @@ { "cell_type": "markdown", "id": "b1c2ddf1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Summary of Ridge and Lasso Regression and start Resampling methods" ] @@ -67,9 +59,7 @@ { "cell_type": "markdown", "id": "cbd33297", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Deriving OLS from a probability distribution\n", "\n", @@ -90,9 +80,7 @@ { "cell_type": "markdown", "id": "64ad454e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y_i\\sim \\mathcal{N}(\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta}, \\sigma^2)=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -102,9 +90,7 @@ { "cell_type": "markdown", "id": "b7f8c8f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Independent and Identically Distrubuted (iid)\n", "\n", @@ -115,9 +101,7 @@ { "cell_type": "markdown", "id": "00767b73", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(y_i, \\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]},\n", @@ -127,9 +111,7 @@ { "cell_type": "markdown", "id": "6d2bcbc3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which reads as finding the likelihood of an event $y_i$ with the input variables $\\boldsymbol{X}$ given the parameters (to be determined) $\\boldsymbol{\\beta}$.\n", "\n", @@ -139,9 +121,7 @@ { "cell_type": "markdown", "id": "bef396db", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{y},\\boldsymbol{X}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}=\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta}).\n", @@ -151,9 +131,7 @@ { "cell_type": "markdown", "id": "3bc7436d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We will write this in a more compact form reserving $\\boldsymbol{D}$ for the domain of events, including the ouputs (targets) and the inputs. That is\n", "in case we have a simple one-dimensional input and output case" @@ -162,9 +140,7 @@ { "cell_type": "markdown", "id": "6c5c39a0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})].\n", @@ -174,9 +150,7 @@ { "cell_type": "markdown", "id": "58368bc2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the more general case the various inputs should be replaced by the possible features represented by the input data set $\\boldsymbol{X}$. \n", "We can now rewrite the above probability as" @@ -185,9 +159,7 @@ { "cell_type": "markdown", "id": "0626e798", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -197,9 +169,7 @@ { "cell_type": "markdown", "id": "c14a62d7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It is a conditional probability (see below) and reads as the likelihood of a domain of events $\\boldsymbol{D}$ given a set of parameters $\\boldsymbol{\\beta}$." ] @@ -207,9 +177,7 @@ { "cell_type": "markdown", "id": "19c85f4b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Maximum Likelihood Estimation (MLE)\n", "\n", @@ -238,9 +206,7 @@ { "cell_type": "markdown", "id": "a11f8cc1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A new Cost Function\n", "\n", @@ -250,9 +216,7 @@ { "cell_type": "markdown", "id": "937861b8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=-\\log{\\prod_{i=0}^{n-1}p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})}=-\\sum_{i=0}^{n-1}\\log{p(y_i,\\boldsymbol{X}\\vert\\boldsymbol{\\beta})},\n", @@ -262,9 +226,7 @@ { "cell_type": "markdown", "id": "a28e3332", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which becomes" ] @@ -272,9 +234,7 @@ { "cell_type": "markdown", "id": "5e3a32e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{n}{2}\\log{2\\pi\\sigma^2}+\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}.\n", @@ -284,9 +244,7 @@ { "cell_type": "markdown", "id": "b3cd9181", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the derivative of the *new* cost function with respect to the parameters $\\beta$ we recognize our familiar OLS equation, namely" ] @@ -294,9 +252,7 @@ { "cell_type": "markdown", "id": "a5452767", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right) =0,\n", @@ -306,9 +262,7 @@ { "cell_type": "markdown", "id": "3490356b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the well-known OLS equation for the optimal paramters $\\beta$" ] @@ -316,9 +270,7 @@ { "cell_type": "markdown", "id": "22b9edee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\hat{\\boldsymbol{\\beta}}^{\\mathrm{OLS}}=\\left(\\boldsymbol{X}^T\\boldsymbol{X}\\right)^{-1}\\boldsymbol{X}^T\\boldsymbol{y}!\n", @@ -328,9 +280,7 @@ { "cell_type": "markdown", "id": "27507c33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Bayes' Theorem\n", "\n", @@ -340,9 +290,7 @@ { "cell_type": "markdown", "id": "46dbb3df", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{p(Y)},\n", @@ -352,9 +300,7 @@ { "cell_type": "markdown", "id": "1d2b22da", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we can rewrite as" ] @@ -362,9 +308,7 @@ { "cell_type": "markdown", "id": "e7bcf267", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(X\\vert Y)= \\frac{p(X,Y)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)}=\\frac{p(Y\\vert X)p(X)}{\\sum_{i=0}^{n-1}p(Y\\vert X=x_i)p(x_i)},\n", @@ -374,9 +318,7 @@ { "cell_type": "markdown", "id": "87dfa89f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is Bayes' theorem. It allows us to evaluate the uncertainty in in $X$ after we have observed $Y$. We can easily interchange $X$ with $Y$." ] @@ -384,9 +326,7 @@ { "cell_type": "markdown", "id": "722016af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Interpretations of Bayes' Theorem\n", "\n", @@ -401,9 +341,7 @@ { "cell_type": "markdown", "id": "befa12e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Test Function for what happens with OLS, Ridge and Lasso\n", "\n", @@ -420,10 +358,7 @@ "cell_type": "code", "execution_count": 1, "id": "1192fb77", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -496,9 +431,7 @@ { "cell_type": "markdown", "id": "3db643cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "How can we understand this?" ] @@ -506,9 +439,7 @@ { "cell_type": "markdown", "id": "754feb02", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Rerunning the above code\n", "\n", @@ -536,10 +467,7 @@ "cell_type": "code", "execution_count": 2, "id": "5fe37551", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -585,9 +513,7 @@ { "cell_type": "markdown", "id": "10c42e4f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Invoking Bayes' theorem\n", "\n", @@ -599,9 +525,7 @@ { "cell_type": "markdown", "id": "fb1b5d4f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{D}=[(x_0,y_0), (x_1,y_1),\\dots, (x_{n-1},y_{n-1})],\n", @@ -611,9 +535,7 @@ { "cell_type": "markdown", "id": "a84af26c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "is given by" ] @@ -621,9 +543,7 @@ { "cell_type": "markdown", "id": "a8b79d61", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}.\n", @@ -633,9 +553,7 @@ { "cell_type": "markdown", "id": "d8152df8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In Bayes' theorem this function plays the role of the so-called likelihood. We could now ask the question what is the posterior probability of a parameter set $\\boldsymbol{\\beta}$ given a domain of events $\\boldsymbol{D}$? That is, how can we define the posterior probability" ] @@ -643,9 +561,7 @@ { "cell_type": "markdown", "id": "6039e4c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D}).\n", @@ -655,9 +571,7 @@ { "cell_type": "markdown", "id": "10e567d6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Bayes' theorem comes to our rescue here since (omitting the normalization constant)" ] @@ -665,9 +579,7 @@ { "cell_type": "markdown", "id": "8436a81d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})\\propto p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})p(\\boldsymbol{\\beta}).\n", @@ -677,9 +589,7 @@ { "cell_type": "markdown", "id": "eb906810", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We have a model for $p(\\boldsymbol{D}\\vert\\boldsymbol{\\beta})$ but need one for the **prior** $p(\\boldsymbol{\\beta}$!" ] @@ -687,9 +597,7 @@ { "cell_type": "markdown", "id": "1521f03f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and Bayes\n", "\n", @@ -703,9 +611,7 @@ { "cell_type": "markdown", "id": "c4186e4b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -715,9 +621,7 @@ { "cell_type": "markdown", "id": "7f86d581", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] @@ -725,9 +629,7 @@ { "cell_type": "markdown", "id": "8fcc90ac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta\\vert\\boldsymbol{D})}=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\beta_j^2}{2\\tau^2}\\right)}.\n", @@ -737,9 +639,7 @@ { "cell_type": "markdown", "id": "f8069c5e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can now optimize this quantity with respect to $\\boldsymbol{\\beta}$. As we\n", "did for OLS, this is most conveniently done by taking the negative\n", @@ -750,9 +650,7 @@ { "cell_type": "markdown", "id": "7d31ce6b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{2\\tau^2}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -762,9 +660,7 @@ { "cell_type": "markdown", "id": "bd0d383f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and replacing $1/2\\tau^2$ with $\\lambda$ we have" ] @@ -772,9 +668,7 @@ { "cell_type": "markdown", "id": "5de8e77b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta})=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_2^2,\n", @@ -784,9 +678,7 @@ { "cell_type": "markdown", "id": "51f277ee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is our Ridge cost function! Nice, isn't it?" ] @@ -794,9 +686,7 @@ { "cell_type": "markdown", "id": "0da3b881", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lasso and Bayes\n", "\n", @@ -806,9 +696,7 @@ { "cell_type": "markdown", "id": "756063ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta})=\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -818,9 +706,7 @@ { "cell_type": "markdown", "id": "f4ec070b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Our posterior probability becomes then (omitting the normalization factor which is just a constant)" ] @@ -828,9 +714,7 @@ { "cell_type": "markdown", "id": "78effef0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\vert\\boldsymbol{D})=\\prod_{i=0}^{n-1}\\frac{1}{\\sqrt{2\\pi\\sigma^2}}\\exp{\\left[-\\frac{(y_i-\\boldsymbol{X}_{i,*}\\boldsymbol{\\beta})^2}{2\\sigma^2}\\right]}\\prod_{j=0}^{p-1}\\exp{\\left(-\\frac{\\vert\\beta_j\\vert}{\\tau}\\right)}.\n", @@ -840,9 +724,7 @@ { "cell_type": "markdown", "id": "12a472ad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the negative\n", "logarithm of the posterior probability and leaving out the\n", @@ -852,9 +734,7 @@ { "cell_type": "markdown", "id": "2446397c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\frac{1}{\\tau}\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -864,9 +744,7 @@ { "cell_type": "markdown", "id": "88cc7598", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and replacing $1/\\tau$ with $\\lambda$ we have" ] @@ -874,9 +752,7 @@ { "cell_type": "markdown", "id": "94da09f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{\\beta}=\\frac{\\vert\\vert (\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta})\\vert\\vert_2^2}{2\\sigma^2}+\\lambda\\vert\\vert\\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -886,9 +762,7 @@ { "cell_type": "markdown", "id": "c738c8a5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is our Lasso cost function!" ] @@ -896,9 +770,7 @@ { "cell_type": "markdown", "id": "4b809aa9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Why resampling methods\n", "\n", @@ -915,9 +787,7 @@ { "cell_type": "markdown", "id": "0422d689", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods\n", "Resampling methods are an indispensable tool in modern\n", @@ -943,9 +813,7 @@ { "cell_type": "markdown", "id": "edcbbc94", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling approaches can be computationally expensive\n", "\n", @@ -969,9 +837,7 @@ { "cell_type": "markdown", "id": "bb8bc020", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Why resampling methods ?\n", "**Statistical analysis.**\n", @@ -986,9 +852,7 @@ { "cell_type": "markdown", "id": "4b5e6fe4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Statistical analysis\n", "\n", @@ -1006,9 +870,7 @@ { "cell_type": "markdown", "id": "e6ead513", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods\n", "\n", @@ -1035,9 +897,7 @@ { "cell_type": "markdown", "id": "6116b137", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods: Jackknife and Bootstrap\n", "\n", @@ -1060,9 +920,7 @@ { "cell_type": "markdown", "id": "810a80fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods: Jackknife\n", "\n", @@ -1074,9 +932,7 @@ { "cell_type": "markdown", "id": "ac902598", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_i = (x_1,x_2,\\cdots,x_{i-1},x_{i+1},\\cdots,x_n),\n", @@ -1086,9 +942,7 @@ { "cell_type": "markdown", "id": "6d1c87c9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which equals the vector $\\boldsymbol{x}$ with the exception that observation\n", "number $i$ is left out. Using this notation, define\n", @@ -1099,9 +953,7 @@ { "cell_type": "markdown", "id": "81e9f625", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Jackknife code example" ] @@ -1110,10 +962,7 @@ "cell_type": "code", "execution_count": 3, "id": "2108cf3e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from numpy import *\n", @@ -1149,9 +998,7 @@ { "cell_type": "markdown", "id": "cd1dbd77", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods: Bootstrap\n", "Bootstrapping is a [non-parametric approach](https://en.wikipedia.org/wiki/Nonparametric_statistics) to statistical inference\n", @@ -1174,9 +1021,7 @@ { "cell_type": "markdown", "id": "dd8a6a19", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Central Limit Theorem\n", "\n", @@ -1194,9 +1039,7 @@ { "cell_type": "markdown", "id": "11b09a03", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "z=\\frac{x_1+x_2+\\dots+x_m}{m},\n", @@ -1206,9 +1049,7 @@ { "cell_type": "markdown", "id": "9cc18d15", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "the question we pose is which is the PDF of the new variable $z$." ] @@ -1216,9 +1057,7 @@ { "cell_type": "markdown", "id": "c8f3836b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Finding the Limit\n", "\n", @@ -1231,9 +1070,7 @@ { "cell_type": "markdown", "id": "35f6d279", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{p}(z)=\\int dx_1p(x_1)\\int dx_2p(x_2)\\dots\\int dx_mp(x_m)\n", @@ -1244,9 +1081,7 @@ { "cell_type": "markdown", "id": "954e305b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where the $\\delta$-function enbodies the constraint that the mean is $z$.\n", "All measurements that lead to each individual $x_i$ are expected to\n", @@ -1257,9 +1092,7 @@ { "cell_type": "markdown", "id": "5004dfbd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Rewriting the $\\delta$-function\n", "\n", @@ -1269,9 +1102,7 @@ { "cell_type": "markdown", "id": "bc1494a8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\delta(z-\\frac{x_1+x_2+\\dots+x_m}{m})=\\frac{1}{2\\pi}\\int_{-\\infty}^{\\infty}\n", @@ -1282,9 +1113,7 @@ { "cell_type": "markdown", "id": "051dfc4a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and inserting $e^{i\\mu q-i\\mu q}$ where $\\mu$ is the mean value\n", "we arrive at" @@ -1293,9 +1122,7 @@ { "cell_type": "markdown", "id": "1aee8ced", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{p}(z)=\\frac{1}{2\\pi}\\int_{-\\infty}^{\\infty}\n", @@ -1307,9 +1134,7 @@ { "cell_type": "markdown", "id": "115eb936", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with the integral over $x$ resulting in" ] @@ -1317,9 +1142,7 @@ { "cell_type": "markdown", "id": "d8095730", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\int_{-\\infty}^{\\infty}dxp(x)\\exp{\\left(iq(\\mu-x)/m\\right)}=\n", @@ -1331,9 +1154,7 @@ { "cell_type": "markdown", "id": "8e3d928c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Identifying Terms\n", "\n", @@ -1344,9 +1165,7 @@ { "cell_type": "markdown", "id": "7a3ebe33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\int_{-\\infty}^{\\infty}dxp(x)e^{\\left(iq(\\mu-x)/m\\right)}=\n", @@ -1357,9 +1176,7 @@ { "cell_type": "markdown", "id": "ba88dac3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "resulting in" ] @@ -1367,9 +1184,7 @@ { "cell_type": "markdown", "id": "5b700a0c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left[\\int_{-\\infty}^{\\infty}dxp(x)\\exp{\\left(iq(\\mu-x)/m\\right)}\\right]^m\\approx\n", @@ -1380,9 +1195,7 @@ { "cell_type": "markdown", "id": "680e8a19", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and in the limit $m\\rightarrow \\infty$ we obtain" ] @@ -1390,9 +1203,7 @@ { "cell_type": "markdown", "id": "37ec4c6b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\tilde{p}(z)=\\frac{1}{\\sqrt{2\\pi}(\\sigma/\\sqrt{m})}\n", @@ -1403,9 +1214,7 @@ { "cell_type": "markdown", "id": "d852821b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is the normal distribution with variance\n", "$\\sigma^2_m=\\sigma^2/m$, where $\\sigma$ is the variance of the PDF $p(x)$\n", @@ -1415,9 +1224,7 @@ { "cell_type": "markdown", "id": "0ae2df37", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Wrapping it up\n", "\n", @@ -1434,9 +1241,7 @@ { "cell_type": "markdown", "id": "0f9730d2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sigma_m=\n", @@ -1447,9 +1252,7 @@ { "cell_type": "markdown", "id": "4b928715", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The latter is true only if the average value is known exactly. This is obtained in the limit\n", "$m\\rightarrow \\infty$ only. Because the mean and the variance are measured quantities we obtain \n", @@ -1459,9 +1262,7 @@ { "cell_type": "markdown", "id": "336d1b60", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\sigma_m\\approx \n", @@ -1472,9 +1273,7 @@ { "cell_type": "markdown", "id": "48e0d6fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In many cases however the above estimate for the standard deviation,\n", "in particular if correlations are strong, may be too simplistic. Keep\n", @@ -1492,9 +1291,7 @@ { "cell_type": "markdown", "id": "9b054871", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Confidence Intervals\n", "\n", @@ -1515,9 +1312,7 @@ { "cell_type": "markdown", "id": "1b4aca73", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Standard Approach based on the Normal Distribution\n", "\n", @@ -1530,9 +1325,7 @@ { "cell_type": "markdown", "id": "692bbecc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\mu_{\\beta}\\pm \\frac{z\\sigma_{\\beta}}{\\sqrt{n}}\\right),\n", @@ -1542,9 +1335,7 @@ { "cell_type": "markdown", "id": "6c3510cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $z$ defines the level of certainty (or confidence). For a normal\n", "distribution typical parameters are $z=2.576$ which corresponds to a\n", @@ -1562,9 +1353,7 @@ { "cell_type": "markdown", "id": "ddee3616", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods: Bootstrap background\n", "\n", @@ -1582,9 +1371,7 @@ { "cell_type": "markdown", "id": "caf51b28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods: More Bootstrap background\n", "\n", @@ -1606,9 +1393,7 @@ { "cell_type": "markdown", "id": "bc59d5da", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods: Bootstrap approach\n", "\n", @@ -1627,9 +1412,7 @@ { "cell_type": "markdown", "id": "017fc1aa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Resampling methods: Bootstrap steps\n", "\n", @@ -1657,9 +1440,7 @@ { "cell_type": "markdown", "id": "95ffe579", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code example for the Bootstrap method\n", "\n", @@ -1681,10 +1462,7 @@ "cell_type": "code", "execution_count": 4, "id": "806253ab", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1718,9 +1496,7 @@ { "cell_type": "markdown", "id": "2a32924c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We see that our new variance and from that the standard deviation, agrees with the central limit theorem." ] @@ -1728,9 +1504,7 @@ { "cell_type": "markdown", "id": "63282d77", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plotting the Histogram" ] @@ -1739,10 +1513,7 @@ "cell_type": "code", "execution_count": 5, "id": "45083031", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# the histogram of the bootstrapped data (normalized data if density = True)\n", @@ -1759,9 +1530,7 @@ { "cell_type": "markdown", "id": "c76d90f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The bias-variance tradeoff\n", "\n", @@ -1777,9 +1546,7 @@ { "cell_type": "markdown", "id": "261a1910", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}=f(\\boldsymbol{x}) + \\boldsymbol{\\epsilon}\n", @@ -1789,9 +1556,7 @@ { "cell_type": "markdown", "id": "a5a3c08b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\epsilon$ is normally distributed with mean zero and standard deviation $\\sigma^2$.\n", "\n", @@ -1806,9 +1571,7 @@ { "cell_type": "markdown", "id": "db14284a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta}) =\\frac{1}{n}\\sum_{i=0}^{n-1}(y_i-\\tilde{y}_i)^2=\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right].\n", @@ -1818,9 +1581,7 @@ { "cell_type": "markdown", "id": "12e08977", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can rewrite this as" ] @@ -1828,9 +1589,7 @@ { "cell_type": "markdown", "id": "5d82351e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\frac{1}{n}\\sum_i(f_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\frac{1}{n}\\sum_i(\\tilde{y}_i-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2+\\sigma^2.\n", @@ -1840,9 +1599,7 @@ { "cell_type": "markdown", "id": "3c56182a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The three terms represent the square of the bias of the learning\n", "method, which can be thought of as the error caused by the simplifying\n", @@ -1857,9 +1614,7 @@ { "cell_type": "markdown", "id": "d42243aa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\mathbb{E}\\left[(\\boldsymbol{f}+\\boldsymbol{\\epsilon}-\\boldsymbol{\\tilde{y}})^2\\right],\n", @@ -1869,9 +1624,7 @@ { "cell_type": "markdown", "id": "716cf198", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and adding and subtracting $\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]$ we get" ] @@ -1879,9 +1632,7 @@ { "cell_type": "markdown", "id": "ca80292a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\mathbb{E}\\left[(\\boldsymbol{f}+\\boldsymbol{\\epsilon}-\\boldsymbol{\\tilde{y}}+\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right]-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2\\right],\n", @@ -1891,9 +1642,7 @@ { "cell_type": "markdown", "id": "61026a02", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which, using the abovementioned expectation values can be rewritten as" ] @@ -1901,9 +1650,7 @@ { "cell_type": "markdown", "id": "30ae2571", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbb{E}\\left[(\\boldsymbol{y}-\\boldsymbol{\\tilde{y}})^2\\right]=\\mathbb{E}\\left[(\\boldsymbol{y}-\\mathbb{E}\\left[\\boldsymbol{\\tilde{y}}\\right])^2\\right]+\\mathrm{Var}\\left[\\boldsymbol{\\tilde{y}}\\right]+\\sigma^2,\n", @@ -1913,9 +1660,7 @@ { "cell_type": "markdown", "id": "dcd2508b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "that is the rewriting in terms of the so-called bias, the variance of the model $\\boldsymbol{\\tilde{y}}$ and the variance of $\\boldsymbol{\\epsilon}$." ] @@ -1923,9 +1668,7 @@ { "cell_type": "markdown", "id": "62ddee8d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A way to Read the Bias-Variance Tradeoff\n", "\n", @@ -1939,9 +1682,7 @@ { "cell_type": "markdown", "id": "8cca508b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Example code for Bias-Variance tradeoff" ] @@ -1950,10 +1691,7 @@ "cell_type": "code", "execution_count": 6, "id": "15a3aa89", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -2015,22 +1753,186 @@ { "cell_type": "markdown", "id": "b15ad9a8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Understanding what happens" ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 3, "id": "b12fbc71", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Polynomial degree: 0\n", + "Error: 0.2937910450030775\n", + "Bias^2: 0.2929212799917661\n", + "Var: 0.0008697650113114114\n", + "0.2937910450030775 >= 0.2929212799917661 + 0.0008697650113114114 = 0.2937910450030775\n", + "Polynomial degree: 1\n", + "Error: 0.06894146856540673\n", + "Bias^2: 0.06832043024896824\n", + "Var: 0.0006210383164384982\n", + "0.06894146856540673 >= 0.06832043024896824 + 0.0006210383164384982 = 0.06894146856540674\n", + "Polynomial degree: 2\n", + "Error: 0.06106765054837855\n", + "Bias^2: 0.06054765422099532\n", + "Var: 0.0005199963273832353\n", + "0.06106765054837855 >= 0.06054765422099532 + 0.0005199963273832353 = 0.06106765054837855\n", + "Polynomial degree: 3\n", + "Error: 0.03346202229536658\n", + "Bias^2: 0.033140956468054594\n", + "Var: 0.00032106582731199066\n", + "0.03346202229536658 >= 0.033140956468054594 + 0.00032106582731199066 = 0.033462022295366586\n", + "Polynomial degree: 4\n", + "Error: 0.03352778717048319\n", + "Bias^2: 0.033116075385773665\n", + "Var: 0.00041171178470952977\n", + "0.03352778717048319 >= 0.033116075385773665 + 0.00041171178470952977 = 0.03352778717048319\n", + "Polynomial degree: 5\n", + "Error: 0.025517151530854775\n", + "Bias^2: 0.02496889020925645\n", + "Var: 0.0005482613215983166\n", + "0.025517151530854775 >= 0.02496889020925645 + 0.0005482613215983166 = 0.025517151530854765\n", + "Polynomial degree: 6\n", + "Error: 0.019946076068427937\n", + "Bias^2: 0.01950207688986865\n", + "Var: 0.00044399917855928643\n", + "0.019946076068427937 >= 0.01950207688986865 + 0.00044399917855928643 = 0.019946076068427937\n", + "Polynomial degree: 7\n", + "Error: 0.018695928655417776\n", + "Bias^2: 0.017979840090002395\n", + "Var: 0.0007160885654153815\n", + "0.018695928655417776 >= 0.017979840090002395 + 0.0007160885654153815 = 0.018695928655417776\n", + "Polynomial degree: 8\n", + "Error: 0.010736105188369705\n", + "Bias^2: 0.010376602508045108\n", + "Var: 0.0003595026803245951\n", + "0.010736105188369705 >= 0.010376602508045108 + 0.0003595026803245951 = 0.010736105188369703\n", + "Polynomial degree: 9\n", + "Error: 0.011013290652731648\n", + "Bias^2: 0.010539027867198339\n", + "Var: 0.0004742627855333097\n", + "0.011013290652731648 >= 0.010539027867198339 + 0.0004742627855333097 = 0.011013290652731648\n", + "Polynomial degree: 10\n", + "Error: 0.010972468815260695\n", + "Bias^2: 0.010593565969982321\n", + "Var: 0.0003789028452783743\n", + "0.010972468815260695 >= 0.010593565969982321 + 0.0003789028452783743 = 0.010972468815260695\n", + "Polynomial degree: 11\n", + "Error: 0.010840555937749019\n", + "Bias^2: 0.010348475861970803\n", + "Var: 0.0004920800757782156\n", + "0.010840555937749019 >= 0.010348475861970803 + 0.0004920800757782156 = 0.010840555937749019\n", + "Polynomial degree: 12\n", + "Error: 0.010192472149420886\n", + "Bias^2: 0.009610568640077932\n", + "Var: 0.0005819035093429566\n", + "0.010192472149420886 >= 0.009610568640077932 + 0.0005819035093429566 = 0.010192472149420888\n", + "Polynomial degree: 13\n", + "Error: 0.010312285920735095\n", + "Bias^2: 0.009802534263878348\n", + "Var: 0.0005097516568567488\n", + "0.010312285920735095 >= 0.009802534263878348 + 0.0005097516568567488 = 0.010312285920735097\n", + "Polynomial degree: 14\n", + "Error: 0.010722455299400673\n", + "Bias^2: 0.010088916760115184\n", + "Var: 0.0006335385392854925\n", + "0.010722455299400673 >= 0.010088916760115184 + 0.0006335385392854925 = 0.010722455299400677\n", + "Polynomial degree: 15\n", + "Error: 0.01115543750295569\n", + "Bias^2: 0.01031176122774682\n", + "Var: 0.0008436762752088676\n", + "0.01115543750295569 >= 0.01031176122774682 + 0.0008436762752088676 = 0.011155437502955688\n", + "Polynomial degree: 16\n", + "Error: 0.011028026783572502\n", + "Bias^2: 0.01022357238456908\n", + "Var: 0.0008044543990034224\n", + "0.011028026783572502 >= 0.01022357238456908 + 0.0008044543990034224 = 0.011028026783572502\n", + "Polynomial degree: 17\n", + "Error: 0.011628743128010138\n", + "Bias^2: 0.010533948727421355\n", + "Var: 0.0010947944005887838\n", + "0.011628743128010138 >= 0.010533948727421355 + 0.0010947944005887838 = 0.01162874312801014\n", + "Polynomial degree: 18\n", + "Error: 0.01437168218988819\n", + "Bias^2: 0.010922362277115589\n", + "Var: 0.0034493199127726\n", + "0.01437168218988819 >= 0.010922362277115589 + 0.0034493199127726 = 0.014371682189888189\n", + "Polynomial degree: 19\n", + "Error: 0.02698630625511262\n", + "Bias^2: 0.012141764328955737\n", + "Var: 0.01484454192615689\n", + "0.02698630625511262 >= 0.012141764328955737 + 0.01484454192615689 = 0.026986306255112627\n", + "Polynomial degree: 20\n", + "Error: 0.01224924367469358\n", + "Bias^2: 0.01006785242252074\n", + "Var: 0.00218139125217284\n", + "0.01224924367469358 >= 0.01006785242252074 + 0.00218139125217284 = 0.012249243674693579\n", + "Polynomial degree: 21\n", + "Error: 0.014973172935081075\n", + "Bias^2: 0.01015437128796444\n", + "Var: 0.004818801647116639\n", + "0.014973172935081075 >= 0.01015437128796444 + 0.004818801647116639 = 0.014973172935081078\n", + "Polynomial degree: 22\n", + "Error: 0.014186611568393607\n", + "Bias^2: 0.00959413326440947\n", + "Var: 0.004592478303984134\n", + "0.014186611568393607 >= 0.00959413326440947 + 0.004592478303984134 = 0.014186611568393605\n", + "Polynomial degree: 23\n", + "Error: 0.025574557075127447\n", + "Bias^2: 0.00947751730999793\n", + "Var: 0.016097039765129516\n", + "0.025574557075127447 >= 0.00947751730999793 + 0.016097039765129516 = 0.025574557075127444\n", + "Polynomial degree: 24\n", + "Error: 0.03147297560808652\n", + "Bias^2: 0.009565257541285182\n", + "Var: 0.021907718066801328\n", + "0.03147297560808652 >= 0.009565257541285182 + 0.021907718066801328 = 0.03147297560808651\n", + "Polynomial degree: 25\n", + "Error: 0.03929022375934006\n", + "Bias^2: 0.009776230006649464\n", + "Var: 0.029513993752690603\n", + "0.03929022375934006 >= 0.009776230006649464 + 0.029513993752690603 = 0.03929022375934007\n", + "Polynomial degree: 26\n", + "Error: 0.15813475868985663\n", + "Bias^2: 0.013239747711811887\n", + "Var: 0.14489501097804466\n", + "0.15813475868985663 >= 0.013239747711811887 + 0.14489501097804466 = 0.15813475868985655\n", + "Polynomial degree: 27\n", + "Error: 0.13603193678604983\n", + "Bias^2: 0.013286496181044733\n", + "Var: 0.12274544060500511\n", + "0.13603193678604983 >= 0.013286496181044733 + 0.12274544060500511 = 0.13603193678604986\n", + "Polynomial degree: 28\n", + "Error: 0.7213612740362524\n", + "Bias^2: 0.044334550324851923\n", + "Var: 0.6770267237114005\n", + "0.7213612740362524 >= 0.044334550324851923 + 0.6770267237114005 = 0.7213612740362524\n", + "Polynomial degree: 29\n", + "Error: 0.4847969941238023\n", + "Bias^2: 0.011774752533506906\n", + "Var: 0.4730222415902955\n", + "0.4847969941238023 >= 0.011774752533506906 + 0.4730222415902955 = 0.4847969941238024\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAD4CAYAAAATpHZ6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxq0lEQVR4nO3de3xT933w8c9XkmX5fsMGgg02hBiIuQVD2uZG7kmXhmxJk7B2T9O0TZc0fbpL9oKse9I07bas69rsaenWLO3SPVtC0rRLWEugSXNPc8FQCGBDYsBgY3zB2JZtSZYs/Z4/JBnZ+CIbWze+b15+Sefop6Pv0TFf/fw7P32PGGNQSimVGizxDkAppdTU0aSulFIpRJO6UkqlEE3qSimVQjSpK6VUCrHF64VnzJhhysvL4/XySimVlHbu3HnSGFM82uNxS+rl5eXU1NTE6+WVUiopicjRsR7X4RellEohmtSVUiqFaFJXSqkUErcx9ZH4fD6amprweDzxDiWpOBwOSktLSUtLi3coSqk4S6ik3tTURE5ODuXl5YhIvMNJCsYYOjo6aGpqoqKiIt7hKKXiLKGGXzweD0VFRZrQJ0BEKCoq0r9ulFJAgiV1QBP6JOh7ppQKS7ikrpRSyabhZB+vf9ge7zAATepnsFqtrFixYvDn0UcfjXdISqkE9+M3DnPvf+4kEIj/9SkS6kRpIsjIyGD37t1jtvH7/Vit1lGXo32eUio1dPZ5cXn9HO9yU1aYGddYtKcepfLycjZs2MBFF13Ez3/+8zOWn376aZYuXUpVVRUbNmwYfF52djZ/+Zd/yfLly3nnnXfiuAdKqeni9PgAqG/vjXMkUfbUReQG4J8BK/CEMebRYY9/H7gytJgJlBhj8s8msG/+z35qm51ns4kzLDkvl2986sIx27jdblasWDG4/OCDD3LHHXcAUFRUxK5duwDYuHHj4HJzczMf+9jH2LlzJwUFBVx33XU8//zz3HLLLfT19XHxxRfzT//0T1O6L0qpxNHtDiX11l6urCyJayzjJnURsQKbgGuBJmCHiGwxxtSG2xhj/jyi/VeBldMQa0yMNfwSTu7Dl3fs2MHatWspLg4WTvvMZz7DG2+8wS233ILVauXWW2+d1piVUvE1mNTbkqOnvgaoN8YcBhCRzcA6oHaU9uuBb5xtYOP1qOMhKytrzOWROBwOHUdXKsUNJvUEGH6JZkx9DtAYsdwUWncGEZkHVACvjPL4PSJSIyI17e2JMf1nKqxZs4bXX3+dkydP4vf7efrpp7niiiviHZZSKgYCAUNv/wAAH7X2YEx8Z8BM9YnSO4HnjDH+kR40xjxujKk2xlSHhyoSTXhMPfyzcePGcZ8ze/ZsHn30Ua688kqWL1/OqlWrWLduXQyiVUrFW49nAGNgbmEmTs8A7b39cY0nmuGX40BZxHJpaN1I7gS+crZBxZPfP+LnEQ0NDWMur1+/nvXr15/xvN7e+P85ppSaPuGhl1XzCjh2ykV9ay8lOY64xRNNT30HsFBEKkTETjBxbxneSEQWAQWAzttTSp0zwkn9onkFQPzH1cdN6saYAeB+YDtQBzxrjNkvIo+IyM0RTe8ENpt4DygppVQMheeoX1CSTU66jY9a45vUo5qnbozZCmwdtu6hYcsPT11YSimVHLrdPsTaR0/gOAtKsuM+rVG/UaqUUmeh2+3DPuO3fGPHfSwozkr84RellFKj63b7EFsPTm83swpdtPf00+3yxS0eTepKKXUWnG4fFqsLAFvGCQDq23viFo8m9WEaGhqoqqo6Y/0Xv/hFamtH+xKtUupc1e32YUsLXnnMLcHvacbzZKmW3o3SE088Ee8QlFIJKHii1I0BTrgPkW6rjOvJUu2pj2BgYIDPfOYzLF68mNtuuw2Xy8XatWupqakB4N5776W6upoLL7yQb3zjdJmbjRs3smTJEpYtW8YDDzwQr/CVUjHU7fZhLMHhl4OdB1lQnB3Xk6WJ21N/cSO07J3abc5aCjeOfyWjgwcP8pOf/IRLLrmEu+++mx/96EdDHv/bv/1bCgsL8fv9XH311XzwwQfMmTOH//7v/+bAgQOICF1dXVMbu1IqIXV7+jHZHnLScjjRd4JLi+GDY9pTTyhlZWVccsklAHz2s5/lrbfeGvL4s88+y0UXXcTKlSvZv38/tbW15OXl4XA4+MIXvsAvf/lLMjPje/UTpVRsdHu6AVg1axUAeXntHO9y4/IOxCWexO2pR9Gjni4iMurykSNH+O53v8uOHTsoKCjgrrvuwuPxYLPZeP/99/ntb3/Lc889xw9/+ENeeWXEYpVKqRTi9AYv5rNm1hpea3wNSW8GKjjU1sfS0ryYx6M99REcO3Zs8NJzTz31FJdeeungY06nk6ysLPLy8mhtbeXFF18EgoW7uru7+eQnP8n3v/999uzZE5fYlVKxY4yh1xdM6hV5FRRnFNNrjgHxm9aoSX0ElZWVbNq0icWLF9PZ2cm99947+Njy5ctZuXIlixYt4o//+I8Hh2l6enq46aabWLZsGZdeeinf+9734hW+UipGXF4/AQmeJM2z51FZWMlx1yFsFonbtMbEHX6Jk/Lycg4cOHDG+tdee23w/pNPPjnic99///1pikoplYjC0xkBctNzWVS4iHeb32VuUVrcpjVqUldKqUkKJvXTPfVFhYsYMAOcV+ykvj0tLjHp8ItSSk1SZE89x57DosJFAGTltHK0w4V3IBDzmDSpK6XUJDlDPfVMWzZWi5WynDIybBkE0o7jDxgaOvpiHpMmdaWUmqRwTz3XnguARSxUFlTS5T8KxKcGjCZ1pZSapG63D7G4yUs/PR+9srCSxt56RAJxOVmqSV0ppSYpPPxSEJHUFxUuom+gj9kz3HzUFvu56lEldRG5QUQOiki9iGwcpc3tIlIrIvtF5KmpDTM2rrzySrZv3z5k3WOPPTZknvpYHnroIV5++eXpCE0plYCcngGsNjd5jqFJHaCksCMxe+oiYgU2ATcCS4D1IrJkWJuFwIPAJcaYC4E/m/pQp9/69evZvHnzkHWbN29m/fr14z7X7/fzyCOPcM0110xXeEqpBBMeU8+zn07q5+efj0UsOLJbOXyyD3/AxDSmaHrqa4B6Y8xhY4wX2AysG9bmS8AmY0wngDGmbWrDjI3bbruNX//613i9XiB4wYzm5maefvrpEUvtlpeXs2HDBi666CJ+/vOfc9ddd/Hcc88B8Mgjj7B69Wqqqqq45557MCZ4YNeuXcuGDRtYs2YNF1xwAW+++SYQ/FB44IEHqKqqYtmyZfzgBz8AYOfOnVxxxRWsWrWK66+/nhMnTsTyLVFKjaHL7cVYXOSm5w6uc9gcVORW4LU04R0I0HjKFdOYovny0RygMWK5Cbh4WJsLAETkbcAKPGyM2XY2gf3D+//AgVNnfrPzbCwqXMSGNRtGfbywsJA1a9bw4osvsm7dOjZv3sztt9/OX//1X59RanfZsmUAFBUVsWvXLgC2bTu9y/fffz8PPfQQAH/yJ3/Cr371Kz71qU8BwXrt77//Plu3buWb3/wmL7/8Mo8//jgNDQ3s3r0bm83GqVOn8Pl8fPWrX+WFF16guLiYZ555hq9//ev89Kc/ndL3RSk1OV3uXsgMDOmpQ/Bk6bvNwesv1Lf1Uj4jK2YxTdWJUhuwEFgLrAf+TUTyhzcSkXtEpEZEatrb26fopadW5BBMeOhlpFK7YXfccceI23n11Ve5+OKLWbp0Ka+88gr79+8ffOyP/uiPAFi1ahUNDQ0AvPzyy3z5y1/GZgt+zhYWFnLw4EH27dvHtddey4oVK/j2t79NU1PTdOy2UmoSuvuDZXcjZ79AsAN5qr8NrH18FONx9Wh66seBsojl0tC6SE3Ae8YYH3BERD4kmOR3RDYyxjwOPA5QXV095kDTWD3q6bRu3Tr+/M//nF27duFyuSgsLByx1G5YVtaZn8Aej4f77ruPmpoaysrKePjhh4c8Jz09HQCr1crAwOg1l40xXHjhhYMVI5VSiSVcdjc8Tz2ssrASgKKC2J8sjaanvgNYKCIVImIH7gS2DGvzPMFeOiIyg+BwzOGpCzN2srOzufLKK7n77rtZv379qKV2xxJO4DNmzKC3t3dwnH0s1157LT/+8Y8Hk/ypU6eorKykvb19MKn7fL4hPX6lVHyFy+5GjqkDVBaEknphO/UxntY4blI3xgwA9wPbgTrgWWPMfhF5RERuDjXbDnSISC3wKvBXxpiO6Qp6uq1fv549e/awfv36UUvtjiU/P58vfelLVFVVcf3117N69epxn/PFL36RuXPnsmzZMpYvX85TTz2F3W7nueeeY8OGDSxfvpwVK1bwu9/9bip2USl1ljw+PwMEywAMH34pyiiiJKOENEcLh9r7BidKxILE8sUiVVdXm/CFnMPq6upYvHhxXOJJdvreKRVbbT0eLtn09zhm/5KXb3uZmVkzhzx+38v3caC9kcN7/pR3HryK2XkZU/K6IrLTGFM92uP6jVKllJoEZ0TZ3eHDLxA8WdrhawTxxbQGjCZ1pZSahG63DyxubJKGw+o44/HKwkoCxo8lvS2mJ0sTLqnHazgomel7plTshb9Nmp2We8bF6uF0uYDsnNaYTmtMqKTucDjo6OjQJDUBxhg6OjpwOM7sKSilpo/TPYBYXeTYzxx6ASjLKSPTlkleXjuHYpjUE+pydqWlpTQ1NZGoX0xKVA6Hg9LS0niHodQ5JXwpu/xhM1/CLGKhsrCSox3N1B85R5N6WloaFRUV8Q5DKaXGFR5+KcyYO2qbyoJK9re/wKk+Dx29/RRlp097XAk1/KKUUsmi2+3DYnWTN8LMl7BFhYvwGjeS1hmzk6Wa1JVSahKc4bK7owy/wOmTpVbHiZidLNWkrpRSk9DpcoOl/4wKjZEW5C/AKlYcmS3aU1dKqUTW6QlWaBzpi0dhDpuDirwKsnJbNakrpVQi6wqX3R2jpw6hLyHZmjWpK6VUIuv1BasvjjWmDrCoYBEeOmjp66DH45v2uDSpK6XUJPT5Rq6lPly4tro1/URMeuua1JVSaoIG/AE8gWCCHq+nHk7qFkdshmA0qSul1AQ5PQOI1Q2Mn9QLHYWUZJaQlhGbGTCa1JVSaoIiy+5mp2WP235R4SLSszSpK6VUQgqXCMiwZmO1WMdtX1lQyYC1hQ/bOqc9Nk3qSik1QZFld6MR/GZpgGbXETw+/7TGpkldKaUmyOkJDr+MN/MlLFwuwJJ+gsPtfdMZWnRJXURuEJGDIlIvIhtHePwuEWkXkd2hny9OfahKKZUYut0+xOIm3zH2SdKw0pxSHNYMLI5mPmrrmdbYxi29KyJWYBNwLdAE7BCRLcaY2mFNnzHG3D8NMSqlVEIJ11IvdORH1T5cW31Xz4lpv2BGND31NUC9MeawMcYLbAbWTWtUSimVwMJj6gVR9tQBFhcuwuZo4cNW5zRGFl1SnwM0Riw3hdYNd6uIfCAiz4lI2UgbEpF7RKRGRGr06kZKqWTV7Rq/7O5wi4sWYyweDnYcncbIpu5E6f8A5caYZcBLwM9GamSMedwYU22MqS4uLp6il1ZKqdjqdPeABCaU1MPfLG12H8LnD0xXaFEl9eNAZM+7NLRukDGmwxjTH1p8Alg1NeEppVTiGSy7G+XsF4Dz889HsGDszRztcE1XaFEl9R3AQhGpEBE7cCewJbKBiMyOWLwZqJu6EJVSKrGEy+6OVUt9uHRrOnOyyrGmT28NmHFnvxhjBkTkfmA7YAV+aozZLyKPADXGmC3A/xaRm4EB4BRw17RFrJRScdbri66W+nBVMxZz3Pk2/QPT9wWkcZM6gDFmK7B12LqHIu4/CDw4taEppVRi6hsIzjWfSE8doKp4MduO/prLF2VOR1iAfqNUKaUmJBAwuENJfaI99fDJ0oOdB6c8rjBN6kopNQG93gGIsuzucJUFwaT+UedHUx5XWFTDL0oppYKcbh9Y3NjEjsPmmNBzCxwFvHTbS8zMnDlN0WlSV0qpCQmXCMi0jV9HfSSzsmZNcURD6fCLUkpNQDipR1t2N9Y0qSul1AQ4Q3Vfcid4kjRWNKkrpdQEON3B65PmO7SnrpRSSW+iZXdjTZO6UkpNQPgCGTMyC+Idyog0qSul1AR0utyI1TuhYl6xpEldKaUm4JSnC5j4F49iRZO6UkpNgCZ1pZRKId39wcvR6fCLUkqlgB5vMKlrT10ppVKAa5IVGmNFk7pSSkXJGIPHH7xq0URrqceKJnWllIqSxxfAL32AkJ02uYJe002TulJKRSn8bVKHJQurxRrvcEYUVVIXkRtE5KCI1IvIxjHa3SoiRkSqpy5EpZRKDE5PsJhXpi0n3qGMatykLiJWYBNwI7AEWC8iS0ZolwN8DXhvqoNUSqlE0B2q0JioZXchup76GqDeGHPYGOMFNgPrRmj3LeAfAM8UxqeUUgmj2xUcfslL0JOkEF1SnwM0Riw3hdYNEpGLgDJjzK/H2pCI3CMiNSJS097ePuFglVIqnsLFvPITdI46TMGJUhGxAN8D/nK8tsaYx40x1caY6uLi4rN9aaWUiimnxwdWN0UZ+fEOZVTRJPXjQFnEcmloXVgOUAW8JiINwMeALXqyVCmVarpcXsSauGV3IbqkvgNYKCIVImIH7gS2hB80xnQbY2YYY8qNMeXAu8DNxpiaaYlYKaXipMPtRCRAviOJh1+MMQPA/cB2oA541hizX0QeEZGbpztApZRKFB2uLiBxi3kB2KJpZIzZCmwdtu6hUdquPfuwlFIq8XT1dwOJW8wL9BulSikVNWeCl90FTepKKRW1RC+7C5rUlVIqan0DmtSVUipleAKhsrs6/KKUUsnN5w8wQB9W7DhsjniHMypN6kopFYVutw+sLjKsiVlHPUyTulJKRSFc9yUzgSs0giZ1pZSKijNUdjcnLXFrqYMmdaWUikr4qke5CXrB6TBN6kopFYXwBTIKErjuC2hSV0qpqISHXwozNKkrpVTS63S7EYuX4qzELbsLmtSVUioq7aEKjUUOTepKKZX0ToXL7ibw9UlBk7pSSkWl0xMqu6uzX5RSKvk5vYlfSx00qSulVFR6fIlfSx00qSulVFRcAz2AjqkrpVRK8Ph7ACHHngJlAkTkBhE5KCL1IrJxhMf/VET2ishuEXlLRJZMfahKKRUfgYDBSx92ycIiid0XHjc6EbECm4AbgSXA+hGS9lPGmKXGmBXAd4DvTXWgSikVLz2eAcTiTviyuxBdT30NUG+MOWyM8QKbgXWRDYwxzojFLMBMXYhKKRVf4WJeWQledheiS+pzgMaI5abQuiFE5CsicohgT/1/j7QhEblHRGpEpKa9vX0y8dLc6eKVXbWTeq5SSk1GuJhXdoKX3YUpPFFqjNlkjFkAbAD+ZpQ2jxtjqo0x1cXFxZN6naYt3+KKFz5Bd0/PWUSrlFLRc3qCF8jIT/A56hBdUj8OlEUsl4bWjWYzcMtZxDSmjNmVWMXQ9OGe6XoJpZQaInwpuwJHfrxDGVc0SX0HsFBEKkTEDtwJbIlsICILIxb/APho6kIcqnjBCgC6j2pSV0rFRperPynK7gLYxmtgjBkQkfuB7YAV+KkxZr+IPALUGGO2APeLyDWAD+gEPjddAc+ctwSfsRJo1XF1pVRsnHQ5ETEUZyZ2hUaIIqkDGGO2AluHrXso4v7XpjiuUYnNznFbGZnd0/bHgFJKDdHu6gRIiqSe2LPoR9GZfT4zPUfiHYZS6hxxyt0FJH6JAEjSpO4vqmQObXSc6oh3KEqpc0CXJ/hVnESv0AhJmtQzSqsAaP5IT5YqpaafM1ShMdFrqUOSJvWZC1YC4DymSV0pNf16vdpTn1ZFpQtxY8e01cU7FKXUOSBZyu5CkiZ1sdpots0jW2fAKKViwBPowYqddGt6vEMZV1ImdYDunAXM9h7BGK0dppSaPsYYvIFe0i2JX6ERkjipm+LFzKST9vbWeIeilEphLq8fY3HjSIKyu5DEST2zdCkAzR/+Ps6RKKVSWTKV3YUkTuqzzl8BQG/jB/ENRCmV0sJld3M1qU+vgtnz6SUDaT8Q71CUUinMGUrqeUkw8wWSOKkjwom0cnJ76uMdiVIqhYWHX5Kh7C4kc1IHnLnnc563gUBAZ8AopaZHR18fYvFRlJH4xbwgyZM6JUsoEicnmhvHb6uUUpPQ1heq0JiVH99AopTUST1nbrAGTNuhXXGORCmVqjrc3QCUZGlPfdrNOv8iAHob98c5EqVUqjrl6QIg35H4dV8gyZN67ow5dJGD7aTWgFFKTY8uT7CnngwVGiHJkzoitKSXk9erM2CUUtPDGarQmAzFvCDKpC4iN4jIQRGpF5GNIzz+FyJSKyIfiMhvRWTe1Ic6st68hZT6juL3B2L1kkqpc0ivL3nK7kIUSV1ErMAm4EZgCbBeRJYMa/Z7oNoYswx4DvjOVAc6anwlS8gVF8cbD8XqJZVS55Bg2V0hOy11ar+sAeqNMYeNMV5gM7AusoEx5lVjjCu0+C5QOrVhji5v7jIA2uq1BoxSaup5Ar2kkYVFkmO0Opoo5wCRE8GbQutG8wXgxZEeEJF7RKRGRGra29ujj3IM510QvAqS57jOgFFKTT2v6SU9SSo0whSfKBWRzwLVwD+O9Lgx5nFjTLUxprq4uHhKXjMzv4QOCrB1aA0YpdTU8vj8GHGRkURJ3RZFm+NAWcRyaWjdECJyDfB14ApjTP/UhBedVkcFBX06A0YpNbWcnmAxr2zb1HRCYyGanvoOYKGIVIiIHbgT2BLZQERWAj8GbjbGtE19mGNz5V9A2UAjXt9ArF9aKZXCnKFiXjn25JjOCFEkdWPMAHA/sB2oA541xuwXkUdE5OZQs38EsoGfi8huEdkyyuamhXXWEjKln+MNOgSjlJo63W4fYkmesrsQ3fALxpitwNZh6x6KuH/NFMc1IfnzlsNuaD+8h4qFVfEMRSmVQrpcXrC6k6bsLiT7N0pDZi9cDkD/8X1xjkQplUra+roRMczIzI93KFFLiaTuyC6gVYpJP3Uw3qEopVLIYNndzOSo0AgpktQB2jLmU+g6HO8wlFIp5KSrC4BZ2YXxDWQCUiapewoqKfM34umP6WxKpVQK63R3AVCYkR/XOCYiZZJ62qwlpMsAjYd0XF0pNbpAwOCP8hKYXf2hsrtJUswLUiipF1YET5aeOrInzpEopRLZ9176kBXf/A1PvHkY78DY1V2d3mBSz02leerJYvb5ywgYwXeiNt6hKKUSVCBgeG5nEwb49q/ruOGxN3jlQCvGjNxz7/X1AMlTSx1SKKmnObI5YZ2FQ2fAKKVGsaepixanh0fWXci/37UaBO5+soa7/n0H9W09Z7R3+3uwYCfdmh6HaCcnZZI6wMnMBRS5dQaMUmpk2/a1kGbvxJq9j8svKGL7n13O/7lpCbuOdXL9Y2/yzf/ZT7fLN9i+P1R2N5mkVFL3FlxAWaCZvr6+eIeilEowxhi27W9h5vzt/M07f8Wdv76TPe27+MKlFbz2wFruWF3Gz37XwNrvvsp/vnuUAX8Ar+nDYc2Jd+gTklJJ3X5eFTYJ0Fi/N96hKKUSzIGWHo52nqLXsp+LZ11MV38Xn9/+eR54/QH66eDv/nApv/rqZVTOyuFvnt/HTT94Cz99ZGhSj58Z81cA0NmgM2CUUkO9uK+FtJw6/MbH/SvvZ8stW7hvxX283vg6Nz9/M5t2b6K82MbTX/oY//KZi+jtHwhWaEzTpB43s+ZX4TNW/Cf0KkhKqaG272thxswDlGSWsKx4GRm2DO5dfi9bbtnCVWVX8a97/pWbn7+ZF4+8yA1Vs3j5L66gIMdPZcnMeIc+ISmV1K1p6TTb5pDR/VG8Q1FKJZDD7b0cbG/HbavlunnXDbne6Ozs2Xzniu/w5A1PUugoZMObG/jcts9xyHkAb6CXGUlU9wVSLKkDdGbNp0RnwCilImzb34It+wB+4+O68utGbLNq5iqe/oOnefjjD3PUeZT1v1qPx+9Jqi8eQQomdV/hIuaYVrqd3fEORSmVILbta6FoZh0lmSUsL14+ajurxcqtF9zKr/7wV/yvJf8Lm9hYkL8ghpGevZRL6o45VVjEcPzD3fEORSmVAI53ufmguY1+Wy3Xzrt2yNDLaHLsOTyw+gF2/slOriy7MgZRTp2US+rFC4Kfwl1HdQaMUirYS7dl1+HHx3XzRh56GY1FLIjINEU2PaJK6iJyg4gcFJF6Edk4wuOXi8guERkQkdumPszozZy3mH6Thmmti2cYSqkEsX1fCwXFdZRklLCiZEW8w5l24yZ1EbECm4AbgSXAehFZMqzZMeAu4KmpDnCixJrG8bQyMnUGjFLnvPaefnYcO4HPXsc1866Jaugl2UWzh2uAemPMYWOMF9gMrItsYIxpMMZ8AIxdxzJGurIWMLv/SLzDUErF2W9qW7BmH8CPd9RZL6kmmqQ+B2iMWG4KrUtY/uLFzOIkHR0n4x2KUiqOtu1rIa+oluKMYlaWrIx3ODER079FROQeEakRkZr29vZpe53MOVUAHP9o17S9hlIqsXW7fLxzuBm/o/acGXqB6JL6caAsYrk0tG7CjDGPG2OqjTHVxcXFk9lEVGaevwKA3mNa2Eupc9XLda2QOblZL8ksmqS+A1goIhUiYgfuBLZMb1hnp2jO+bhIx7TpDBilzlUv7mshu6iWGRkzzpmhF4giqRtjBoD7ge1AHfCsMWa/iDwiIjcDiMhqEWkCPg38WETiWlFLLFaa0+aR49QZMEqdi/r6B3ij/jhk1HHN3GuwWqzxDilmbNE0MsZsBbYOW/dQxP0dBIdlEkZ3zvnMO/U2xpik+/KAUursvHqwjYCj9pya9RKWsmcOAsWLmUE3ba3N8Q5FKRVj2/a1kFW4nyJHEReVXBTvcGIqZZN6dulSAFo++n2cI1FKxZLH5+fVg02QGfzC0bk09AIpnNRnLQx+OnfseJY3332Xli4Xxpg4R6WUmm5vfXSSfvt+AnjPqVkvYVGNqSejgplzaUkr5SrnC7DtBZwvZlJjWcDJ3CWY2SspWLiGysoqCrPT4x2qUmoKbdvfQkb+PgodhayauSre4cRcyiZ1RJi1cQ/9J2o5UfcO7qM7mHlyL8u7f4G9+xk4AJ1bsnnPej6deRdiW3A5a2+8HZs1Zf94USrl+fwBflPXiGXeAa6Ze8s5N/QCqZzUAaw20kuXUV66DPhycN1AP31NH9BS9y7eYzs579ReVnVuxlbzX7yx92nm3/VjSmfPimvYSqnJee/wKVzWfWScg7NewlI7qY/Elk5W+WoWlK8+vc7npu6Xf8cnan9Iy79eyltXPMalV90UvxiVUpPy4r4TpOftoyC94JwceoEUPlE6IWkZLL7jW3TcvgWb1crHX/8sv9n0NfrcnnhHppSKkj9g2FbbiC3nANfMuwab5dzrs4Im9SFmXngZMx54j4MlN3Bd+5Mc+ccrqKvT+jFKJYNdxzrpZi8B+s/ZoRfQpH4GW2Y+S76ymfrLHqM8cIzSzdfy22d/SCCg0yGVSmTb9rVgz91Lfno+1TOrY/fCfh9sexB+9inonlStwymlSX0U51/9eQJffovWjAVcXft1fvfdW5nOcsFKqclp7+nnlQOt/OqDo6TFeuiltx3+Yx28+yNofB+euBqad8fmtUdxbg46RSl39gJy/uo19m5+iI9/+C+c2PQJdlz9A1ZfdkO8Q1PqnNTZ52Xv8W4+aOrig6Zu9h7v5kR38NyXPXc/6dIfuy8cndgDmz8Dfe3wR09AyWJ46g749xvh1idg0R/EJo5hJF7fsqyurjY1NTVxee3JaNrzKrYXvswMfzvHZDaGM4uEhdeF39F+cdCWvZiBWReRf8HHWLh4JQXZjhhGrVRy8gcMzV1uDrX3crClhw+auvngeBeNp9yDbebPyGJpaR5L5+SxrDSfZxr+nh2t7/HK7a9Mf0993y/g+a9AZiHc+V9wXqi0b08LPL0emn8P130LPn4/THFBQRHZaYwZdXxJk/oE9Pd2UvfM/8HW0wQwmNaFkd9Du7eL2a6DZBH8RXSaTA5aF3IqfylSuoriRZdQef4CMu36B5OaGl0uLzmONKyW5KhM2ts/wOH2Xg6193K4vW/w9sjJPvoHTl/yuLQgg+Wl+SwtzWPZnDwunJOHPW2Ag6cOsr9jP3UddWxr2MZN82/i4U88PH0BB/zwyrfgre/D3I/D7f8B2SVD23hd8PyfQu0LsOou+OR3wZo2ZSFoUo+3gJ++47WcqH2b/qM7yOnYzXn9h7GFrtF93MzgsL2SntyFULSAzFkLKZq7hLlzziMvY+p+EVTqOtbhYuu+E7y49wR7mro5L8/BbatK+XR1GWWFmfEOb4iGk328XNfK6x+282FrD63O/sHHLAJzCzNZUJzN/OKs0G02C0uySbcPcODUAepO1VHbUUttRy2Huw8TMMH/R4WOQqpmVPHgmgcpzZmmKuDuLvjFF6H+JVj1ebjxO2Czj9w2EAgl/+/B/LXw6Z9BRv6UhKFJPRF5XXQerqH9wO8INNZQ2LWXGf5WLBE9/k6TTZPM5lR6KZ7ccqRwPpmzLyBvVgX2rFzs6Vk40tNIt1lJt1lwpFmTpnemzl7DyT5+vfcEL+47wb7jTgCWl+axtrKE3Y1dvPFRO8bAJxYUccfqMq6/cBaOtNh/Zd7nD1DT0MkrB1r57YE2Drf3AbCwJJtlpfmDyXtBcRZzizLxBdw0OBs40n2EBmcDDd0N1HfVc6T7CCb0/6PIUcSSoiVDfmZmzpze6ya0fwib10NnQzCZr/5CdM/7/X/C//wZFM6HP34GCivOOhRN6snC56G//RAdxw7Q03wQf8ch7N1HyHM3UuRvH5LwwzwmDTfpuLHjNun0Y6df0ukXB16LA2dmGYFZy8lfsJrzFy1ndn6mXjAkiR1q72XrByfYuq+FuhPBRL5ybj6frJrNDVWzhvTKm7vcPLeziZ/vbKTxlJtch411K+Zwe3UZVXNyp/X3oLPPy+sftg/2yHs8A9itFi6eX8jVi0q4evFMrPauwWQdTt4NzgZOuk8ObsciFuZkz2F+3vwhCbwks2SMV58GH24P9tCtdrjj/8G8T0zs+UfehGc+CxYr3Pk0zL34rMLRpJ4KfB7c7Yc4ebQO18ljmH4XxucCnwt8bsTnwjLgxuL3YBlwY/O7SfO7KOpvwo4PAKfJ4COZz8ncxZjZKyg8v1pP3CaAQMDQ4xmgy+2l0+Wjy+Wl2+2jy+Wj0+Wly+Wj2+2jttnJwdYeAKrnFXDj0mAin5OfgTGGupO1vN74W95v2UFFXgXXVdxI9azVWLDy7uEOnq1p5MV9LfQPBFg8O5fbq0u5/IJiXP3+wdfuDr1eV+j1u1xeutzB1zfGkGa1kGa1YLNK6L5gswRvg+sttHS72Xm0k4CBGdnpXLWomKsWzeRjC/I55NzP642v81rTaxzpPjL4HuSn51OeW055XjnzcudRkVtBeV45ZTll2K2jDG9MhDHg7YXetuBMld426HcOffz0wtDndtTD2/8XZi+DO/4L8ssmF8PJenjq08F57Lf8CJbeNrntoEn93Ob30d+8n5aD7+I+uouMk3uZ5a4nHS8AvcZBvXU+3ZnzMFgAg5gAYEK/24HTv/DGIASGbD7yBPFI/T6DYERC2w7fFwitMwSXjcWGERsBiw1jSYPBW2vw1pqGsdjAYgvGZwxmMFbABBBjQnEH4zTGYEwAMQFMIDC4L8YEQu0DwfsBc3o/TfB5mPB+n14/+D5IeJbT6dPkkbMbBmdFhV4n+Hx/KO7AsG0HMP4Bgu9GACsBLKH7FgxWAqRZId1iyLRBcYYh3x4gLdCPe6Cf9yxeXkuDN9NttNmsiDFUen0cTbPhtlgoCAS4ymfhOpPJ6rR8sGXT5LJxsBOO9VlxYx88hhJxPO1WIcNmIT3NgiPNQrrNAljwmdAPVnyB8LIM3vcaC/Y0G5Wzcll8Xh75uRbe6T3C68563nQeotvvxiYWqrPncUXeBVRlzqE8vZB8W+bgsQu/i+HfucFlE3Eswm2H3A++z7hOQV9bcP54X1swgfe2wcDpWTMTVnUb3PwDsJ/l+QnXqWCP/ejbcNNjUP35SW1mSpK6iNwA/DNgBZ4wxjw67PF04D+AVUAHcIcxpmGsbWpSjxO/j77j+2k+8C6eo7vI7NhLQX8zAgQklHwhdBv6ry6cTsxnbDByGmdkajeDrS0Ek25wi4Fh6wJY8WM1fqwMYMM/4lDTdAiEP1SG/FiG7G9gMGkH92bI3oY/SIavxwTTslhC2wtvN7gc/lBDBCxWxGLFEv6xWrFarVitNqzW4GOIBSw2WqxW3rB4ec308X6gh34CZImNT6TP5IqsuVyaXUFReh5uby9vO+v5Te8RXve24SJAPhau8qdxncfPGlcfVo8Ty4B78EOW8F6IhPYl/OELfSIETAAJDITeldM/FmOGLDfbrLyemclrmRnsdKQzIEK+389lLg9XuFxc4vaQPa0dSYGsGZBVAtnFodsSyCoO3maXBNc5coPva+TzhmwmtGxJg5yZUxfeQD+8/DB87F7InzupTZx1UhcRK/AhcC3QBOwA1htjaiPa3AcsM8b8qYjcCfyhMeaOsbarSV2NKuAPfvU64IPAAPgHQvf9gwkWsYxyP/xjOfOHyPVDe9jTyYT+sgiE/hIIECBggj8un4sebw9Or5NeXy893p7Ty97Ty4e7D3Ow8yAApdmlrC1by+Wll1M9s5q0MabLeQY8vN38Ni8dfYnXGl+jz9dHrj2Xq+ZexcqSlfT5+uju78bpdQ7eOvuddHu7cfY7cXqd+I1/wvs8P7uMK2auZu3M1SwvuACrWMNvBoO968gPExh6PM54zDLyMR88rqH76blgTe0pwlOR1D8OPGyMuT60/CCAMebvI9psD7V5R0RsQAtQbMbY+GST+pP7nuSxXY8hIgiCRSxYQp+4FrFgwQJy+n7kCaHBPkjouYPrwh/KYkHC/+T0bXh9uA2AwQy5PF74zHz4P/CY90PPNYPDHBFk9DjD+xJOCOHthO8PXx9+jlWsQ24tYgneR7BagreR+zMkvvD+DTuUw9/X8HLk/dH2P3J5xMcNZ7xXw0W+XuTtsD8WBl8jct8i92cw2YaORSA8dDPsvY2MPXK/R7oPp7czuO1JsFls5NpzybHnMDNzJpfNuYzLyy6nIrdiUic6+/39vNP8Dr9p+A2vNr5Kr693MPYcew659lzy0vPIS88bvJ9rzyXXnovVYh364RS6BYZ8WOXZ87hszmWU5U5y7FmNa7ykHs1H2hygMWK5CRh++nawjTFmQES6gSLgZGQjEbkHuAdg7tzJ/emxtHgpd1fdPeSXa8h/PnPmL91ICTVseFKJfM7w/+Dhx8f7cDgjyQ1bjnxu5DZGSqyRMYbjsohlyIdN+INtpA8hv/EPJvvw++IP+IO3ocfCHwAjfaBELkcaEucoiTm8vdHeixHfq2Hv0/D3efgxGxJTxPrR3u/hrwMMeR/PWJaR4x7pdyhyOfyhOXhsRLBgGXLsLFjITMskx55Djj2H7LTswSSebc/GYXVM6SyVdGs6a8vWsrZsLV6/l1ZXK7n2XLLTss/JKwSlqpj+nWKMeRx4HII99clsY9XMVeds8XulpordaqcsR3vTqSiaKo3HgcijXxpaN2Kb0PBLHsETpkoppWIomqS+A1goIhUiYgfuBLYMa7MF+Fzo/m3AK2ONpyullJoe4w6/hMbI7we2E5zS+FNjzH4ReQSoMcZsAX4C/D8RqQdOEUz8SimlYiyqMXVjzFZg67B1D0Xc9wCfntrQlFJKTZRe+UgppVKIJnWllEohmtSVUiqFaFJXSqkUErcqjSLSDhyd5NNnMOzbqikg1fYp1fYHUm+fUm1/IPX2aaT9mWeMKR7tCXFL6mdDRGrGqn2QjFJtn1JtfyD19inV9gdSb58msz86/KKUUilEk7pSSqWQZE3qj8c7gGmQavuUavsDqbdPqbY/kHr7NOH9ScoxdaWUUiNL1p66UkqpEWhSV0qpFJJ0SV1EbhCRgyJSLyIb4x3P2RKRBhHZKyK7RSQpL9oqIj8VkTYR2RexrlBEXhKRj0K3BfGMcSJG2Z+HReR46DjtFpFPxjPGiRKRMhF5VURqRWS/iHwttD4pj9MY+5O0x0lEHCLyvojsCe3TN0PrK0TkvVDOeyZUAn307STTmHo0F8FONiLSAFQbY5L2CxMicjnQC/yHMaYqtO47wCljzKOhD98CY8yGeMYZrVH252Gg1xjz3XjGNlkiMhuYbYzZJSI5wE7gFuAukvA4jbE/t5Okx0mC1y7MMsb0ikga8BbwNeAvgF8aYzaLyL8Ce4wx/zLadpKtp74GqDfGHDbGeIHNwLo4x3TOM8a8QbCOfqR1wM9C939G8D9cUhhlf5KaMeaEMWZX6H4PUEfw2sJJeZzG2J+kZYJ6Q4tpoR8DXAU8F1o/7jFKtqQ+0kWwk/pAEjxovxGRnaELc6eKmcaYE6H7LcDMeAYzRe4XkQ9CwzNJMUwxEhEpB1YC75ECx2nY/kASHycRsYrIbqANeAk4BHQZYwZCTcbNecmW1FPRpcaYi4Abga+E/vRPKaFLGybPON/I/gVYAKwATgD/FNdoJklEsoFfAH9mjHFGPpaMx2mE/Unq42SM8RtjVhC8FvQaYNFEt5FsST2ai2AnFWPM8dBtG/DfBA9kKmgNjXuGxz/b4hzPWTHGtIb+wwWAfyMJj1NonPYXwH8ZY34ZWp20x2mk/UmF4wRgjOkCXgU+DuSLSPgqdePmvGRL6tFcBDtpiEhW6CQPIpIFXAfsG/tZSSPyYuSfA16IYyxnLZz4Qv6QJDtOoZNwPwHqjDHfi3goKY/TaPuTzMdJRIpFJD90P4PghJA6gsn9tlCzcY9RUs1+AQhNUXqM0xfB/tv4RjR5IjKfYO8cgteLfSoZ90dEngbWEiwT2gp8A3geeBaYS7DE8u3GmKQ4+TjK/qwl+Ce9ARqAL0eMRSc8EbkUeBPYCwRCq/+a4Dh00h2nMfZnPUl6nERkGcEToVaCHe5njTGPhPLEZqAQ+D3wWWNM/6jbSbakrpRSanTJNvyilFJqDJrUlVIqhWhSV0qpFKJJXSmlUogmdaWUSiGa1JVSKoVoUldKqRTy/wGVhnoGUwYzXgAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "import matplotlib.pyplot as plt\n", "import numpy as np\n", @@ -2042,9 +1944,9 @@ "\n", "np.random.seed(2018)\n", "\n", - "n = 40\n", + "n = 400\n", "n_boostraps = 100\n", - "maxdegree = 14\n", + "maxdegree = 30\n", "\n", "\n", "# Make data set.\n", @@ -2083,9 +1985,7 @@ { "cell_type": "markdown", "id": "0edc7ea3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Summing up\n", "\n", @@ -2121,9 +2021,7 @@ { "cell_type": "markdown", "id": "dc1b4e05", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Another Example from Scikit-Learn's Repository" ] @@ -2132,10 +2030,7 @@ "cell_type": "code", "execution_count": 8, "id": "01de9323", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\"\"\"\n", @@ -2214,9 +2109,7 @@ { "cell_type": "markdown", "id": "5ad3f9ac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Various steps in cross-validation\n", "\n", @@ -2239,9 +2132,7 @@ { "cell_type": "markdown", "id": "5916aa2e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Cross-validation in brief\n", "\n", @@ -2267,9 +2158,7 @@ { "cell_type": "markdown", "id": "ea8461c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code Example for Cross-validation and $k$-fold Cross-validation\n", "\n", @@ -2280,10 +2169,7 @@ "cell_type": "code", "execution_count": 9, "id": "1a35e97f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -2380,9 +2266,7 @@ { "cell_type": "markdown", "id": "109bd9f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More examples on bootstrap and cross-validation and errors" ] @@ -2391,10 +2275,7 @@ "cell_type": "code", "execution_count": 10, "id": "a00dca0c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Common imports\n", @@ -2480,9 +2361,7 @@ { "cell_type": "markdown", "id": "8a218b0e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that we kept the intercept column in the fitting here. This means that we need to set the **intercept** in the call to the **Scikit-Learn** function as **False**. Alternatively, we could have set up the design matrix $X$ without the first column of ones." ] @@ -2490,9 +2369,7 @@ { "cell_type": "markdown", "id": "c883f354", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The same example but now with cross-validation\n", "\n", @@ -2503,10 +2380,7 @@ "cell_type": "code", "execution_count": 11, "id": "a399bb68", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Common imports\n", @@ -2579,7 +2453,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.14" + } + }, "nbformat": 4, "nbformat_minor": 5 }