From 4f117041b11574fd80744be6d4cbad5ebe39470a Mon Sep 17 00:00:00 2001 From: Morten Hjorth-Jensen Date: Wed, 18 Sep 2024 15:26:11 +0200 Subject: [PATCH] Update week38.ipynb --- doc/pub/week38/ipynb/week38.ipynb | 763 +++++++++--------------------- 1 file changed, 216 insertions(+), 547 deletions(-) diff --git a/doc/pub/week38/ipynb/week38.ipynb b/doc/pub/week38/ipynb/week38.ipynb index 68998261e..0f6bb91d0 100644 --- a/doc/pub/week38/ipynb/week38.ipynb +++ b/doc/pub/week38/ipynb/week38.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "a811ba80", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "e5014a9c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 38: Logistic Regression and Optimization\n", "**Morten Hjorth-Jensen**, Department of Physics and Center for Computing in Science Education, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -28,9 +24,7 @@ { "cell_type": "markdown", "id": "023eb6d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plans for week 38, lecture Monday September 16\n", "\n", @@ -48,9 +42,7 @@ { "cell_type": "markdown", "id": "e981c015", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Suggested reading and videos\n", " * Readings and Videos:\n", @@ -71,9 +63,7 @@ { "cell_type": "markdown", "id": "11590c09", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plans for the lab sessions\n", "\n", @@ -93,9 +83,7 @@ { "cell_type": "markdown", "id": "57e011be", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Material for lecture Monday September 16" ] @@ -103,9 +91,7 @@ { "cell_type": "markdown", "id": "0896e712", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Logistic Regression\n", "\n", @@ -125,9 +111,7 @@ { "cell_type": "markdown", "id": "44bb3650", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Classification problems\n", "\n", @@ -151,9 +135,7 @@ { "cell_type": "markdown", "id": "921c6771", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization and Deep learning\n", "\n", @@ -175,9 +157,7 @@ { "cell_type": "markdown", "id": "f80e9666", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Basics\n", "\n", @@ -200,9 +180,7 @@ { "cell_type": "markdown", "id": "952f8119", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\ 1 & \\mathrm{yes} \\end{bmatrix}.\n", @@ -212,9 +190,7 @@ { "cell_type": "markdown", "id": "9b587b40", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linear classifier\n", "\n", @@ -230,9 +206,7 @@ { "cell_type": "markdown", "id": "bfb711d7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -248,9 +222,7 @@ { "cell_type": "markdown", "id": "0acaaf3c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n", "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors." @@ -259,9 +231,7 @@ { "cell_type": "markdown", "id": "73564ce7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Some selected properties\n", "\n", @@ -286,9 +256,7 @@ { "cell_type": "markdown", "id": "ef6011fd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example\n", "\n", @@ -299,10 +267,7 @@ "cell_type": "code", "execution_count": 1, "id": "3444ad7b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -366,9 +331,7 @@ { "cell_type": "markdown", "id": "01d01242", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plotting the mean value for each group\n", "\n", @@ -379,10 +342,7 @@ "cell_type": "code", "execution_count": 2, "id": "143c59fe", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "agegroupmean = np.array([0.1, 0.133, 0.250, 0.333, 0.462, 0.625, 0.765, 0.800])\n", @@ -398,9 +358,7 @@ { "cell_type": "markdown", "id": "42136436", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We are now trying to find a function $f(y\\vert x)$, that is a function which gives us an expected value for the output $y$ with a given input $x$.\n", "In standard linear regression with a linear dependence on $x$, we would write this in terms of our model" @@ -409,9 +367,7 @@ { "cell_type": "markdown", "id": "e8a7f059", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n", @@ -421,9 +377,7 @@ { "cell_type": "markdown", "id": "f1c0bcf8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This expression implies however that $f(y_i\\vert x_i)$ could take any\n", "value from minus infinity to plus infinity. If we however let\n", @@ -440,9 +394,7 @@ { "cell_type": "markdown", "id": "e4fd2845", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The logistic function\n", "\n", @@ -462,9 +414,7 @@ { "cell_type": "markdown", "id": "f4bb77ad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n", @@ -474,9 +424,7 @@ { "cell_type": "markdown", "id": "47fc800d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that $1-p(t)= p(-t)$." ] @@ -484,9 +432,7 @@ { "cell_type": "markdown", "id": "0fe9154b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Examples of likelihood functions used in logistic regression and nueral networks\n", "\n", @@ -497,10 +443,7 @@ "cell_type": "code", "execution_count": 3, "id": "150c4acd", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\"\"\"The sigmoid function (or the logistic curve) is a\n", @@ -562,9 +505,7 @@ { "cell_type": "markdown", "id": "9c1d64b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Two parameters\n", "\n", @@ -574,9 +515,7 @@ { "cell_type": "markdown", "id": "d1929423", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -589,9 +528,7 @@ { "cell_type": "markdown", "id": "1698b9e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", "\n", @@ -601,9 +538,7 @@ { "cell_type": "markdown", "id": "eff2f862", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n", @@ -613,9 +548,7 @@ { "cell_type": "markdown", "id": "640f9f45", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Maximum likelihood\n", "\n", @@ -630,9 +563,7 @@ { "cell_type": "markdown", "id": "f94fafba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -644,9 +575,7 @@ { "cell_type": "markdown", "id": "5d457b2e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "from which we obtain the log-likelihood and our **cost/loss** function" ] @@ -654,9 +583,7 @@ { "cell_type": "markdown", "id": "683657ba", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n", @@ -666,9 +593,7 @@ { "cell_type": "markdown", "id": "3d17d95b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The cost function rewritten\n", "\n", @@ -678,9 +603,7 @@ { "cell_type": "markdown", "id": "76cd7541", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", @@ -690,9 +613,7 @@ { "cell_type": "markdown", "id": "b88061e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n", "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that" @@ -701,9 +622,7 @@ { "cell_type": "markdown", "id": "3c95fe37", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", @@ -713,9 +632,7 @@ { "cell_type": "markdown", "id": "4f573bed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n", "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression." @@ -724,9 +641,7 @@ { "cell_type": "markdown", "id": "08a700a8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Minimizing the cross entropy\n", "\n", @@ -740,9 +655,7 @@ { "cell_type": "markdown", "id": "9bd6709b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n", @@ -752,9 +665,7 @@ { "cell_type": "markdown", "id": "98c81b67", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -762,9 +673,7 @@ { "cell_type": "markdown", "id": "5540b76a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n", @@ -774,9 +683,7 @@ { "cell_type": "markdown", "id": "0018d823", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A more compact expression\n", "\n", @@ -789,9 +696,7 @@ { "cell_type": "markdown", "id": "ee63f4f9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -801,9 +706,7 @@ { "cell_type": "markdown", "id": "413ff641", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -812,9 +715,7 @@ { "cell_type": "markdown", "id": "337a2c56", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -824,9 +725,7 @@ { "cell_type": "markdown", "id": "8c3e92fe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Extending to more predictors\n", "\n", @@ -836,9 +735,7 @@ { "cell_type": "markdown", "id": "ba84fae7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n", @@ -848,9 +745,7 @@ { "cell_type": "markdown", "id": "bddd73d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to" ] @@ -858,9 +753,7 @@ { "cell_type": "markdown", "id": "fce6aba6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n", @@ -870,9 +763,7 @@ { "cell_type": "markdown", "id": "63325aad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Including more classes\n", "\n", @@ -884,9 +775,7 @@ { "cell_type": "markdown", "id": "1c5878f6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n", @@ -896,9 +785,7 @@ { "cell_type": "markdown", "id": "2c8a1b85", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -906,9 +793,7 @@ { "cell_type": "markdown", "id": "cced4ec8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n", @@ -918,9 +803,7 @@ { "cell_type": "markdown", "id": "6efd1ce1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and so on till the class $C=K-1$ class" ] @@ -928,9 +811,7 @@ { "cell_type": "markdown", "id": "933753b8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n", @@ -940,9 +821,7 @@ { "cell_type": "markdown", "id": "ba94450f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and the model is specified in term of $K-1$ so-called log-odds or\n", "**logit** transformations." @@ -951,9 +830,7 @@ { "cell_type": "markdown", "id": "8f174f5d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More classes\n", "\n", @@ -974,9 +851,7 @@ { "cell_type": "markdown", "id": "9ba36ed7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n", @@ -986,9 +861,7 @@ { "cell_type": "markdown", "id": "b5b5ecc6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It is easy to extend to more predictors. The final class is" ] @@ -996,9 +869,7 @@ { "cell_type": "markdown", "id": "e6b33699", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n", @@ -1008,9 +879,7 @@ { "cell_type": "markdown", "id": "b49a6a23", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and they sum to one. Our earlier discussions were all specialized to\n", "the case with two classes only. It is easy to see from the above that\n", @@ -1025,9 +894,7 @@ { "cell_type": "markdown", "id": "e9bfd38c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Searching for Optimal Regularization Parameters $\\lambda$\n", "\n", @@ -1044,10 +911,7 @@ "cell_type": "code", "execution_count": 4, "id": "5dc4fd0e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1099,9 +963,7 @@ { "cell_type": "markdown", "id": "70944449", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n", "By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n", @@ -1111,9 +973,7 @@ { "cell_type": "markdown", "id": "1538973c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Grid Search\n", "\n", @@ -1126,10 +986,7 @@ "cell_type": "code", "execution_count": 5, "id": "1c1fdba0", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1179,9 +1036,7 @@ { "cell_type": "markdown", "id": "dd6f78eb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "By default the grid search function includes cross validation with\n", "five folds. The [Scikit-Learn\n", @@ -1194,9 +1049,7 @@ { "cell_type": "markdown", "id": "20b7afcb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Randomized Grid Search\n", "\n", @@ -1214,10 +1067,7 @@ "cell_type": "code", "execution_count": 6, "id": "4810b670", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1268,9 +1118,7 @@ { "cell_type": "markdown", "id": "0696dfc9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Wisconsin Cancer Data\n", "\n", @@ -1283,10 +1131,7 @@ "cell_type": "code", "execution_count": 7, "id": "c55d1159", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1310,9 +1155,7 @@ { "cell_type": "markdown", "id": "b83cd520", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using the correlation matrix\n", "\n", @@ -1324,10 +1167,7 @@ "cell_type": "code", "execution_count": 8, "id": "5497a1d8", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1369,9 +1209,7 @@ { "cell_type": "markdown", "id": "e9552a3c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Discussing the correlation data\n", "\n", @@ -1394,10 +1232,7 @@ "cell_type": "code", "execution_count": 9, "id": "623ddee7", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)" @@ -1406,9 +1241,7 @@ { "cell_type": "markdown", "id": "7a61e306", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and then" ] @@ -1417,10 +1250,7 @@ "cell_type": "code", "execution_count": 10, "id": "859552c6", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "correlation_matrix = cancerpd.corr().round(1)" @@ -1429,9 +1259,7 @@ { "cell_type": "markdown", "id": "43d915d7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Diagonalizing this matrix we can in turn say something about which\n", "features are of relevance and which are not. This leads us to\n", @@ -1442,9 +1270,7 @@ { "cell_type": "markdown", "id": "5c8e892e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Other measures in classification studies: Cancer Data again" ] @@ -1453,10 +1279,7 @@ "cell_type": "code", "execution_count": 11, "id": "08b680f2", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1496,9 +1319,7 @@ { "cell_type": "markdown", "id": "fe0c7fda", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -1517,9 +1338,7 @@ { "cell_type": "markdown", "id": "9df4ecc4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -1534,9 +1353,7 @@ { "cell_type": "markdown", "id": "a0a65501", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -1549,9 +1366,7 @@ { "cell_type": "markdown", "id": "8dc1194c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] @@ -1559,9 +1374,7 @@ { "cell_type": "markdown", "id": "62d70952", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations to solve\n", "\n", @@ -1575,9 +1388,7 @@ { "cell_type": "markdown", "id": "41787d1e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -1587,9 +1398,7 @@ { "cell_type": "markdown", "id": "86fc7282", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -1598,9 +1407,7 @@ { "cell_type": "markdown", "id": "8f4c640e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -1610,9 +1417,7 @@ { "cell_type": "markdown", "id": "f37d28ca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This defines what is called the Hessian matrix." ] @@ -1620,9 +1425,7 @@ { "cell_type": "markdown", "id": "9b5cb6dd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -1634,9 +1437,7 @@ { "cell_type": "markdown", "id": "f474d68a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -1646,9 +1447,7 @@ { "cell_type": "markdown", "id": "ff928190", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or in matrix form as" ] @@ -1656,9 +1455,7 @@ { "cell_type": "markdown", "id": "a9e9efc2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -1668,9 +1465,7 @@ { "cell_type": "markdown", "id": "93061994", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -1680,9 +1475,7 @@ { "cell_type": "markdown", "id": "08acd443", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -1700,9 +1493,7 @@ { "cell_type": "markdown", "id": "caa94b50", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations\n", "\n", @@ -1716,9 +1507,7 @@ { "cell_type": "markdown", "id": "ac3e7ef2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -1732,9 +1521,7 @@ { "cell_type": "markdown", "id": "6bd1aafd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -1743,9 +1530,7 @@ { "cell_type": "markdown", "id": "699697a1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -1755,9 +1540,7 @@ { "cell_type": "markdown", "id": "4efbdd72", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "yielding" ] @@ -1765,9 +1548,7 @@ { "cell_type": "markdown", "id": "4bd64a59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -1777,9 +1558,7 @@ { "cell_type": "markdown", "id": "358dc6db", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] @@ -1787,9 +1566,7 @@ { "cell_type": "markdown", "id": "8a007c48", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -1799,9 +1576,7 @@ { "cell_type": "markdown", "id": "e0828d1d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple geometric interpretation\n", "\n", @@ -1821,9 +1596,7 @@ { "cell_type": "markdown", "id": "26efa0c4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Extending to more than one variable\n", "\n", @@ -1834,9 +1607,7 @@ { "cell_type": "markdown", "id": "8af30001", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -1847,9 +1618,7 @@ { "cell_type": "markdown", "id": "77528641", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we Taylor expand to obtain" ] @@ -1857,9 +1626,7 @@ { "cell_type": "markdown", "id": "d10154f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -1875,9 +1642,7 @@ { "cell_type": "markdown", "id": "58a6cb05", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] @@ -1885,9 +1650,7 @@ { "cell_type": "markdown", "id": "87917443", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -1900,9 +1663,7 @@ { "cell_type": "markdown", "id": "316440eb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we can rephrase Newton's method as" ] @@ -1910,9 +1671,7 @@ { "cell_type": "markdown", "id": "4ec22184", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -1924,9 +1683,7 @@ { "cell_type": "markdown", "id": "9da35b82", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined" ] @@ -1934,9 +1691,7 @@ { "cell_type": "markdown", "id": "61c4f7fc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -1948,9 +1703,7 @@ { "cell_type": "markdown", "id": "ffd39c16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -1963,9 +1716,7 @@ { "cell_type": "markdown", "id": "de590520", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent\n", "\n", @@ -1980,9 +1731,7 @@ { "cell_type": "markdown", "id": "6a0e0292", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -1992,9 +1741,7 @@ { "cell_type": "markdown", "id": "ec6877a5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -2006,9 +1753,7 @@ { "cell_type": "markdown", "id": "b7e72c2f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on Steepest descent\n", "\n", @@ -2021,9 +1766,7 @@ { "cell_type": "markdown", "id": "cae90d84", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -2033,9 +1776,7 @@ { "cell_type": "markdown", "id": "1ab31b86", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -2044,9 +1785,7 @@ { "cell_type": "markdown", "id": "87d0d18e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The ideal\n", "\n", @@ -2072,9 +1811,7 @@ { "cell_type": "markdown", "id": "c92a82a1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -2094,9 +1831,7 @@ { "cell_type": "markdown", "id": "b5a9af46", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex functions\n", "\n", @@ -2116,9 +1851,7 @@ { "cell_type": "markdown", "id": "77ee5272", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex function\n", "\n", @@ -2128,9 +1861,7 @@ { "cell_type": "markdown", "id": "282df4c7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conditions on convex functions\n", "\n", @@ -2165,9 +1896,7 @@ { "cell_type": "markdown", "id": "e435596b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on convex functions\n", "\n", @@ -2193,9 +1922,7 @@ { "cell_type": "markdown", "id": "7bc1bf29", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Some simple problems\n", "\n", @@ -2223,9 +1950,7 @@ { "cell_type": "markdown", "id": "90fef1a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our first homework\n", "\n", @@ -2247,10 +1972,7 @@ "cell_type": "code", "execution_count": 12, "id": "1c59342a", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -2260,9 +1982,7 @@ { "cell_type": "markdown", "id": "79d0e3da", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -2271,9 +1991,7 @@ { "cell_type": "markdown", "id": "ec79a08a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -2283,9 +2001,7 @@ { "cell_type": "markdown", "id": "fa4910ae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "such that" ] @@ -2293,9 +2009,7 @@ { "cell_type": "markdown", "id": "e7665e13", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -2305,9 +2019,7 @@ { "cell_type": "markdown", "id": "90ffc363", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent example\n", "\n", @@ -2319,9 +2031,7 @@ { "cell_type": "markdown", "id": "3aa073fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -2335,9 +2045,7 @@ { "cell_type": "markdown", "id": "e1ddc571", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The cost/loss/risk function is given by (" ] @@ -2345,9 +2053,7 @@ { "cell_type": "markdown", "id": "5709f3d7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -2357,9 +2063,7 @@ { "cell_type": "markdown", "id": "b7b3b90f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] @@ -2367,9 +2071,7 @@ { "cell_type": "markdown", "id": "6651ef6c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -2379,9 +2081,7 @@ { "cell_type": "markdown", "id": "646be0cc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2393,9 +2093,7 @@ { "cell_type": "markdown", "id": "b6f528c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $X$ is the design matrix defined above." ] @@ -2403,9 +2101,7 @@ { "cell_type": "markdown", "id": "ae40f47b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -2414,9 +2110,7 @@ { "cell_type": "markdown", "id": "592c656d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2429,9 +2123,7 @@ { "cell_type": "markdown", "id": "aaff093b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] @@ -2439,9 +2131,7 @@ { "cell_type": "markdown", "id": "dd177bee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple program\n", "\n", @@ -2451,9 +2141,7 @@ { "cell_type": "markdown", "id": "94ead835", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -2463,9 +2151,7 @@ { "cell_type": "markdown", "id": "75c4e856", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -2478,9 +2164,7 @@ { "cell_type": "markdown", "id": "228edb14", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient Descent Example\n", "\n", @@ -2491,10 +2175,7 @@ "cell_type": "code", "execution_count": 13, "id": "46647c95", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -2548,9 +2229,7 @@ { "cell_type": "markdown", "id": "e0bb3c65", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -2559,10 +2238,7 @@ "cell_type": "code", "execution_count": 14, "id": "d29a0ccf", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Importing various packages\n", @@ -2586,9 +2262,7 @@ { "cell_type": "markdown", "id": "7d20e2cc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -2598,9 +2272,7 @@ { "cell_type": "markdown", "id": "52a46927", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -2610,9 +2282,7 @@ { "cell_type": "markdown", "id": "446851f9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] @@ -2620,9 +2290,7 @@ { "cell_type": "markdown", "id": "dc10da38", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2634,9 +2302,7 @@ { "cell_type": "markdown", "id": "e15d77aa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] @@ -2644,9 +2310,7 @@ { "cell_type": "markdown", "id": "89cd7379", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -2656,9 +2320,7 @@ { "cell_type": "markdown", "id": "20a6a0b6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -2667,9 +2329,7 @@ { "cell_type": "markdown", "id": "2bcf31af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2682,9 +2342,7 @@ { "cell_type": "markdown", "id": "3f9a5445", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -2696,9 +2354,7 @@ { "cell_type": "markdown", "id": "003f6d0d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Program example for gradient descent with Ridge Regression" ] @@ -2707,10 +2363,7 @@ "cell_type": "code", "execution_count": 15, "id": "bb679580", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from random import random, seed\n", @@ -2768,9 +2421,7 @@ { "cell_type": "markdown", "id": "2050684c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -2790,9 +2441,7 @@ { "cell_type": "markdown", "id": "6b20b26d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Challenge yourself the coming weekend\n", "\n", @@ -2802,9 +2451,7 @@ { "cell_type": "markdown", "id": "3570021a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Lab session: Material from last week and relevant for the first project" ] @@ -2812,9 +2459,7 @@ { "cell_type": "markdown", "id": "c5f36ff0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Various steps in cross-validation\n", "\n", @@ -2837,9 +2482,7 @@ { "cell_type": "markdown", "id": "a6e47b16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## How to set up the cross-validation for Ridge and/or Lasso\n", "\n", @@ -2853,9 +2496,7 @@ { "cell_type": "markdown", "id": "5b7545c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -2869,9 +2510,7 @@ { "cell_type": "markdown", "id": "fa3a49a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "* Evaluate the prediction performance of these models on the test set by $C[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n", "\n", @@ -2883,9 +2522,7 @@ { "cell_type": "markdown", "id": "685304e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Cross-validation in brief\n", "\n", @@ -2911,9 +2548,7 @@ { "cell_type": "markdown", "id": "9cac2104", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code Example for Cross-validation and $k$-fold Cross-validation\n", "\n", @@ -2922,13 +2557,21 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 6, "id": "1134c2ed", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGwCAYAAACHJU4LAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABmvklEQVR4nO3dd1yV5f/H8ddhHTYIAoJMQUUUHLhQc+U2s+EqM7VpqWnDyva2rNTKhlrZ0LLSLHOPcu+BIjhQUVBUFJDNYZz79wff+EWOFIHrnMPn+Xicx4Nzn3u87+PB8+G6r/u6dJqmaQghhBBCmCAr1QGEEEIIIa5GChUhhBBCmCwpVIQQQghhsqRQEUIIIYTJkkJFCCGEECZLChUhhBBCmCwpVIQQQghhsmxUB7gZRqOR1NRUXFxc0Ol0quMIIYQQ4jpomkZOTg5+fn5YWV27zcSsC5XU1FQCAgJUxxBCCCFEJaSkpODv73/Ndcy6UHFxcQHKTtTV1VVxGiGEEEJcj+zsbAICAsq/x6/FrAuVvy/3uLq6SqEihBBCmJnr6bYhnWmFEEIIYbKkUBFCCCGEyZJCRQghhBAmy6z7qFyv0tJSiouLVccQFs7W1hZra2vVMYQQwqJYdKGiaRrnzp3j0qVLqqOIWsLd3Z169erJuD5CCFFFLLpQ+btI8fb2xtHRUb48RLXRNI38/HzS0tIA8PX1VZxICCEsg8UWKqWlpeVFiqenp+o4ohZwcHAAIC0tDW9vb7kMJIQQVcBiO9P+3SfF0dFRcRJRm/z9eZM+UUIIUTUstlD5m1zuETVJPm9CCFG1LL5QEUIIIYT5kkJFCCGEECZLChWhlE6n47ffflMdQwghhImSQkUIIcR1KzVqXDiVSsGlHNVRRC0hhYqZk7tLqkdRUZHqCEKYlPPZhUz+NY4Wr69m+V2P4FDHlcMNIomd+R2a0ag6nrBgtapQ0TSN/KKSGn9omnZDOY1GI++99x5hYWHo9XoCAwN5++23OXnyJDqdjp9//pmuXbtib2/PvHnzMBqNvPHGG/j7+6PX62nRogUrV64s319RURHjxo3D19cXe3t7goODmTJlSvnrr732GoGBgej1evz8/HjiiSf+M+PkyZNp3779ZcujoqJ49dVXAdi1axc9e/akbt26uLm50aVLF/bu3XtD78X1nsOlS5d45JFH8PHxwd7enmbNmrF06dLy1xctWkTTpk3R6/UEBwfz4YcfVth/cHAwb731FqNGjcLNzY2HH34YgK1bt9K5c2ccHBwICAjgiSeeIC8vr1LnIIS52j9/Cbe/u5IfdyaTYyjhjKs3AOFJB2kxfiS7u9+BIS9fcUphqSx2wLcrKSguJeKVVTV+3IQ3euNod/1v9eTJk5kzZw7Tp0+nU6dOnD17lsOHD5e//txzz/Hhhx8yd+5c9Ho9H330ER9++CGzZs2iZcuWfP3119x+++3Ex8fTsGFDPv74Y5YsWcLPP/9MYGAgKSkppKSkALBw4UKmT5/OggULaNq0KefOnWP//v3/mXH48OG8++67HD9+nNDQUADi4+OJi4tj4cKFAOTk5DBy5Eg+/vhjAD788EP69etHYmIiLi4u1/1+ANc8B6PRSN++fcnJyWHevHmEhoaSkJBQPuDanj17GDJkCK+99hpDhw5l69atPP7443h6ejJq1KjyY7z//vu8/PLLvPTSSwDExcXRu3dv3nzzTb766isuXLjAuHHjGDduHHPnzr2h/EKYqwNfLqDJoyN4Nawtn499lxf6R9DmrT6kJb7I8Vffpe3Cr2mz4Q9iO/am6fa12NrrVUcWFkan3eif+yYkOzsbNzc3srKycHV1rfBaYWEhSUlJhISEYG9vD0B+UYnJFyo5OTl4eXkxc+ZMHnrooQqvnTx5kpCQEGbMmMGECRPKl9evX5+xY8fywgsvlC9r27Ytbdq04dNPP+WJJ54gPj6etWvXXjbOx7Rp05g1axYHDx7E1tb2hs6refPmDBo0iJdffhmAF154gbVr17Jz584rrl9aWkqdOnX44YcfuO2224CyzrSLFy/mjjvuuOaxrnUOq1evpm/fvhw6dIhGjRpdtu3w4cO5cOECq1evLl/27LPPsmzZMuLj44GyFpWWLVuyePHi8nXuv/9+HBwcmDVrVvmyzZs306VLF/Ly8so/V/90pc+dEObq1JY91O1+C05FBexteytN1y9F71Dxcx339c80fHQE9iVF7OgzlHYrFihKK8zJtb6//61Wtag42FqT8EZvJce9XocOHcJgMHDrrbdedZ3WrVuX/5ydnU1qaiodO3assE7Hjh3LW0ZGjRpFz549ady4MX369OG2226jV69eAAwePJgZM2bQoEED+vTpQ79+/RgwYAA2Nv/90Rg+fDhff/01L7/8Mpqm8eOPPzJx4sTy19PS0njllVf4888/OX/+PKWlpeTn55OcnHzd78ffrnUOsbGx+Pv7X7FIgbL3dODAgZe9PzNmzKC0tLS85eWf7yuUtcQcO3aM+fPnly/TNA2j0UhSUhJNmjS54fMQwlyUGIooHnYPTkUFJDRsQbO/lmLncHnxHfnAEGLzC4kaP4rWq35hy5LH6Hh7FwWJhaWqVYWKTqe7oUswKvw9X8y1ODk5Xbbs360MmqaVL2vVqhVJSUmsWLGCtWvXMmTIEHr06MHChQsJCAjgyJEjrFmzhrVr1/L444/z/vvvs2HDhv9sYbn33nt5/vnn2bt3LwUFBaSkpDBs2LDy10eNGsWFCxeYMWMGQUFB6PV6YmJiKtVR9Vrn8F/v2T/fi38u+7d/v69Go5FHH330in12AgMDb/gchDAnu599i/anE8m2d8Z7+WLsHK/eQthi3P2sOnSMmbkepB0sZk2vYlztb6yFVoirqVWdac1Bw4YNcXBwYN26dde1vqurK35+fmzevLnC8q1bt1b4i9/V1ZWhQ4cyZ84cfvrpJxYtWkRGRgZQVhzdfvvtfPzxx6xfv55t27YRFxf3n8f29/enc+fOzJ8/n/nz59OjRw98fHzKX9+0aRNPPPEE/fr1K+/IevHixes6r6ud65XOISoqitOnT3P06NErbhcREXHF96dRo0bXnDiwVatWxMfHExYWdtnDzs6u0uchhKlLO3ScZl+UdTg/NPFF6oYF/+c2XWa8Sm5kC85nG5iy/PB/ri/E9TLt5oVayN7enueee45nn30WOzs7OnbsyIULF4iPj7/q5aBJkybx6quvEhoaSosWLZg7dy6xsbHllyymT5+Or68vLVq0wMrKil9++YV69erh7u7ON998Q2lpKe3atcPR0ZHvv/8eBwcHgoKCrivv8OHDee211ygqKmL69OkVXgsLC+P777+ndevWZGdnM2nSpOtqMbqSa51Dly5d6Ny5M3fffTfTpk0jLCyMw4cPo9Pp6NOnD08//TRt2rThzTffZOjQoWzbto2ZM2fy2WefXfOYzz33HO3bt2fs2LE8/PDDODk5cejQIdasWcMnn3xSqfMQwhycHPsMbYvyORLSlDZvTrqubextrXn3rkiGzt7O5jU7OeEPDdpGVnNSUStoCmVnZ2sTJkzQAgMDNXt7ey0mJkbbuXPndW+flZWlAVpWVtZlrxUUFGgJCQlaQUFBVUauEaWlpdpbb72lBQUFaba2tlpgYKD2zjvvaElJSRqg7du377L1X3/9da1+/fqara2t1rx5c23FihXlr8+ePVtr0aKF5uTkpLm6umq33nqrtnfvXk3TNG3x4sVau3btNFdXV83JyUlr3769tnbt2uvOmpmZqen1es3R0VHLycmp8NrevXu11q1ba3q9XmvYsKH2yy+/aEFBQdr06dPL1wG0xYsX/+dxrnUOmqZp6enp2ujRozVPT0/N3t5ea9asmbZ06dLy1xcuXKhFRESUv5/vv/9+hf3/O9ffdu7cqfXs2VNzdnbWnJyctKioKO3tt9++ak5z/twJoWmalpKRp90z/F1tY1AL7fCvK294++8feFErsrLW9rTpXg3phKW41vf3vym962fo0KEcPHiQzz//HD8/P+bNm8f06dNJSEigfv36/7n9jd71I0R1k8+dMHev/H6Q77adokOoJz88fPlYSf8laf0OgrrFYIXG8VUbCe11SzWkFObuRu76UdZHpaCggEWLFjF16lQ6d+5MWFgYr732GiEhIXz++eeqYgkhRK2VllPIgl1l4xON6xZWqX2EdG3HvpiyO/KyJ79SZdlE7aWsUCkpKaG0tPSyvzodHBwu6/j4N4PBQHZ2doWHqB6bNm3C2dn5qo+q9s4771z1WH379q3y4wkhLpcw+W0mrvuaXk4FxIR6Vno/dae+BUDzvRs4sye+quKJWkpZZ1oXFxdiYmJ48803adKkCT4+Pvz444/s2LGDhg0bXnGbKVOm8Prrr9dw0tqpdevWxMbG1tjxxowZw5AhQ674WmU74Aohrl9pcQnh82bRNesCu2/vctkt/TciqFNrDjRtR1T8DlLe/oD6v8pIzqLylPZROX78OA888AAbN27E2tqaVq1a0ahRI/bu3UtCQsJl6xsMBgwGQ/nz7OxsAgICpI+KMBnyuRPmav+cH2n+yL1ccnDB/vxZ7F0uH6/pRsR+/j0tHr+fbHtnbFJP41jHrYqSCktgFn1UAEJDQ9mwYQO5ubmkpKSwc+dOiouLCQkJueL6er0eV1fXCg8hhBA3r3TWbAAO97zjposUgKiH7uGMhy/WpSVs+XntTe9P1F4mMY6Kk5MTTk5OZGZmsmrVKqZOnao6khBC1BoXjycTtXcjAL5Pj6+SfVrZ2rBz6ixeOVhAg5J69KySvYraSGmLyqpVq1i5ciVJSUmsWbOGbt260bhxY0aPHq0ylhBC1CrHP52LjWbkaFATgjq3qbL93jK0F/kOzuxPucSxtNwq26+oXZQWKllZWYwdO5bw8HDuv/9+OnXqxOrVq294Fl8hhBCV57rkVwAyBtxVpfut66ynayMvAFb8eaBK9y1qD6WXfoYMGXLVOz2EEEJUv3OZ+exxqY+X4ylCHhtZ5fsf4VXM08+Nx9OQS+ndqVjbmkSPA2FGZFLCWmjUqFHccccd11yna9euTJw4sUbyCCHUWRZ/npd6j2Xs+3/gE3HloSFuRkyX5vhnX8An6wIJ83+v8v0LyyeFigm6UiGxcOFC7O3tmTp1Kq+99ho6ne6yx9q10rNeCHFjlh5IBaBvc/9q2b/e2YlDncsGbSyY/2O1HENYNilUzMCXX37J8OHDmTlzJs8++ywATZs25ezZsxUenTt3VpxUCGFOLpxMpXTHTnSakX6RvtV2HKfhwwBouHUtJYaiajuOsEy1s1DJy7v6o7Dw+tctKPjvdW/S1KlTGTduHD/88AMPPfRQ+XIbGxvq1atX4WFnZwdAXFwc3bt3x8HBAU9PTx555BFyc6/e4z4vL4/7778fZ2dnfH19+fDDD286txDC9CV9OY8l3z3FL0vfwdu1+gYoDB96G5mOrtTJz+LwT0ur7TjCMtXOQsXZ+eqPu++uuK6399XX/fccNMHBl69zE55//nnefPNNli5dyt3/znUV+fn59OnThzp16rBr1y5++eUX1q5dy7hx4666zaRJk/jrr79YvHgxq1evZv369ezZs+emsgshTJ/tihUAFLdtV63HsdHbkRjTA4C8HxZU67GE5ZHu1yZqxYoV/P7776xbt47u3btf9npcXFyFyQEjIiLYuXMn8+fPp6CggO+++w4np7LRJWfOnMmAAQN477338PHxqbCf3NxcvvrqK7777jt69iwbkunbb7/F3796rlcLIUxDYU4ejeO2A+A1rGpvS74S+3uGwLpfCdu8htLiErn7R1y32vlJucZlEKytKz5PS7v6ulb/apA6ebLSkf4tKiqKixcv8sorr9CmTRtcXFwqvN64cWOWLFlS/lyv1wNw6NAhmjdvXl6kAHTs2BGj0ciRI0cuK1SOHz9OUVERMTEx5cs8PDxo3LhxlZ2LEML0HP15KVHFhaS5eBLas1O1H6/J8DtY9FkfVgVF82BSOu0a+fz3RkJQWwsVpxuYx6K61v0P9evXZ9GiRXTr1o0+ffqwcuXKCsWKnZ0dYWFhl22nadpVZz290nKFc1IKIRQq+LXsVuGk9t3w/vcfXdXA1l7PpknvsDo2lZDjGVKoiOtWO/uomInAwEA2bNhAWloavXr1Ijs7+z+3iYiIIDY2lrx/dOTdsmULVlZWNGrU6LL1w8LCsLW1Zfv27eXLMjMzOXr0aNWchBDCJNXfsQEA/e0DauyY3cK9Afjr8DVaqoX4FylUTJy/vz/r168nPT2dXr16kZWVdc31hw8fjr29PSNHjuTgwYP89ddfjB8/nhEjRlx22QfA2dmZBx98kEmTJrFu3ToOHjzIqFGjsKqBv7CEEGqk7kvAPz2VEp0VYcNqrlDp0siLyPPH6LtoNmfj5I8hcX3k28gM1K9fnw0bNnDp0iV69uzJpUuXrrquo6Mjq1atIiMjgzZt2jBo0CBuvfVWZs6cedVt3n//fTp37sztt99Ojx496NSpE9HR0dVwJkIIU7A5346h90zhq2FP41y3To0d193RjqmbvubJLT+Q/O1PNXZcYd50mhl3UsjOzsbNzY2srCxcXV0rvFZYWEhSUhIhISHY21ff+ABC/JN87oQ5GPfDXpYeOMuEWxvyZM/LLwlXp22PTCJmzgcciIwh6sDWGj22MB3X+v7+N2lREUKIWsRo1Nh6PB2ATg3r1vjx6907CIBGh/ZQcCmnxo8vzI8UKkIIUYuc2LiT8b9/Qu/kfbQIcK/x4wd3bsM5d2/sS4o48uNvNX58YX6kUBFCiFrk4s+/MXrPHzyesBJb65r/CtBZWXGqfVcADL//UePHF+bH4gsVM+6CI8yQfN6EqXPYugmAgk5dlGWwH1h2p1HQjg1oRqOyHMI8WGyhYmtrC5TNfSNETfn78/b3508IU2IsKSXkyH4APPrcqixHo2EDKbK2wSk/h+SEE8pyCPNgsSPTWltb4+7uTtr/hsB3dHS86oitQtwsTdPIz88nLS0Nd3d3rP89FYMQJuDUpp2EFOaSb2tPgxoYNv9qHNxdeOalb1ic78JrubaMUJZEmAOLLVQA6tWrB1BerAhR3dzd3cs/d0KYmrQV6wgBjodFEqm3U5oluHNbSlcfZUviRUa0D1KaRZg2iy5UdDodvr6+eHt7U1xcrDqOsHC2trbSkiJMmvWWLQDktmmvOAl0DKvLB6uPsvX4RUpLjVgr6NgrzINFFyp/s7a2li8QIUStZ59yEgCXHt3UBgEi67vxyqZv6J6wheOt59Hotu6qIwkTJSWsEELUAqcz87ntnvfp8tjXNBjYU3UcbKytaFl4keBLZ0n/fbnqOMKESaEihBC1wO6TmaDT4d60EY6uzqrjAFDUpSsALpvXK80hTJsUKkIIUQvsPJkBQJugmpuE8L/Uu/s2ABomHqAwO1dxGmGqpFARQohaYOikEcz+9S06W2erjlIuMKYlaa510ZcWk/jrKtVxxL+czy5kw9ELZOQVKc0hhYoQQli4S8lnaX58P70St9Ms3F91nHI6KytOtYwBIHf5SsVpxL/tXr6ZGW9+x9PfblOaQwoVIYSwcCeXrQUg2SsAjxDTKVQA6FZ2t0+dXWq/DMXl6n49i8XznuGR9fOU5pBCRQghLFzh5rIi4FzTVoqTXM7/zj4c8/Bnl0cIeYUy3pUp8TgcB4C+XVulOWrFOCpCCFGbOe7fC4DWpo3iJJfzjQqn46RvOXOpgJCULDo1rKs6kgAMRcVQUABAve4dlWaRFhUhhLBgmtFI8PF4ADy7q5vf51rahngAsCMpXXES8bejafn0fPAzbnl+Ib7Nw5VmkUJFCCEs2OldcbgW5mKwtiWoq/qh86+kXYgHNqUlnNmyW3UU8T8HzlwCIDjMH52V2lJBLv0IIYQFO3b8LOfqR2Dn7EBze73qOFcUY1/IgY+GYm00UvjMHdi7OKmOVOvFnc4CyqY6UE1aVIQQwoJtcA5g8H1T+W3qN6qjXFVgszDy9Y7oS4s5sfwv1XEEcO/zo/jhxxfoUHBWdRS1hUpJSQkvvfQSISEhODg40KBBA9544w2MRqPKWEIIYTH2n74EQItA0xmR9t90VlacatoagKyV6xSnEYU5eTQ5cYAOyQcIDfNTHUftpZ/33nuPL774gm+//ZamTZuye/duRo8ejZubGxMmTFAZTQghzF6RoZiTSefA2p7m/u6q41xTSadOsGMNzju3qo5S6yVv2EEjYymZjm7Ua9pQdRy1LSrbtm1j4MCB9O/fn+DgYAYNGkSvXr3YvVs6VAkhxM06tW4rez4YzMIFkwnydFQd55q8+5fN6ByaeIDiQoPiNLVb5p+bAEgOjVDekRYUFyqdOnVi3bp1HD16FID9+/ezefNm+vXrd8X1DQYD2dnZFR5CCCGuLGP9JqzQsHNxRqfTqY5zTUG3tOWSgwuOxYWcWLVRdZxazWr3LgDyW7ZWnKSM0kLlueee45577iE8PBxbW1tatmzJxIkTueeee664/pQpU3Bzcyt/BAQE1HBiIYQwH1a7yr5w8lqY3oi0/2ZlY01Sk7KcmdJPRal6hw8A4NgpRnGSMkoLlZ9++ol58+bxww8/sHfvXr799ls++OADvv322yuuP3nyZLKyssofKSkpNZxYCCHMh9eh/33hdDTN8VP+7eJdQ5na+X6W1W+hOkqtlXUmjYALZd+tQb26KE5TRmln2kmTJvH8888zbNgwACIjIzl16hRTpkxh5MiRl62v1+vR601zHAAhhDAluRczCTx/CgD/3qbxhfNf6o28l89yAnEpseE1o4a1lWlfrrJERw6fIj8kGp+SXJoEqb/jBxS3qOTn52P1r4461tbWcnuyEELcpJS/tmGFxnk3LzxDzOMyeRNfF5z1NuQUlnDorPRBVGE77owa8jpfTP1RdZRySguVAQMG8Pbbb7Ns2TJOnjzJ4sWLmTZtGnfeeafKWEIIYfaytuwA4GyI2nlaboSNtRXd3I3cdmgjp39fqTpOrbQ/5RKASd3OrvTSzyeffMLLL7/M448/TlpaGn5+fjz66KO88sorKmMJIYTZ2+/iy+lmt+LeubPqKDdk+P6VtF/yMbsv9oHHh6qOU6toRiOnDiWBzokWge6q45TTaZqmqQ5RWdnZ2bi5uZGVlYWrq6vqOEIIYTL6zNjI4XM5zB4RTa+m9VTHuW4Hv/uVZiPv5py7N/Uyz6uOU6ukxh7Cr2UESR5++J5Jwt7ertqOdSPf3+pHchFCCFGlDCWlHEvLBaCZCUwqdyNC+nenRGdFvUtpnE9IVB2nVjm7ej0AJc6u1Vqk3CgpVIQQwsIcjz9J6Pkk6uqt8HWzVx3nhjh5unOyfhgAp5euVZymdineuh2AjKbNFSepSAoVIYSwMLk/LWTV1+P46tc3TX5E2itJj4oGoGTjZsVJahe3g7EAWLUzrXF3pFARQggLo+3dA0B+eITiJJVj3fkWADz3y7xvNaUov5CQU4cB8OllWh2wpVARQggL434kHgC7NtGKk1SOf/8eAASfOUZe+iW1YWqJk+u2YF9SxCUHFwLayqUfIYQQ1aTEUETQ6WMAeHc2rSb861WvWUMm3/sqt4z5iv2ZJarj1AoZq/8C4GSj5uisrRWnqUgKFSGEsCCnt8diX1JEnp0D9VtHqY5TaTn9b+esqxd7TmWqjlIrbHEP5ruW/Unvc5vqKJeRQkUIISzIxY3bAEgOaIiVjWn9ZXwjWgfVAWC3FCrVTtM0frIP5pVej+H8yEOq41xG6ci0QgghqlbJnr0AZIc3U5zk5rTxseex7b/Q4o8TGO//y6yLLlN3OrOAtBwDNlY6mge4q45zGSlUhBDCgiwLv4XNGUaibzO9Jvwb0TjAk5CtP+FYXMiJzbto0NU8+9uYg6N/7SD6dAK61tHY25peQSiXfoQQwkJomsZSx0BmdhiG14A+quPcFBu9HSdCy1qFLqz8U3Eay+b05SwWzX+WZ9Z/qzrKFUmhIoQQFiItx0BmfjFWOgjzdlYd56bltmoLgNXWrYqTWDavA2Xj7tjd0klxkiuTQkUIISzEye376XV0G+2tc02yCf9GOXYvG3jML36v4iSWKyctneCzJwAIuO1WxWmuTAoVIYSwENrixcxe/DZPr/1KdZQqEXzbrRjRUT/jLBePnVQdxyKdXP4n1pqR1Dr18GrcQHWcK5JCRQghLITNwYMAFDU17zt+/ubqU5eTfmVfnil/yASF1SF33UYAUiNaKk5ydVKoCCGEhfA4UTZXi0Mr0/3SuVEXIltRaGPH+cNJqqNYJJc9ZTMml7aPUZzk6uT2ZCGEsADFhQb8z50CwKdTa8Vpqs6F518mMvJemgZ7Yd73MZmeovxCQhPjAPC5rZfiNFcnLSpCCGEBTm+Pxc5YQq6dI/UiG6uOU2WiIkMptrYlPjWLwuJS1XEsSty5HO655x0+7P0IQbeYbnErhYoQQliA9G27AEgJCENnZTn/tQd4OODloqe4VONAyiXVcSzK9lNZxPo1JnH4IyY3EeE/Wc6nWQgharHiffsByA6znNYUAJ1Ox8Tjf7Hqq8cpfe9d1XEsyo6kDADaNfBQnOTapI+KEEJYgEVt+jPf4EH/2023U2RlhTrpaHwxmdgd21VHsRglhiL6fv4mdeo1pt1jpj09gbSoCCGEBdhS4sLSJp3xurWz6ihVrk6PLgAEH92PsUT6qVSFE6s3cc/upby5dhbhfm6q41yTFCpCCGHmLuUXcTarEIBG9VwUp6l6IT1vodDGDveCHFK271MdxyJkLF8DwInwlljZmvbFFSlUhBDCzJ3cvIeHd/xK30vHcLW3VR2nytk52nMiJAKA8yvWKU5jGRy2bQagsOMtipP8NylUhBDCzBlWrubF9V/z6I5FqqNUm6z/TVCokwkKb1ppcQkNDpe1THn2M93xU/4mhYoQQpg5XVzZoF0F4U0VJ6k+jl3L/vKvd1AmKLxZSeu24GLIJ0fvSIOeHVXH+U9SqAghhJlzO3YIALuWUYqTVJ/gAT057lGfbb7hXLyUpzqOWUv/YxUAJxq3wNrE+6eA3J4shBBmzVhSSsCZEwDU7dBGcZrq41bfh0EvzicxLZdZZ3Lo7e6kOpLZyo8vK2wLOndTnOT6SIuKEEKYsbOxCTgWF2KwtqV+G8ttUQFoHVwHgD2nMhUnMV+GklIev+VR2j7+LR6PPaQ6znWRQkUIIczY+S3/GzrfNxgbvZ3iNNUrOsgDK2MpqTv3q45itvYlX6KguBSjry8NmwSpjnNd5NKPEEKYMcOBgwBkhjRSnKT6tdMbOPDRMOxKiyl8egD2LnL550ZtSbwAQMewuuh0OsVpro8UKkIIYcYW3zKIt0uDGN65EZbbQ6WMf0QIGbZ6nIsKOLxyA+GD+6mOZHZ6PnEfrYs1Clq8pzrKdZNLP0IIYcbis0s5WC8Mz3YtVUepdjorK5LDWwBwad0GtWHMUPb5izRN3EeXpL20aBqoOs51U1qoBAcHo9PpLnuMHTtWZSwhhDALRqPG8Qu5AIR5OytOUzMM7comXdTvlAkKb9Txn5dirRlJ8QqgXqT5XCpUWqjs2rWLs2fPlj/WrCmbe2Dw4MEqYwkhhFk4l3CcF5fNZMT+lQR6OKqOUyPce3YFIPjIfjSjUW0YM1O0cjUAZ1qb/iBv/6S0j4qXl1eF5++++y6hoaF06dLliusbDAYMBkP58+zs7GrNJ4QQpuzi5h2M2LecpHoh2Fh/ojpOjWjQqzOFNnbUyc8ieUcsgTGtVEcyG767yub30fcx/WHz/8lk+qgUFRUxb948Hnjggav2RJ4yZQpubm7lj4CAgBpOKYQQpiP/f3f8ZASFKU5Sc+wc7TkR3ASA88v/VJzGfJyLO0rghRRKdVY0GHyb6jg3xGQKld9++41Lly4xatSoq64zefJksrKyyh8pKSk1F1AIIUyM1eHDABQ1ClecpGYl3T6UaZ2Gs9EjRHUUs3Fq/kIAjjZohpuv13+sbVpM5vbkr776ir59++Ln53fVdfR6PXq9vgZTCSGE6XI7mQiAbbMIxUlqlv1DD/Cx7W4aGJ14WnUYM7Gz0J6ioBZY9eitOsoNM4lC5dSpU6xdu5Zff/1VdRQhhDALmtGIb+pJADzatlCapaZFB5UNpX/iQh4ZeUV4OFn2iLw3y1BSyueuEeQPe4ul4zupjnPDTOLSz9y5c/H29qZ///6qowghhFm4mHgSV0MepTori5/j59/cHe1o7VBE7yNbObx2q+o4Jm9XUib5RaV4u+hp6ueqOs4NU16oGI1G5s6dy8iRI7GxMYkGHiGEMHnndscBcKZuffROtePW5H96dtM8Zv32Drp581RHMXmH/1iHV24GXRp5mc2w+f+kvFBZu3YtycnJPPDAA6qjCCGE2dgb0pyoCQuY9dQ01VHU6Fg2Foj7vl2Kg5i+nu89y65P72dweoLqKJWivFDp1asXmqbRqJH5jJInhBCqJablkG3vjGuLZqqjKOHbtxsADU4ewpCXrziN6TqzJ56gtGRKdFaE39lTdZxKUV6oCCGEuHHH0v43dL5X7Rg6/9/82zYn3ckdfWkxJ5avVx3HZJ3+cREAR8Ka4+pTV3GaypFCRQghzNBDHz/Hq2tnEW5r+O+VLZDOyopTTaMBuLRyreI0pst+zSoAcrqbZ2sKmMjtyUIIIa7fpeSz9IjfBEBe/e8Up1GnuOMtsHMdTjvkzp8ryb2YSXh8WR8en2F3KU5TedKiIoQQZubsjn0AnHP3xsnTXW0Yhbz6l7UShB3dT3Fh7WxZupaj3y5EX1rMGU8/gju3UR2n0qRQEUIIM5Oz9wAA5/1DFSdRK7hLO168cxK9HvyUg+fzVMcxOaW//QZASude6KzM9+teLv0IIYSZMSaU3WZaENpQcRK1rGysOT9wCKcPnWfnyUxaBnmojmQyikuNPNlhNG3cwnnwUfO97APSoiKEEGbH8UTZHD+6prVrjp8rad+grDjZkZShOIlp2ZmUwWmdAxvb9aFJzw6q49wUKVSEEMLMeKccB8C1Ze0aOv9K2tdz5MGdixnywTOUFpeojmMyVsefA6BHEx+srcxvNNp/kks/QghhRnKzcrEtKus46tuupeI06oUHevDklh9xLsrn2J9bCevdWXUk5TSjke4vjMHeuyExg15SHeemSYuKEEKYkePZJUSPn0+P53/GPaCe6jjK2ejtON64OQAXl61RnMY0HFu5iS4JW5iwZQHtG/mojnPTpFARQggzciwtF3Q66ob4q45iMvJjOgGg37pZcRLTcPH7HwE43DwGe1fzH7lYChUhhDAjiX8Pne9t/l9AVaVO71sBCD60F2NJqeI0amlGIwFrlgJgvOtuxWmqhhQqQghhRtp98BJfLXydDueOqI5iMhr06UK+rZ46+dkkb9mjOo5Sx9dsxj/9DIU2doQ/fK/qOFVCChUhhDAjYQd2cOvxXQQ6WauOYjLsHO05ERYJwPk/VilOo9aFr74HIKFlJ5zr1lGcpmpIoSKEEGaiMCcPv/RUAOq1a6E2jInJadcRg7UNF5POqI6ijGY0ErS27LKPNmSo4jRVR25PFkIIM3F29wFCNCPZeic8w4JUxzEp+icn0NzjFhzdXelr1LAy87FDKiPu8GlOezXA3pBPxIPDVMepMtKiIoQQZiJjVywAqX4hZj13S3VoFhGElZMTGXlFHD6XozqOEktO5PL4nS/w+uercajjqjpOlZFPuhBCmImiuLI5frKDwxQnMT12Nla0CykbTn/r4XOK09Q8o1FjedxZAPq2sqzWNilUhBDCTOiPHgbA2KSJ4iSm6e7cEyyb+wRtnhytOkqNO7h5Hw4nEnHW29C1sZfqOFVK+qgIIYSZyCzRkWPngENUM9VRTFJERCAN0k6Qn3mGovxC7BztVUeqMQVvTWHdmoWsGvQo9ra9VcepUtKiIoQQZqCk1MiYPhOJnPgzHncNUB3HJAV3aUeGkxuOxQaO/bFWdZwaU5iTR5ONKwCoP9CyihSQQkUIIcxCckY+xaUaDnY21PeUUWmvxMrGmqSo9gBkLV2pOE3NiZ81H1dDHufcvIgYdrvqOFVOChUhhDADfw+d38DLqVbeenu9Srt3B6DO1o2Kk9Qc63llg7wl9b0LKxvLGwhQChUhhDADzp/MYO2cMTy4Z4nqKCbNf1DZZbGwpHhy0tIVp6l+F48n0+zAVgD8xj2sOE31kEJFCCHMgF3CQcIyTuNrU7sn3fsvfi2acNrTDxvNyPFFy1XHqXbHPpqDjWbkaFATgjpGq45TLeSuHyGEMAPuJ48BoJc7fv7TsS592XnkBBfybGmhOkw1068q64uTOchyRqL9NylUhBDCxBlLSvE7dwoAzzYt1IYxA8Vvvs1T3+0m0OjII5qGTmeZfXriTmcx5PYX6XdsB68/YZmXfUAu/QghhMk7n5CIU3EhRVY2+EVLi8p/6RDqiZ21FckZ+Zy4mKc6TrX5YWcyxda2MHQI7oG+quNUGylUhBDCxKXtjAUg1csfW72d2jBmwElvQ9vgOjRJO8HBxZY5nkpubgF/7EsB4J62gYrTVC8pVIQQwsQVxMYBkBEUqjiJ+XgsYRUr5j5Bg4+mqI5SLeLf/ojlnzzA40kbyuc4slRSqAghhIk7U2JDvHcD8iMiVUcxG/6DbgOg8ZG95KVfUhumGnjM/4bArPN09rK12D44f5NCRQghTNwPzXvTf/THZEycpDqK2QiMaUmqhy92pSUkLvhDdZwqlbh8Aw1TjlBkbUPjSWNVx6l2yguVM2fOcN999+Hp6YmjoyMtWrRgz549qmMJIYRJ0DSNY/8blTbMS4bOv146KytS2nYGwPCHZRUqmdM+BuBA+57UCfJTnKb6KS1UMjMz6dixI7a2tqxYsYKEhAQ+/PBD3N3dVcYSQgiTcSErn5y8Qqx0ZcPni+tnP7Ds8k/Qjg1oRqPiNFUj81QqURuWAuA43vJbU0DxOCrvvfceAQEBzJ07t3xZcHCwukBCCGFi0pau4dC0Iexu1Br7Kf1VxzErjYYNxDDOlnqX0jixfgcNuseojnTTDr81jZiSIo75N6LJ4L6q49QIpS0qS5YsoXXr1gwePBhvb29atmzJnDlzrrq+wWAgOzu7wkMIISxZXmwc+tJinPQyPueNcnB34VBk2WzKZ7//WXGam1dsKCL0528ByHzoMXRWyntv1AilZ3nixAk+//xzGjZsyKpVqxgzZgxPPPEE33333RXXnzJlCm5ubuWPgICAGk4shBA17FACAAVhjRQHMU/p459i+NC3eKfZANVRbtqKwxd5ZOBkfm3dj6hnHlEdp8YoLVSMRiOtWrXinXfeoWXLljz66KM8/PDDfP7551dcf/LkyWRlZZU/UlJSajixEELULKfjiQBYN41QnMQ8tRraj+0NWhJ/oYCTZj5K7debk4j1a0zylOnonRxVx6kxSgsVX19fIiIq/vI1adKE5OTkK66v1+txdXWt8BBCCEtW70wSAO7RzRUnMU91nOyIaeAJwKr4c4rTVN7eUxnEplzCztqK4e2CVMepUUovenbs2JEjR45UWHb06FGCgmrXP4IQQlxJ1pk06uZmAODbvqXiNObrDm/oNmsO4auzYe+fquNUSsnQe3izyIakRyfg5aJXHadGKS1UnnzySTp06MA777zDkCFD2LlzJ7Nnz2b27NkqYwkhhEk4u2MvbsB5Ny98vCx7mPTq1DXcG4/dS7BCI+3QcbybmNdUBCc37KTtjtW0RkdKszdUx6lxSi/9tGnThsWLF/Pjjz/SrFkz3nzzTWbMmMHw4cNVxhJCCJNwKqeU5Y06EN+8o+ooZq1uoxCONiibdTppzjzFaW7chZfLipPYNt0I6hitOE3NU36/22233cZtt92mOoYQQpicXZ4hfHnnC4zuGEx31WHM3KU+t8FncTgvWwLTXlUd57ql7kug5eYVALi89rLiNGrUjpuwhRDCDCX+PXS+twydf7MCHx4BQJOj+0g7dFxxmuuX8sLr2GhGDjRtR8N+XVXHUUIKFSGEMFGXEpNA02jo7aI6itnza9GEQ6FRWKFx4pMvVce5LucTEmmxZjEA1pMnK06jjhQqQghhgvIysvj93WHETx9MQ32p6jgWIXvQUAC8lixSnOT6HJ/0GvrSYg6FRhFxj/kPWFdZUqgIIYQJSt0eC4DBTk8dPy+1YSxE+LgHSHX1YkO9JhxJTlcd55pOpefxRMMBfNH2LrS336k1w+VfSe09cyGEMGFZe/cDcNavgeIklsPNvx6vfryMN3o8wuL4C6rjXNOMtYlctHdh66PPETG0dk9GKYWKEEKYoOKD8QDkNghTnMSy3B3tD8CivacpLjUqTnNliUnn+W3faQAm9WqsOI16UqgIIYQJsk88CoAW3kRxEsvSPdwHL0cbGsXtYPdPK1XHuaLcQUP54ccXeMAtl0h/N9VxlFM+jooQQojL1U0pu4XWqUWU4iSWxc7Gig+Or6DLTx+xPyEG7u2nOlIFB79fTMu9GyjRWeHXUS77wU20qJSUlLB27VpmzZpFTk4OAKmpqeTm5lZZOCGEqI0Mefn4XjwDgE+7FmrDWKDQx0cDEBm3nXNxRxWn+X8lhiKcnnsGgD19hxLUuY3iRKahUoXKqVOniIyMZODAgYwdO5YLF8o6JU2dOpVnnnmmSgMKIURtk5yayZy2d7Ey4ha8GoeojmNx/Ns1J75xNFZoJL0zTXWccntenkrI2RNk2TvT+PMPVccxGZUqVCZMmEDr1q3JzMzEwcGhfPmdd97JunXrqiycEELURkcLrHiv6yi+eHxKrb4ttToVjXkMgCa//0DBpRzFacpmym706fsAHBrzNO6BvooTmY5K/QZs3ryZl156CTs7uwrLg4KCOHPmTJUEE0KI2ioxreyLU4bOrz5RY0eS6uGLe0EOB975WHUcDj80njr52Zz0CSZ6Su0dhfZKKlWoGI1GSksvHynx9OnTuLjIUM9CCHEzcvfuxys3k4ZeTqqjWCxrWxuSRzwMgO83szCWqBv9d9fhVJz27QEg9/1p2NrrlWUxRZUqVHr27MmMGTPKn+t0OnJzc3n11Vfp18+0elALIYS5ufejyez6dATtjuxUHcWiNXtpItl6J3J1NmzYEq8kQ2FxKc8tPcodIz7k68kzaTbiTiU5TFmlbk+ePn063bp1IyIigsLCQu69914SExOpW7cuP/74Y1VnFEKIWqOkqJj655MB8GrdXHEay+Zctw5zP/6JN44bCY/NosstGlZWuhrNMPPPY5y4mIeXuxN3P/lIjR7bXFSqRcXPz4/Y2FgmTZrEo48+SsuWLXn33XfZt28f3t7eVZ1RCCFqjbOxh9CXFlNgo6delIxKWt3uvK8nzvZ2HDqbzar4czV67MRlf6F/83XsSop5c2BT3Bxta/T45qLSA745ODgwevRoRo8eXZV5hBCiVkvfGUsAkOoTSKitjMlZ3dwd7RjdKYQ5K+I4+sYH9PrhPaxr4H3PSUvH8f7hjM84SxM3a3o0u6Paj2muKtWi8u2337Js2bLy588++yzu7u506NCBU6dOVVk4IYSobQri4gDIDApVnKT2eLBDEMu/m8iEhdPY/fLUaj+eZjRy5M77qJ9xlrPuPrSZLWOmXEulCpV33nmnfPyUbdu2MXPmTKZOnUrdunV58sknqzSgEELUJjaHDwNQ3ChccZLaw81Jz/n7HgCg0SfvkXW6ei8B7XptOq23rqREZ0XWV9/iVl+6TFxLpQqVlJQUwsLKZvT87bffGDRoEI888ghTpkxh06ZNVRpQCCFqE/eTxwDQRzZVnKR2af3eiyTVC6FOfjaHH55Qbcc5+sc6oqa8AMCuB58k/K7e1XYsS1GpQsXZ2Zn09HQAVq9eTY8ePQCwt7enoKCg6tIJIUQtomkaX7YcwJetB+LRub3qOLWKjd6O/A+mA9Bu5c/EfbOoyo9xPiERj3uHYF9SRGzzTrT7/L0qP4YlqvQ4Kg899BAPPfQQR48epX///gDEx8cTFBRUpQGFEKK2OJtVyE/hXXi35yPUbxmhOk6t03T4QHb0HgyAz4QxVXoJ6FJ+Ee/MWou+qICkeiGErV2ClY11le3fklWqUPn000+JiYnhwoULLFq0CE9PTwD27NnDvffeW6UBhRCitkhMK5t9PriuE7bWMsePCpE/ziHFKwDv7IvsGTmeUqN20/vMKSxm5Nc7+d0hkHEPvo9+xTKc69apgrS1Q6XuwXJ3d+eDDz7gwIEDpKWlsWTJEgCio6OrNJwQQtQm6dv3EH06gcDgNqqj1FqOddw48813bB73FE83vZOBSxN47fbK9xfKSErhrc9Wst/ajzqOtrz45P34+chUMzeiUoXKypUruf/++0lPT0fTKlabOp3uivMACSGEuLb6875k0ZpFbDM+BmO6qo5TazXs15Wji5eS+eM+vtl6Ejd7Gyb2aHjDM1mf2rIHqzvu4NWcTE4/NI1Xxg+lkRQpN6xSbYvjxo1j8ODBpKamYjQaKzykSBFCiMpxPZEIgG2k9E9RrX9zP17oV3aLeMGU99jZbxhF+YXXta1mNLL7rY/x6taJgIunKXBw4oPhrWlW3606I1usSrWopKWl8dRTT+Hj41PVeYQQolbSjEZ8U5MAcI+WOX5MwSOdQ/E+l8yAqd9irRk50XAvJbNm0+i27lfd5vjqTRRMfIrWh3YDEN84Gp+lC6kXFlxDqS1PpQqVQYMGsX79ekJDZeREIYSoChmnzuBZkIMRHfXbtVQdR/zPHUO6se/8VwQ/P5EGqcdhwK3EN2xJ7h134dahLbqWLcnWrNmfcommEx4kZt96AAzWtuwd8ThtZ39QI0PyWzKd9u9OJtchPz+fwYMH4+XlRWRkJLa2FSdSeuKJJ6os4LVkZ2fj5uZGVlYWrq6uNXJMIYSoDvHzf6fpfXdwxsOX+umpquOIf8lIOs2J+8fQYssKbDRj+fJbH/qc454BADy98XvG7FjI/nY98Pt0Gn5yi/lV3cj3d6XKvB9++IFVq1bh4ODA+vXr0en+f1psnU5XY4WKEEJYitz9ZXP8XAxoQH3FWcTlPEL88di0lPMJiZx4/1Ocdu+g3qlE6jjYUt/dgSa+rrjGPM2lJlNo3ShEdVyLUqlC5aWXXuKNN97g+eefx+oGe0ELIYS4gvgEAApCGykOIq7FJ6IhPnNnlD9fqC5KrVGpQqWoqIihQ4dKkSKEEFXk9+jeLC12o2t/mftFiH+qVKUxcuRIfvrpp6rOIoQQtdZafX2+b3UbdXp0UR1FCJNSqRaV0tJSpk6dyqpVq4iKirqsM+20adOuaz+vvfYar7/+eoVlPj4+nDtXvVNsCyGEKcnKLyYtxwBAQ29nxWmEMC2VKlTi4uJo2bLs9rmDBw9WeO2fHWuvR9OmTVm7dm35c2trmaRJCFG7JO89yN1x6zgf2gQXe9v/3kCIWqRShcpff/1VdQFsbKhXr16V7U8IIcxN/orVfLh8OgeatgceVh1HCJOivDdsYmIifn5+hISEMGzYME6cOHHVdQ0GA9nZ2RUeQghh7rSD8QDkh8kdP0L8m9JCpV27dnz33XesWrWKOXPmcO7cOTp06EB6evoV158yZQpubm7lj4CAgBpOLIQQVc/x+FEArJrKAGFC/FulRqatLnl5eYSGhvLss8/y1FNPXfa6wWDAYDCUP8/OziYgIEBGphVCmLXz7t74ZF3g8KIVhN/VR3UcIapdtY9MW12cnJyIjIwkMTHxiq/r9Xr0en0NpxJCiOqTnZaOT9YFAHzbRytOI4TpUd5H5Z8MBgOHDh3C19dXdRQhhKgRqdv2ApDm4ombn5fiNEKYHqWFyjPPPMOGDRtISkpix44dDBo0iOzsbEaOHKkylhBC1Jic3fsBOO/fQHESIUyT0ks/p0+f5p577uHixYt4eXnRvn17tm/fTlBQkMpYQghRYzZFdOCTwa/TPSqASNVhhDBBSguVBQsWqDy8EEIod6DQho0Nounds5nqKEKYJJPqoyKEELVN4vlcABp6uyhOIoRpMqm7foQQojbJTb/E0CWzSawbSEOvHqrjCGGSpFARQghFUrft5YltP3HRuQ51nKeqjiOESZJLP0IIoUjWnv/d8VNf7vgR4mqkUBFCCEVK4srm+MkJlTl+hLgaKVSEEEIRh2NHANDJHD9CXJUUKkIIoYh3ynEAnFtGKU4ihOmSQkUIIRTIz8yiXsY5AHzbt1KcRgjTJYWKEEIokLojFis0Mh3d8AjxVx1HCJMltycLIYQCBzyDefCR2XR2LeVN1WGEMGFSqAghhAKJ6QWcquOH1j5QdRQhTJpc+hFCCAVk6Hwhro+0qAghhAJ9vpxCuM6RJne9pDqKECZNChUhhKhhhdm53LnlN6w1Ixc831IdRwiTJpd+hBCihp3Zvg9rzcglBxfqhgSojiOESZNCRQghaljmrlgAzvqFoLOS/4aFuBb5DRFCiBpWsv8AANlhjRUnEcL0SaEihBA1zOHoobIfmkWqDSKEGZBCRQghapjPqUQAXFq3UBtECDMghYoQQtSgnPRLeGRnAFD/lnaK0whh+uT2ZCGEqEFHc40Me+oXooszWFDfW3UcIUyeFCpCCFGDjpzLpdjaFrsmzVRHEcIsyKUfIYSoQUfOZQMQXk+GzhfiekiLihBC1KCO014hMiMbl+gXgCaq4whh8qRQEUKIGqIZjbTZvY46+dkcc3lZdRwhzIJc+hFCiBqSfuwUdfKzKdVZ4d8hWnUcIcyCFCpCCFFDUrfsBuBM3frYuzorTiOEeZBCRQghakj+nlgALgY3VBtECDMihYoQQtQQ6/iDABiaNFWcRAjzIYWKEELUEPcTRwHQt2iuOIkQ5kMKFSGEqAGlpUZKDEUY0eEV00p1HCHMhtyeLIQQNSA5s4C+oz7G3VjEntZRquMIYTakRUUIIWrAkXM5APgH1MXaxlpxGiHMh8kUKlOmTEGn0zFx4kTVUYQQosr9Xag09nFVnEQI82ISl3527drF7NmziYqS5lAhhGVq/faz/HzsKBe8nwOkM60Q10t5i0pubi7Dhw9nzpw51KlT55rrGgwGsrOzKzyEEMIc+Cfspe3pBALc9KqjCGFWlBcqY8eOpX///vTo0eM/150yZQpubm7lj4CAgBpIKIQQN6cwOxf/C6cB8O3URnEaIcyL0kJlwYIF7N27lylTplzX+pMnTyYrK6v8kZKSUs0JhRDi5iVv3Im1ZiTDyY26DYNVxxHCrCjro5KSksKECRNYvXo19vb217WNXq9Hr5dmUyGEebm0dRcAZ4Ia42GlvCFbCLOirFDZs2cPaWlpREf//wyipaWlbNy4kZkzZ2IwGLC2llv4hBDmzxgbC0Bek2ZqgwhhhpQVKrfeeitxcXEVlo0ePZrw8HCee+45KVKEEBbD7WgCADYtW6gNIoQZUlaouLi40KxZxb8unJyc8PT0vGy5EEKYK6NR46TenTrOHtTt1E51HCHMjkmMoyKEEJYqOSOfxwY8i52NFQkdo/97AyFEBSZVqKxfv151BCGEqFIJZ8vGewqv54KNDJ0vxA2T7udCCFGNjpy8AECErwydL0RlSKEihBDVqNvLY9k5cwS9ju9QHUUIsySFihBCVCPfk0fwzsvEt4G/6ihCmCUpVIQQoppknkrFJ6vs0o9/F7njR4jKkEJFCCGqyZkN2wE47emHi5eH4jRCmCcpVIQQoprk7tgNQFqDcMVJhDBfUqgIIUQ1sY47AIChaZTiJEKYLylUhBCimngeOwSAQ9tWipMIYb5MasA3IYSwFIXFpfwZ0Jw0G0eCO7dXHUcIsyWFihBCVINjabm81e1B3B1t2RcRpjqOEGZLLv0IIUQ1SEgtGzo/wtcVnU6nOI0Q5ksKFSGEqAapew/iZMiniQydL8RNkUJFCCGqQZ/3niVuxlB6JW5XHUUIsyaFihBCVLESQxFBKYlYoeHbvoXqOEKYNSlUhBCiiiVv2YNDiYFcO0f82zRXHUcIsyaFihBCVLGL67cCcCo4HCsba8VphDBvUqgIIUQV03btAiC7mbSmCHGzpFARQogq5n44DgDbtm0UJxHC/EmhIoQQVai40EBwSiIA9bp1VJxGCPMnI9MKIUQVSjyTyY/dHyQq/RSDWjdTHUcIsyeFihBCVKED6UV83+o2OoR6MthKGq2FuFnyWySEEFUo7kwWAJH+boqTCGEZpEVFCCGqkOPyP2hS6kKUT6TqKEJYBJ2maZrqEJWVnZ2Nm5sbWVlZuLrKfBpCCLWK8gvB1QW70hLO7D5I/eimqiMJYZJu5PtbLv0IIUQVSd64A7vSErLsnfFr2UR1HCEsghQqQghRRdL/3ARAcoMIdNKRVogqIb9JQghRRax27AAgt5UM9CZEVZFCRQghqki9Q/sBcOwsA70JUVWkUBFCiCpwKfksARdSAAjq3VVtGCEsiBQqQghRBU6tWg9AilcA7oG+asMIYUFkHBUhhKgCG70b88GQN+ge4MRo1WGEsCDSoiKEEFVg58ViNoW0wmbQ3aqjCGFRlBYqn3/+OVFRUbi6uuLq6kpMTAwrVqxQGUkIIW6Y0agRm3IJgJYB7kqzCGFplBYq/v7+vPvuu+zevZvdu3fTvXt3Bg4cSHx8vMpYQghxQ5L3HOTxlV/S69Qewuu5qI4jhEVR2kdlwIABFZ6//fbbfP7552zfvp2mTWXoaSGEeUj7YzWP7VhIQsYxbKxfUR1HCItiMp1pS0tL+eWXX8jLyyMmJuaK6xgMBgwGQ/nz7OzsmoonhBBXpW3fDkBW82jFSYSwPMo708bFxeHs7Ixer2fMmDEsXryYiIiIK647ZcoU3Nzcyh8BAQE1nFYIIS7nFb8PAH2nDoqTCGF5lM+eXFRURHJyMpcuXWLRokV8+eWXbNiw4YrFypVaVAICAmT2ZCGEMtnnL+Lk64O1ZuTC4eN4NW6gOpIQJu9GZk9WfunHzs6OsLAwAFq3bs2uXbv46KOPmDVr1mXr6vV69Hp9TUcUQoirSlqyhuaakTMevtSXIkWIKqf80s+/aZpWodVECCFMWf669QCkRslEhEJUB6UtKi+88AJ9+/YlICCAnJwcFixYwPr161m5cqXKWEIIcd1sE/43nEKnW9QGEcJCKS1Uzp8/z4gRIzh79ixubm5ERUWxcuVKevbsqTKWEEJcl8LiUu697QXqtz/NNyP7qY4jhEVSWqh89dVXKg8vhBA3ZX/KJYqMGrlBDQgMra86jhAWyeT6qAghhLnYdTIDgLbBHuh0OsVphLBMyu/6EUIIc9XipQl8mpFNacSLqqMIYbGkUBFCiEooMRTRcs96nIoKOO4r4zgJUV3k0o8QQlTCybVbcSoqIFvvRHDXdqrjCGGxpFARQohKuLhyLQBJ4S2wtpXGaSGqixQqQghRCfqtWwDIbyfz+whRnaRQEUKIG1RiKCI0fhcAngP6KE4jhGWT9kohLFRG0mlSVm8kraCEPY3bciHHQInRyKC572FrZwP+/jg0b4Z/7y54hshM5Dfi2Ir1hBvyyLJ3JrR3Z9VxhLBoUqgIYSEKc/I4/N2vFP26GL/9O/FPP4MHsNM/gi+GTy1f78Utq/HOy6yw7Qm/UM5370vA+IfxbxtVw8nNT8LxNAp9G1ESGERr6Z8iRLWS3zAhzJimaew+mYHxgQeJ3LKKFsWFFV4/5R1IQcNwRnUIxsfVHjsbK44UTeT4mdPYJZ/CM+kowedP0iD1OA3mzeTQqqWMfncBj3VrSNsQD0VnZfoWOIey8/5pvHV7BK1VhxHCwkmhIoQZys3J5+f95/lhZzLH0nL5IjUNx+JCzrl5capTT+wH3kZwv1sJqu9NENDlnxt3eqXCvjKSTnN83iL0Py/g65BO/HX0In8dvUivMA9eamZPYPuWNXlqJi/PUMK+5LIWqU6NvBWnEcLySWdaIcxIRtJptg1/nKJ6fnw970+OpeXiYGvN4Yef5PDiVfhknKPd0vk0f/ge3Opf35eoR4g/bV6eQFTcNiZ+/Rr3tA3ExkpH/QVzqdepLdsfepoSQ1E1n5n52LfnCPr8PPzrOBDk6ag6jhAWT6dpmqY6RGVlZ2fj5uZGVlYWrq4yMqSwXOlJKSROfJHmy3/GocQAwHc9RqB7803uaOGHi71tlR7v+IVczt8+mA7bVwJwJKQp7ksX4xPRsEqPY4623zGS1kvmsfaecfSZ/5HqOEKYpRv5/pYWFSFMWPb5i2wb/jgOjRvRfsn3OJQYSAwMZ9/0Lxm+/GtGtA+q8iIFINTLmZgty9j1xgyy7Z1pnBSPTdu2JCz4o8qPZW58dm7GRjPi1aKJ6ihC1ArSoiKECSosLuX7jYn0v7sLfllpABwNbILh1ddoNmoQOqua+xvjzJ54Cm+/g9DUY5TorNjz3Nu0m/J8jR3flFw8dpK6DUMAyDx5hjpBfooTCWGepEVFCDNVbChi/o5TdHn/L95ec5yFEd045RPEvulf0jDpIJEPDKnRIgWgfnRT/OL3srtjX2w0I63fe5EfvluNGf+NU2knFywB4Jh/QylShKghctePECbAWFLK3ve/wPfDd/ij1zjOB0ZR390B/w/ewr9NMEGKx+pwcHcheuNSto0Yx28ZNvyUUMzJFYeZ3DccnU6nNFtN0i1fDsCFDl0JU5xFiNpCChUhFNKMRg589RPOr71C69RjADy+93f6jB3GPe0C0dtYK074/3RWVsTM/4z4TSdg2SFmbzyBdV4uz94dXeOtPCqUGIpouK9sfp86Q+9SnEaI2sPy/3cRwkQdXriCQ41b0fyRewlNPUaO3pFtoyYSvWUlozqGmFSR8k8P3dKAd++KxDsnnTsfG8SOkU+ojlQjji5ehWthLpmOrjQc0EN1HCFqDWlREaKGHT6XTep9D9F93S8AFNrYETvwPsKnvUVMoK/idNdnWNtA/H87R6P0ZJj3KdtdXGj/2RTVsarVajyY3+tx2vg4cIcMmy9EjZEWFSFqyIkLuTz5Uyx9P9rEEodASnRW7Oh5N1kH4mm/8CvczaRI+Vund55l20NPA9D+83fZPeVTxYmq17LUEua37If100+pjiJErSKFihDV7OSGnezu2Je59z3H4n1n0DQoGTqU1G17abd6IT5NzLdbZsycD9h+5ygAol6eSMJPy9QGqiYpGfkkpuVibaWjcyMv1XGEqFWk/VKIanJ8zRayXnqVFjv/JBgNXxcvzg+9j/G9Ioj0d1Mdr8q0/flL9sacotXuv/AbfQ8pQRsIsLD5gY7MnMv9e+K40Os23ByqfoA9IcTVSYuKEFVIMxo5+P1i9rfoRGivTrTauQ4rNPa26U7BwkXMfiDGoooUACsba5qs+Z2jQU1wL8ghY/gosguLVceqUgHffM4ba2cx6twe1VGEqHWkUBGiChhKSvlldwoLewyn2f130Xz/Fozo2BPTm6S/ttNq5zrCet2iOma1cXB3wWPdSv5q2olHez/FkwtiMRotY0C4tEPHaZwUD0CDh+9TnEaI2kcu/QhxE87sOciSg+f5KkXjYq6BFv5t6G+7mLied+L3yvNEt2uuOmKNqRsaiOfKP8j4YhvrDqfx0bpEnuzZSHWsm5Y0Zx7ewOGQZoSHh6qOI0StIy0qQtygovxC9nwwm7iIttRvHYnLjA+5mGugnqs9vR8YSFHyadot+5GAWlSk/C3K352372gGwJHPvmHfp98rTnTznJeVDZuf1XeA4iRC1E7SoiLEdTAWl3Do52XkffM9jTevIrowt2w5OhraFPHFfa24tYkPttZS+w9uHUDRb0sY/tsUcpc7ktyqKYExrVTHqpSLx04SnhgLQMBDctlHCBWkUBHiKkpKjew5lcmahPMMGDeE5qfiy19Lc/HkxG2DCZw0nnYtIxSmNE1DXnqIhJ+/JCIxlot33k1e/D6cPN1Vx7phxz7+ivaaseyyj/w7C6GEFCpC/EP2+YskfreQkmXLGX/LI6QVl024V8c3gpBzJznSqRcOI0fQZNgAvGV00quytdfjvXwxF1pFE3z+JHsGDKPV5uVmNyfQmYTjFFtZk3XXYNVRhKi1dJoZz9WenZ2Nm5sbWVlZuLq6qo4jzFBhTh7Hf19D9qq1uG3fTNiJeOyMJQCMGvQasZExdG/sTf8gRzpF+qN3clSc2LwcXriC0KEDsDWWsn3CK7Sf8brqSNftWFouPaZtwLMwh9XP9cDTt67qSEJYjBv5/pY/CUWtkpFrYP+ZLGKTL6EtWsjYL1+jaWnFMT+SvQJIvaUHT43pQ0TXNthIv5NKCx/Ul+3jXqD9x28S/cnbHO7UnvBBfVXHui6/x54BoHmLUClShFBIChVhkTSjkfRjpzi7dS95u/Zgu3cPvkfj+Cz6Dua16g9Ao2I3niot5qKzByej2mLs0pX6d/UjsHUkgYrzW5J2019jz84dRG9fzfpZP+PZuzteLnrVsa6ptLiEXau2g21dBrbwUx1HiFpNaaEyZcoUfv31Vw4fPoyDgwMdOnTgvffeo3HjxipjCTNSYiji3LlMThVbc/xCLmlxR+j3/nP4nTlB3YIc/v13cIuzR9nqNYQW/u60CmxK8kMxBLRrQV0z6zthTnRWVoQv/YmXJ33G997NWf/jXuY92M6kW6riv13IgmmjWBfekY5vblAdR4haTWmhsmHDBsaOHUubNm0oKSnhxRdfpFevXiQkJODk5KQymjARuYYSzl/MpuCvjRSeTKbk9Bl0J5NwOJ2Mx7kUfDLPsyGqJy/2HgeAkyGfZ47tB6BUZ0Vq3fpcCArD0KIVLrd0oGfPWxjk+89J5YJr/qRqISdPd0a++wS/ztzM9hMZvL/yEJP7N1Ud66pKv5gFgFPDEOxtrRWnEaJ2U1qorFy5ssLzuXPn4u3tzZ49e+jcubOiVKK6aJpGrqGES/nFZGblwZ9/UpR2kdILFzGmp2OVno71pUwcLqaxNyCC9zoMJ9dQgkNRIYemD7rqfgOyLxDq5USwpxON6rmw0+8LPJtHUD+mFQEuTgTU4DmKqwvzdub9wc15ddZauj/2PPuemkjLiQ+qjnWZC0dOELl3IwD1nh6vOI0QwqT6qGRlZQHg4eFxxdcNBgMGg6H8eXZ2do3ksnSa0UhJUTGG3HyK8wowoKPQyRVDiZGiAgO2e3ZRkl9AaUEhxoICSgsKMBYY0AoLyPAJ4FiLDuQWlVCUnUP/mW9gk5+LTUEedvl56AvzsS/Mx8FQwKpGMUzqOwEAu5Jijn447KqZLhRBbnTZ3Tc2Ls4c9m9MibMzhZ7eFAcEYd0wDKcmDanbvAmdGjVgnc0//urtE16t75eovH6RvjhnbKddykFyn3uClPatTG6m5WPvfkKMZuRQaBRNurRVHUeIWs9kChVN03jqqafo1KkTzZo1u+I6U6ZM4fXXq//2xkNns9m2dDNN/lwCmgZGDdDQGY2gaWhGjUMde3K6cQs0NDxSkmi77Afg/9f9ezudpnGgQy8SI9tj1DQ8ziXTY9Gcste1/+2z7A1AZzSyp31P9rfujoZGnbRUBs2fBhrl+9QZjejQ0JUa2d76Vv6M6UepUcP94lkmzXkZK2MJ1qWlWBlLsSotxdpY9vMfrXozq9sISowaHpcusnTmA9gYS7E2GrHWjNgCf09eP79Fn/JLKW4FOez/+J6rvle/RXThwwFlPUFsS4t5dfOyq67rmZsJgL2tFXXcXDgUFIHR3gGDmzvF7h4Y69RB5+mJjb8fHuGN+bNTDN6u9jjrbeD1wzfzTypMSMzc6STs3kpEYiwX7ryb/IR9ONYxjRmlDXn5NFz4HQB5o0yvtUeI2shkCpVx48Zx4MABNm/efNV1Jk+ezFNPPVX+PDs7m4CAqm/YP3o+h60rt/PAr7Ouus7SfEfmXyz7z7XDyf2MX/7jVdddVerOT4b6ALQ6c4TJG/646rrrbbxYpm8CQHjaKd7es/6q625xqs92n/YABGZm0ij50FXXtb2UycXcIgBsio04Fhuuuq7eWIqTnTV6W2tcHVw47VmfYls7Suz0lNjaUWpnR4mtHqOdHcYmrbinbQCOdjY46W3Ylj8ZKxdnrN1csXFxxcbdFX0dN/QebjTz8eJwgN//X/OfHH/VDMJy2ert8F76KxejWxNyLondA+4heuNSkxgMbv/UL2ibm0GaiydRTz2iOo4QAhMZ8G38+PH89ttvbNy4kZCQkOverroGfIs7ncXGX/8kes0i0OlAp0OzsgJ06KzKnh/p2JOz4S3Q6aDOuRQiV/0K/3sNndX/frYCnY6UVh240Kysedsp/TyN1ywp3w86HVhZlf98oWlL0pu2QKfTYZ+dSfCfywAdWOnQWVmDDtDp0NnaktuwCblNo7Cx0mFjKMBr91Z0NrZY2dpiZWuDla0NOhsbrGxt0Xy80QICsLHSYa1p6FNPY2Vjg7WdDVZ2ttjY2mLn4oydkwPWNtJ5UFS/hJ+W0eie27HRjGx/6nXaf/iK0jyaprE3vC3RR3ez/dFnaf/Fe0rzCGHJbuT7W2mhomka48ePZ/Hixaxfv56GDRve0PYyMq0Q5m37+JdoP/NtiqxsOLFwGeF39lKWZf2RNB6Zs5VBiZt47ovncatwd5gQoirdyPe30rbWsWPHMm/ePH744QdcXFw4d+4c586do6CgQGUsIUQNaffRG+xteyvJ7vV4bcNpLuZe/ZJkddI0jelrjlJkY4vDww9KkSKECVHaoqLT6a64fO7cuYwaNeo/t5cWFSHMX+7FTIbN2sbBHI2YBp58/2DbGh8MbtPa3YxedQZbez0bn+1m8iPnCmHuzKZFRdO0Kz6up0gRQlgG57p1mPHwLTjaWbPtRDpz5v1Vo8c3lpRSf+Qw1n05hud98qRIEcLEqO9mL4So9cK8XZg6KIoHdy7moQd6s+/juTV27N1vzKBB6nHqFORwx52dauy4QojrI4WKEMIk3BblR0+XImyNpTSaNJbEZdXfspJ1+hwNP3wTgIQHxuNW36fajymEuDFSqAghTEb0T3M42KQ1TkUF1B1yJ8nb9lbr8Q4/+AR18rM4WS+Y6GmvVeuxhBCVI4WKEMJk2NrrCd64msSAxtTJz8KuX1/OJyRWy7HivllEu9W/AJA37SNs7aVvihCmSAoVIYRJca5bB88Na0nxCqDepTSKO3cldX/VTqGQkXQa3/FlI8/u6D2YpvfcXqX7F0JUHSlUhBAmxyPEH5u1q0n18MU/PZUFL83k+IXcKtl3SamRl5YkEOsTykmfYKIWfFUl+xVCVA8pVIQQJsk3KhybzZv4dOBYPm7al6GztrEvOfOm9qlpGi8uPsjys8WMG/YaRWvX4eDuUkWJhRDVQQoVIYTJ8m4SyrB5H9LUz5WLuUU8MmMN296eWal9GUtK+f2Zqfy86xRWOvjk3mgaNWtQxYmFEFVNChUhhEnzdNaz4JH29G7syYeL3yXmpfHsbdeTjKTT172Pgks57LulH3dMe57Jf83ljYHN6BkhtyILYQ6kUBFCmDwXe1s+H9EG265dKLayptXOtVhHNGHbo8+Sn5l11e00o5G4r38mI7Qx0dtXU2RlQ8vbu3Jf+6AaTC+EuBlK5/q5WTLXjxC1z7GVG7B68EEapB4HIM/OgYQOPcl56FG8u3ZEb2NF3v6DFP7+B16/LyT0TNntzefdvLgwczbN7rtDYXohBNzY97dNDWUSQogqEdanC6UnD7PrnZn4fvI+/umptFm/hCdcGrMkruzvrgd3/cbLf34JgMHaln19h9D0q49o5u2pMroQohKkUBFCmB1rWxvavDoR7eUnSPhlBdnzF1DSrh11NT1GTSO5cQv2XeqCoeMthD/zOO0DfVVHFkJUklz6EUIIIUSNupHvb+lMK4QQQgiTJYWKEEIIIUyWFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiTJYWKEEIIIUyWFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiTJYWKEEIIIUyWFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiTJYWKEEIIIUyWjeoAN0PTNKBsumghhBBCmIe/v7f//h6/FrMuVHJycgAICAhQnEQIIYQQNyonJwc3N7drrqPTrqecMVFGo5HU1FRcXFzQ6XRVuu/s7GwCAgJISUnB1dW1SvdtCuT8zJ+ln6Olnx9Y/jnK+Zm/6jpHTdPIycnBz88PK6tr90Ix6xYVKysr/P39q/UYrq6uFvsBBDk/S2Dp52jp5weWf45yfuavOs7xv1pS/iadaYUQQghhsqRQEUIIIYTJkkLlKvR6Pa+++ip6vV51lGoh52f+LP0cLf38wPLPUc7P/JnCOZp1Z1ohhBBCWDZpURFCCCGEyZJCRQghhBAmSwoVIYQQQpgsKVSEEEIIYbKkULkOt99+O4GBgdjb2+Pr68uIESNITU1VHatKnDx5kgcffJCQkBAcHBwIDQ3l1VdfpaioSHW0KvX222/ToUMHHB0dcXd3Vx3npn322WeEhIRgb29PdHQ0mzZtUh2pymzcuJEBAwbg5+eHTqfjt99+Ux2pSk2ZMoU2bdrg4uKCt7c3d9xxB0eOHFEdq0p9/vnnREVFlQ8SFhMTw4oVK1THqhZTpkxBp9MxceJE1VGqzGuvvYZOp6vwqFevnrI8Uqhch27duvHzzz9z5MgRFi1axPHjxxk0aJDqWFXi8OHDGI1GZs2aRXx8PNOnT+eLL77ghRdeUB2tShUVFTF48GAee+wx1VFu2k8//cTEiRN58cUX2bdvH7fccgt9+/YlOTlZdbQqkZeXR/PmzZk5c6bqKNViw4YNjB07lu3bt7NmzRpKSkro1asXeXl5qqNVGX9/f9599112797N7t276d69OwMHDiQ+Pl51tCq1a9cuZs+eTVRUlOooVa5p06acPXu2/BEXF6cujCZu2O+//67pdDqtqKhIdZRqMXXqVC0kJER1jGoxd+5czc3NTXWMm9K2bVttzJgxFZaFh4drzz//vKJE1QfQFi9erDpGtUpLS9MAbcOGDaqjVKs6depoX375peoYVSYnJ0dr2LChtmbNGq1Lly7ahAkTVEeqMq+++qrWvHlz1THKSYvKDcrIyGD+/Pl06NABW1tb1XGqRVZWFh4eHqpjiCsoKipiz5499OrVq8LyXr16sXXrVkWpxM3IysoCsNjfudLSUhYsWEBeXh4xMTGq41SZsWPH0r9/f3r06KE6SrVITEzEz8+PkJAQhg0bxokTJ5RlkULlOj333HM4OTnh6elJcnIyv//+u+pI1eL48eN88sknjBkzRnUUcQUXL16ktLQUHx+fCst9fHw4d+6colSisjRN46mnnqJTp040a9ZMdZwqFRcXh7OzM3q9njFjxrB48WIiIiJUx6oSCxYsYO/evUyZMkV1lGrRrl07vvvuO1atWsWcOXM4d+4cHTp0ID09XUmeWluoXKmz0L8fu3fvLl9/0qRJ7Nu3j9WrV2Ntbc3999+PZsKD+t7o+QGkpqbSp08fBg8ezEMPPaQo+fWrzDlaCp1OV+G5pmmXLROmb9y4cRw4cIAff/xRdZQq17hxY2JjY9m+fTuPPfYYI0eOJCEhQXWsm5aSksKECROYN28e9vb2quNUi759+3L33XcTGRlJjx49WLZsGQDffvutkjw2So5qAsaNG8ewYcOuuU5wcHD5z3Xr1qVu3bo0atSIJk2aEBAQwPbt2022KfNGzy81NZVu3boRExPD7Nmzqzld1bjRc7QEdevWxdra+rLWk7S0tMtaWYRpGz9+PEuWLGHjxo34+/urjlPl7OzsCAsLA6B169bs2rWLjz76iFmzZilOdnP27NlDWloa0dHR5ctKS0vZuHEjM2fOxGAwYG1trTBh1XNyciIyMpLExEQlx6+1hcrfhUdl/N2SYjAYqjJSlbqR8ztz5gzdunUjOjqauXPnYmVlHg1tN/NvaK7s7OyIjo5mzZo13HnnneXL16xZw8CBAxUmE9dL0zTGjx/P4sWLWb9+PSEhIaoj1QhN00z6/8zrdeutt152B8zo0aMJDw/nueees7giBcq+6w4dOsQtt9yi5Pi1tlC5Xjt37mTnzp106tSJOnXqcOLECV555RVCQ0NNtjXlRqSmptK1a1cCAwP54IMPuHDhQvlrKu+br2rJyclkZGSQnJxMaWkpsbGxAISFheHs7Kw23A166qmnGDFiBK1bty5vAUtOTraYfkW5ubkcO3as/HlSUhKxsbF4eHgQGBioMFnVGDt2LD/88AO///47Li4u5a1jbm5uODg4KE5XNV544QX69u1LQEAAOTk5LFiwgPXr17Ny5UrV0W6ai4vLZf2J/u6/aCn9jJ555hkGDBhAYGAgaWlpvPXWW2RnZzNy5Eg1gVTecmQODhw4oHXr1k3z8PDQ9Hq9FhwcrI0ZM0Y7ffq06mhVYu7cuRpwxYclGTly5BXP8a+//lIdrVI+/fRTLSgoSLOzs9NatWplUbe2/vXXX1f8txo5cqTqaFXiar9vc+fOVR2tyjzwwAPln08vLy/t1ltv1VavXq06VrWxtNuThw4dqvn6+mq2traan5+fdtddd2nx8fHK8ug0zYR7hAohhBCiVjOPzghCCCGEqJWkUBFCCCGEyZJCRQghhBAmSwoVIYQQQpgsKVSEEEIIYbKkUBFCCCGEyZJCRQghhBAmSwoVIYQQQpgsKVSEsEBdu3Zl4sSJqmNcUXp6Ot7e3pw8eRKA9evXo9PpuHTpUrUet7LH+eabb3B3d7+hbdq0acOvv/56Q9sIIa5MChUhxH86e/Ys9957L40bN8bKyuqqRdCiRYuIiIhAr9cTERHB4sWLL1tnypQpDBgwwOJmtv6nl19+meeffx6j0ag6ihBmTwoVIcR/MhgMeHl58eKLL9K8efMrrrNt2zaGDh3KiBEj2L9/PyNGjGDIkCHs2LGjfJ2CggK++uorHnrooZqKrkT//v3Jyspi1apVqqMIYfakUBHCwmVmZnL//fdTp04dHB0d6du3L4mJiRXWmTNnDgEBATg6OnLnnXcybdq0Cpc7goOD+eijj7j//vtxc3O74nFmzJhBz549mTx5MuHh4UyePJlbb72VGTNmlK+zYsUKbGxsrjnzeHp6Ovfccw/+/v44OjoSGRnJjz/+WGGdrl27Mn78eCZOnEidOnXw8fFh9uzZ5OXlMXr0aFxcXAgNDWXFihWX7X/Lli00b94ce3t72rVrR1xcXIXXv/nmGwIDA8vfi/T09AqvHz9+nIEDB+Lj44OzszNt2rRh7dq1FdaxtramX79+l+UWQtw4KVSEsHCjRo1i9+7dLFmyhG3btqFpGv369aO4uBgo++IeM2YMEyZMIDY2lp49e/L222/f8HG2bdtGr169Kizr3bs3W7duLX++ceNGWrdufc39FBYWEh0dzdKlSzl48CCPPPIII0aMqNAyA/Dtt99St25ddu7cyfjx43nssccYPHgwHTp0YO/evfTu3ZsRI0aQn59fYbtJkybxwQcfsGvXLry9vbn99tvL34sdO3bwwAMP8PjjjxMbG0u3bt146623Kmyfm5tLv379WLt2Lfv27aN3794MGDCA5OTkCuu1bduWTZs2Xd+bJ4S4OmXzNgshqs3f084fPXpUA7QtW7aUv3bx4kXNwcFB+/nnnzVNK5vSvX///hW2Hz58uObm5nbNff+bra2tNn/+/ArL5s+fr9nZ2ZU/HzhwoPbAAw9UWOevv/7SAC0zM/Oq59OvXz/t6aefrpChU6dO5c9LSko0JycnbcSIEeXLzp49qwHatm3bKhxnwYIF5eukp6drDg4O2k8//aRpmqbdc889Wp8+fSoce+jQoVd9L/4WERGhffLJJxWW/f7775qVlZVWWlp6zW2FENcmLSpCWLBDhw5hY2NDu3btypd5enrSuHFjDh06BMCRI0do27Zthe3+/fx66XS6Cs81TauwrKCgAHt7+2vuo7S0lLfffpuoqCg8PT1xdnZm9erVl7VYREVFlf9sbW2Np6cnkZGR5ct8fHwASEtLq7DdPy87eXh4VHgvDh06dNllqX8/z8vL49lnnyUiIgJ3d3ecnZ05fPjwZfkcHBwwGo0YDIZrnq8Q4tpsVAcQQlQfTdOuuvzvAuLfxcS1truWevXqce7cuQrL0tLSygsGgLp165KZmXnN/Xz44YdMnz6dGTNmEBkZiZOTExMnTqSoqKjCera2thWe63S6Csv+PqfrufPmn+/Ff5k0aRKrVq3igw8+ICwsDAcHBwYNGnRZvoyMDBwdHXFwcPjPfQohrk5aVISwYBEREZSUlFTo35Gens7Ro0dp0qQJAOHh4ezcubPCdrt3777hY8XExLBmzZoKy1avXk2HDh3Kn7ds2ZKEhIRr7mfTpk0MHDiQ++67j+bNm9OgQYPLOv/ejO3bt5f/nJmZydGjRwkPDwfK3q9/vv7v9f/ON2rUKO68804iIyOpV69e+Zgw/3Tw4EFatWpVZbmFqK2kUBHCgjVs2JCBAwfy8MMPs3nzZvbv3899991H/fr1GThwIADjx49n+fLlTJs2jcTERGbNmsWKFSsua2WJjY0lNjaW3NxcLly4QGxsbIWiY8KECaxevZr33nuPw4cP895777F27doKY6707t2b+Pj4a7aqhIWFsWbNGrZu3cqhQ4d49NFHL2upuRlvvPEG69at4+DBg4waNYq6detyxx13APDEE0+wcuVKpk6dytGjR5k5cyYrV668LN+vv/5KbGws+/fv5957771iq82mTZsu61wshLhxUqgIYeHmzp1LdHQ0t912GzExMWiaxvLly8svk3Ts2JEvvviCadOm0bx5c1auXMmTTz55WV+Sli1b0rJlS/bs2cMPP/xAy5Yt6devX/nrHTp0YMGCBcydO5eoqCi++eYbfvrppwr9YyIjI2ndujU///zzVfO+/PLLtGrVit69e9O1a1fq1atXXkhUhXfffZcJEyYQHR3N2bNnWbJkCXZ2dgC0b9+eL7/8kk8++YQWLVqwevVqXnrppQrbT58+nTp16tChQwcGDBhA7969L2s5OXPmDFu3bmX06NFVlluI2kqnVeZitBDCoj388MMcPny4Wm6vXb58Oc888wwHDx7Eysoy/1aaNGkSWVlZzJ49W3UUIcyedKYVQvDBBx/Qs2dPnJycWLFiBd9++y2fffZZtRyrX79+JCYmcubMGQICAqrlGKp5e3vzzDPPqI4hhEWQFhUhBEOGDGH9+vXk5OTQoEEDxo8fz5gxY1THEkIIKVSEEEIIYbos8wKxEEIIISyCFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiTJYWKEEIIIUyWFCpCCCGEMFlSqAghhBDCZEmhIoQQQgiT9X+fZkvEqg05YAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", @@ -3020,9 +2663,35 @@ "\n", "plt.show()" ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "884e5e5a", + "metadata": {}, + "outputs": [], + "source": [] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.18" + } + }, "nbformat": 4, "nbformat_minor": 5 }