diff --git a/doc/pub/week38/ipynb/week38.ipynb b/doc/pub/week38/ipynb/week38.ipynb index 8e33a53e5..7d0cbbbf0 100644 --- a/doc/pub/week38/ipynb/week38.ipynb +++ b/doc/pub/week38/ipynb/week38.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "e1437fb7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "7edae964", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Data Analysis and Machine Learning: Logistic Regression\n", "**Morten Hjorth-Jensen**, Department of Physics and Center for Computing in Science Education, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -28,9 +24,7 @@ { "cell_type": "markdown", "id": "b080984b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plans for week 38\n", "\n", @@ -54,9 +48,7 @@ { "cell_type": "markdown", "id": "8766162e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Ridge and LASSO Regression, reminder\n", "\n", @@ -67,9 +59,7 @@ { "cell_type": "markdown", "id": "a0f26a88", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n", @@ -79,9 +69,7 @@ { "cell_type": "markdown", "id": "2ab8174d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or we can state it as" ] @@ -89,9 +77,7 @@ { "cell_type": "markdown", "id": "7310c495", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -102,9 +88,7 @@ { "cell_type": "markdown", "id": "cf23c167", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have used the definition of a norm-2 vector, that is" ] @@ -112,9 +96,7 @@ { "cell_type": "markdown", "id": "99d4c9b7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n", @@ -124,9 +106,7 @@ { "cell_type": "markdown", "id": "bd386f6d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "By minimizing the above equation with respect to the parameters\n", "$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n", @@ -137,9 +117,7 @@ { "cell_type": "markdown", "id": "182d5d41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -150,9 +128,7 @@ { "cell_type": "markdown", "id": "0d3682e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to the Ridge regression minimization problem where we\n", "require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n", @@ -162,9 +138,7 @@ { "cell_type": "markdown", "id": "25248610", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n", @@ -174,9 +148,7 @@ { "cell_type": "markdown", "id": "da238137", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we have a new optimization equation" ] @@ -184,9 +156,7 @@ { "cell_type": "markdown", "id": "f9bb791c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n", @@ -197,9 +167,7 @@ { "cell_type": "markdown", "id": "01cb254e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n", "\n", @@ -209,9 +177,7 @@ { "cell_type": "markdown", "id": "b6ab9d2d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n", @@ -221,9 +187,7 @@ { "cell_type": "markdown", "id": "a5fd2fb1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Various steps in cross-validation\n", "\n", @@ -246,9 +210,7 @@ { "cell_type": "markdown", "id": "567db724", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## How to set up the cross-validation for Ridge and/or Lasso\n", "\n", @@ -262,9 +224,7 @@ { "cell_type": "markdown", "id": "db50c5d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -278,9 +238,7 @@ { "cell_type": "markdown", "id": "77644e97", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "* Evaluate the prediction performance of these models on the test set by $C[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n", "\n", @@ -292,9 +250,7 @@ { "cell_type": "markdown", "id": "222c145f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Cross-validation in brief\n", "\n", @@ -320,9 +276,7 @@ { "cell_type": "markdown", "id": "1906df63", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code Example for Cross-validation and $k$-fold Cross-validation\n", "\n", @@ -333,10 +287,7 @@ "cell_type": "code", "execution_count": 1, "id": "c152fd7e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -435,9 +386,7 @@ { "cell_type": "markdown", "id": "5d7ccc04", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Logistic Regression\n", "\n", @@ -457,9 +406,7 @@ { "cell_type": "markdown", "id": "b84fbc4f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Classification problems\n", "\n", @@ -483,9 +430,7 @@ { "cell_type": "markdown", "id": "495a8bd0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization and Deep learning\n", "\n", @@ -507,9 +452,7 @@ { "cell_type": "markdown", "id": "48b58d37", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Basics\n", "\n", @@ -532,9 +475,7 @@ { "cell_type": "markdown", "id": "cf4ea236", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\ 1 & \\mathrm{yes} \\end{bmatrix}.\n", @@ -544,9 +485,7 @@ { "cell_type": "markdown", "id": "eeb9a563", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Linear classifier\n", "\n", @@ -562,9 +501,7 @@ { "cell_type": "markdown", "id": "5d676ec0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -580,9 +517,7 @@ { "cell_type": "markdown", "id": "4f1c8890", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n", "$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors." @@ -591,9 +526,7 @@ { "cell_type": "markdown", "id": "95d3b55e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Some selected properties\n", "\n", @@ -618,9 +551,7 @@ { "cell_type": "markdown", "id": "a58cb406", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example\n", "\n", @@ -631,10 +562,7 @@ "cell_type": "code", "execution_count": 2, "id": "cccbdeef", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Common imports\n", @@ -696,9 +624,7 @@ { "cell_type": "markdown", "id": "eda7b9f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plotting the mean value for each group\n", "\n", @@ -709,10 +635,7 @@ "cell_type": "code", "execution_count": 3, "id": "c657b457", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "agegroupmean = np.array([0.1, 0.133, 0.250, 0.333, 0.462, 0.625, 0.765, 0.800])\n", @@ -728,9 +651,7 @@ { "cell_type": "markdown", "id": "c027575d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We are now trying to find a function $f(y\\vert x)$, that is a function which gives us an expected value for the output $y$ with a given input $x$.\n", "In standard linear regression with a linear dependence on $x$, we would write this in terms of our model" @@ -739,9 +660,7 @@ { "cell_type": "markdown", "id": "02de7a3d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n", @@ -751,9 +670,7 @@ { "cell_type": "markdown", "id": "64969b5a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This expression implies however that $f(y_i\\vert x_i)$ could take any\n", "value from minus infinity to plus infinity. If we however let\n", @@ -770,9 +687,7 @@ { "cell_type": "markdown", "id": "c8870523", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The logistic function\n", "\n", @@ -792,9 +707,7 @@ { "cell_type": "markdown", "id": "17cebf36", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n", @@ -804,9 +717,7 @@ { "cell_type": "markdown", "id": "ef1c3060", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that $1-p(t)= p(-t)$." ] @@ -814,9 +725,7 @@ { "cell_type": "markdown", "id": "65471b2a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Examples of likelihood functions used in logistic regression and nueral networks\n", "\n", @@ -827,10 +736,7 @@ "cell_type": "code", "execution_count": 4, "id": "e9a4591d", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\"\"\"The sigmoid function (or the logistic curve) is a\n", @@ -892,9 +798,7 @@ { "cell_type": "markdown", "id": "b988efeb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Two parameters\n", "\n", @@ -904,9 +808,7 @@ { "cell_type": "markdown", "id": "cfb6a943", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -919,9 +821,7 @@ { "cell_type": "markdown", "id": "2361aee9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n", "\n", @@ -931,9 +831,7 @@ { "cell_type": "markdown", "id": "847fd47b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n", @@ -943,9 +841,7 @@ { "cell_type": "markdown", "id": "be4b11d1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Maximum likelihood\n", "\n", @@ -960,9 +856,7 @@ { "cell_type": "markdown", "id": "0c649e6a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -974,9 +868,7 @@ { "cell_type": "markdown", "id": "a2382368", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "from which we obtain the log-likelihood and our **cost/loss** function" ] @@ -984,9 +876,7 @@ { "cell_type": "markdown", "id": "8e56dda5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n", @@ -996,9 +886,7 @@ { "cell_type": "markdown", "id": "c77515e4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The cost function rewritten\n", "\n", @@ -1008,9 +896,7 @@ { "cell_type": "markdown", "id": "d0d27dc3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", @@ -1020,9 +906,7 @@ { "cell_type": "markdown", "id": "4955b6b1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n", "Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that" @@ -1031,9 +915,7 @@ { "cell_type": "markdown", "id": "c3e91e52", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n", @@ -1043,9 +925,7 @@ { "cell_type": "markdown", "id": "3b3d97b6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n", "in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression." @@ -1054,9 +934,7 @@ { "cell_type": "markdown", "id": "47890d23", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Minimizing the cross entropy\n", "\n", @@ -1070,9 +948,7 @@ { "cell_type": "markdown", "id": "71d34eed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n", @@ -1082,9 +958,7 @@ { "cell_type": "markdown", "id": "243773d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1092,9 +966,7 @@ { "cell_type": "markdown", "id": "81fd7337", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n", @@ -1104,9 +976,7 @@ { "cell_type": "markdown", "id": "38fbde35", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## A more compact expression\n", "\n", @@ -1119,9 +989,7 @@ { "cell_type": "markdown", "id": "bd7644e6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -1131,9 +999,7 @@ { "cell_type": "markdown", "id": "0fcc74e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -1142,9 +1008,7 @@ { "cell_type": "markdown", "id": "7b5aa78f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -1154,9 +1018,7 @@ { "cell_type": "markdown", "id": "45ee534e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Extending to more predictors\n", "\n", @@ -1166,9 +1028,7 @@ { "cell_type": "markdown", "id": "f62d8583", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n", @@ -1178,9 +1038,7 @@ { "cell_type": "markdown", "id": "837cc6c1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to" ] @@ -1188,9 +1046,7 @@ { "cell_type": "markdown", "id": "75b9139f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n", @@ -1200,9 +1056,7 @@ { "cell_type": "markdown", "id": "372ee8ea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Including more classes\n", "\n", @@ -1214,9 +1068,7 @@ { "cell_type": "markdown", "id": "b171dd51", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n", @@ -1226,9 +1078,7 @@ { "cell_type": "markdown", "id": "7816121e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and" ] @@ -1236,9 +1086,7 @@ { "cell_type": "markdown", "id": "b58d0bc3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n", @@ -1248,9 +1096,7 @@ { "cell_type": "markdown", "id": "0d75989b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and so on till the class $C=K-1$ class" ] @@ -1258,9 +1104,7 @@ { "cell_type": "markdown", "id": "af883d63", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n", @@ -1270,9 +1114,7 @@ { "cell_type": "markdown", "id": "6727eebf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and the model is specified in term of $K-1$ so-called log-odds or\n", "**logit** transformations." @@ -1281,9 +1123,7 @@ { "cell_type": "markdown", "id": "c8713334", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More classes\n", "\n", @@ -1304,9 +1144,7 @@ { "cell_type": "markdown", "id": "7819ea46", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n", @@ -1316,9 +1154,7 @@ { "cell_type": "markdown", "id": "278c2527", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "It is easy to extend to more predictors. The final class is" ] @@ -1326,9 +1162,7 @@ { "cell_type": "markdown", "id": "376c873f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n", @@ -1338,9 +1172,7 @@ { "cell_type": "markdown", "id": "389cdb39", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and they sum to one. Our earlier discussions were all specialized to\n", "the case with two classes only. It is easy to see from the above that\n", @@ -1355,9 +1187,7 @@ { "cell_type": "markdown", "id": "5a3810ce", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Friday September 23" ] @@ -1365,9 +1195,7 @@ { "cell_type": "markdown", "id": "f1e08704", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Searching for Optimal Regularization Parameters $\\lambda$\n", "\n", @@ -1384,10 +1212,7 @@ "cell_type": "code", "execution_count": 5, "id": "d2e8e94e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1439,9 +1264,7 @@ { "cell_type": "markdown", "id": "d9054061", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n", "By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n", @@ -1451,9 +1274,7 @@ { "cell_type": "markdown", "id": "e8d8872f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Grid Search\n", "\n", @@ -1466,10 +1287,7 @@ "cell_type": "code", "execution_count": 6, "id": "212e1c6b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1519,9 +1337,7 @@ { "cell_type": "markdown", "id": "91b3b83c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "By default the grid search function includes cross validation with\n", "five folds. The [Scikit-Learn\n", @@ -1534,9 +1350,7 @@ { "cell_type": "markdown", "id": "7e0ab91a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Randomized Grid Search\n", "\n", @@ -1554,10 +1368,7 @@ "cell_type": "code", "execution_count": 7, "id": "92c30256", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", @@ -1608,9 +1419,7 @@ { "cell_type": "markdown", "id": "fb8e4f5a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Wisconsin Cancer Data\n", "\n", @@ -1623,10 +1432,7 @@ "cell_type": "code", "execution_count": 8, "id": "d040e14f", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1650,9 +1456,7 @@ { "cell_type": "markdown", "id": "47009c95", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using the correlation matrix\n", "\n", @@ -1664,10 +1468,7 @@ "cell_type": "code", "execution_count": 9, "id": "40128247", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1709,9 +1510,7 @@ { "cell_type": "markdown", "id": "dc60a6ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Discussing the correlation data\n", "\n", @@ -1734,10 +1533,7 @@ "cell_type": "code", "execution_count": 10, "id": "47ec15b8", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)" @@ -1746,9 +1542,7 @@ { "cell_type": "markdown", "id": "eb8b4198", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and then" ] @@ -1757,10 +1551,7 @@ "cell_type": "code", "execution_count": 11, "id": "58486f42", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "correlation_matrix = cancerpd.corr().round(1)" @@ -1769,9 +1560,7 @@ { "cell_type": "markdown", "id": "07834b9b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Diagonalizing this matrix we can in turn say something about which\n", "features are of relevance and which are not. This leads us to\n", @@ -1782,9 +1571,7 @@ { "cell_type": "markdown", "id": "9908e2e3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Other measures in classification studies: Cancer Data again" ] @@ -1793,10 +1580,7 @@ "cell_type": "code", "execution_count": 12, "id": "645bfbca", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", @@ -1836,9 +1620,7 @@ { "cell_type": "markdown", "id": "f528c325", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -1857,9 +1639,7 @@ { "cell_type": "markdown", "id": "ff30edd5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -1874,9 +1654,7 @@ { "cell_type": "markdown", "id": "847724bd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -1889,9 +1667,7 @@ { "cell_type": "markdown", "id": "f1969cb5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] @@ -1899,9 +1675,7 @@ { "cell_type": "markdown", "id": "e163f74c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations to solve\n", "\n", @@ -1915,9 +1689,7 @@ { "cell_type": "markdown", "id": "9b49b101", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -1927,9 +1699,7 @@ { "cell_type": "markdown", "id": "f19697ae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -1938,9 +1708,7 @@ { "cell_type": "markdown", "id": "3f641f13", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -1950,9 +1718,7 @@ { "cell_type": "markdown", "id": "d82f5c8e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This defines what is called the Hessian matrix." ] @@ -1960,9 +1726,7 @@ { "cell_type": "markdown", "id": "f8d50532", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -1974,9 +1738,7 @@ { "cell_type": "markdown", "id": "81e002c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -1986,9 +1748,7 @@ { "cell_type": "markdown", "id": "9ec52629", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or in matrix form as" ] @@ -1996,9 +1756,7 @@ { "cell_type": "markdown", "id": "d0b02495", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -2008,9 +1766,7 @@ { "cell_type": "markdown", "id": "b1475d07", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -2020,9 +1776,7 @@ { "cell_type": "markdown", "id": "7297eb74", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -2040,9 +1794,7 @@ { "cell_type": "markdown", "id": "5ca9494b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations\n", "\n", @@ -2056,9 +1808,7 @@ { "cell_type": "markdown", "id": "ec42264c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -2072,9 +1822,7 @@ { "cell_type": "markdown", "id": "0d0c3ae9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -2083,9 +1831,7 @@ { "cell_type": "markdown", "id": "df03a2fd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -2095,9 +1841,7 @@ { "cell_type": "markdown", "id": "f6b5cd5a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "yielding" ] @@ -2105,9 +1849,7 @@ { "cell_type": "markdown", "id": "251ef281", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -2117,9 +1859,7 @@ { "cell_type": "markdown", "id": "84013f15", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] @@ -2127,9 +1867,7 @@ { "cell_type": "markdown", "id": "0498425e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -2139,9 +1877,7 @@ { "cell_type": "markdown", "id": "096433ab", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple geometric interpretation\n", "\n", @@ -2161,9 +1897,7 @@ { "cell_type": "markdown", "id": "b0851d8d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Extending to more than one variable\n", "\n", @@ -2174,9 +1908,7 @@ { "cell_type": "markdown", "id": "010769ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -2187,9 +1919,7 @@ { "cell_type": "markdown", "id": "d853f614", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we Taylor expand to obtain" ] @@ -2197,9 +1927,7 @@ { "cell_type": "markdown", "id": "4bac06cd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -2215,9 +1943,7 @@ { "cell_type": "markdown", "id": "85987829", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] @@ -2225,9 +1951,7 @@ { "cell_type": "markdown", "id": "e0708642", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -2240,9 +1964,7 @@ { "cell_type": "markdown", "id": "8c47428d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we can rephrase Newton's method as" ] @@ -2250,9 +1972,7 @@ { "cell_type": "markdown", "id": "5b1a1b9b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -2264,9 +1984,7 @@ { "cell_type": "markdown", "id": "688f39ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined" ] @@ -2274,9 +1992,7 @@ { "cell_type": "markdown", "id": "0540c80a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -2288,9 +2004,7 @@ { "cell_type": "markdown", "id": "8749a5d9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -2303,9 +2017,7 @@ { "cell_type": "markdown", "id": "10468ee8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent\n", "\n", @@ -2320,9 +2032,7 @@ { "cell_type": "markdown", "id": "fd5204db", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -2332,9 +2042,7 @@ { "cell_type": "markdown", "id": "3643016f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -2346,9 +2054,7 @@ { "cell_type": "markdown", "id": "3f3e0600", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on Steepest descent\n", "\n", @@ -2361,9 +2067,7 @@ { "cell_type": "markdown", "id": "51927cc7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -2373,9 +2077,7 @@ { "cell_type": "markdown", "id": "62148b2a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -2384,9 +2086,7 @@ { "cell_type": "markdown", "id": "0aaedf94", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The ideal\n", "\n", @@ -2412,9 +2112,7 @@ { "cell_type": "markdown", "id": "13569ae6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -2434,9 +2132,7 @@ { "cell_type": "markdown", "id": "bfdc53c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex functions\n", "\n", @@ -2456,9 +2152,7 @@ { "cell_type": "markdown", "id": "a73c982b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex function\n", "\n", @@ -2468,9 +2162,7 @@ { "cell_type": "markdown", "id": "a56a2ae9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conditions on convex functions\n", "\n", @@ -2505,9 +2197,7 @@ { "cell_type": "markdown", "id": "3090b99d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on convex functions\n", "\n", @@ -2533,9 +2223,7 @@ { "cell_type": "markdown", "id": "093b11b8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Some simple problems\n", "\n", @@ -2563,9 +2251,7 @@ { "cell_type": "markdown", "id": "6f15fc92", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our first homework\n", "\n", @@ -2587,10 +2273,7 @@ "cell_type": "code", "execution_count": 13, "id": "ce786ea2", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -2600,9 +2283,7 @@ { "cell_type": "markdown", "id": "02fce565", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -2611,9 +2292,7 @@ { "cell_type": "markdown", "id": "f2ae83a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -2623,9 +2302,7 @@ { "cell_type": "markdown", "id": "9016e945", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "such that" ] @@ -2633,9 +2310,7 @@ { "cell_type": "markdown", "id": "9195a113", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -2645,9 +2320,7 @@ { "cell_type": "markdown", "id": "e66e1135", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent example\n", "\n", @@ -2659,9 +2332,7 @@ { "cell_type": "markdown", "id": "4e5fe9a8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -2675,9 +2346,7 @@ { "cell_type": "markdown", "id": "4178bf2b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The cost/loss/risk function is given by (" ] @@ -2685,9 +2354,7 @@ { "cell_type": "markdown", "id": "91dbd291", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -2697,9 +2364,7 @@ { "cell_type": "markdown", "id": "b36bcd33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] @@ -2707,9 +2372,7 @@ { "cell_type": "markdown", "id": "dcfc85c6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -2719,9 +2382,7 @@ { "cell_type": "markdown", "id": "cd6314b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2733,9 +2394,7 @@ { "cell_type": "markdown", "id": "54f6c9ae", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $X$ is the design matrix defined above." ] @@ -2743,9 +2402,7 @@ { "cell_type": "markdown", "id": "9b2c6838", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -2754,9 +2411,7 @@ { "cell_type": "markdown", "id": "46b26877", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2769,9 +2424,7 @@ { "cell_type": "markdown", "id": "4972c71f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] @@ -2779,9 +2432,7 @@ { "cell_type": "markdown", "id": "f8ed6356", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple program\n", "\n", @@ -2791,9 +2442,7 @@ { "cell_type": "markdown", "id": "643c173d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -2803,9 +2452,7 @@ { "cell_type": "markdown", "id": "c5629d6e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -2818,9 +2465,7 @@ { "cell_type": "markdown", "id": "ec12f593", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient Descent Example\n", "\n", @@ -2829,13 +2474,34 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 17, "id": "27555cc9", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Eigenvalues of Hessian Matrix:[0.28065531 4.4834507 ]\n", + "[[4.31499792]\n", + " [2.76774725]]\n", + "[[4.31499792]\n", + " [2.76774725]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtxklEQVR4nO3deXxcdb3/8dcnadM2bWnpQhdKE5ACArIWkCW1iktFEJSrggUB0Uq5KoqKSy+bWsUrKq4/RFZpRRBcALlXuJCSlK2mQFlKW1q60kLTDdqma/L5/XFOyjSdmZxJzsyZSd7Px2MemTnr55yZnM/5Luccc3dERETaU5Z0ACIiUhqUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMKTgzW2JmHwzff8/Mbk4ojvFmtiKJdXcXqd+1lD4lDNmNmZ1jZs+Y2WYzWx2+v9TMLB/rc/cfufsXOrscM6s2MzezHnHElTQzu93Mfph0HCKplDBkFzP7BvBL4KfAcGAYcAlwMlCRYZ7yggUoIolSwhAAzGwA8H3gUne/1903euA5d5/o7tvC6W43s/9nZg+Z2Wbg/Wb2MTN7zszeNrPlZnZNm2Wfb2ZLzWytmU1pM+4aM5uW8vm9ZvakmW0wszlmNj5l3Awz+4GZPWFmG83sYTMbEo6uC/9uMLNNZnZimm3sE8a/3szmAse1GT/SzO4zs0YzW2xmX00Zd7yZNYTb+KaZ/Txl3CkpMS83swvD4b3M7HozWxbOc6OZ9QnHjTezFWb2jbAkt8rMLgrHTQImAleE2/JAhu/sEDN7xMzWmdl8M/t0OPxd4bBjUrarsXVfmtlFZvZKuA9fM7MvpSyzNa4rUuI6y8xOM7MF4XK/1+b7u9fM7g6X96yZHZkh3jIz+46ZLQp/C/eY2aB000qRcne99AKYAOwEerQz3e3AWwSljjKgNzAeeE/4+QjgTeCscPpDgU3AOKAX8PNwPR8Mx18DTAvf7wusBU4Ll/Wh8PPQcPwMYBFwENAn/HxdOK4a8GzxA9cB9cAgYD/gJWBFOK4MmA1cRVCaOgB4DfhIOP4p4PzwfT/gveH7KmAjcC7QExgMHBWO+wVwf7i+/sADwI/DcePD/fD9cL7TgCZg75T9/MMs29IXWA5cBPQAjgbWAIeG478IzAUqgX8B16fM+zHgXYAB7wvXe0ybuK4K4/oi0Aj8KdyGw4AtwP4p398O4D/C6b8JLAZ6huOXpHzXlwFPA6MIfgu/B+5K+revVw7HiaQD0Ks4XsB5wBtthj0JbAgPEOPCYbcDf2xnWTcAvwjfXwX8OWVcX2A76RPGt4E72yzrX8AF4fsZwH+ljLsU+N/wfTXtJ4zXgAkpnyfxTsI4AVjWZvrvAreF7+uAa4Ehaab5W5p1GbAZeFfKsBOBxeH78eF+7ZEyfjXvJKLbyZ4wPgPUtxn2e+DqlM/3Ay8CLwC9sizr78BlbeIqDz/3D/frCSnTz+adE4JrgKdTxpUBq4Ca8POSlO/6FeDUlGlHECSbrCcpehXPq0s0EEos1gJDzKyHu+8EcPeTAMKeRKnVl8tTZzSzEwjO3g8nODvvBfwlHD0ydXp332xmazPEUAV8yszOSBnWE6hN+fxGyvsmgrP9qHaLBVjaZt0jzWxDyrByghIJwMUEpYF5ZrYYuNbdHyQoqSxKs66hBGf3s1P6C1i4zFZrW/d1B7anCjihTbw9gDtTPv+BIGlM8rBKEcDMPgpcTVBSKwvjfLFNXM3h+y3h3zdTxm9pE2fq99sS/l5GZoj5b2bWkjKsmaCt7PX0mynFRAlDWj0FbAPOBO5rZ9q2tzj+E/Ab4KPuvtXMbgBa2xZWAe9undDMKgmqbdJZTlDC+GJuoaeNKZ1VBAf4l8PPo9use7G7j0m7cPdXgXPNrAz4JHCvmQ0O5zs+zSxrCA6sh7l7Rw6G7W3PcuBxd/9QupFm1o+gpHcLcI2Z3efu68ysF8H3+zngH+6+w8z+TpDMOmq/lPWWEVQ5rcwQ8+fd/YlOrEsSpEZvAcDdNxBUufzOzP7DzPqHjZRHEVQjZdMfWBcmi+OBz6aMuxc4PWwYriA4S8/0u5sGnGFmHzGzcjPrHTbCjoqwCY1AC0HbQyb3AN81s73DZX4lZdwsYKOZfTtsHC83s8PN7DgAMzvPzIa6ewtBNR3h+qYDHzSzT5tZDzMbbGZHhdP9AfiFme0TLmNfM/tIhG2B4Iw+27Y8CBxkQYeCnuHrODNrTc6/BBo86LL8T+DGcHhrCbAR2BmWNj4cMaZMjjWzT1rQpflrBCceT6eZ7kZgqplVAZjZUDM7s5PrlgJSwpBd3P2/gcuBKwgOWG8S1It/m6A9I5NLge+b2UaCNot7Upb5MvCfBKWQVcB6IO3Fcu6+nKCE8z2CA9py4FtE+J26exMwFXgi7K303jSTXUtQDbUYeJiU6puwCuZ04Khw/BrgZmBAOMkE4GUz20RwMD7H3be4+zKCButvAOuA54HWXkLfBhYCT5vZ28D/AQe3ty2hW4BDw235e5rt3UhwoD+H4Gz+DeAnQK/wIDwBmBxOfjlwjJlNDOf7KsF3tJ4gud8fMaZM/kHQprIeOB/4pLvvSDPdL8N1PRz+Vp4maDuSEmHueoCSiHSMBV2oD3T385KORfJPJQwREYlECUNERCJRlZSIiESiEoaIiERSUtdhDBkyxKurq5MOQ0SkpMyePXuNuw/t7HJKKmFUV1fT0NCQdBgiIiXFzJa2P1X7VCUlIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJHlPGGZ2q5mtNrOX0oz7hpm5mQ1JN6+IiBSPQpQwbid4XORuzGw/gkdMLitADCIi0kl5TxjuXkfwrOO2fkHw7Gg9kENEpAQk0oYRPqT+dXefE2HaSWbWYGYNjY2NBYhORETSKXjCMLNK4HvAVVGmd/eb3H2su48dOrTTt3MXEZEOSqKE8S5gf2COmS0BRgHPmtnwBGIREZGICv4AJXd/Edin9XOYNMa6+5pCxyIiItEVolvtXcBTwMFmtsLMLs73OkVEJH55L2G4+7ntjK/OdwwiItJ5utJbREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0S6j+nToboaysqCv9OnJx1RSSn4vaRERBIxfTpMmgRNTcHnpUuDzwATJyYXVwlRCUNEuocpU95JFq2amoLhEokShoh0D8syPA0603DZgxKGiHQPo0fnNlz2oIQhIt3D1KlQWbn7sMrKYLhEooQhIt3DxIlw001QVQVmwd+bblKDdw7US0pEuo+JE5UgOkElDBERiUQJQ0REIlHCEBGRSJQwREQkkrwnDDO71cxWm9lLKcN+ambzzOwFM/ubmQ3MdxwiItI5hShh3A5MaDPsEeBwdz8CWAB8twBxiIhIJ+Q9Ybh7HbCuzbCH3X1n+PFpYFS+4xARkc4phjaMzwP/k2mkmU0yswYza2hsbCxgWCIikirRhGFmU4CdQMab0rv7Te4+1t3HDh06tHDBiYjIbhK70tvMLgROB051d08qDhERiSaRhGFmE4ArgPe5e1N704uISPIK0a32LuAp4GAzW2FmFwO/AfoDj5jZ82Z2Y77jEBGRzilEL6lz3X2Eu/d091Hufou7H+ju+7n7UeHrknzHISJSNEr02eLF0EtKRKR0dPZg3/ps8aVLwf2dZ4uXQNJQwhARiSqOg30JP1tcCUNEJKo4DvYl/GxxJQwRkajiONiX8LPFlTBERKKK42DfkWeLF0kjuRKGiEhUHTnYt5Xrs8WLqJHcSuki67Fjx3pDQ0PSYYhIdzZ9etBmsWxZULKYOjW/zwmvrg6SRFtVVbBkSaRFmNlsdx/b2VCUMEREillZWVCyaMsMWloiLSKuhKEqKRGRYlZEjeRKGCIixSyOdpOYKGGIiBSzXBvJ8yix25uLiEhEEycmkiDaUglDREQiUcIQEZFIlDBERCQSJQwRkVJQBLcHUaO3iEixa709SOudcltvDwIFbQxXCUNEuq8iOGuPpEieoaEShoh0T0Vy1h5JkTxDQyUMEemeiuSsPZIiuT1I3hOGmd1qZqvN7KWUYYPM7BEzezX8u3e+4xAR2U2RnLVHUiS3BylECeN2YEKbYd8BHnX3McCj4WcRkcIpkrP2SIrk9iB5TxjuXgesazP4TOCO8P0dwFn5jkNEZDdFctYe2cSJwfMvWlqCvwm0syTVhjHM3VeF798AhmWa0MwmmVmDmTU0NjYWJjoR6fqK5Ky9lCTeS8rd3cwyPsXJ3W8CboLgAUoFC0xEur4iualf3La9vY2G6fOp/8c66p/tG9tyk0oYb5rZCHdfZWYjgNUJxSEiUvI2vbGJp26fT/1DG6l7YSDPvHUwWzkCgEMqFsW2nqQSxv3ABcB14d9/JBSHiEjJWTN/LTNvX0j9I1uonzeUZzcfTDPHUkYzR1fOZ/Ixz1Dzwd6ccuGBDH33uzCLZ715TxhmdhcwHhhiZiuAqwkSxT1mdjGwFPh0vuMQkTamTw+uOVi2LOgZNHVql6ye6QqWP7OS+j8upq62mfrXRjJ324HAYHqxlRMGzOM7J8+k5qP9OPGCg9hr1KHAoXmJI+8Jw93PzTDq1HyvW0QyKKWrnLsZnzadnd+4gh6rV7G2xz78oPl7/Mq/CoykP29z8pD5nPe+GdScOYjjzjuYXnsdVbDYzL102pHHjh3rDQ0NSYchUvqqq4Mk0VZVVdBlUwqmeXszc/6ygPr7VtNSO4MvbfgJlWzZNX6r9aL22G8y/PLzOOLsMZRXlOe8DjOb7e5jOxurEoZId1RWBun+982Cfv6lqgSq2bZu2Mq/p82n/v711D/XlyfWHMxG9gJgOaMYxet7ztTJRB5XwtC9pES6o1K6yjmq1mq2pUuDZNhazZbLHWjzcPfajSs38q+pDUw5eQbjBsxh4N4w7itHMuWR8SzfOJCJhz7Pn778JMtnrWKUrUy/kKVLi+JuuiphiHRHbdswILjKuZQvXOtoNVtrqWTp0qCElXpM7MA+aXxlDfW3LaT+/7ZRN28fnt9yEC2UU85Ojuk7n3GHNFLz4T6cctEYBo8ZFG0bOhEPxFfCwN1L5nXssce6iMRk2jT3qip3s+DvtGlJR9S5mMzcg8P97i+z7OurrEw/X+urqirrapfMXO53XjLTv3jI435IxcJds/WmyccPfNavrKn1R34y2zeu2hht+zsZTzpAg8dwDFYJQ0SKQ2dLPR0pYbR3Rg+7tet4i/PKg4uov3sl9U+WU7e8muXN+wIwgLc4eegCxh27mZqzBnPsuQfRa69e7cfdVmqJp514olKjt4h0LUlUKWVq/E+xbfAIfnfKn6ib1ZuZbxzIGh8CwPCyN6kZ+Ro1J2xn3KeHc/hZB3aoB1NGMfZkiythJH4vKRERoGPPp2hbKnF/J2lUVbXfS2r06KwljCb68IW1P+Wuf4zngB5LOf1dr1BzCtR8dj8OPLUKK8t439TOmzo1fYkrwbvpqpeUiEST7+dfd6TnVrqn5rUmiwi3AG+6/L/Y2bP3bsNaMFqAlYzk7lGXc+ZX92fFv1exaEcVt71aw+dvq2HMh6qxspjut5FJEd5NV1VSItK+QvSq6sg6crye5M2XGoMeTI9up27BMF7YMobP8Gd+zPfYj+W8VbEPr512Kftf/xUGvavrPAhU12GISOEU4vnXHTmjzlIq8RZncd1y7vjiTL5wcD0HVyxm+HuG8qmfn8gf5hzHoN5NXPm+er5w/aEMeXMuZd7C3rf+jGOfu5VBYwYXxXUPRSeOrlaFeqlbrUhCOtJltRDSdEPdXt7Lfz34Kt+3bOWuwQNtvZ8x7Gn/79Nq/ak/vOjbNm6LtCyvrIyvu3GC3ZiJqVutShgi0r4ivDJ8R9MOZm0/igcP+jqry4bTgrGEKi5ovoUfr7+EU/ZdzG8/8zgv3LuAtdv34v43TuBb/xzPe79wOBX9KvZcYD5LUR29Cj3f7Ua5iiPrFOqlEoZIQvJ99h3B5sbN/tjPnvVr31/rp+492yvZtCuUA3su9s+PqfPbLq73hY8u8ZbmluwLS3e2H2cpqu3yBw9Ov+xsF+HFuM+JqYSReBLI5aWEISWjGK+i7qzUbRo8OHjlcfvWvbbeH7jyGb/i+Fo/sd8L3pNtwfGbZj+y9zz/8ntm+D1ff9JXPvdG7tuR7kDct2/uB/Woy8/0ypaMqqriiceVMESKVxGcjXdatoSXp+1b+dwbfvfXnvD/fM8MP6L3PDeaHdx7ss1P6j/Hv31CrT949Sxfv2RDp9aT8UCc7lVRkXm7Mu2jXJaf7eAfY4lHCUOkWMV4ZpizOEo27SWEzm7ftGneMnq0t5j5xr7D/MZhV/qBPRfvWkxfNvoHBzX49z9Q67W/eM43N27OfRvabk/qPol6MIegFJXrPsp0oG/7ai/JqoShhCHdQFI9iuI682/vQNWB7Wve0exz/jLf/3X8FN9ivXabbzN9/Pq9rvHrT6/1Z259ybdv3t7hXbCHdPsk6gE92zZl20eZxg0enFsyVxuGEoZ0A0mVMOJab3sJIcJ6tm3c5gsuvs439B7mzZgvZZSfyzRfTEwxRpUp1rbbmGmbM8WVbR/FWWUXU1uYEoZIsYp6wGivnSDXA0VcJZts1TZVVe6TJ++xfS19+viL5/zQr35frb9/4LN+Abf6JnafZkePXt6S65l8Z2UrTaTu3zTblPUg317SLLJOD10iYQBfB14GXgLuAnpnm14Jowsqsn+s2LS3XdmSSkfPUOMqYbTXy6ey0rdM/Lxv3mu4t2C+0kb4edzh4F7GTj+6z1xf23NY+nnLy+OJMapcDuy59PyaPHnPZFTEHRtKPmEA+wKLgT7h53uAC7PNo4TRxXSF3kQd1ZE68PYOqvmoCsmQNBZT5eBewVY/Za/n/bsn1vpD187yDUs3BPNnO7NPF+Pkyfk5cchHYs7ULjJ5cjwx50FXSRjLgUEEt1l/EPhwtnmUMLqYJHsTJS1b9VFnqpZiKLG1NLf4gocX+80X1HkL6WNpwfzxXz3vW9ZvSb+QTN9tawmj9W9HqoNylWv31/Z+fyX4uy35hBFsA5cBm4BGYHqGaSYBDUDD6NGjY92JkrBivT9RIeSjhNGedAfOlC6ub/UZ5jcMvNqHlb25a5XLGJX5wJ+tGifKxWutSSGpA3BHf38dfRRsglWvJZ8wgL2Bx4ChQE/g78B52eZRCaOLKZUztXz8s+ejqiTH9e2wHr6NHrsN20wf/82Qq/z3Ex/3uQ8s9JY7c7hquW2cqfstW9tFUicOhSphFEHVa1dIGJ8Cbkn5/Dngd9nmUcJIUKEPmsUinzHG3Usqg42rNnrTwOHRD/ptD3xRDvztHTizJYUoB+Bi+v21l+zbxlkEJ0ZdIWGcEPaQqgQMuAP4SrZ5lDASku4fpGfPeO4lVOy9pDrTAJ3rdqUeXFLr+HPcJ43z1vjfvvO0X35srR/X9yUvZ4c3Z2iLyPnMPspFb+nmz7Yf2ztwJ5W0c50vU5wd2c8xK/mEEWwD1wLzwm61dwK9sk2vhJGQKLdTKLaSQVw6Wl+d6wEuW51/O/Mue/p1n37pTP/Sux/3Q3u9umu2XmzxcQOe8ykn13rTgE6UMFJF+S2kmz9KUsh04C6CM/RI2mvoVwmjsC8ljIREvZVCIf4B8lU1EefBKs552szb0tzir/xzkd90/uN+/gH1XlW+fNck/XnLJwyZ5T/6cK3X/3aOb31r6+7b2PaAXVERlBTjSmztzd/R765UOkfk2pW4K7ZhAI8AR8axss6+lDASEuWsshD/wAVqDN6tT31H1tmRA1w7SbkF80+OfMqH2updg/ex1X72vk/6DZ+Y4c/+6RXfuW1n+9uarhqlo1VnBbrNecmXMFLbMrp6LyngGKAWuA0YEcdKO/pSwkhIlC6S+f4HnjYtP0X7TP/krfcEal13Lv/seShhLKbKq3ss888dUO9/+Fydz3toUfsPCeoqSqFzhHtRx1nwKingbGAOcHXr1dmFfilhJKjtWWVFReH+MdpLWJ0p2bR3r6G44s2wf95+/W3/3x/+2+8eM8W30DttHDt69PI13/9Nx+Io5g4FuSiVbSnSOAuaMMJeTIcDlwBrgBXA+XEEkMtLCaOIFPIfI2L9fuzL7kwiyrB/Vs9t9Pu+9ZR/7egZfkyfuV7GTgf3cnb4f1Vc52srhnkLeEsnekntWn8hznaL9AApuytkldQTwMqwLeMHwOnAgcCvgZviCCLqSwmjm2qvMbGzbRi53to6B0tmLvc7L5npXzzkcT+kYuGuRfemyccPfNavrKn1R34y2zeu2tjpde2mEPX+RVwFk1Y3Tm6FTBiHAZZh3CtxBBH1pYTRTWXrrhjHP31Mdx5taW7xl//xqt/42cd9YvVM3698xa7FDWCDnzZ0ll83odafuPGF3Xsw5UMhehaVSmO0e+klt5gVRbda4IA4goj6UsLopgrxz97e2Wea8Tu27PB/3/Gy/+zjtX7WiKd8iDXuCm942Rv+qVFP+q/OnuHP3z2v/R5McSvEwbxUuru6l1Zyy4OiSBiFfilhdGO5VifEWf2QJmFtobdfxM27Bh3QY4lfeGCd33JhnS94eHHyPZgKkWRL6SBcSsktD5QwpFvXyWYV08Fyw9IN/tC1s3xdRfqHAa3pOcz//NUn/PXZq/K3HZ35fvP9+yilap5SSm55oITR3ZXSP2uhdfDg8MaLq/0vlz/pXz1yhh/V55VdPZgy3ocpn2enpfL9lspJS6nszzyJK2FYsKzSMHbsWG9oaEg6jOJQXQ1Ll+45vKoKliwpdDTFpawsOCS0ZQYtLQB4i7Nk5grq7lxKfZ1Tv3gUC3bsD0Afmjhx73nUHPE2484YwPgbzqRsxfI9l5fPfa3vN37Tp8OUKbBsGYweDVOnwsSJSUdVEGY2293HdnY5PeIIRhKwbFluw7uT0aPTHmy3DxnBLefWUf9UOXXLD2BcSx0/Ygrns4w1ZcN45siLGfrlczjmnIOo6HfMOzMO/zFMmgRNTe8Mq6wMDjhxSHcg0/cbv4kTu02CyJs4iimFeqlKKkVXqpONu1pj2jRv6bN79cNm+vi5THNwH1m20n816GrfXt4rehVFvqpeMlWVDB7cdb5fSRxqw+jmSqlONtvBNqbt2Ny42R/72bN+7ftr/dS9Z/uF3OKLqfJmzFewr/9+2JV+28X1vvDRJUEPpmJJuJniGDy4dL5fKXpKGJKXM/O83Do824Gvgwfuda+t9weufMavOL7WT+z3gvdkW9AOTbMf2Xuef/k9M/yerz/pK597I/0CiqWbZbY4SqVBWYpeXAlDjd4SmD49fT39TTd1rt43U+NteXnQAJ3p95fSQA2w6vk3qb9jEXWP7qD+1eG8uHUMThk92c5x/edRc+g6aib05eSLDmJg1YCOx1XoRuViiUO6tLgavZUwJJCvA1emHkvt2LHPvkz72F3Uz4T6JfuxcEc1AH3ZxImD5jPuqI3UnDGQ4887iMohlbnHla8EWapxSJcWV8JIvJopl5eqpPIoX1U0UR++lKGBepCt9TOHP+3Xn17rz9z6km/fvD2WzXX3wlzYFmX5qnqSPENVUhKrfJUw0p1Bp+GAY6xkJH8ffDE9PnwqNZ8Zybs/dgBlPco6vv6k5KPk0I2vI5DOUQlD4pXHXldbfneLNw0c7i2Y76AsbaliY79hvrh+efL3YIpL3L2wSqlXnBQdYiphJHrqZmYDzexeM5tnZq+Y2YlJxtOtTZwYnP1WVQUNzlVVHT4bXrdoPfdPeYZvHTeD9/Z7if6Xnk/lhlX0YAdX9byO7dZr9xkqK+l348+oPmUUVmbpFzp9elAKKisL/k6fnnNcBRX3hXdTpuxZSmtqCoaLFEiiVVJmdgdQ7+43m1kFUOnuGzJNryqp4vR6wyrq/7iYusd2Ur9wBC9tGwNABds4fq951By2npoJfTnpwoMYMHpA7lUrpdgwHHcVX4TbnYhkUvK9pMxsAPA8wTM1IgXRrRJGkdZXe4uz8NGl1E1fTv1Mo27paBbvHA1APzZy0uD5jDt6EzUf35vjzz+Y3gN7d36lpdj1NO4kV4r7QIpGybdhAEcBs4DbgeeAm4G+aaabBDQADaNHj46pRq/IFVF99c5tO/25P8/zX509w/9j3yd9WNmbu0IaYo3+iRFP+c/PrPWGO+f6ji078hNEsVxkl6s8P5NDbRgSFaV+pTcwFtgJnBB+/iXwg2zzJNLoHfc/fZRlJXjbiq1vbfWZv5vjP/5IrZ82dJYPYMOu1Y8uX+4Tq2f67yc+7nMfWBitgbq924IU+f4oKup+Kx3UFRLGcGBJyuca4J/Z5tktYRTinyfOs7pclpXpjBpi386Nqzb6wz9u8Ctrav19A57z3jTtWtW7Kxb6pEMe9zsvmelLZi7PfeHZtjmX/aGza5FOKfmEEWwD9cDB4ftrgJ9mm35XwijUASTOM9tclpXtYrdObmfjvDX+t+887ZcfW+vH9X3Jy9nh4F7GTj+28mX/+jG1/tcrnvLVcxvbX1h7STvbNue6b3V2LdJhcSWMpHtJHUXQdlEBvAZc5O7rM02/q9G7UA2AcfZMyWVZ7V3slm072zSWr73om/xr9dHU1TZT/9pI5m47EIBebOWEAfOoOXwDNR/tx4kXHMReo/aKvj1RGnWzbTOo149IgZR8o3dHXrtKGIVqBM126+lcz3Y7ckadqZSRYTtbLpnsLW0eJ7qJSj+Xad6ft3zCkFn+ow/Xev1v5/jWt7Z2bJ/ksj1xljBEpMPoClVSub52JYxCHWzSVX1VVLj37Ln7sCjVRB2pRmtnO3du2+mzp831X5w1w//Z55PekiHBbBsy0ndu2xnbbnH3aEk7rjaMJKgKTLqQ7p0w8nGwyXSAaDu8M09Cy/UgNG3aHsmpuayH/+3wKT5hyCzvz1sO7ucyzZvJ0lCej+6nUZN2HL2kCq3Yk5lIjrp3wnBPro97AR940/Trm725fPeEsZUKP5dp/q3y631Nj2HegnlLWXnmZJGvap6ufFCNqwRbrAlRuh0ljDjF0YMphkdqrp7b6Pd96yn/2tEz/Jg+c30Jo9Ouq3ngoD3Xla10ka8DVVc9IMbRRtaVE6qUnLgShm5vDp3vwVRZCX36wNq1ey4jS4+mpU+soH7aUrb+43/56KpbGcEqljGaq7maZQOP4rENx2J08vuZPBl+97vOLaO7iaMXnm7lIUUkrl5SJfiggTwYPTr68Ex3dV23Lv0ywruTeosz9/6F/H5iHeft/wSje7xO9SmjeOjGJXx21c/Yl5WU4VSzlNv7fJna38zFqjLElQsli9xNnRqcBKSqrAyGRxX33WpFikEcxZRCvfJWJRVH9UGGqqq3+gzzs0Y85UOscdfg4WVv+KdGPem/OnuGbxsyMnN1WKa4MjW8q849Pp3dF+o2LEUEtWHErJMHiG2/v9V39uy928GhBXw1g/0yu8EvPLDOb7mwzhc8vHj3ezC1V1+eLq50iaTtK9eEpzr3eGl/ShFRwkjYhqUb/KFrZ/l33lvrJ/ef4xVs9XOZ5qsZnP56iMGD0x8sOnom2jaRTJ6sM+JioxKbFIm4EoYavSN686VG6m9bSP2j26lbMIwXtoyhhXJ6sIOx/eZR8+611HykktNuPZvylSvSLyTd8xDSNaL37Al77RW0ixTqWRh6QI9Il1XyD1DqiEIlDG9xlsxcQd2dS6mvc+oXj2LBjv0B6EMTJ+49j5oj3mbcGQM44fyD6LtP33dmznTgbZWul0zq/Z8GDYKNG2H79nfGF+LpcurVI9JlKWHEqGVnC3MfWET9Pauof6qcuuUH8HrLCAAG2gZq9plPzbFbqPnEEI455yAq+lVkXlimA2+r9s7Ykzpwl+JjUEUkkrgSRo84gik1O5p28NzdC6i7r5H62X2Y+eYY1vkYYAwjy1ZRs+9ixp20gJpPjeCwMw+krMcJ0Rc+dWr2O81m6sLbKqnumK1JoQgfCysixaFbJIymNU0888f51D/4FnXP78VT6w+micMAOLDnEs468GVqxhk1n92PA8aPxspGdHxlrQfYyy7b80K+KH35R49OX8JoL9HEYeJEJQgRyahLVkmtX7yBJ25bQP2/mqifO5iGTQezgwqMFo7o/So1Y95g3AcrOOVzBzDiqGH5C7jNsykinbGrakhEYqY2jBSrnn+T+jsWUffoDupfHc6LW8fglNGT7RzXfx41h66jZkJfTr7oIAZWDUgg8hx1JNGIiGTQbRPGv2f9m9dmLKNu2jLqZ0L9kv1YuKMagL5s4sRB8xl31EZqzhjI8ecdROWQyuwLFRHp4rplwhhUcYj3bp7BqpbhwWdbR82wV6kZu4WaTw7l6M8cRM/KnglHmTCVTkSkjW7ZS2rTjl5MqFoU9GD6zEje/bEDcuvB1NW1bf9YujT4DEoaItJpJVXCSPJK75Kgi+9EJI0uc3tzMys3s+fM7MGkYyl5uqW2iORR4gkDuAx4Jekg8mL69OCsv6ws+Dt9en7Xl8tzPUREcpRowjCzUcDHgJuTjCMvWtsTli4N7i3V2p6Qz6QRx4N/REQySLqEcQNwBZDx5kpmNsnMGsysobGxsWCBddqUKXveHqSpKRieL5meBqgGbxGJQWKN3mZ2OnCau19qZuOBb7r76dnmKalGb90uXESKRFdo9D4Z+LiZLQH+DHzAzKYlGE+81J4gIl1MYgnD3b/r7qPcvRo4B3jM3c/LeUGFbliOSu0JItLFJN2G0TnZGpaTTiRqTxCRLqa0L9zLdKHa4MGwZYvu+CoiQtdow+i8TBekrV1b+B5KIiJdXGknjFwbkHO94jnpai0RkSJS2gkjU8Py4MHpp88lwSRx4Z2ISBEr7YSRqWH5l7/sfA+lJC68ExEpYiV1e/O0sj2HujPPhdCN/EREdlPaJYxsJk4Mbund0hL8zbV3VKbqq7IytWmISLfUdRNGZ6VrHwFoblabhoh0S0oYmbRtHykv33MatWmISDeihJFNarVWphsGqk1DRLoJJYyodDNBEenmlDCi0s0ERaSbU8KISjcTFJFurvSvwyikbNd8iIh0cSphiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESSWMIws/3MrNbM5prZy2Z2WVKxiIhI+5K8cG8n8A13f9bM+gOzzewRd5+bYEwiIpJBYiUMd1/l7s+G7zcCrwD7JhWPiIhkVxRtGGZWDRwNPJNm3CQzazCzhsbGxoLHJiIigcQThpn1A+4Dvubub7cd7+43uftYdx87dOjQwgcoIiJAwgnDzHoSJIvp7v7XJGMREZHskuwlZcAtwCvu/vOk4hARkWiSLGGcDJwPfMDMng9fpyUYj4iIZJFYt1p3nwlYUusXEZHcJN7oLSIipUEJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCSSRBOGmU0ws/lmttDMvpNkLCIikl1iCcPMyoHfAh8FDgXONbNDk4pHRESyS7KEcTyw0N1fc/ftwJ+BMxOMR0REsuiR4Lr3BZanfF4BnNB2IjObBEwKP24zs5cKEFtnDQHWJB1EBIozPqUQIyjOuJVKnAfHsZAkE0Yk7n4TcBOAmTW4+9iEQ2qX4oxXKcRZCjGC4oxbKcUZx3KSrJJ6Hdgv5fOocJiIiBShJBPGv4ExZra/mVUA5wD3JxiPiIhkkViVlLvvNLMvA/8CyoFb3f3ldma7Kf+RxUJxxqsU4iyFGEFxxq1bxWnuHsdyRESki9OV3iIiEokShoiIRFI0CaO924SYWS8zuzsc/4yZVaeM+244fL6ZfSTBGC83s7lm9oKZPWpmVSnjms3s+fCV18b9CHFeaGaNKfF8IWXcBWb2avi6IOE4f5ES4wIz25AyriD708xuNbPVma7/scCvwm14wcyOSRlXyH3ZXpwTw/heNLMnzezIlHFLwuHPx9X9shNxjjezt1K+26tSxhXsVkIR4vxWSowvhb/HQeG4guxPM9vPzGrDY87LZnZZmmni/X26e+IvgkbvRcABQAUwBzi0zTSXAjeG788B7g7fHxpO3wvYP1xOeUIxvh+oDN9Pbo0x/LypiPblhcBv0sw7CHgt/Lt3+H7vpOJsM/1XCDpGFHp/jgOOAV7KMP404H8AA94LPFPofRkxzpNa109wO55nUsYtAYYUyf4cDzzY2d9LvuNsM+0ZwGOF3p/ACOCY8H1/YEGa//VYf5/FUsKIcpuQM4E7wvf3AqeamYXD/+zu29x9MbAwXF7BY3T3WndvCj8+TXBtSaF15pYrHwEecfd17r4eeASYUCRxngvcladYMnL3OmBdlknOBP7ogaeBgWY2gsLuy3bjdPcnwzggud9mlP2ZSUFvJZRjnEn9Nle5+7Ph+43AKwR30EgV6++zWBJGutuEtN3wXdO4+07gLWBwxHkLFWOqiwkye6veZtZgZk+b2Vl5iK9V1DjPDouo95pZ6wWUhdqXOa0rrNrbH3gsZXCh9md7Mm1HIfdlrtr+Nh142MxmW3ArnqSdaGZzzOx/zOywcFhR7k8zqyQ40N6XMrjg+9OCKvqjgWfajIr191n0twYpRWZ2HjAWeF/K4Cp3f93MDgAeM7MX3X1RMhHyAHCXu28zsy8RlNw+kFAsUZwD3OvuzSnDiml/lgwzez9BwjglZfAp4b7cB3jEzOaFZ9hJeJbgu91kZqcBfwfGJBRLFGcAT7h7ammkoPvTzPoRJKyvufvb+VoPFE8JI8ptQnZNY2Y9gAHA2ojzFipGzOyDwBTg4+6+rXW4u78e/n0NmEFwNpAP7cbp7mtTYrsZODbqvIWMM8U5tCnyF3B/tifTdhTdrW/M7AiC7/tMd1/bOjxlX64G/kZ+qnQjcfe33X1T+P4hoKeZDaEI92co228z7/vTzHoSJIvp7v7XNJPE+/vMd8NMxMabHgSNLvvzToPWYW2m+U92b/S+J3x/GLs3er9Gfhq9o8R4NEHD3Jg2w/cGeoXvhwCvkqcGu4hxjkh5/wngaX+nIWxxGO/e4ftBScUZTncIQSOiJbE/w3VUk7mR9mPs3qg4q9D7MmKcowna905qM7wv0D/l/ZPAhATjHN76XRMcaJeF+zbS76VQcYbjBxC0c/RNYn+G++WPwA1Zpon195m3nd2BjT+NoJV/ETAlHPZ9gjN1gN7AX8If/SzggJR5p4TzzQc+mmCM/we8CTwfvu4Ph58EvBj+yF8ELk54X/4YeDmMpxY4JGXez4f7eCFwUZJxhp+vAa5rM1/B9ifB2eMqYAdBPe/FwCXAJeF4I3gQ2KIwlrEJ7cv24rwZWJ/y22wIhx8Q7sc54W9iSsJxfjnlt/k0KQku3e8lqTjDaS4k6HCTOl/B9idBtaIDL6R8r6fl8/epW4OIiEgkxdKGISIiRU4JQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBEOiF8HsGHwvc/NLNfJx2TSL7o5oMinXM18P3wRnNHAx9POB6RvNGV3iKdZGaPA/2A8R48l0CkS1KVlEgnmNl7CJ58tl3JQro6JQyRDgqfXDad4Klmm8wsb0/UEykGShgiHRA+ae2vwDfc/RXgBwTtGSJdltowREQkEpUwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCL5/7039OU3Ysw9AAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "\n", "# Importing various packages\n", @@ -2888,22 +2554,27 @@ { "cell_type": "markdown", "id": "b48c2f03", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And a corresponding example using **scikit-learn**" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 18, "id": "54641cf5", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[[3.55513228]\n", + " [3.3552839 ]]\n", + "[3.5182936] [3.30404461]\n" + ] + } + ], "source": [ "# Importing various packages\n", "from random import random, seed\n", @@ -2926,9 +2597,7 @@ { "cell_type": "markdown", "id": "1ab2de11", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -2938,9 +2607,7 @@ { "cell_type": "markdown", "id": "ebf9e606", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -2950,9 +2617,7 @@ { "cell_type": "markdown", "id": "27d8b0b2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] @@ -2960,9 +2625,7 @@ { "cell_type": "markdown", "id": "afa59358", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2974,9 +2637,7 @@ { "cell_type": "markdown", "id": "c8b5852f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] @@ -2984,9 +2645,7 @@ { "cell_type": "markdown", "id": "8e13c9e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -2996,9 +2655,7 @@ { "cell_type": "markdown", "id": "4e94f377", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -3007,9 +2664,7 @@ { "cell_type": "markdown", "id": "2b2beaeb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -3022,9 +2677,7 @@ { "cell_type": "markdown", "id": "36f7c79f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -3036,22 +2689,41 @@ { "cell_type": "markdown", "id": "fb66e23f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Program example for gradient descent with Ridge Regression" ] }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 19, "id": "aeaf35cf", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Eigenvalues of Hessian Matrix:[0.29052231 4.94613286]\n", + "[[4.05416408]\n", + " [2.8806349 ]]\n", + "[[4.05191339]\n", + " [2.88235092]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAo00lEQVR4nO3de5xdZX3v8c9vkhAYwuEyiYhAZsALFigKjDdqLd4KRiyeXokDBxRNCdrikXojrVIkrbYe0WNPixEpSMZYi4rowQpVLK0a7KCCXESFTEKUSwighFguya9/rGcna3b25dl7r9ve832/Xus1e6/rb6+9Z/3Wep5nPcvcHRERkXaGyg5ARET6gxKGiIhEUcIQEZEoShgiIhJFCUNERKIoYYiISBQljFnOzKbN7FXh9XlmdklJcRxvZhvL2PZskf6uu1h2uZndb2ZbzGwk69gabO9iM/uLFtPdzJ6VdxwykxJGhZnZKWZ2o5k9ZmYPhNdnm5nlsT13/yt3f3Ov6zGzsfAPPTeLuMpmZpeZ2YVlx1EWM5sHfAT4bXdf4O6bM1jntJn9KiSg+8I+XlCb7u5nufsHet2OZEsJo6LM7FzgY8DfAk8H9gfOAn4D2K3JMnMKC1Bmk/2B3YHbOl3QEs2OM69z9wXA84Gjgfd2HaEUQgmjgsxsb+AC4Gx3v9LdH/XE9919wt0fD/NdZmb/YGbXmNljwMvN7LVm9n0z+6WZ3WNm59et+zQzW29mm81sRd20881sder9i83s22b2iJndbGbHp6Z908w+YGbfMrNHzexaM1sYJt8Q/j4SziBf0uAz7hHif9jMbgdeUDf9GWb2eTPbZGbrzOxPU9NeaGZT4TPeb2YfSU17aSrme8zsjDB+vpl92Mw2hGUuNrM9wrTjzWyjmZ0bruTuNbM3hmnLgAngXeGzfLnJd/ZcM7vOzB4yszvN7A/D+GeGccekPtem2r40szea2R1hH95tZn+cWmctrnel4nq9mS0xsx+H9Z5X9/1daWb/FNb3PTN7XpN4h8zsPWZ2V/gtfM7M9msw33OAO1Pf5zfC+OPM7D/N7Bfh73GpZb5pZivN7FvAVuDQRjHUuPt9wNdIEkdtHTOu6szsneHz/9zM3lQX44iZfTn8Hv7TzC40s/9o991IF9xdQ8UG4ETgKWBum/kuA35BctUxRHIWeDzw6+H9UcD9wOvD/IcDW4CXAfNJihmeAl4Vpp8PrA6vDwQ2A0vCul4d3i8K078J3AU8B9gjvP9gmDYGeKv4gQ8C/w7sBxwM3ApsDNOGgJuA95FcTR0K3A2cEKZ/BzgtvF4AvDi8HgUeBZYC84AR4Plh2kXA1WF7ewFfBv46TDs+7IcLwnJLSA50+6b284UtPsuewD3AG4G5JGfLDwKHh+lvAW4HhkkOjB9OLfta4JmAAb8VtntMXVzvC3G9BdgEfCZ8hiOAXwGHpL6/J4HfD/P/GbAOmBemT6e+63OAtcBBJL+FTwBrmny+Gd9n2IcPA6eFz7s0vB9J/TY2hPjm1rZft850LAcBPwQ+VvfbvjD1/3A/cGTY158J8TwrTP9sGIZJfuP3AP8R891o6PDYVHYAGhp8KXAqcF/duG8Dj4QDxMvCuMuAT7dZ10eBi8Lr9wGfTU3bE3iCxgnj3cAVdev6GnB6eP1N4M9T084G/iW8nnGAaRLX3cCJqffL2JkwXgRsqJv/vcA/htc3AH8JLGwwzxcbbMuAx4Bnpsa9BFgXXh8f9uvc1PQH2JmIdhy8mnyWPwL+vW7cJ4D3p95fHQ6KtwDzW6zrKuCcurjmhPd7hf36otT8N7HzhOB8YG1q2hBwL/Cb4f106ru+A3hlat4DSJLNLt9Z/fdJkii+WzfPd4AzUr+NC9r8LqdJTl4eDev+OrBPavqOfQ5cSjgZCe+fE5Z5FjAnxH1YavqF7EwYbb8bDfGDiqSqaTOw0FKVxu5+nLvvE6alv7d70gua2YvM7PpQ7PELknqPWlHRM9Lzu/tjYX2NjAJ/EIp2HjGzR4CXkhxYau5Lvd5KcrYfa0YswPq6bT+jbtvnkZSlA5xJctD4USiCOCmMP5jkqqfeIpKzz5tS6/uXML5ms7s/1eXnGQVeVBfvBEndU80nSc6QP+6hSBHAzF5jZmtDcckjJFc3C1PLbXb3beH1r8Lf+1PTf1UXZ/r73Q5sJNnXjWL+YireO4Bt7NzHrTyDmd8X4f2BjeJo4fXuvhdJYnwuMz93/faa/VYWkVw5pKenX8d8NxJJCaOavgM8DpwcMW99d8OfITmbPdjd9wYuJjnDhuRs8+DajGY2TFJs08g9JFcY+6SGPd39g13E1MiMWIDFddteV7ftvdx9CYC7/8TdlwJPAz4EXGlmtaKHZzbY1oMkB9YjUuvb25MK1xjtPs89wL/VxbvA3ZcDWNL656PAp4Dza3UFZjYf+DzwYWD/cEJwDTu/r26kv98hkuKenzeJ+TV1Me/u7j+L2MbPSQ7EaYuB9LLR3WC7+7+RXFF8uMksrX4rm0iK7Q5KjUvP2/K7kc4oYVSQuz9CUuTy92b2+2a2V6ikfD5JMVIrewEPuft/mdkLgTekpl0JnBQqhncjKbNv9htYDbzOzE4wszlmtnuohD2oyfxpm4DttK7s/BzwXjPbN6zzT1LTvgs8ambvtqRyfI6ZHWlmLwAws1PNbFE4g34kLLMdmAReZWZ/aGZzQ2Xo88N8nwQuMrOnhXUcaGYnRHwWSM7oW32WrwDPsaRBwbwwvMDMfi1M/xgw5UmT5f9PksQhqZ+ZTzjomdlrgN+OjKmZY83sd8PV6dtJTjzWNpjvYmClmY0CmNkiM4s5QYEkqT3HzN4Q9vMfkdQdfKWHuD8KvLpJJf3ngDPM7PBwkvP+2oRw9fUFkkQ8bGbPBf5Xatl23410QAmjotz9b4B3AO8iOWDdT1L2+m6S+oxmzgYuMLNHSeosPpda523AW0muQu4lqahseLOcu99DcoVzHskB7R7gnUT8Ztx9K7AS+FYoBnhxg9n+kqRoYR1wLXBFavltwEkkrWbWkVwhXALsHWY5EbjNzLaQHIxPcfdfufsGkiKdc4GHgB8AtQPQu4GfAmvN7JfAvwKHtfsswaeAw8NnuarB532U5EB/CsnZ930kVz7zw0H4RKB2RvsO4BgzmwjL/SnJd/QwSXK/OjKmZr5EUm5fq5T+XXd/ssF8Hwvbujb8VtaS1B215cl9GCeR7OfNJL/Rk9z9wW6DdvdNwKdJfrP1075KklC+QfIdfqNulreR/DbuI/kdrSFJlC2/m25jnc3MXQ9QEhkEljShfpa7n1p2LGUysw8BT3f308uOZdDoCkNE+lq4z+IoS7yQpFHEF8uOaxANRNcNIjKr7UVSDPUMkqLb/0NSNCcZU5GUiIhEUZGUiIhE6asiqYULF/rY2FjZYYiI9JWbbrrpQXdf1H7O1voqYYyNjTE1NVV2GCIifcXM6u/M74qKpEREJIoShoiIRFHCEBGRKEoYIiISRQlDRESiKGGIiEgUJQwREYmihCEiIlGUMEREJIoShoiIRMk9YZjZpWb2gJnd2mDauWbmZtbs4e8iIlIRRVxhXEbyiMoZzOxgkkcnbiggBhER6VHuCcPdbyB5vnK9i0ieBawHcoiI9IFS6jDM7GTgZ+5+c8S8y8xsysymNm3aVEB0IiLSSOEJw8yGgfOA98XM7+6r3H3c3ccXLeq5O3cREelSGVcYzwQOAW42s2ngIOB7Zvb0EmIREZFIhT9Ayd1/CDyt9j4kjXF3f7DoWEREJF4RzWrXAN8BDjOzjWZ2Zt7bFBGR7OV+heHuS9tMH8s7BhER6Z3u9BYRkShKGCIiEkUJQ0REoihhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REoihhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REoihhiIhIlNwThpldamYPmNmtqXF/a2Y/MrNbzOyLZrZP3nGIiEhvirjCuAw4sW7cdcCR7n4U8GPgvQXEISIiPcg9Ybj7DcBDdeOudfenwtu1wEF5xyEiIr2pQh3Gm4CvNptoZsvMbMrMpjZt2lRgWCIiFTE5CWNjMDSU/J2cLCWMUhOGma0AngKafnp3X+Xu4+4+vmjRouKCExGpgslJWLYM1q8H9+TvsmWlJI3SEoaZnQGcBEy4u5cVh4hIpa1YAVu3zhy3dWsyvmBzC98iYGYnAu8Cfsvdt7abX0Rk1tqwobPxOSqiWe0a4DvAYWa20czOBP4O2Au4zsx+YGYX5x2HiEhfWry4s/E5yv0Kw92XNhj9qby3KyIyEFauTOos0sVSw8PJ+IJVoZWUiIg0MzEBq1bB6CiYJX9XrUrGF6yUOgwREenAxEQpCaKerjBERCSKEoaIiERRwhARqaKK3N2dpjoMEZGqqd3dXWsZVbu7G0qty9AVhohI1VTo7u40JQwRkappdhf3+vWlFlEpYYiIVE2ru7hL7IBQCUNEpGpWrkzu5m6lhCIqJQwRkaqpv7u7mYI7IFTCEBHpVR5NYCcmYHoatm9PEkcjBXdAqIQhItKLIh5w1KiIqoQOCJUwRER6UUQT2Ip0QKiEISLSi6IecFQrorriiuT9aacV3rxWCUNE+k99ncHZZ+98v3BhMmR9v0KzeooiH3BU8vO9rZ8epz0+Pu5TU1NlhyEiZarvNqOd4eHei28abbO2Xmg+Lesio7GxJEnUGx1Nrj6aMLOb3H28180rYYhIf2l20GylzQG1623W1js5mdRZbNiQXFmsXJlP/cLQUHJlUc8saU3VhBKGiMxOzQ6arbQ5oHa9zV7X26mSrzBUhyEi/aWbuoFe6xOKrKdopVnz2iVLCukKPfeEYWaXmtkDZnZratx+Znadmf0k/N037zhEZEDEdJuRlsX9ChW5D6Jh89rTT4fLLy+kIryIK4zLgBPrxr0H+Lq7Pxv4engvIrNNN3dINzpoLl++8/3ISDJkeb9CRe6D2BFL7Q7w6Wm45priukJ399wHYAy4NfX+TuCA8PoA4M6Y9Rx77LEuIgNi9Wr34WH35Lw4GYaHk/GDYPVq99FRd7Pkb1afq3696f2XHsx2LAJMeQbH8rLqMPZ393vD6/uA/ZvNaGbLzGzKzKY2bdpUTHQikr8yHxKU9+NP87pfotF6m3VOmEP9SiGtpMxsDPiKux8Z3j/i7vukpj/s7m3rMdRKSmSAlNXyqNU9FVkVMXXZmqnr9ZrN3Jfh8zxx0u/yq4f/i30O2a+vW0ndb2YHAIS/D5QUh4iUpayWRzFXNr1egeTVXUiz5d1hzhwc2G5DXLr1D7BTJ5i/zx7sc0h2bYrKShhXA6eH16cDXyopDhEpS1ktj9odzLMoTsorGTZZfjsG27ZhwJBv54/4Z5aSxHvhCf/e2zbTsqgIaTUAa4B7gSeBjcCZwAhJ66ifAP8K7BezLlV6iwyYvCqGW2lWUTw6Gjc9Rk4V+ned9j7fwsz1bsPaxktGld65J4wsByUMEelZu4O5NTkAp1odRW+nx2R4w/lf3yWMpaz2dYz6Niz8bRDrgLWSEhEpR7t7KtoVJ8XWb9TfLxFRoX7tuV/DjB3Dy85/xS7zrGGCQ5hmyLcz5tMMFfg0PiUMEclO3s1Vs9LqYN6qbiXj5rJfePM1MxLECR85oem89ZcQUfFmLYvLlKIGFUmJVNgg3YjXrDipx/qNT//eVU3vs6svduqoOKtN8RcZFUmpt1oRyUZe9x5USYf3jlx68pc48+qTo1btq0MX6bWb8dLb6fE+EfVWKyLVUtSjSsvUpn7jE0u+NKOIqVWymHFNsTpV1FWbmFbUHfBtKGGISDaq0gV4nhrUFzzGMG9YvxIzOOurkQmi/iKl0c2E9SqQeJUwRCQbVekCPEd//J59eMPWVUwzynaMaUZ5C6tYw65FRS0TRL2YZNBpK60cKGGISDbqm6uOjMAee8Bpp1W7xVQLbzrgqzOKmFZtfO2OZq1z2M4hTO9IFlEJotnBvt1VWE6ttDqlSm8RyV4RHfzl4OS9r+fqX748at6OD52t9gnsOq1W8T06uvMZ4SU/olUJQ0Sy1yctpk7Y8z+4dutLo+bt+VDZbp9MhlZSGzYkVxy1JJHWSSut1Pqe5/7Eze7ze/wEKpISkRxk2WIqwzL7F8z7/owiplbJoqM6iBjt9knMneGxDQvqiq7mwW7dhp2mhCEi2cuqxVSPZfZHzrl9RoKYeuropvNmniDqZbFPYhsWxLS66oIShsggqUrXHFm1mOrwqXzp5GAGt20/vOmqc08Q9bLYJ7HPFs+pCa4ShsigKLkFzQyxB7Z22hTj1CeIVnJLEJ10RpjFPqkvuoJdt5/TvS9KGCKDosxnZDfSRW+tu2hy4Jv2xfkliE6u0spO0s22v2TJrlczWciiQ6qiBnU+KNJAreO5iOciFBJHp89/aLHcUlbv8sCgLQz7Ulbv8jEz+wyddKDYSWeEeXTO2Gr7qf16FDzuGRyDS08CnQxKGCJ1Gh2EuuxJNfM4Yg6GDZarTwj1DwyqTctFswPwyEjj+Tt52FIWT/LrcvvoiXsi0vLKIosz2F7jaHMwXEfj5dYx2vsVRDdXPM0OwNB4+U4+d1ZP8uti+1klDNVhiPSzVq1huq1UzTKOuvH1ldSLabzcYjbschTsSLd1C60qixvVBXXS8inLpsa1OpYtW2DevLjtZyGLrFPUoCsMkTp5FHNkGEejK4X00OwKo+f4u90vq1c3D7bZlUDslUwWdRiN1rHbbkmRWYvtMwhFUsD/Bm4DbgXWALu3ml8JQ6ROVZ5yF1EX0bCIqdP4Yw/OvRT/jIzkl4S7bRhQ02Ui7PuEARwIrAP2CO8/B5zRahklDJn1Gh1wej0I9SB9zGpWOd22DiKPM/RerryqkoQb6TIRDkrCuAfYD5gLfAX47VbLKGHIrFaBA1mr4qWGVxBZ6rUJq5n78uVx2yoxCbc0W68wks/AOcAWYBMw2WSeZcAUMLV48eJOd6/I4CihvqLUBFGvVQumRgf15ct3XaYqVwrd6vKkIauEUVorKTPbFzgZOAR4BrCnmZ1aP5+7r3L3cXcfX7RoUdFhilRHAc/M7rirjdWT+OgYbgX0XdWqNVGjVlDXXJMEmVaRZ2N3LavuRbpUZrPaVwHr3H2Tuz8JfAE4rsR4RMoR2xVFDs/M7qkvpqK7xWjUhDVt61Y455yd7wtIsKXIosuVLpWZMDYALzazYTMz4JXAHSXGU4yq9CYqncnre+vkoJtBb6eZdtZXdN9V6bPrZjZvTvbd5GTyXTWSU8d8s0IW5VrdDsBfAj8iaVZ7BTC/1fx9X+ldgUrLTFW1YjBreX5vndZLdLjPc62DyOPO5Vit7nAfGWneXUo//7/1gEGo9O506PuEUZWbrLIwaMmvlVYHp14TZcYH3UIrqSM7vsvlZKLVDXbNhjlzBvP3GaGwhAFcBzwvi431OvR9wijzjCxrg5T82mnVOqfXRNnjfqzN3u4eiEwSRL1mJw3LlxdzMtHsBrtmQz/+n2WkyIRxDHA98I/AAVlstNuh7xPGIB1kq5r8OjmzjZ23XQd/tWKQbrv27uDg2mjTzboAb7iOrM/8G62vqN95s323YEHz72iWKrxICvg94Gbg/YS7s4se+j5hDFIxThWTXyf7t9d52w2dfK8tDuIxm4ruj6mo31+RJxON9l2zKw8ljGISBmDAkcBZwIPARuC0LALoZOj7hOE+OBXFvRx88toHnSSxbiubO0kaXSTPTla/Q+wBuqgkX/bJRFWvfktUZJHUt4Cfh7qMDwAnAc8CPg6syiKI2GEgEsYg6ebAn+dZbicHim4PKp1cbUQcoLpKEPViD9BFHUib7aORkUo/m2OQFZkwjgCsybQ7sggidlDCGAB5/jPneYWRVp8oO+jdtF1SSFdeZ56EizyQNisaKqIIdpCKfjNSiWa1wKFZBBE7KGEMgDzPcvOqw+hhux1dQeRdzFf0gbTMM/1BKfrNSCUSRtGDEsYAyPsgkq5rmDNn57rzbjEU1tWqWWvbIqYiDrCdfuZe9pHqEipDCUP6U15nuekD28iI+7x52W+jgY6uINqp2gG21+9KdQmVkVXC0DO9pVh59LZZ3x/T5s3w5JMz58moj6NM+2Kql0Pngj3pta+oDPq+kmpRwpDiZd3bZqMDWyNd9FLac2+unXRYWLUDbK+9vZbcFbdkb27ZAYj0LPYAFnGm3i4ppLW8aqhd9dQSWa0XWmh+wKyNX7Ei+UyLFyfJoqwD7OLFSdyNxseamFCCGCC6wpD+F3sAa3CmnlsRU7fFOSU+62AXVbvikdIpYfSTfn2WRt5xt3uwDsDICExM5FsHkdauOKcfvksVKUm9LGrOixpmdSupfr0Zqai4081p61obbWG4u2auvWjX9Xc/fpf9QPdfNISa1c4yZTVR7PUfsOC4Ia6r78wTRL1WSUHNTfOhRNxUVgnDknX1h/HxcZ+amio7jHIMDTUuFzFLyrvzUF9xC0nRTyfFEjnHnVkldR4mJxtXYJfxXc4GY2ONK+lHR5P6oFnMzG5y9/Fe16M6jH5RRhv9LJ7ZnHHchdVB1OumzqFZBXbV7rcYFL02A5a2lDD6RRktVrL4B+wx7tISRFr9jYG1JrIxSaNRolHro3woEecvi3KtoobM6zD6rYKs2+7Eu/2MWfXoOjKSDBExZNrVRla63Q+tytT77bfXD1SH0RSDUOkN7ANcCfwIuAN4Sav5M00Ys+HH1etn7Hb5DpaLTQ7Gti52QEa67eNJldvFUyJuaFASxuXAm8Pr3YB9Ws2facKYDf/MWXzGbv4BW2w3NkEcwM+yiaVTnTyjes6c1rFUrTNBmbX6PmEAewPraPJwpkZDpgljNvwzl/UZm2x3G9Y0QRzCXa3XWcQVYbNtLF/e/il7ZT+wSKSFrBJGmZXehwCbgH80s++b2SVmtmf9TGa2zMymzGxq06ZN2W19NlSQ7bdfZ+MzYAbT3ngfbmDn+CO4dcZR9G4/tPWKs2ix1U6zbVxzzcw7nufM2XXZRrGoclsGTJkJYy5wDPAP7n408BjwnvqZ3H2Vu4+7+/iiRYuy27r+mTPRqBXTeazkMWbu2/9iPmOrV+5IELf6kZ1tqIgmk622kW4i2+xeifrl1bWGDJgyE8ZGYKO73xjeX0mSQIoxG/6ZH3qos/ERYpq5rmGCj/O2Gft29+VvSs7Au+07qYgrwthtdBJLlToTFOlRaQnD3e8D7jGzw8KoVwK3FxpE1f6Zs+6QLoODbOx9ECfb1TOKmN7jH9q5b1euhMsv7+4+hpoirghjt6GrU5mtsqgI6XYAng9MAbcAVwH7tpp/oPuSyqNSt4t1xrZiOn2Pz8bHkVXlb1mtpPKKRU1ApSD0eyupboaBThh5tahpc1CKTRBn7XVF9zFk2VprUA6ys+E+IKmMrBKGOh+sgslJOPXUxtMy7pAutrO+d+1/OR+67/RsNppVp3BZdIZYFeooTwqkzgcHRe0g2EyPlbqxdRB/ceClM07/M0sW0LjMf9482LKls/qaIprWFkUd5UkfUsIoW6ODYE0XFanRCeKAS2YkiAs2vqmj7XSkvkXayEjyd/PmzirBB+kgOxvuA5KBo4RRtlYHu4iiltgE8TfPWjUzQfz8zcmEoh4Vmm6RtmABPPHEzOkxVwqDdJBVSyvpQ0oYRWl2YG521/XoaMNkEZsgLj72kzMSxDt/0qDYq5duu3vR7ZXCIB1kZ8N9QDJ4sqg5L2ro21ZSrfoo2m23XVsPzZu3o7VMbCumS170ic7jKquvo6y6Te/nVlIiBWIA+pLKV1FFLTGaVdauWrVr0Qyw6cn/gZ060fIK4tO/ObOI6cy1LSrOa+r3SaNWOpB/nUAvVwpVu9lSZBYZzIRRVlFLM80OwNu2NRw9wq5dd/zzCTMrqU+7ISJBpDXaJ80yUt51AiqOEelLg5kw8mh+2csVS5MD8FM06PWUpFfXq06amSB+/1/e3HnMaY32ifuuSaOoOgFdKYj0ncFMGFk3v+zhisUM3rB+195bH2OYi1m2y3iGhxlbvZKTv9xjgqjX7LO760xfRKIMZsLIuvllB1csjVoxrWGCt7CKaUbZjjHNKPefvIy3+d+z5+qCimbaffYrrtCZvoi0NJgJI+vmly2uWGKbua748AhjPs2Qb2fMpzn0qouSCUUVzTTaJzVl1/GISF8YnISRrmNYsQJOPz27M/cmZ+fNniwHcPtFX5tRB3HEuSd2t+2spCuaG+nXLjZEpDCDkTAa1TFcfnlyVt3jmXurOojz2HnFcuffXTcjQfza20/o5RPlo3Y10+xSqEpdbFSpWbSIAIOSMDJsFRVbB7HljX/KZ3xiR4J4zltfncEHKUjVu9ioWrNoEQEGJWH00CqqUYJYyiTrGGMbQ6xjjKVM8qHJxTPqIPa/9K8z/AAFq3oXG72cAOjKRCQ3g5EwOjhjbldJvZRJPskyxljPEM4Y6/nM8DIO9goV1/Sq6jfOdXsCoCsTkVwNxgOUGj1YB2BkhDds/hhraH8g3PT1W1j4iqP0YJsq6PY70Hcn0pAeoJRWO2MeGWFG+tu8mU+yjKXseob5yI13zqikXviKo5IJg/TMhX7VbZGZvjuRXPV1wnCHP3/t95PipVMnmN68gPpSpj3Zyl+xgkdvXT8jQez9wsMar7TqFcKzQbdFZvruRHJVesIwszlm9n0z+0rbmR96CF88ynYbYtrGmBiaZOU1R++YvJjGZ5JjtoEFRzS5/6Be1SuEZ4tubmjUdyeSq9ITBnAOcEfUnOvXY/ds2FEZXStu+oN9r+PJh7cwNJrBGWbVK4SlOX13IrkqtdLbzA4CLgdWAu9w95NazT9u5rtUeacrNBtVfg8P66AhIrPaoFR6fxR4F7C92QxmtszMpsysQfMoZlZo6gxTRCQ3pSUMMzsJeMDdb2o1n7uvcvfxptmxvripUdl3pzdz6eYvEZFdlHmF8RvA75jZNPBZ4BVmtrrlEkN14cZUaHZ6M1dVb/5SEhORklXixj0zOx74s7Z1GIce6lPbtyfFUIsXJ8miXXFTpzdzVfHmL9XNiEgPsqrD6K+E0exO71aGhpIrhV03mhRb9Tp/EaqYxESkbwxKpTcA7v7Ndsmia53ezFXFm790B7OIVEAlEkauOr2Zq4o3f1UxiYnIrDP4CaPTprZVbJpbxSQmIrNOJeowYnVVhzEoJieT50F0UuEvIkJ2dRhzswhGCjAxoQQhIqUa/CIpERHJhBKGiIhEUcIQEZEoShgiIhJFCUNERKIoYYiISBQlDBERiaKEISIiUZQwREQkihKGiIhEUcIQEZEoShgiIhJFCUNERKIoYYiISBQlDBERiaKEISIiUUpLGGZ2sJldb2a3m9ltZnZOWbGIiEh7ZV5hPAWc6+6HAy8G3mpmh0ctOTkJY2MwNJT8nZzML0oREQFKfESru98L3BteP2pmdwAHAre3XHByEpYtg61bk/fr1yfvQY8wFRHJkbl72TFgZmPADcCR7v7LumnLgGUAixcvPna9WZIk6o2OwvR07rGKiPQbM7vJ3cd7XU/pld5mtgD4PPD2+mQB4O6r3H3c3ccXLVoEGzY0XlGz8SIikolSE4aZzSNJFpPu/oWohRYv7my8iIhkosxWUgZ8CrjD3T8SveDKlTA8PHPc8HAyXkREclPmFcZvAKcBrzCzH4RhSdulJiZg1aqkzsIs+btqlSq8RURyVolK71jj4+M+NTVVdhgiIn1lYCq9RUSkPyhhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REoihhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REoihhiIhIFCUMERGJooQhIiJRSk0YZnaimd1pZj81s/eUGYuIiLRWWsIwsznA/wNeAxwOLDWzw8uKR0REWivzCuOFwE/d/W53fwL4LHByifGIiEgLc0vc9oHAPan3G4EX1c9kZsuAZeHt42Z2awGx9Woh8GDZQURQnNnphxhBcWatX+I8LIuVlJkworj7KmAVgJlNuft4ySG1pTiz1Q9x9kOMoDiz1k9xZrGeMoukfgYcnHp/UBgnIiIVVGbC+E/g2WZ2iJntBpwCXF1iPCIi0kJpRVLu/pSZvQ34GjAHuNTdb2uz2Kr8I8uE4sxWP8TZDzGC4szarIrT3D2L9YiIyIDTnd4iIhJFCUNERKJUJmG06ybEzOab2T+F6Tea2Vhq2nvD+DvN7IQSY3yHmd1uZreY2dfNbDQ1bZuZ/SAMuVbuR8R5hpltSsXz5tS0083sJ2E4veQ4L0rF+GMzeyQ1rZD9aWaXmtkDze7/scT/DZ/hFjM7JjWtyH3ZLs6JEN8PzezbZva81LTpMP4HWTW/7CHO483sF6nv9n2paYV1JRQR5ztTMd4afo/7hWmF7E8zO9jMrg/HnNvM7JwG82T7+3T30geSSu+7gEOB3YCbgcPr5jkbuDi8PgX4p/D68DD/fOCQsJ45JcX4cmA4vF5eizG831KhfXkG8HcNlt0PuDv83Te83resOOvm/xOShhFF78+XAccAtzaZvgT4KmDAi4Ebi96XkXEeV9s+SXc8N6amTQMLK7I/jwe+0uvvJe846+Z9HfCNovcncABwTHi9F/DjBv/rmf4+q3KFEdNNyMnA5eH1lcArzczC+M+6++Puvg74aVhf4TG6+/XuvjW8XUtyb0nReuly5QTgOnd/yN0fBq4DTqxInEuBNTnF0pS73wA81GKWk4FPe2ItsI+ZHUCx+7JtnO7+7RAHlPfbjNmfzRTalVCHcZb127zX3b8XXj8K3EHSg0Zapr/PqiSMRt2E1H/wHfO4+1PAL4CRyGWLijHtTJLMXrO7mU2Z2Voze30O8dXExvl74RL1SjOr3UBZ1L7saFuhaO8Q4Bup0UXtz3aafY4i92Wn6n+bDlxrZjdZ0hVP2V5iZjeb2VfN7IgwrpL708yGSQ60n0+NLnx/WlJEfzRwY92kTH+fle8apB+Z2anAOPBbqdGj7v4zMzsU+IaZ/dDd7yonQr4MrHH3x83sj0mu3F5RUiwxTgGudPdtqXFV2p99w8xeTpIwXpoa/dKwL58GXGdmPwpn2GX4Hsl3u8XMlgBXAc8uKZYYrwO+5e7pq5FC96eZLSBJWG9391/mtR2ozhVGTDchO+Yxs7nA3sDmyGWLihEzexWwAvgdd3+8Nt7dfxb+3g18k+RsIA9t43T3zanYLgGOjV22yDhTTqHukr/A/dlOs89Rua5vzOwoku/7ZHffXBuf2pcPAF8knyLdKO7+S3ffEl5fA8wzs4VUcH8GrX6bue9PM5tHkiwm3f0LDWbJ9veZd8VMZOXNXJJKl0PYWaF1RN08b2VmpffnwusjmFnpfTf5VHrHxHg0ScXcs+vG7wvMD68XAj8hpwq7yDgPSL3+n8Ba31kRti7Eu294vV9ZcYb5nktSiWhl7M+wjTGaV9K+lpmVit8tel9GxrmYpH7vuLrxewJ7pV5/GzixxDifXvuuSQ60G8K+jfq9FBVnmL43ST3HnmXsz7BfPg18tMU8mf4+c9vZXXz4JSS1/HcBK8K4C0jO1AF2B/45/Oi/CxyaWnZFWO5O4DUlxvivwP3AD8JwdRh/HPDD8CP/IXBmyfvyr4HbQjzXA89NLfumsI9/CryxzDjD+/OBD9YtV9j+JDl7vBd4kqSc90zgLOCsMN1IHgR2V4hlvKR92S7OS4CHU7/NqTD+0LAfbw6/iRUlx/m21G9zLakE1+j3UlacYZ4zSBrcpJcrbH+SFCs6cEvqe12S5+9TXYOIiEiUqtRhiIhIxSlhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCI9CM8jeHV4faGZfbzsmETyos4HRXrzfuCC0NHc0cDvlByPSG50p7dIj8zs34AFwPGePJdAZCCpSEqkB2b26yRPPntCyUIGnRKGSJfCk8smSZ5qtsXMcnuinkgVKGGIdCE8ae0LwLnufgfwAZL6DJGBpToMERGJoisMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REovw3IJsTD8NS93IAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "from random import random, seed\n", "import numpy as np\n", @@ -3108,9 +2780,7 @@ { "cell_type": "markdown", "id": "41ef6f59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -3130,9 +2800,7 @@ { "cell_type": "markdown", "id": "14c608fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Challenge yourself this weekend\n", "\n", @@ -3140,7 +2808,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.14" + } + }, "nbformat": 4, "nbformat_minor": 5 }