diff --git a/doc/pub/week38/html/._week38-bs001.html b/doc/pub/week38/html/._week38-bs001.html
index 4ec102c37..ba7096458 100644
--- a/doc/pub/week38/html/._week38-bs001.html
+++ b/doc/pub/week38/html/._week38-bs001.html
@@ -306,8 +306,15 @@ MathJax.Hub.Config({
- Lab Wednesday and Thursday: work on project 1
-- Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression Video of lecture Video on Bootstrapping
+- Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression
+
- Friday: Classification problems and Logistic Regression, from binary cases to several categories. Start optimization methods
+
- Reading recommendations:
- See lecture notes for week 37 on cross-validation and week 38 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
diff --git a/doc/pub/week38/html/week38-reveal.html b/doc/pub/week38/html/week38-reveal.html
index c9568789e..f83edb523 100644
--- a/doc/pub/week38/html/week38-reveal.html
+++ b/doc/pub/week38/html/week38-reveal.html
@@ -199,8 +199,20 @@ MathJax.Hub.Config({
- Lab Wednesday and Thursday: work on project 1
-- Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression Video of lecture Video on Bootstrapping
+- Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression
+
+
- Friday: Classification problems and Logistic Regression, from binary cases to several categories. Start optimization methods
+
+
- Reading recommendations:
- See lecture notes for week 37 on cross-validation and week 38 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
diff --git a/doc/pub/week38/html/week38-solarized.html b/doc/pub/week38/html/week38-solarized.html
index 4f2a2b95c..5fc3f0c67 100644
--- a/doc/pub/week38/html/week38-solarized.html
+++ b/doc/pub/week38/html/week38-solarized.html
@@ -269,8 +269,15 @@ MathJax.Hub.Config({
- Lab Wednesday and Thursday: work on project 1
-- Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression Video of lecture Video on Bootstrapping
+- Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression
+
- Friday: Classification problems and Logistic Regression, from binary cases to several categories. Start optimization methods
+
- Reading recommendations:
- See lecture notes for week 37 on cross-validation and week 38 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
diff --git a/doc/pub/week38/html/week38.html b/doc/pub/week38/html/week38.html
index ec6f6a458..542bee19f 100644
--- a/doc/pub/week38/html/week38.html
+++ b/doc/pub/week38/html/week38.html
@@ -346,8 +346,15 @@ MathJax.Hub.Config({
- Lab Wednesday and Thursday: work on project 1
-- Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression Video of lecture Video on Bootstrapping
+- Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression
+
- Friday: Classification problems and Logistic Regression, from binary cases to several categories. Start optimization methods
+
- Reading recommendations:
- See lecture notes for week 37 on cross-validation and week 38 at https://compphysics.github.io/MachineLearning/doc/web/course.html.
diff --git a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz
index 51effdeb0..a1cbd6098 100644
Binary files a/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz and b/doc/pub/week38/ipynb/ipynb-week38-src.tar.gz differ
diff --git a/doc/pub/week38/ipynb/week38.ipynb b/doc/pub/week38/ipynb/week38.ipynb
index 7d0cbbbf0..a2f3e6d54 100644
--- a/doc/pub/week38/ipynb/week38.ipynb
+++ b/doc/pub/week38/ipynb/week38.ipynb
@@ -2,8 +2,10 @@
"cells": [
{
"cell_type": "markdown",
- "id": "e1437fb7",
- "metadata": {},
+ "id": "9dc09e5f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
@@ -12,8 +14,10 @@
},
{
"cell_type": "markdown",
- "id": "7edae964",
- "metadata": {},
+ "id": "611a1193",
+ "metadata": {
+ "editable": true
+ },
"source": [
"# Data Analysis and Machine Learning: Logistic Regression\n",
"**Morten Hjorth-Jensen**, Department of Physics and Center for Computing in Science Education, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
@@ -23,17 +27,25 @@
},
{
"cell_type": "markdown",
- "id": "b080984b",
- "metadata": {},
+ "id": "a8ce5806",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Plans for week 38\n",
"\n",
"* Lab Wednesday and Thursday: work on project 1\n",
"\n",
- "* Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression [Video of lecture](https://youtu.be/sdt_BFla8uA) [Video on Bootstrapping](https://www.youtube.com/watch?v=Xz0x-8-cgaQ)\n",
+ "* Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression\n",
+ "\n",
+ " * [Video of lecture](https://youtu.be/sdt_BFla8uA)\n",
+ "\n",
+ " * [Video on Bootstrapping](https://www.youtube.com/watch?v=Xz0x-8-cgaQ)\n",
"\n",
"* Friday: Classification problems and Logistic Regression, from binary cases to several categories. Start optimization methods\n",
"\n",
+ " * [Video of lecture](https://youtu.be/7OdqoLOphTA)\n",
+ "\n",
"* Reading recommendations:\n",
"\n",
"a. See lecture notes for week 37 on cross-validation and week 38 at \n",
@@ -47,8 +59,10 @@
},
{
"cell_type": "markdown",
- "id": "8766162e",
- "metadata": {},
+ "id": "521805e9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Ridge and LASSO Regression, reminder\n",
"\n",
@@ -58,8 +72,10 @@
},
{
"cell_type": "markdown",
- "id": "a0f26a88",
- "metadata": {},
+ "id": "edb95e43",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in {\\mathbb{R}}^{p}}}\\frac{1}{n}\\left\\{\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)^T\\left(\\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\right)\\right\\}.\n",
@@ -68,16 +84,20 @@
},
{
"cell_type": "markdown",
- "id": "2ab8174d",
- "metadata": {},
+ "id": "b1122f96",
+ "metadata": {
+ "editable": true
+ },
"source": [
"or we can state it as"
]
},
{
"cell_type": "markdown",
- "id": "7310c495",
- "metadata": {},
+ "id": "9fc4e95a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
@@ -87,16 +107,20 @@
},
{
"cell_type": "markdown",
- "id": "cf23c167",
- "metadata": {},
+ "id": "69ab6ac4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have used the definition of a norm-2 vector, that is"
]
},
{
"cell_type": "markdown",
- "id": "99d4c9b7",
- "metadata": {},
+ "id": "793ac2b3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\vert\\vert \\boldsymbol{x}\\vert\\vert_2 = \\sqrt{\\sum_i x_i^2}.\n",
@@ -105,8 +129,10 @@
},
{
"cell_type": "markdown",
- "id": "bd386f6d",
- "metadata": {},
+ "id": "b41ba30e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"By minimizing the above equation with respect to the parameters\n",
"$\\boldsymbol{\\beta}$ we could then obtain an analytical expression for the\n",
@@ -116,8 +142,10 @@
},
{
"cell_type": "markdown",
- "id": "182d5d41",
- "metadata": {},
+ "id": "716392d3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
@@ -127,8 +155,10 @@
},
{
"cell_type": "markdown",
- "id": "0d3682e5",
- "metadata": {},
+ "id": "da4da7d2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which leads to the Ridge regression minimization problem where we\n",
"require that $\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_2^2\\le t$, where $t$ is\n",
@@ -137,8 +167,10 @@
},
{
"cell_type": "markdown",
- "id": "25248610",
- "metadata": {},
+ "id": "97ca4c56",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\boldsymbol{X},\\boldsymbol{\\beta})=\\frac{1}{n}\\vert\\vert \\boldsymbol{y}-\\boldsymbol{X}\\boldsymbol{\\beta}\\vert\\vert_2^2+\\lambda\\vert\\vert \\boldsymbol{\\beta}\\vert\\vert_1,\n",
@@ -147,16 +179,20 @@
},
{
"cell_type": "markdown",
- "id": "da238137",
- "metadata": {},
+ "id": "3633c01c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"we have a new optimization equation"
]
},
{
"cell_type": "markdown",
- "id": "f9bb791c",
- "metadata": {},
+ "id": "da11d1d4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\displaystyle \\min_{\\boldsymbol{\\beta}\\in\n",
@@ -166,8 +202,10 @@
},
{
"cell_type": "markdown",
- "id": "01cb254e",
- "metadata": {},
+ "id": "9aa702c4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which leads to Lasso regression. Lasso stands for least absolute shrinkage and selection operator. \n",
"\n",
@@ -176,8 +214,10 @@
},
{
"cell_type": "markdown",
- "id": "b6ab9d2d",
- "metadata": {},
+ "id": "cf763fe2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\vert\\vert \\boldsymbol{x}\\vert\\vert_1 = \\sum_i \\vert x_i\\vert.\n",
@@ -186,8 +226,10 @@
},
{
"cell_type": "markdown",
- "id": "a5fd2fb1",
- "metadata": {},
+ "id": "3c9abf9a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Various steps in cross-validation\n",
"\n",
@@ -209,8 +251,10 @@
},
{
"cell_type": "markdown",
- "id": "567db724",
- "metadata": {},
+ "id": "66b6d57a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## How to set up the cross-validation for Ridge and/or Lasso\n",
"\n",
@@ -223,8 +267,10 @@
},
{
"cell_type": "markdown",
- "id": "db50c5d3",
- "metadata": {},
+ "id": "5c0e346e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -237,8 +283,10 @@
},
{
"cell_type": "markdown",
- "id": "77644e97",
- "metadata": {},
+ "id": "b2ddf9d3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"* Evaluate the prediction performance of these models on the test set by $C[y_i, \\boldsymbol{X}_{i, \\ast}; \\boldsymbol{\\beta}_{-i}(\\lambda), \\boldsymbol{\\sigma}_{-i}^2(\\lambda)]$. Or, by the prediction error $|y_i - \\boldsymbol{X}_{i, \\ast} \\boldsymbol{\\beta}_{-i}(\\lambda)|$, the relative error, the error squared or the R2 score function.\n",
"\n",
@@ -249,8 +297,10 @@
},
{
"cell_type": "markdown",
- "id": "222c145f",
- "metadata": {},
+ "id": "8c99c07a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Cross-validation in brief\n",
"\n",
@@ -275,8 +325,10 @@
},
{
"cell_type": "markdown",
- "id": "1906df63",
- "metadata": {},
+ "id": "542a6705",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Code Example for Cross-validation and $k$-fold Cross-validation\n",
"\n",
@@ -286,8 +338,11 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "c152fd7e",
- "metadata": {},
+ "id": "89e68714",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -385,8 +440,10 @@
},
{
"cell_type": "markdown",
- "id": "5d7ccc04",
- "metadata": {},
+ "id": "53085e52",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Logistic Regression\n",
"\n",
@@ -405,8 +462,10 @@
},
{
"cell_type": "markdown",
- "id": "b84fbc4f",
- "metadata": {},
+ "id": "f137ae6a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Classification problems\n",
"\n",
@@ -429,8 +488,10 @@
},
{
"cell_type": "markdown",
- "id": "495a8bd0",
- "metadata": {},
+ "id": "91ce21f8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Optimization and Deep learning\n",
"\n",
@@ -451,8 +512,10 @@
},
{
"cell_type": "markdown",
- "id": "48b58d37",
- "metadata": {},
+ "id": "2d951eb0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Basics\n",
"\n",
@@ -474,8 +537,10 @@
},
{
"cell_type": "markdown",
- "id": "cf4ea236",
- "metadata": {},
+ "id": "f98791eb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"y_i = \\begin{bmatrix} 0 & \\mathrm{no}\\\\ 1 & \\mathrm{yes} \\end{bmatrix}.\n",
@@ -484,8 +549,10 @@
},
{
"cell_type": "markdown",
- "id": "eeb9a563",
- "metadata": {},
+ "id": "5ccf9ac5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Linear classifier\n",
"\n",
@@ -500,8 +567,10 @@
},
{
"cell_type": "markdown",
- "id": "5d676ec0",
- "metadata": {},
+ "id": "7cdf3ea1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -516,8 +585,10 @@
},
{
"cell_type": "markdown",
- "id": "4f1c8890",
- "metadata": {},
+ "id": "1fa46430",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\boldsymbol{y}$ is a vector representing the possible outcomes, $\\boldsymbol{X}$ is our\n",
"$n\\times p$ design matrix and $\\boldsymbol{\\beta}$ represents our estimators/predictors."
@@ -525,8 +596,10 @@
},
{
"cell_type": "markdown",
- "id": "95d3b55e",
- "metadata": {},
+ "id": "e4b93ac2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Some selected properties\n",
"\n",
@@ -550,8 +623,10 @@
},
{
"cell_type": "markdown",
- "id": "a58cb406",
- "metadata": {},
+ "id": "f261d0bb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Simple example\n",
"\n",
@@ -561,8 +636,11 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "cccbdeef",
- "metadata": {},
+ "id": "eec653ca",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Common imports\n",
@@ -623,8 +701,10 @@
},
{
"cell_type": "markdown",
- "id": "eda7b9f2",
- "metadata": {},
+ "id": "3ffd775a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Plotting the mean value for each group\n",
"\n",
@@ -634,8 +714,11 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "c657b457",
- "metadata": {},
+ "id": "03e71e8a",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"agegroupmean = np.array([0.1, 0.133, 0.250, 0.333, 0.462, 0.625, 0.765, 0.800])\n",
@@ -650,8 +733,10 @@
},
{
"cell_type": "markdown",
- "id": "c027575d",
- "metadata": {},
+ "id": "4f6dffd3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We are now trying to find a function $f(y\\vert x)$, that is a function which gives us an expected value for the output $y$ with a given input $x$.\n",
"In standard linear regression with a linear dependence on $x$, we would write this in terms of our model"
@@ -659,8 +744,10 @@
},
{
"cell_type": "markdown",
- "id": "02de7a3d",
- "metadata": {},
+ "id": "28288336",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(y_i\\vert x_i)=\\beta_0+\\beta_1 x_i.\n",
@@ -669,8 +756,10 @@
},
{
"cell_type": "markdown",
- "id": "64969b5a",
- "metadata": {},
+ "id": "92b7048a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This expression implies however that $f(y_i\\vert x_i)$ could take any\n",
"value from minus infinity to plus infinity. If we however let\n",
@@ -686,8 +775,10 @@
},
{
"cell_type": "markdown",
- "id": "c8870523",
- "metadata": {},
+ "id": "43f1fa08",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The logistic function\n",
"\n",
@@ -706,8 +797,10 @@
},
{
"cell_type": "markdown",
- "id": "17cebf36",
- "metadata": {},
+ "id": "b9976820",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(t) = \\frac{1}{1+\\mathrm \\exp{-t}}=\\frac{\\exp{t}}{1+\\mathrm \\exp{t}}.\n",
@@ -716,16 +809,20 @@
},
{
"cell_type": "markdown",
- "id": "ef1c3060",
- "metadata": {},
+ "id": "73d57620",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Note that $1-p(t)= p(-t)$."
]
},
{
"cell_type": "markdown",
- "id": "65471b2a",
- "metadata": {},
+ "id": "b0bebb27",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Examples of likelihood functions used in logistic regression and nueral networks\n",
"\n",
@@ -735,8 +832,11 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "e9a4591d",
- "metadata": {},
+ "id": "3d3031e1",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"\"\"\"The sigmoid function (or the logistic curve) is a\n",
@@ -797,8 +897,10 @@
},
{
"cell_type": "markdown",
- "id": "b988efeb",
- "metadata": {},
+ "id": "ebe0d43e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Two parameters\n",
"\n",
@@ -807,8 +909,10 @@
},
{
"cell_type": "markdown",
- "id": "cfb6a943",
- "metadata": {},
+ "id": "0954b684",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -820,8 +924,10 @@
},
{
"cell_type": "markdown",
- "id": "2361aee9",
- "metadata": {},
+ "id": "935a90e4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$. \n",
"\n",
@@ -830,8 +936,10 @@
},
{
"cell_type": "markdown",
- "id": "847fd47b",
- "metadata": {},
+ "id": "2db24426",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(y_i=0\\vert x_i, \\boldsymbol{\\beta}) = 1-p(y_i=1\\vert x_i, \\boldsymbol{\\beta}).\n",
@@ -840,8 +948,10 @@
},
{
"cell_type": "markdown",
- "id": "be4b11d1",
- "metadata": {},
+ "id": "53b7f663",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Maximum likelihood\n",
"\n",
@@ -855,8 +965,10 @@
},
{
"cell_type": "markdown",
- "id": "0c649e6a",
- "metadata": {},
+ "id": "8585f2d4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -867,16 +979,20 @@
},
{
"cell_type": "markdown",
- "id": "a2382368",
- "metadata": {},
+ "id": "19e94129",
+ "metadata": {
+ "editable": true
+ },
"source": [
"from which we obtain the log-likelihood and our **cost/loss** function"
]
},
{
"cell_type": "markdown",
- "id": "8e56dda5",
- "metadata": {},
+ "id": "9147fec7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left( y_i\\log{p(y_i=1|x_i,\\boldsymbol{\\beta})} + (1-y_i)\\log\\left[1-p(y_i=1|x_i,\\boldsymbol{\\beta}))\\right]\\right).\n",
@@ -885,8 +1001,10 @@
},
{
"cell_type": "markdown",
- "id": "c77515e4",
- "metadata": {},
+ "id": "1be24830",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The cost function rewritten\n",
"\n",
@@ -895,8 +1013,10 @@
},
{
"cell_type": "markdown",
- "id": "d0d27dc3",
- "metadata": {},
+ "id": "121ff8ed",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta}) = \\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
@@ -905,8 +1025,10 @@
},
{
"cell_type": "markdown",
- "id": "4955b6b1",
- "metadata": {},
+ "id": "b6ce724f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The maximum likelihood estimator is defined as the set of parameters that maximize the log-likelihood where we maximize with respect to $\\beta$.\n",
"Since the cost (error) function is just the negative log-likelihood, for logistic regression we have that"
@@ -914,8 +1036,10 @@
},
{
"cell_type": "markdown",
- "id": "c3e91e52",
- "metadata": {},
+ "id": "db562b81",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathcal{C}(\\boldsymbol{\\beta})=-\\sum_{i=1}^n \\left(y_i(\\beta_0+\\beta_1x_i) -\\log{(1+\\exp{(\\beta_0+\\beta_1x_i)})}\\right).\n",
@@ -924,8 +1048,10 @@
},
{
"cell_type": "markdown",
- "id": "3b3d97b6",
- "metadata": {},
+ "id": "9f22edac",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This equation is known in statistics as the **cross entropy**. Finally, we note that just as in linear regression, \n",
"in practice we often supplement the cross-entropy with additional regularization terms, usually $L_1$ and $L_2$ regularization as we did for Ridge and Lasso regression."
@@ -933,8 +1059,10 @@
},
{
"cell_type": "markdown",
- "id": "47890d23",
- "metadata": {},
+ "id": "2013ef41",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Minimizing the cross entropy\n",
"\n",
@@ -947,8 +1075,10 @@
},
{
"cell_type": "markdown",
- "id": "71d34eed",
- "metadata": {},
+ "id": "4ffe3c25",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_0} = -\\sum_{i=1}^n \\left(y_i -\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right),\n",
@@ -957,16 +1087,20 @@
},
{
"cell_type": "markdown",
- "id": "243773d3",
- "metadata": {},
+ "id": "9b9049e2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "81fd7337",
- "metadata": {},
+ "id": "f1f17458",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\beta_1} = -\\sum_{i=1}^n \\left(y_ix_i -x_i\\frac{\\exp{(\\beta_0+\\beta_1x_i)}}{1+\\exp{(\\beta_0+\\beta_1x_i)}}\\right).\n",
@@ -975,8 +1109,10 @@
},
{
"cell_type": "markdown",
- "id": "38fbde35",
- "metadata": {},
+ "id": "30a6df8f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## A more compact expression\n",
"\n",
@@ -988,8 +1124,10 @@
},
{
"cell_type": "markdown",
- "id": "bd7644e6",
- "metadata": {},
+ "id": "6fa70ee0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
@@ -998,8 +1136,10 @@
},
{
"cell_type": "markdown",
- "id": "0fcc74e2",
- "metadata": {},
+ "id": "4b20a583",
+ "metadata": {
+ "editable": true
+ },
"source": [
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
@@ -1007,8 +1147,10 @@
},
{
"cell_type": "markdown",
- "id": "7b5aa78f",
- "metadata": {},
+ "id": "9692b511",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
@@ -1017,8 +1159,10 @@
},
{
"cell_type": "markdown",
- "id": "45ee534e",
- "metadata": {},
+ "id": "b237c3a6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Extending to more predictors\n",
"\n",
@@ -1027,8 +1171,10 @@
},
{
"cell_type": "markdown",
- "id": "f62d8583",
- "metadata": {},
+ "id": "35a5200d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\log{ \\frac{p(\\boldsymbol{\\beta}\\boldsymbol{x})}{1-p(\\boldsymbol{\\beta}\\boldsymbol{x})}} = \\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p.\n",
@@ -1037,16 +1183,20 @@
},
{
"cell_type": "markdown",
- "id": "837cc6c1",
- "metadata": {},
+ "id": "a7651f8a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Here we defined $\\boldsymbol{x}=[1,x_1,x_2,\\dots,x_p]$ and $\\boldsymbol{\\beta}=[\\beta_0, \\beta_1, \\dots, \\beta_p]$ leading to"
]
},
{
"cell_type": "markdown",
- "id": "75b9139f",
- "metadata": {},
+ "id": "72b25b77",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(\\boldsymbol{\\beta}\\boldsymbol{x})=\\frac{ \\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}{1+\\exp{(\\beta_0+\\beta_1x_1+\\beta_2x_2+\\dots+\\beta_px_p)}}.\n",
@@ -1055,8 +1205,10 @@
},
{
"cell_type": "markdown",
- "id": "372ee8ea",
- "metadata": {},
+ "id": "6f5b25ea",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Including more classes\n",
"\n",
@@ -1067,8 +1219,10 @@
},
{
"cell_type": "markdown",
- "id": "b171dd51",
- "metadata": {},
+ "id": "04e9135f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\log{\\frac{p(C=1\\vert x)}{p(K\\vert x)}} = \\beta_{10}+\\beta_{11}x_1,\n",
@@ -1077,16 +1231,20 @@
},
{
"cell_type": "markdown",
- "id": "7816121e",
- "metadata": {},
+ "id": "e9876fbf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and"
]
},
{
"cell_type": "markdown",
- "id": "b58d0bc3",
- "metadata": {},
+ "id": "7b79566f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\log{\\frac{p(C=2\\vert x)}{p(K\\vert x)}} = \\beta_{20}+\\beta_{21}x_1,\n",
@@ -1095,16 +1253,20 @@
},
{
"cell_type": "markdown",
- "id": "0d75989b",
- "metadata": {},
+ "id": "1cf783dc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and so on till the class $C=K-1$ class"
]
},
{
"cell_type": "markdown",
- "id": "af883d63",
- "metadata": {},
+ "id": "7f8f8164",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\log{\\frac{p(C=K-1\\vert x)}{p(K\\vert x)}} = \\beta_{(K-1)0}+\\beta_{(K-1)1}x_1,\n",
@@ -1113,8 +1275,10 @@
},
{
"cell_type": "markdown",
- "id": "6727eebf",
- "metadata": {},
+ "id": "995aaa2b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and the model is specified in term of $K-1$ so-called log-odds or\n",
"**logit** transformations."
@@ -1122,8 +1286,10 @@
},
{
"cell_type": "markdown",
- "id": "c8713334",
- "metadata": {},
+ "id": "18d8bdf8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More classes\n",
"\n",
@@ -1143,8 +1309,10 @@
},
{
"cell_type": "markdown",
- "id": "7819ea46",
- "metadata": {},
+ "id": "05a64fde",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(C=k\\vert \\mathbf {x} )=\\frac{\\exp{(\\beta_{k0}+\\beta_{k1}x_1)}}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}}.\n",
@@ -1153,16 +1321,20 @@
},
{
"cell_type": "markdown",
- "id": "278c2527",
- "metadata": {},
+ "id": "5ed1d535",
+ "metadata": {
+ "editable": true
+ },
"source": [
"It is easy to extend to more predictors. The final class is"
]
},
{
"cell_type": "markdown",
- "id": "376c873f",
- "metadata": {},
+ "id": "a4a5ec15",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"p(C=K\\vert \\mathbf {x} )=\\frac{1}{1+\\sum_{l=1}^{K-1}\\exp{(\\beta_{l0}+\\beta_{l1}x_1)}},\n",
@@ -1171,8 +1343,10 @@
},
{
"cell_type": "markdown",
- "id": "389cdb39",
- "metadata": {},
+ "id": "30066906",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and they sum to one. Our earlier discussions were all specialized to\n",
"the case with two classes only. It is easy to see from the above that\n",
@@ -1186,16 +1360,20 @@
},
{
"cell_type": "markdown",
- "id": "5a3810ce",
- "metadata": {},
+ "id": "4260f4cf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Friday September 23"
]
},
{
"cell_type": "markdown",
- "id": "f1e08704",
- "metadata": {},
+ "id": "c59626d9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Searching for Optimal Regularization Parameters $\\lambda$\n",
"\n",
@@ -1211,8 +1389,11 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "d2e8e94e",
- "metadata": {},
+ "id": "5f04cd97",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -1263,8 +1444,10 @@
},
{
"cell_type": "markdown",
- "id": "d9054061",
- "metadata": {},
+ "id": "09b867a4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n",
"By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n",
@@ -1273,8 +1456,10 @@
},
{
"cell_type": "markdown",
- "id": "e8d8872f",
- "metadata": {},
+ "id": "93a80ac4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Grid Search\n",
"\n",
@@ -1286,8 +1471,11 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "212e1c6b",
- "metadata": {},
+ "id": "f5a684e8",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -1336,8 +1524,10 @@
},
{
"cell_type": "markdown",
- "id": "91b3b83c",
- "metadata": {},
+ "id": "ad6ffe63",
+ "metadata": {
+ "editable": true
+ },
"source": [
"By default the grid search function includes cross validation with\n",
"five folds. The [Scikit-Learn\n",
@@ -1349,8 +1539,10 @@
},
{
"cell_type": "markdown",
- "id": "7e0ab91a",
- "metadata": {},
+ "id": "044c2638",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Randomized Grid Search\n",
"\n",
@@ -1367,8 +1559,11 @@
{
"cell_type": "code",
"execution_count": 7,
- "id": "92c30256",
- "metadata": {},
+ "id": "4e4b21fc",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np\n",
@@ -1418,8 +1613,10 @@
},
{
"cell_type": "markdown",
- "id": "fb8e4f5a",
- "metadata": {},
+ "id": "eee8e0b2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Wisconsin Cancer Data\n",
"\n",
@@ -1431,8 +1628,11 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "d040e14f",
- "metadata": {},
+ "id": "31c74e62",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
@@ -1455,8 +1655,10 @@
},
{
"cell_type": "markdown",
- "id": "47009c95",
- "metadata": {},
+ "id": "2219fff5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Using the correlation matrix\n",
"\n",
@@ -1467,8 +1669,11 @@
{
"cell_type": "code",
"execution_count": 9,
- "id": "40128247",
- "metadata": {},
+ "id": "2895fc8d",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
@@ -1509,8 +1714,10 @@
},
{
"cell_type": "markdown",
- "id": "dc60a6ef",
- "metadata": {},
+ "id": "f2af8ff5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Discussing the correlation data\n",
"\n",
@@ -1532,8 +1739,11 @@
{
"cell_type": "code",
"execution_count": 10,
- "id": "47ec15b8",
- "metadata": {},
+ "id": "f6dc5f57",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"cancerpd = pd.DataFrame(cancer.data, columns=cancer.feature_names)"
@@ -1541,8 +1751,10 @@
},
{
"cell_type": "markdown",
- "id": "eb8b4198",
- "metadata": {},
+ "id": "a4240f80",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and then"
]
@@ -1550,8 +1762,11 @@
{
"cell_type": "code",
"execution_count": 11,
- "id": "58486f42",
- "metadata": {},
+ "id": "e75a34b6",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"correlation_matrix = cancerpd.corr().round(1)"
@@ -1559,8 +1774,10 @@
},
{
"cell_type": "markdown",
- "id": "07834b9b",
- "metadata": {},
+ "id": "f58d6745",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Diagonalizing this matrix we can in turn say something about which\n",
"features are of relevance and which are not. This leads us to\n",
@@ -1570,8 +1787,10 @@
},
{
"cell_type": "markdown",
- "id": "9908e2e3",
- "metadata": {},
+ "id": "cd31f5fc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Other measures in classification studies: Cancer Data again"
]
@@ -1579,8 +1798,11 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "645bfbca",
- "metadata": {},
+ "id": "8df5ad77",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
@@ -1619,8 +1841,10 @@
},
{
"cell_type": "markdown",
- "id": "f528c325",
- "metadata": {},
+ "id": "e41f6d0c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Optimization, the central part of any Machine Learning algortithm\n",
"\n",
@@ -1638,8 +1862,10 @@
},
{
"cell_type": "markdown",
- "id": "ff30edd5",
- "metadata": {},
+ "id": "86345111",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Revisiting our Logistic Regression case\n",
"\n",
@@ -1653,8 +1879,10 @@
},
{
"cell_type": "markdown",
- "id": "847724bd",
- "metadata": {},
+ "id": "a36f83c1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -1666,16 +1894,20 @@
},
{
"cell_type": "markdown",
- "id": "f1969cb5",
- "metadata": {},
+ "id": "66eb6c64",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$."
]
},
{
"cell_type": "markdown",
- "id": "e163f74c",
- "metadata": {},
+ "id": "a31c7c6c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The equations to solve\n",
"\n",
@@ -1688,8 +1920,10 @@
},
{
"cell_type": "markdown",
- "id": "9b49b101",
- "metadata": {},
+ "id": "29c83af2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
@@ -1698,8 +1932,10 @@
},
{
"cell_type": "markdown",
- "id": "f19697ae",
- "metadata": {},
+ "id": "b644c823",
+ "metadata": {
+ "editable": true
+ },
"source": [
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
@@ -1707,8 +1943,10 @@
},
{
"cell_type": "markdown",
- "id": "3f641f13",
- "metadata": {},
+ "id": "ca7021e3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
@@ -1717,16 +1955,20 @@
},
{
"cell_type": "markdown",
- "id": "d82f5c8e",
- "metadata": {},
+ "id": "d7d62e65",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This defines what is called the Hessian matrix."
]
},
{
"cell_type": "markdown",
- "id": "f8d50532",
- "metadata": {},
+ "id": "3b671ed2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Solving using Newton-Raphson's method\n",
"\n",
@@ -1737,8 +1979,10 @@
},
{
"cell_type": "markdown",
- "id": "81e002c2",
- "metadata": {},
+ "id": "1b54bea0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n",
@@ -1747,16 +1991,20 @@
},
{
"cell_type": "markdown",
- "id": "9ec52629",
- "metadata": {},
+ "id": "c57119c0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"or in matrix form as"
]
},
{
"cell_type": "markdown",
- "id": "d0b02495",
- "metadata": {},
+ "id": "1d9a1845",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n",
@@ -1765,8 +2013,10 @@
},
{
"cell_type": "markdown",
- "id": "b1475d07",
- "metadata": {},
+ "id": "58ce29a7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The right-hand side is computed with the old values of $\\beta$. \n",
"\n",
@@ -1775,8 +2025,10 @@
},
{
"cell_type": "markdown",
- "id": "7297eb74",
- "metadata": {},
+ "id": "870a36ad",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Brief reminder on Newton-Raphson's method\n",
"\n",
@@ -1793,8 +2045,10 @@
},
{
"cell_type": "markdown",
- "id": "5ca9494b",
- "metadata": {},
+ "id": "1fd088ac",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The equations\n",
"\n",
@@ -1807,8 +2061,10 @@
},
{
"cell_type": "markdown",
- "id": "ec42264c",
- "metadata": {},
+ "id": "47115609",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"\n",
@@ -1821,8 +2077,10 @@
},
{
"cell_type": "markdown",
- "id": "0d0c3ae9",
- "metadata": {},
+ "id": "695bcabe",
+ "metadata": {
+ "editable": true
+ },
"source": [
"For small enough values of the function and for well-behaved\n",
"functions, the terms beyond linear are unimportant, hence we obtain"
@@ -1830,8 +2088,10 @@
},
{
"cell_type": "markdown",
- "id": "df03a2fd",
- "metadata": {},
+ "id": "74dcca6e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(x)+(s-x)f'(x)\\approx 0,\n",
@@ -1840,16 +2100,20 @@
},
{
"cell_type": "markdown",
- "id": "f6b5cd5a",
- "metadata": {},
+ "id": "462a186c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"yielding"
]
},
{
"cell_type": "markdown",
- "id": "251ef281",
- "metadata": {},
+ "id": "fc0eb634",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"s\\approx x-\\frac{f(x)}{f'(x)}.\n",
@@ -1858,16 +2122,20 @@
},
{
"cell_type": "markdown",
- "id": "84013f15",
- "metadata": {},
+ "id": "4212ab1b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Having in mind an iterative procedure, it is natural to start iterating with"
]
},
{
"cell_type": "markdown",
- "id": "0498425e",
- "metadata": {},
+ "id": "374357e5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n",
@@ -1876,8 +2144,10 @@
},
{
"cell_type": "markdown",
- "id": "096433ab",
- "metadata": {},
+ "id": "69418f00",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Simple geometric interpretation\n",
"\n",
@@ -1896,8 +2166,10 @@
},
{
"cell_type": "markdown",
- "id": "b0851d8d",
- "metadata": {},
+ "id": "4bee6c6a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Extending to more than one variable\n",
"\n",
@@ -1907,8 +2179,10 @@
},
{
"cell_type": "markdown",
- "id": "010769ef",
- "metadata": {},
+ "id": "b69516cc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n",
@@ -1918,16 +2192,20 @@
},
{
"cell_type": "markdown",
- "id": "d853f614",
- "metadata": {},
+ "id": "73989e3b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which we Taylor expand to obtain"
]
},
{
"cell_type": "markdown",
- "id": "4bac06cd",
- "metadata": {},
+ "id": "bd2ac66c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n",
@@ -1942,16 +2220,20 @@
},
{
"cell_type": "markdown",
- "id": "85987829",
- "metadata": {},
+ "id": "8a93ea94",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have"
]
},
{
"cell_type": "markdown",
- "id": "e0708642",
- "metadata": {},
+ "id": "1c535fd6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n",
@@ -1963,16 +2245,20 @@
},
{
"cell_type": "markdown",
- "id": "8c47428d",
- "metadata": {},
+ "id": "11de1257",
+ "metadata": {
+ "editable": true
+ },
"source": [
"we can rephrase Newton's method as"
]
},
{
"cell_type": "markdown",
- "id": "5b1a1b9b",
- "metadata": {},
+ "id": "dcebe4b0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n",
@@ -1983,16 +2269,20 @@
},
{
"cell_type": "markdown",
- "id": "688f39ed",
- "metadata": {},
+ "id": "e64184d6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have defined"
]
},
{
"cell_type": "markdown",
- "id": "0540c80a",
- "metadata": {},
+ "id": "e593fdec",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n",
@@ -2003,8 +2293,10 @@
},
{
"cell_type": "markdown",
- "id": "8749a5d9",
- "metadata": {},
+ "id": "20aa2562",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We need thus to compute the inverse of the Jacobian matrix and it\n",
"is to understand that difficulties may\n",
@@ -2016,8 +2308,10 @@
},
{
"cell_type": "markdown",
- "id": "10468ee8",
- "metadata": {},
+ "id": "f517e5e1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Steepest descent\n",
"\n",
@@ -2031,8 +2325,10 @@
},
{
"cell_type": "markdown",
- "id": "fd5204db",
- "metadata": {},
+ "id": "3cb0ab6d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n",
@@ -2041,8 +2337,10 @@
},
{
"cell_type": "markdown",
- "id": "3643016f",
- "metadata": {},
+ "id": "25679ba7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with $\\gamma_k > 0$.\n",
"\n",
@@ -2053,8 +2351,10 @@
},
{
"cell_type": "markdown",
- "id": "3f3e0600",
- "metadata": {},
+ "id": "71de6599",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More on Steepest descent\n",
"\n",
@@ -2066,8 +2366,10 @@
},
{
"cell_type": "markdown",
- "id": "51927cc7",
- "metadata": {},
+ "id": "5dc34c7a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n",
@@ -2076,8 +2378,10 @@
},
{
"cell_type": "markdown",
- "id": "62148b2a",
- "metadata": {},
+ "id": "c621dec2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The parameter $\\gamma_k$ is often referred to as the step length or\n",
"the learning rate within the context of Machine Learning."
@@ -2085,8 +2389,10 @@
},
{
"cell_type": "markdown",
- "id": "0aaedf94",
- "metadata": {},
+ "id": "04f46782",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The ideal\n",
"\n",
@@ -2111,8 +2417,10 @@
},
{
"cell_type": "markdown",
- "id": "13569ae6",
- "metadata": {},
+ "id": "95dedf75",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The sensitiveness of the gradient descent\n",
"\n",
@@ -2131,8 +2439,10 @@
},
{
"cell_type": "markdown",
- "id": "bfdc53c0",
- "metadata": {},
+ "id": "58b9a1d8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Convex functions\n",
"\n",
@@ -2151,8 +2461,10 @@
},
{
"cell_type": "markdown",
- "id": "a73c982b",
- "metadata": {},
+ "id": "b10f43aa",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Convex function\n",
"\n",
@@ -2161,8 +2473,10 @@
},
{
"cell_type": "markdown",
- "id": "a56a2ae9",
- "metadata": {},
+ "id": "e9f0af64",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conditions on convex functions\n",
"\n",
@@ -2196,8 +2510,10 @@
},
{
"cell_type": "markdown",
- "id": "3090b99d",
- "metadata": {},
+ "id": "aa712729",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More on convex functions\n",
"\n",
@@ -2222,8 +2538,10 @@
},
{
"cell_type": "markdown",
- "id": "093b11b8",
- "metadata": {},
+ "id": "52e7c59f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Some simple problems\n",
"\n",
@@ -2250,8 +2568,10 @@
},
{
"cell_type": "markdown",
- "id": "6f15fc92",
- "metadata": {},
+ "id": "593c68ce",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Revisiting our first homework\n",
"\n",
@@ -2272,8 +2592,11 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "ce786ea2",
- "metadata": {},
+ "id": "e36bcca7",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"x = 2*np.random.rand(m,1)\n",
@@ -2282,8 +2605,10 @@
},
{
"cell_type": "markdown",
- "id": "02fce565",
- "metadata": {},
+ "id": "69515af1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n",
"The linear regression model is given by"
@@ -2291,8 +2616,10 @@
},
{
"cell_type": "markdown",
- "id": "f2ae83a6",
- "metadata": {},
+ "id": "276fa711",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n",
@@ -2301,16 +2628,20 @@
},
{
"cell_type": "markdown",
- "id": "9016e945",
- "metadata": {},
+ "id": "ac536ef4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"such that"
]
},
{
"cell_type": "markdown",
- "id": "9195a113",
- "metadata": {},
+ "id": "83fb4d48",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n",
@@ -2319,8 +2650,10 @@
},
{
"cell_type": "markdown",
- "id": "e66e1135",
- "metadata": {},
+ "id": "4c9daea7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Gradient descent example\n",
"\n",
@@ -2331,8 +2664,10 @@
},
{
"cell_type": "markdown",
- "id": "4e5fe9a8",
- "metadata": {},
+ "id": "2e19aed3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"X \\equiv \\begin{bmatrix}\n",
@@ -2345,16 +2680,20 @@
},
{
"cell_type": "markdown",
- "id": "4178bf2b",
- "metadata": {},
+ "id": "301f952d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The cost/loss/risk function is given by ("
]
},
{
"cell_type": "markdown",
- "id": "91dbd291",
- "metadata": {},
+ "id": "b0b74f0b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n",
@@ -2363,16 +2702,20 @@
},
{
"cell_type": "markdown",
- "id": "b36bcd33",
- "metadata": {},
+ "id": "2df6a47e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and we want to find $\\beta$ such that $C(\\beta)$ is minimized."
]
},
{
"cell_type": "markdown",
- "id": "dcfc85c6",
- "metadata": {},
+ "id": "0e074abe",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The derivative of the cost/loss function\n",
"\n",
@@ -2381,8 +2724,10 @@
},
{
"cell_type": "markdown",
- "id": "cd6314b3",
- "metadata": {},
+ "id": "37ae56ec",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
@@ -2393,16 +2738,20 @@
},
{
"cell_type": "markdown",
- "id": "54f6c9ae",
- "metadata": {},
+ "id": "ee992f83",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $X$ is the design matrix defined above."
]
},
{
"cell_type": "markdown",
- "id": "9b2c6838",
- "metadata": {},
+ "id": "8e90490b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The Hessian matrix\n",
"The Hessian matrix of $C(\\beta)$ is given by"
@@ -2410,8 +2759,10 @@
},
{
"cell_type": "markdown",
- "id": "46b26877",
- "metadata": {},
+ "id": "1bdad037",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{H} \\equiv \\begin{bmatrix}\n",
@@ -2423,16 +2774,20 @@
},
{
"cell_type": "markdown",
- "id": "4972c71f",
- "metadata": {},
+ "id": "0a47bdec",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite."
]
},
{
"cell_type": "markdown",
- "id": "f8ed6356",
- "metadata": {},
+ "id": "d23e8493",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Simple program\n",
"\n",
@@ -2441,8 +2796,10 @@
},
{
"cell_type": "markdown",
- "id": "643c173d",
- "metadata": {},
+ "id": "0adab660",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n",
@@ -2451,8 +2808,10 @@
},
{
"cell_type": "markdown",
- "id": "c5629d6e",
- "metadata": {},
+ "id": "a8a5f2dc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We can use the expression we computed for the gradient and let use a\n",
"$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n",
@@ -2464,8 +2823,10 @@
},
{
"cell_type": "markdown",
- "id": "ec12f593",
- "metadata": {},
+ "id": "7ba3ee87",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Gradient Descent Example\n",
"\n",
@@ -2474,34 +2835,13 @@
},
{
"cell_type": "code",
- "execution_count": 17,
- "id": "27555cc9",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Eigenvalues of Hessian Matrix:[0.28065531 4.4834507 ]\n",
- "[[4.31499792]\n",
- " [2.76774725]]\n",
- "[[4.31499792]\n",
- " [2.76774725]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtxklEQVR4nO3deXxcdb3/8dcnadM2bWnpQhdKE5ACArIWkCW1iktFEJSrggUB0Uq5KoqKSy+bWsUrKq4/RFZpRRBcALlXuJCSlK2mQFlKW1q60kLTDdqma/L5/XFOyjSdmZxJzsyZSd7Px2MemTnr55yZnM/5Luccc3dERETaU5Z0ACIiUhqUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMKTgzW2JmHwzff8/Mbk4ojvFmtiKJdXcXqd+1lD4lDNmNmZ1jZs+Y2WYzWx2+v9TMLB/rc/cfufsXOrscM6s2MzezHnHElTQzu93Mfph0HCKplDBkFzP7BvBL4KfAcGAYcAlwMlCRYZ7yggUoIolSwhAAzGwA8H3gUne/1903euA5d5/o7tvC6W43s/9nZg+Z2Wbg/Wb2MTN7zszeNrPlZnZNm2Wfb2ZLzWytmU1pM+4aM5uW8vm9ZvakmW0wszlmNj5l3Awz+4GZPWFmG83sYTMbEo6uC/9uMLNNZnZimm3sE8a/3szmAse1GT/SzO4zs0YzW2xmX00Zd7yZNYTb+KaZ/Txl3CkpMS83swvD4b3M7HozWxbOc6OZ9QnHjTezFWb2jbAkt8rMLgrHTQImAleE2/JAhu/sEDN7xMzWmdl8M/t0OPxd4bBjUrarsXVfmtlFZvZKuA9fM7MvpSyzNa4rUuI6y8xOM7MF4XK/1+b7u9fM7g6X96yZHZkh3jIz+46ZLQp/C/eY2aB000qRcne99AKYAOwEerQz3e3AWwSljjKgNzAeeE/4+QjgTeCscPpDgU3AOKAX8PNwPR8Mx18DTAvf7wusBU4Ll/Wh8PPQcPwMYBFwENAn/HxdOK4a8GzxA9cB9cAgYD/gJWBFOK4MmA1cRVCaOgB4DfhIOP4p4PzwfT/gveH7KmAjcC7QExgMHBWO+wVwf7i+/sADwI/DcePD/fD9cL7TgCZg75T9/MMs29IXWA5cBPQAjgbWAIeG478IzAUqgX8B16fM+zHgXYAB7wvXe0ybuK4K4/oi0Aj8KdyGw4AtwP4p398O4D/C6b8JLAZ6huOXpHzXlwFPA6MIfgu/B+5K+revVw7HiaQD0Ks4XsB5wBtthj0JbAgPEOPCYbcDf2xnWTcAvwjfXwX8OWVcX2A76RPGt4E72yzrX8AF4fsZwH+ljLsU+N/wfTXtJ4zXgAkpnyfxTsI4AVjWZvrvAreF7+uAa4Ehaab5W5p1GbAZeFfKsBOBxeH78eF+7ZEyfjXvJKLbyZ4wPgPUtxn2e+DqlM/3Ay8CLwC9sizr78BlbeIqDz/3D/frCSnTz+adE4JrgKdTxpUBq4Ca8POSlO/6FeDUlGlHECSbrCcpehXPq0s0EEos1gJDzKyHu+8EcPeTAMKeRKnVl8tTZzSzEwjO3g8nODvvBfwlHD0ydXp332xmazPEUAV8yszOSBnWE6hN+fxGyvsmgrP9qHaLBVjaZt0jzWxDyrByghIJwMUEpYF5ZrYYuNbdHyQoqSxKs66hBGf3s1P6C1i4zFZrW/d1B7anCjihTbw9gDtTPv+BIGlM8rBKEcDMPgpcTVBSKwvjfLFNXM3h+y3h3zdTxm9pE2fq99sS/l5GZoj5b2bWkjKsmaCt7PX0mynFRAlDWj0FbAPOBO5rZ9q2tzj+E/Ab4KPuvtXMbgBa2xZWAe9undDMKgmqbdJZTlDC+GJuoaeNKZ1VBAf4l8PPo9use7G7j0m7cPdXgXPNrAz4JHCvmQ0O5zs+zSxrCA6sh7l7Rw6G7W3PcuBxd/9QupFm1o+gpHcLcI2Z3efu68ysF8H3+zngH+6+w8z+TpDMOmq/lPWWEVQ5rcwQ8+fd/YlOrEsSpEZvAcDdNxBUufzOzP7DzPqHjZRHEVQjZdMfWBcmi+OBz6aMuxc4PWwYriA4S8/0u5sGnGFmHzGzcjPrHTbCjoqwCY1AC0HbQyb3AN81s73DZX4lZdwsYKOZfTtsHC83s8PN7DgAMzvPzIa6ewtBNR3h+qYDHzSzT5tZDzMbbGZHhdP9AfiFme0TLmNfM/tIhG2B4Iw+27Y8CBxkQYeCnuHrODNrTc6/BBo86LL8T+DGcHhrCbAR2BmWNj4cMaZMjjWzT1rQpflrBCceT6eZ7kZgqplVAZjZUDM7s5PrlgJSwpBd3P2/gcuBKwgOWG8S1It/m6A9I5NLge+b2UaCNot7Upb5MvCfBKWQVcB6IO3Fcu6+nKCE8z2CA9py4FtE+J26exMwFXgi7K303jSTXUtQDbUYeJiU6puwCuZ04Khw/BrgZmBAOMkE4GUz20RwMD7H3be4+zKCButvAOuA54HWXkLfBhYCT5vZ28D/AQe3ty2hW4BDw235e5rt3UhwoD+H4Gz+DeAnQK/wIDwBmBxOfjlwjJlNDOf7KsF3tJ4gud8fMaZM/kHQprIeOB/4pLvvSDPdL8N1PRz+Vp4maDuSEmHueoCSiHSMBV2oD3T385KORfJPJQwREYlECUNERCJRlZSIiESiEoaIiERSUtdhDBkyxKurq5MOQ0SkpMyePXuNuw/t7HJKKmFUV1fT0NCQdBgiIiXFzJa2P1X7VCUlIiKRKGGIiEgkShgiIhKJEoaIiESihCEiIpEoYYiISCRKGCIiEokShoiIRKKEISIikShhiIhIJHlPGGZ2q5mtNrOX0oz7hpm5mQ1JN6+IiBSPQpQwbid4XORuzGw/gkdMLitADCIi0kl5TxjuXkfwrOO2fkHw7Gg9kENEpAQk0oYRPqT+dXefE2HaSWbWYGYNjY2NBYhORETSKXjCMLNK4HvAVVGmd/eb3H2su48dOrTTt3MXEZEOSqKE8S5gf2COmS0BRgHPmtnwBGIREZGICv4AJXd/Edin9XOYNMa6+5pCxyIiItEVolvtXcBTwMFmtsLMLs73OkVEJH55L2G4+7ntjK/OdwwiItJ5utJbREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0S6j+nToboaysqCv9OnJx1RSSn4vaRERBIxfTpMmgRNTcHnpUuDzwATJyYXVwlRCUNEuocpU95JFq2amoLhEokShoh0D8syPA0603DZgxKGiHQPo0fnNlz2oIQhIt3D1KlQWbn7sMrKYLhEooQhIt3DxIlw001QVQVmwd+bblKDdw7US0pEuo+JE5UgOkElDBERiUQJQ0REIlHCEBGRSJQwREQkkrwnDDO71cxWm9lLKcN+ambzzOwFM/ubmQ3MdxwiItI5hShh3A5MaDPsEeBwdz8CWAB8twBxiIhIJ+Q9Ybh7HbCuzbCH3X1n+PFpYFS+4xARkc4phjaMzwP/k2mkmU0yswYza2hsbCxgWCIikirRhGFmU4CdQMab0rv7Te4+1t3HDh06tHDBiYjIbhK70tvMLgROB051d08qDhERiSaRhGFmE4ArgPe5e1N704uISPIK0a32LuAp4GAzW2FmFwO/AfoDj5jZ82Z2Y77jEBGRzilEL6lz3X2Eu/d091Hufou7H+ju+7n7UeHrknzHISJSNEr02eLF0EtKRKR0dPZg3/ps8aVLwf2dZ4uXQNJQwhARiSqOg30JP1tcCUNEJKo4DvYl/GxxJQwRkajiONiX8LPFlTBERKKK42DfkWeLF0kjuRKGiEhUHTnYt5Xrs8WLqJHcSuki67Fjx3pDQ0PSYYhIdzZ9etBmsWxZULKYOjW/zwmvrg6SRFtVVbBkSaRFmNlsdx/b2VCUMEREillZWVCyaMsMWloiLSKuhKEqKRGRYlZEjeRKGCIixSyOdpOYKGGIiBSzXBvJ8yix25uLiEhEEycmkiDaUglDREQiUcIQEZFIlDBERCQSJQwRkVJQBLcHUaO3iEixa709SOudcltvDwIFbQxXCUNEuq8iOGuPpEieoaEShoh0T0Vy1h5JkTxDQyUMEemeiuSsPZIiuT1I3hOGmd1qZqvN7KWUYYPM7BEzezX8u3e+4xAR2U2RnLVHUiS3BylECeN2YEKbYd8BHnX3McCj4WcRkcIpkrP2SIrk9iB5TxjuXgesazP4TOCO8P0dwFn5jkNEZDdFctYe2cSJwfMvWlqCvwm0syTVhjHM3VeF798AhmWa0MwmmVmDmTU0NjYWJjoR6fqK5Ky9lCTeS8rd3cwyPsXJ3W8CboLgAUoFC0xEur4iualf3La9vY2G6fOp/8c66p/tG9tyk0oYb5rZCHdfZWYjgNUJxSEiUvI2vbGJp26fT/1DG6l7YSDPvHUwWzkCgEMqFsW2nqQSxv3ABcB14d9/JBSHiEjJWTN/LTNvX0j9I1uonzeUZzcfTDPHUkYzR1fOZ/Ixz1Dzwd6ccuGBDH33uzCLZ715TxhmdhcwHhhiZiuAqwkSxT1mdjGwFPh0vuMQkTamTw+uOVi2LOgZNHVql6ye6QqWP7OS+j8upq62mfrXRjJ324HAYHqxlRMGzOM7J8+k5qP9OPGCg9hr1KHAoXmJI+8Jw93PzTDq1HyvW0QyKKWrnLsZnzadnd+4gh6rV7G2xz78oPl7/Mq/CoykP29z8pD5nPe+GdScOYjjzjuYXnsdVbDYzL102pHHjh3rDQ0NSYchUvqqq4Mk0VZVVdBlUwqmeXszc/6ygPr7VtNSO4MvbfgJlWzZNX6r9aL22G8y/PLzOOLsMZRXlOe8DjOb7e5jOxurEoZId1RWBun+982Cfv6lqgSq2bZu2Mq/p82n/v711D/XlyfWHMxG9gJgOaMYxet7ztTJRB5XwtC9pES6o1K6yjmq1mq2pUuDZNhazZbLHWjzcPfajSs38q+pDUw5eQbjBsxh4N4w7itHMuWR8SzfOJCJhz7Pn778JMtnrWKUrUy/kKVLi+JuuiphiHRHbdswILjKuZQvXOtoNVtrqWTp0qCElXpM7MA+aXxlDfW3LaT+/7ZRN28fnt9yEC2UU85Ojuk7n3GHNFLz4T6cctEYBo8ZFG0bOhEPxFfCwN1L5nXssce6iMRk2jT3qip3s+DvtGlJR9S5mMzcg8P97i+z7OurrEw/X+urqirrapfMXO53XjLTv3jI435IxcJds/WmyccPfNavrKn1R34y2zeu2hht+zsZTzpAg8dwDFYJQ0SKQ2dLPR0pYbR3Rg+7tet4i/PKg4uov3sl9U+WU7e8muXN+wIwgLc4eegCxh27mZqzBnPsuQfRa69e7cfdVmqJp514olKjt4h0LUlUKWVq/E+xbfAIfnfKn6ib1ZuZbxzIGh8CwPCyN6kZ+Ro1J2xn3KeHc/hZB3aoB1NGMfZkiythJH4vKRERoGPPp2hbKnF/J2lUVbXfS2r06KwljCb68IW1P+Wuf4zngB5LOf1dr1BzCtR8dj8OPLUKK8t439TOmzo1fYkrwbvpqpeUiEST7+dfd6TnVrqn5rUmiwi3AG+6/L/Y2bP3bsNaMFqAlYzk7lGXc+ZX92fFv1exaEcVt71aw+dvq2HMh6qxspjut5FJEd5NV1VSItK+QvSq6sg6crye5M2XGoMeTI9up27BMF7YMobP8Gd+zPfYj+W8VbEPr512Kftf/xUGvavrPAhU12GISOEU4vnXHTmjzlIq8RZncd1y7vjiTL5wcD0HVyxm+HuG8qmfn8gf5hzHoN5NXPm+er5w/aEMeXMuZd7C3rf+jGOfu5VBYwYXxXUPRSeOrlaFeqlbrUhCOtJltRDSdEPdXt7Lfz34Kt+3bOWuwQNtvZ8x7Gn/79Nq/ak/vOjbNm6LtCyvrIyvu3GC3ZiJqVutShgi0r4ivDJ8R9MOZm0/igcP+jqry4bTgrGEKi5ovoUfr7+EU/ZdzG8/8zgv3LuAtdv34v43TuBb/xzPe79wOBX9KvZcYD5LUR29Cj3f7Ua5iiPrFOqlEoZIQvJ99h3B5sbN/tjPnvVr31/rp+492yvZtCuUA3su9s+PqfPbLq73hY8u8ZbmluwLS3e2H2cpqu3yBw9Ov+xsF+HFuM+JqYSReBLI5aWEISWjGK+i7qzUbRo8OHjlcfvWvbbeH7jyGb/i+Fo/sd8L3pNtwfGbZj+y9zz/8ntm+D1ff9JXPvdG7tuR7kDct2/uB/Woy8/0ypaMqqriiceVMESKVxGcjXdatoSXp+1b+dwbfvfXnvD/fM8MP6L3PDeaHdx7ss1P6j/Hv31CrT949Sxfv2RDp9aT8UCc7lVRkXm7Mu2jXJaf7eAfY4lHCUOkWMV4ZpizOEo27SWEzm7ftGneMnq0t5j5xr7D/MZhV/qBPRfvWkxfNvoHBzX49z9Q67W/eM43N27OfRvabk/qPol6MIegFJXrPsp0oG/7ai/JqoShhCHdQFI9iuI682/vQNWB7Wve0exz/jLf/3X8FN9ivXabbzN9/Pq9rvHrT6/1Z259ybdv3t7hXbCHdPsk6gE92zZl20eZxg0enFsyVxuGEoZ0A0mVMOJab3sJIcJ6tm3c5gsuvs439B7mzZgvZZSfyzRfTEwxRpUp1rbbmGmbM8WVbR/FWWUXU1uYEoZIsYp6wGivnSDXA0VcJZts1TZVVe6TJ++xfS19+viL5/zQr35frb9/4LN+Abf6JnafZkePXt6S65l8Z2UrTaTu3zTblPUg317SLLJOD10iYQBfB14GXgLuAnpnm14Jowsqsn+s2LS3XdmSSkfPUOMqYbTXy6ey0rdM/Lxv3mu4t2C+0kb4edzh4F7GTj+6z1xf23NY+nnLy+OJMapcDuy59PyaPHnPZFTEHRtKPmEA+wKLgT7h53uAC7PNo4TRxXSF3kQd1ZE68PYOqvmoCsmQNBZT5eBewVY/Za/n/bsn1vpD187yDUs3BPNnO7NPF+Pkyfk5cchHYs7ULjJ5cjwx50FXSRjLgUEEt1l/EPhwtnmUMLqYJHsTJS1b9VFnqpZiKLG1NLf4gocX+80X1HkL6WNpwfzxXz3vW9ZvSb+QTN9tawmj9W9HqoNylWv31/Z+fyX4uy35hBFsA5cBm4BGYHqGaSYBDUDD6NGjY92JkrBivT9RIeSjhNGedAfOlC6ub/UZ5jcMvNqHlb25a5XLGJX5wJ+tGifKxWutSSGpA3BHf38dfRRsglWvJZ8wgL2Bx4ChQE/g78B52eZRCaOLKZUztXz8s+ejqiTH9e2wHr6NHrsN20wf/82Qq/z3Ex/3uQ8s9JY7c7hquW2cqfstW9tFUicOhSphFEHVa1dIGJ8Cbkn5/Dngd9nmUcJIUKEPmsUinzHG3Usqg42rNnrTwOHRD/ptD3xRDvztHTizJYUoB+Bi+v21l+zbxlkEJ0ZdIWGcEPaQqgQMuAP4SrZ5lDASku4fpGfPeO4lVOy9pDrTAJ3rdqUeXFLr+HPcJ43z1vjfvvO0X35srR/X9yUvZ4c3Z2iLyPnMPspFb+nmz7Yf2ztwJ5W0c50vU5wd2c8xK/mEEWwD1wLzwm61dwK9sk2vhJGQKLdTKLaSQVw6Wl+d6wEuW51/O/Mue/p1n37pTP/Sux/3Q3u9umu2XmzxcQOe8ykn13rTgE6UMFJF+S2kmz9KUsh04C6CM/RI2mvoVwmjsC8ljIREvZVCIf4B8lU1EefBKs552szb0tzir/xzkd90/uN+/gH1XlW+fNck/XnLJwyZ5T/6cK3X/3aOb31r6+7b2PaAXVERlBTjSmztzd/R765UOkfk2pW4K7ZhAI8AR8axss6+lDASEuWsshD/wAVqDN6tT31H1tmRA1w7SbkF80+OfMqH2updg/ex1X72vk/6DZ+Y4c/+6RXfuW1n+9uarhqlo1VnBbrNecmXMFLbMrp6LyngGKAWuA0YEcdKO/pSwkhIlC6S+f4HnjYtP0X7TP/krfcEal13Lv/seShhLKbKq3ss888dUO9/+Fydz3toUfsPCeoqSqFzhHtRx1nwKingbGAOcHXr1dmFfilhJKjtWWVFReH+MdpLWJ0p2bR3r6G44s2wf95+/W3/3x/+2+8eM8W30DttHDt69PI13/9Nx+Io5g4FuSiVbSnSOAuaMMJeTIcDlwBrgBXA+XEEkMtLCaOIFPIfI2L9fuzL7kwiyrB/Vs9t9Pu+9ZR/7egZfkyfuV7GTgf3cnb4f1Vc52srhnkLeEsnekntWn8hznaL9AApuytkldQTwMqwLeMHwOnAgcCvgZviCCLqSwmjm2qvMbGzbRi53to6B0tmLvc7L5npXzzkcT+kYuGuRfemyccPfNavrKn1R34y2zeu2tjpde2mEPX+RVwFk1Y3Tm6FTBiHAZZh3CtxBBH1pYTRTWXrrhjHP31Mdx5taW7xl//xqt/42cd9YvVM3698xa7FDWCDnzZ0ll83odafuPGF3Xsw5UMhehaVSmO0e+klt5gVRbda4IA4goj6UsLopgrxz97e2Wea8Tu27PB/3/Gy/+zjtX7WiKd8iDXuCm942Rv+qVFP+q/OnuHP3z2v/R5McSvEwbxUuru6l1Zyy4OiSBiFfilhdGO5VifEWf2QJmFtobdfxM27Bh3QY4lfeGCd33JhnS94eHHyPZgKkWRL6SBcSsktD5QwpFvXyWYV08Fyw9IN/tC1s3xdRfqHAa3pOcz//NUn/PXZq/K3HZ35fvP9+yilap5SSm55oITR3ZXSP2uhdfDg8MaLq/0vlz/pXz1yhh/V55VdPZgy3ocpn2enpfL9lspJS6nszzyJK2FYsKzSMHbsWG9oaEg6jOJQXQ1Ll+45vKoKliwpdDTFpawsOCS0ZQYtLQB4i7Nk5grq7lxKfZ1Tv3gUC3bsD0Afmjhx73nUHPE2484YwPgbzqRsxfI9l5fPfa3vN37Tp8OUKbBsGYweDVOnwsSJSUdVEGY2293HdnY5PeIIRhKwbFluw7uT0aPTHmy3DxnBLefWUf9UOXXLD2BcSx0/Ygrns4w1ZcN45siLGfrlczjmnIOo6HfMOzMO/zFMmgRNTe8Mq6wMDjhxSHcg0/cbv4kTu02CyJs4iimFeqlKKkVXqpONu1pj2jRv6bN79cNm+vi5THNwH1m20n816GrfXt4rehVFvqpeMlWVDB7cdb5fSRxqw+jmSqlONtvBNqbt2Ny42R/72bN+7ftr/dS9Z/uF3OKLqfJmzFewr/9+2JV+28X1vvDRJUEPpmJJuJniGDy4dL5fKXpKGJKXM/O83Do824Gvgwfuda+t9weufMavOL7WT+z3gvdkW9AOTbMf2Xuef/k9M/yerz/pK597I/0CiqWbZbY4SqVBWYpeXAlDjd4SmD49fT39TTd1rt43U+NteXnQAJ3p95fSQA2w6vk3qb9jEXWP7qD+1eG8uHUMThk92c5x/edRc+g6aib05eSLDmJg1YCOx1XoRuViiUO6tLgavZUwJJCvA1emHkvt2LHPvkz72F3Uz4T6JfuxcEc1AH3ZxImD5jPuqI3UnDGQ4887iMohlbnHla8EWapxSJcWV8JIvJopl5eqpPIoX1U0UR++lKGBepCt9TOHP+3Xn17rz9z6km/fvD2WzXX3wlzYFmX5qnqSPENVUhKrfJUw0p1Bp+GAY6xkJH8ffDE9PnwqNZ8Zybs/dgBlPco6vv6k5KPk0I2vI5DOUQlD4pXHXldbfneLNw0c7i2Y76AsbaliY79hvrh+efL3YIpL3L2wSqlXnBQdYiphJHrqZmYDzexeM5tnZq+Y2YlJxtOtTZwYnP1WVQUNzlVVHT4bXrdoPfdPeYZvHTeD9/Z7if6Xnk/lhlX0YAdX9byO7dZr9xkqK+l348+oPmUUVmbpFzp9elAKKisL/k6fnnNcBRX3hXdTpuxZSmtqCoaLFEiiVVJmdgdQ7+43m1kFUOnuGzJNryqp4vR6wyrq/7iYusd2Ur9wBC9tGwNABds4fq951By2npoJfTnpwoMYMHpA7lUrpdgwHHcVX4TbnYhkUvK9pMxsAPA8wTM1IgXRrRJGkdZXe4uz8NGl1E1fTv1Mo27paBbvHA1APzZy0uD5jDt6EzUf35vjzz+Y3gN7d36lpdj1NO4kV4r7QIpGybdhAEcBs4DbgeeAm4G+aaabBDQADaNHj46pRq/IFVF99c5tO/25P8/zX509w/9j3yd9WNmbu0IaYo3+iRFP+c/PrPWGO+f6ji078hNEsVxkl6s8P5NDbRgSFaV+pTcwFtgJnBB+/iXwg2zzJNLoHfc/fZRlJXjbiq1vbfWZv5vjP/5IrZ82dJYPYMOu1Y8uX+4Tq2f67yc+7nMfWBitgbq924IU+f4oKup+Kx3UFRLGcGBJyuca4J/Z5tktYRTinyfOs7pclpXpjBpi386Nqzb6wz9u8Ctrav19A57z3jTtWtW7Kxb6pEMe9zsvmelLZi7PfeHZtjmX/aGza5FOKfmEEWwD9cDB4ftrgJ9mm35XwijUASTOM9tclpXtYrdObmfjvDX+t+887ZcfW+vH9X3Jy9nh4F7GTj+28mX/+jG1/tcrnvLVcxvbX1h7STvbNue6b3V2LdJhcSWMpHtJHUXQdlEBvAZc5O7rM02/q9G7UA2AcfZMyWVZ7V3slm072zSWr73om/xr9dHU1TZT/9pI5m47EIBebOWEAfOoOXwDNR/tx4kXHMReo/aKvj1RGnWzbTOo149IgZR8o3dHXrtKGIVqBM126+lcz3Y7ckadqZSRYTtbLpnsLW0eJ7qJSj+Xad6ft3zCkFn+ow/Xev1v5/jWt7Z2bJ/ksj1xljBEpMPoClVSub52JYxCHWzSVX1VVLj37Ln7sCjVRB2pRmtnO3du2+mzp831X5w1w//Z55PekiHBbBsy0ndu2xnbbnH3aEk7rjaMJKgKTLqQ7p0w8nGwyXSAaDu8M09Cy/UgNG3aHsmpuayH/+3wKT5hyCzvz1sO7ucyzZvJ0lCej+6nUZN2HL2kCq3Yk5lIjrp3wnBPro97AR940/Trm725fPeEsZUKP5dp/q3y631Nj2HegnlLWXnmZJGvap6ufFCNqwRbrAlRuh0ljDjF0YMphkdqrp7b6Pd96yn/2tEz/Jg+c30Jo9Ouq3ngoD3Xla10ka8DVVc9IMbRRtaVE6qUnLgShm5vDp3vwVRZCX36wNq1ey4jS4+mpU+soH7aUrb+43/56KpbGcEqljGaq7maZQOP4rENx2J08vuZPBl+97vOLaO7iaMXnm7lIUUkrl5SJfiggTwYPTr68Ex3dV23Lv0ywruTeosz9/6F/H5iHeft/wSje7xO9SmjeOjGJXx21c/Yl5WU4VSzlNv7fJna38zFqjLElQsli9xNnRqcBKSqrAyGRxX33WpFikEcxZRCvfJWJRVH9UGGqqq3+gzzs0Y85UOscdfg4WVv+KdGPem/OnuGbxsyMnN1WKa4MjW8q849Pp3dF+o2LEUEtWHErJMHiG2/v9V39uy928GhBXw1g/0yu8EvPLDOb7mwzhc8vHj3ezC1V1+eLq50iaTtK9eEpzr3eGl/ShFRwkjYhqUb/KFrZ/l33lvrJ/ef4xVs9XOZ5qsZnP56iMGD0x8sOnom2jaRTJ6sM+JioxKbFIm4EoYavSN686VG6m9bSP2j26lbMIwXtoyhhXJ6sIOx/eZR8+611HykktNuPZvylSvSLyTd8xDSNaL37Al77RW0ixTqWRh6QI9Il1XyD1DqiEIlDG9xlsxcQd2dS6mvc+oXj2LBjv0B6EMTJ+49j5oj3mbcGQM44fyD6LtP33dmznTgbZWul0zq/Z8GDYKNG2H79nfGF+LpcurVI9JlKWHEqGVnC3MfWET9Pauof6qcuuUH8HrLCAAG2gZq9plPzbFbqPnEEI455yAq+lVkXlimA2+r9s7Ykzpwl+JjUEUkkrgSRo84gik1O5p28NzdC6i7r5H62X2Y+eYY1vkYYAwjy1ZRs+9ixp20gJpPjeCwMw+krMcJ0Rc+dWr2O81m6sLbKqnumK1JoQgfCysixaFbJIymNU0888f51D/4FnXP78VT6w+micMAOLDnEs468GVqxhk1n92PA8aPxspGdHxlrQfYyy7b80K+KH35R49OX8JoL9HEYeJEJQgRyahLVkmtX7yBJ25bQP2/mqifO5iGTQezgwqMFo7o/So1Y95g3AcrOOVzBzDiqGH5C7jNsykinbGrakhEYqY2jBSrnn+T+jsWUffoDupfHc6LW8fglNGT7RzXfx41h66jZkJfTr7oIAZWDUgg8hx1JNGIiGTQbRPGv2f9m9dmLKNu2jLqZ0L9kv1YuKMagL5s4sRB8xl31EZqzhjI8ecdROWQyuwLFRHp4rplwhhUcYj3bp7BqpbhwWdbR82wV6kZu4WaTw7l6M8cRM/KnglHmTCVTkSkjW7ZS2rTjl5MqFoU9GD6zEje/bEDcuvB1NW1bf9YujT4DEoaItJpJVXCSPJK75Kgi+9EJI0uc3tzMys3s+fM7MGkYyl5uqW2iORR4gkDuAx4Jekg8mL69OCsv6ws+Dt9en7Xl8tzPUREcpRowjCzUcDHgJuTjCMvWtsTli4N7i3V2p6Qz6QRx4N/REQySLqEcQNwBZDx5kpmNsnMGsysobGxsWCBddqUKXveHqSpKRieL5meBqgGbxGJQWKN3mZ2OnCau19qZuOBb7r76dnmKalGb90uXESKRFdo9D4Z+LiZLQH+DHzAzKYlGE+81J4gIl1MYgnD3b/r7qPcvRo4B3jM3c/LeUGFbliOSu0JItLFJN2G0TnZGpaTTiRqTxCRLqa0L9zLdKHa4MGwZYvu+CoiQtdow+i8TBekrV1b+B5KIiJdXGknjFwbkHO94jnpai0RkSJS2gkjU8Py4MHpp88lwSRx4Z2ISBEr7YSRqWH5l7/sfA+lJC68ExEpYiV1e/O0sj2HujPPhdCN/EREdlPaJYxsJk4Mbund0hL8zbV3VKbqq7IytWmISLfUdRNGZ6VrHwFoblabhoh0S0oYmbRtHykv33MatWmISDeihJFNarVWphsGqk1DRLoJJYyodDNBEenmlDCi0s0ERaSbU8KISjcTFJFurvSvwyikbNd8iIh0cSphiIhIJEoYIiISiRKGiIhEooQhIiKRKGGIiEgkShgiIhKJEoaIiESSWMIws/3MrNbM5prZy2Z2WVKxiIhI+5K8cG8n8A13f9bM+gOzzewRd5+bYEwiIpJBYiUMd1/l7s+G7zcCrwD7JhWPiIhkVxRtGGZWDRwNPJNm3CQzazCzhsbGxoLHJiIigcQThpn1A+4Dvubub7cd7+43uftYdx87dOjQwgcoIiJAwgnDzHoSJIvp7v7XJGMREZHskuwlZcAtwCvu/vOk4hARkWiSLGGcDJwPfMDMng9fpyUYj4iIZJFYt1p3nwlYUusXEZHcJN7oLSIipUEJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCJRwhARkUiUMEREJBIlDBERiUQJQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCSSRBOGmU0ws/lmttDMvpNkLCIikl1iCcPMyoHfAh8FDgXONbNDk4pHRESyS7KEcTyw0N1fc/ftwJ+BMxOMR0REsuiR4Lr3BZanfF4BnNB2IjObBEwKP24zs5cKEFtnDQHWJB1EBIozPqUQIyjOuJVKnAfHsZAkE0Yk7n4TcBOAmTW4+9iEQ2qX4oxXKcRZCjGC4oxbKcUZx3KSrJJ6Hdgv5fOocJiIiBShJBPGv4ExZra/mVUA5wD3JxiPiIhkkViVlLvvNLMvA/8CyoFb3f3ldma7Kf+RxUJxxqsU4iyFGEFxxq1bxWnuHsdyRESki9OV3iIiEokShoiIRFI0CaO924SYWS8zuzsc/4yZVaeM+244fL6ZfSTBGC83s7lm9oKZPWpmVSnjms3s+fCV18b9CHFeaGaNKfF8IWXcBWb2avi6IOE4f5ES4wIz25AyriD708xuNbPVma7/scCvwm14wcyOSRlXyH3ZXpwTw/heNLMnzezIlHFLwuHPx9X9shNxjjezt1K+26tSxhXsVkIR4vxWSowvhb/HQeG4guxPM9vPzGrDY87LZnZZmmni/X26e+IvgkbvRcABQAUwBzi0zTSXAjeG788B7g7fHxpO3wvYP1xOeUIxvh+oDN9Pbo0x/LypiPblhcBv0sw7CHgt/Lt3+H7vpOJsM/1XCDpGFHp/jgOOAV7KMP404H8AA94LPFPofRkxzpNa109wO55nUsYtAYYUyf4cDzzY2d9LvuNsM+0ZwGOF3p/ACOCY8H1/YEGa//VYf5/FUsKIcpuQM4E7wvf3AqeamYXD/+zu29x9MbAwXF7BY3T3WndvCj8+TXBtSaF15pYrHwEecfd17r4eeASYUCRxngvcladYMnL3OmBdlknOBP7ogaeBgWY2gsLuy3bjdPcnwzggud9mlP2ZSUFvJZRjnEn9Nle5+7Ph+43AKwR30EgV6++zWBJGutuEtN3wXdO4+07gLWBwxHkLFWOqiwkye6veZtZgZk+b2Vl5iK9V1DjPDouo95pZ6wWUhdqXOa0rrNrbH3gsZXCh9md7Mm1HIfdlrtr+Nh142MxmW3ArnqSdaGZzzOx/zOywcFhR7k8zqyQ40N6XMrjg+9OCKvqjgWfajIr191n0twYpRWZ2HjAWeF/K4Cp3f93MDgAeM7MX3X1RMhHyAHCXu28zsy8RlNw+kFAsUZwD3OvuzSnDiml/lgwzez9BwjglZfAp4b7cB3jEzOaFZ9hJeJbgu91kZqcBfwfGJBRLFGcAT7h7ammkoPvTzPoRJKyvufvb+VoPFE8JI8ptQnZNY2Y9gAHA2ojzFipGzOyDwBTg4+6+rXW4u78e/n0NmEFwNpAP7cbp7mtTYrsZODbqvIWMM8U5tCnyF3B/tifTdhTdrW/M7AiC7/tMd1/bOjxlX64G/kZ+qnQjcfe33X1T+P4hoKeZDaEI92co228z7/vTzHoSJIvp7v7XNJPE+/vMd8NMxMabHgSNLvvzToPWYW2m+U92b/S+J3x/GLs3er9Gfhq9o8R4NEHD3Jg2w/cGeoXvhwCvkqcGu4hxjkh5/wngaX+nIWxxGO/e4ftBScUZTncIQSOiJbE/w3VUk7mR9mPs3qg4q9D7MmKcowna905qM7wv0D/l/ZPAhATjHN76XRMcaJeF+zbS76VQcYbjBxC0c/RNYn+G++WPwA1Zpon195m3nd2BjT+NoJV/ETAlHPZ9gjN1gN7AX8If/SzggJR5p4TzzQc+mmCM/we8CTwfvu4Ph58EvBj+yF8ELk54X/4YeDmMpxY4JGXez4f7eCFwUZJxhp+vAa5rM1/B9ifB2eMqYAdBPe/FwCXAJeF4I3gQ2KIwlrEJ7cv24rwZWJ/y22wIhx8Q7sc54W9iSsJxfjnlt/k0KQku3e8lqTjDaS4k6HCTOl/B9idBtaIDL6R8r6fl8/epW4OIiEgkxdKGISIiRU4JQ0REIlHCEBGRSJQwREQkEiUMERGJRAlDREQiUcIQEZFIlDBEOiF8HsGHwvc/NLNfJx2TSL7o5oMinXM18P3wRnNHAx9POB6RvNGV3iKdZGaPA/2A8R48l0CkS1KVlEgnmNl7CJ58tl3JQro6JQyRDgqfXDad4Klmm8wsb0/UEykGShgiHRA+ae2vwDfc/RXgBwTtGSJdltowREQkEpUwREQkEiUMERGJRAlDREQiUcIQEZFIlDBERCQSJQwREYlECUNERCL5/7039OU3Ysw9AAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
+ "execution_count": 14,
+ "id": "6b2468a8",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"\n",
"# Importing various packages\n",
@@ -2553,28 +2893,23 @@
},
{
"cell_type": "markdown",
- "id": "b48c2f03",
- "metadata": {},
+ "id": "e685e839",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## And a corresponding example using **scikit-learn**"
]
},
{
"cell_type": "code",
- "execution_count": 18,
- "id": "54641cf5",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[[3.55513228]\n",
- " [3.3552839 ]]\n",
- "[3.5182936] [3.30404461]\n"
- ]
- }
- ],
+ "execution_count": 15,
+ "id": "4442c99c",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"# Importing various packages\n",
"from random import random, seed\n",
@@ -2596,8 +2931,10 @@
},
{
"cell_type": "markdown",
- "id": "1ab2de11",
- "metadata": {},
+ "id": "ac26dfe4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Gradient descent and Ridge\n",
"\n",
@@ -2606,8 +2943,10 @@
},
{
"cell_type": "markdown",
- "id": "ebf9e606",
- "metadata": {},
+ "id": "31ef09be",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n",
@@ -2616,16 +2955,20 @@
},
{
"cell_type": "markdown",
- "id": "27d8b0b2",
- "metadata": {},
+ "id": "56a3ffc2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows"
]
},
{
"cell_type": "markdown",
- "id": "afa59358",
- "metadata": {},
+ "id": "afc07c7c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
@@ -2636,16 +2979,20 @@
},
{
"cell_type": "markdown",
- "id": "c8b5852f",
- "metadata": {},
+ "id": "4344f9af",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by"
]
},
{
"cell_type": "markdown",
- "id": "8e13c9e7",
- "metadata": {},
+ "id": "cd1e2bc1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n",
@@ -2654,8 +3001,10 @@
},
{
"cell_type": "markdown",
- "id": "4e94f377",
- "metadata": {},
+ "id": "e14a1bf4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The Hessian matrix for Ridge Regression\n",
"The Hessian matrix of Ridge Regression for our simple example is given by"
@@ -2663,8 +3012,10 @@
},
{
"cell_type": "markdown",
- "id": "2b2beaeb",
- "metadata": {},
+ "id": "11fffb8f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{H} \\equiv \\begin{bmatrix}\n",
@@ -2676,8 +3027,10 @@
},
{
"cell_type": "markdown",
- "id": "36f7c79f",
- "metadata": {},
+ "id": "2e18cb7b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This implies that the Hessian matrix is positive definite, hence the stationary point is a\n",
"minimum.\n",
@@ -2688,42 +3041,23 @@
},
{
"cell_type": "markdown",
- "id": "fb66e23f",
- "metadata": {},
+ "id": "990e0fa6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Program example for gradient descent with Ridge Regression"
]
},
{
"cell_type": "code",
- "execution_count": 19,
- "id": "aeaf35cf",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Eigenvalues of Hessian Matrix:[0.29052231 4.94613286]\n",
- "[[4.05416408]\n",
- " [2.8806349 ]]\n",
- "[[4.05191339]\n",
- " [2.88235092]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAo00lEQVR4nO3de5xdZX3v8c9vkhAYwuEyiYhAZsALFigKjDdqLd4KRiyeXokDBxRNCdrikXojrVIkrbYe0WNPixEpSMZYi4rowQpVLK0a7KCCXESFTEKUSwighFguya9/rGcna3b25dl7r9ve832/Xus1e6/rb6+9Z/3Wep5nPcvcHRERkXaGyg5ARET6gxKGiIhEUcIQEZEoShgiIhJFCUNERKIoYYiISBQljFnOzKbN7FXh9XlmdklJcRxvZhvL2PZskf6uu1h2uZndb2ZbzGwk69gabO9iM/uLFtPdzJ6VdxwykxJGhZnZKWZ2o5k9ZmYPhNdnm5nlsT13/yt3f3Ov6zGzsfAPPTeLuMpmZpeZ2YVlx1EWM5sHfAT4bXdf4O6bM1jntJn9KiSg+8I+XlCb7u5nufsHet2OZEsJo6LM7FzgY8DfAk8H9gfOAn4D2K3JMnMKC1Bmk/2B3YHbOl3QEs2OM69z9wXA84Gjgfd2HaEUQgmjgsxsb+AC4Gx3v9LdH/XE9919wt0fD/NdZmb/YGbXmNljwMvN7LVm9n0z+6WZ3WNm59et+zQzW29mm81sRd20881sder9i83s22b2iJndbGbHp6Z908w+YGbfMrNHzexaM1sYJt8Q/j4SziBf0uAz7hHif9jMbgdeUDf9GWb2eTPbZGbrzOxPU9NeaGZT4TPeb2YfSU17aSrme8zsjDB+vpl92Mw2hGUuNrM9wrTjzWyjmZ0bruTuNbM3hmnLgAngXeGzfLnJd/ZcM7vOzB4yszvN7A/D+GeGccekPtem2r40szea2R1hH95tZn+cWmctrnel4nq9mS0xsx+H9Z5X9/1daWb/FNb3PTN7XpN4h8zsPWZ2V/gtfM7M9msw33OAO1Pf5zfC+OPM7D/N7Bfh73GpZb5pZivN7FvAVuDQRjHUuPt9wNdIEkdtHTOu6szsneHz/9zM3lQX44iZfTn8Hv7TzC40s/9o991IF9xdQ8UG4ETgKWBum/kuA35BctUxRHIWeDzw6+H9UcD9wOvD/IcDW4CXAfNJihmeAl4Vpp8PrA6vDwQ2A0vCul4d3i8K078J3AU8B9gjvP9gmDYGeKv4gQ8C/w7sBxwM3ApsDNOGgJuA95FcTR0K3A2cEKZ/BzgtvF4AvDi8HgUeBZYC84AR4Plh2kXA1WF7ewFfBv46TDs+7IcLwnJLSA50+6b284UtPsuewD3AG4G5JGfLDwKHh+lvAW4HhkkOjB9OLfta4JmAAb8VtntMXVzvC3G9BdgEfCZ8hiOAXwGHpL6/J4HfD/P/GbAOmBemT6e+63OAtcBBJL+FTwBrmny+Gd9n2IcPA6eFz7s0vB9J/TY2hPjm1rZft850LAcBPwQ+VvfbvjD1/3A/cGTY158J8TwrTP9sGIZJfuP3AP8R891o6PDYVHYAGhp8KXAqcF/duG8Dj4QDxMvCuMuAT7dZ10eBi8Lr9wGfTU3bE3iCxgnj3cAVdev6GnB6eP1N4M9T084G/iW8nnGAaRLX3cCJqffL2JkwXgRsqJv/vcA/htc3AH8JLGwwzxcbbMuAx4Bnpsa9BFgXXh8f9uvc1PQH2JmIdhy8mnyWPwL+vW7cJ4D3p95fHQ6KtwDzW6zrKuCcurjmhPd7hf36otT8N7HzhOB8YG1q2hBwL/Cb4f106ru+A3hlat4DSJLNLt9Z/fdJkii+WzfPd4AzUr+NC9r8LqdJTl4eDev+OrBPavqOfQ5cSjgZCe+fE5Z5FjAnxH1YavqF7EwYbb8bDfGDiqSqaTOw0FKVxu5+nLvvE6alv7d70gua2YvM7PpQ7PELknqPWlHRM9Lzu/tjYX2NjAJ/EIp2HjGzR4CXkhxYau5Lvd5KcrYfa0YswPq6bT+jbtvnkZSlA5xJctD4USiCOCmMP5jkqqfeIpKzz5tS6/uXML5ms7s/1eXnGQVeVBfvBEndU80nSc6QP+6hSBHAzF5jZmtDcckjJFc3C1PLbXb3beH1r8Lf+1PTf1UXZ/r73Q5sJNnXjWL+YireO4Bt7NzHrTyDmd8X4f2BjeJo4fXuvhdJYnwuMz93/faa/VYWkVw5pKenX8d8NxJJCaOavgM8DpwcMW99d8OfITmbPdjd9wYuJjnDhuRs8+DajGY2TFJs08g9JFcY+6SGPd39g13E1MiMWIDFddteV7ftvdx9CYC7/8TdlwJPAz4EXGlmtaKHZzbY1oMkB9YjUuvb25MK1xjtPs89wL/VxbvA3ZcDWNL656PAp4Dza3UFZjYf+DzwYWD/cEJwDTu/r26kv98hkuKenzeJ+TV1Me/u7j+L2MbPSQ7EaYuB9LLR3WC7+7+RXFF8uMksrX4rm0iK7Q5KjUvP2/K7kc4oYVSQuz9CUuTy92b2+2a2V6ikfD5JMVIrewEPuft/mdkLgTekpl0JnBQqhncjKbNv9htYDbzOzE4wszlmtnuohD2oyfxpm4DttK7s/BzwXjPbN6zzT1LTvgs8ambvtqRyfI6ZHWlmLwAws1PNbFE4g34kLLMdmAReZWZ/aGZzQ2Xo88N8nwQuMrOnhXUcaGYnRHwWSM7oW32WrwDPsaRBwbwwvMDMfi1M/xgw5UmT5f9PksQhqZ+ZTzjomdlrgN+OjKmZY83sd8PV6dtJTjzWNpjvYmClmY0CmNkiM4s5QYEkqT3HzN4Q9vMfkdQdfKWHuD8KvLpJJf3ngDPM7PBwkvP+2oRw9fUFkkQ8bGbPBf5Xatl23410QAmjotz9b4B3AO8iOWDdT1L2+m6S+oxmzgYuMLNHSeosPpda523AW0muQu4lqahseLOcu99DcoVzHskB7R7gnUT8Ztx9K7AS+FYoBnhxg9n+kqRoYR1wLXBFavltwEkkrWbWkVwhXALsHWY5EbjNzLaQHIxPcfdfufsGkiKdc4GHgB8AtQPQu4GfAmvN7JfAvwKHtfsswaeAw8NnuarB532U5EB/CsnZ930kVz7zw0H4RKB2RvsO4BgzmwjL/SnJd/QwSXK/OjKmZr5EUm5fq5T+XXd/ssF8Hwvbujb8VtaS1B215cl9GCeR7OfNJL/Rk9z9wW6DdvdNwKdJfrP1075KklC+QfIdfqNulreR/DbuI/kdrSFJlC2/m25jnc3MXQ9QEhkEljShfpa7n1p2LGUysw8BT3f308uOZdDoCkNE+lq4z+IoS7yQpFHEF8uOaxANRNcNIjKr7UVSDPUMkqLb/0NSNCcZU5GUiIhEUZGUiIhE6asiqYULF/rY2FjZYYiI9JWbbrrpQXdf1H7O1voqYYyNjTE1NVV2GCIifcXM6u/M74qKpEREJIoShoiIRFHCEBGRKEoYIiISRQlDRESiKGGIiEgUJQwREYmihCEiIlGUMEREJIoShoiIRMk9YZjZpWb2gJnd2mDauWbmZtbs4e8iIlIRRVxhXEbyiMoZzOxgkkcnbiggBhER6VHuCcPdbyB5vnK9i0ieBawHcoiI9IFS6jDM7GTgZ+5+c8S8y8xsysymNm3aVEB0IiLSSOEJw8yGgfOA98XM7+6r3H3c3ccXLeq5O3cREelSGVcYzwQOAW42s2ngIOB7Zvb0EmIREZFIhT9Ayd1/CDyt9j4kjXF3f7DoWEREJF4RzWrXAN8BDjOzjWZ2Zt7bFBGR7OV+heHuS9tMH8s7BhER6Z3u9BYRkShKGCIiEkUJQ0REoihhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REoihhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REoihhiIhIlNwThpldamYPmNmtqXF/a2Y/MrNbzOyLZrZP3nGIiEhvirjCuAw4sW7cdcCR7n4U8GPgvQXEISIiPcg9Ybj7DcBDdeOudfenwtu1wEF5xyEiIr2pQh3Gm4CvNptoZsvMbMrMpjZt2lRgWCIiFTE5CWNjMDSU/J2cLCWMUhOGma0AngKafnp3X+Xu4+4+vmjRouKCExGpgslJWLYM1q8H9+TvsmWlJI3SEoaZnQGcBEy4u5cVh4hIpa1YAVu3zhy3dWsyvmBzC98iYGYnAu8Cfsvdt7abX0Rk1tqwobPxOSqiWe0a4DvAYWa20czOBP4O2Au4zsx+YGYX5x2HiEhfWry4s/E5yv0Kw92XNhj9qby3KyIyEFauTOos0sVSw8PJ+IJVoZWUiIg0MzEBq1bB6CiYJX9XrUrGF6yUOgwREenAxEQpCaKerjBERCSKEoaIiERRwhARqaKK3N2dpjoMEZGqqd3dXWsZVbu7G0qty9AVhohI1VTo7u40JQwRkappdhf3+vWlFlEpYYiIVE2ru7hL7IBQCUNEpGpWrkzu5m6lhCIqJQwRkaqpv7u7mYI7IFTCEBHpVR5NYCcmYHoatm9PEkcjBXdAqIQhItKLIh5w1KiIqoQOCJUwRER6UUQT2Ip0QKiEISLSi6IecFQrorriiuT9aacV3rxWCUNE+k99ncHZZ+98v3BhMmR9v0KzeooiH3BU8vO9rZ8epz0+Pu5TU1NlhyEiZarvNqOd4eHei28abbO2Xmg+Lesio7GxJEnUGx1Nrj6aMLOb3H28180rYYhIf2l20GylzQG1623W1js5mdRZbNiQXFmsXJlP/cLQUHJlUc8saU3VhBKGiMxOzQ6arbQ5oHa9zV7X26mSrzBUhyEi/aWbuoFe6xOKrKdopVnz2iVLCukKPfeEYWaXmtkDZnZratx+Znadmf0k/N037zhEZEDEdJuRlsX9ChW5D6Jh89rTT4fLLy+kIryIK4zLgBPrxr0H+Lq7Pxv4engvIrNNN3dINzpoLl++8/3ISDJkeb9CRe6D2BFL7Q7w6Wm45priukJ399wHYAy4NfX+TuCA8PoA4M6Y9Rx77LEuIgNi9Wr34WH35Lw4GYaHk/GDYPVq99FRd7Pkb1afq3696f2XHsx2LAJMeQbH8rLqMPZ393vD6/uA/ZvNaGbLzGzKzKY2bdpUTHQikr8yHxKU9+NP87pfotF6m3VOmEP9SiGtpMxsDPiKux8Z3j/i7vukpj/s7m3rMdRKSmSAlNXyqNU9FVkVMXXZmqnr9ZrN3Jfh8zxx0u/yq4f/i30O2a+vW0ndb2YHAIS/D5QUh4iUpayWRzFXNr1egeTVXUiz5d1hzhwc2G5DXLr1D7BTJ5i/zx7sc0h2bYrKShhXA6eH16cDXyopDhEpS1ktj9odzLMoTsorGTZZfjsG27ZhwJBv54/4Z5aSxHvhCf/e2zbTsqgIaTUAa4B7gSeBjcCZwAhJ66ifAP8K7BezLlV6iwyYvCqGW2lWUTw6Gjc9Rk4V+ned9j7fwsz1bsPaxktGld65J4wsByUMEelZu4O5NTkAp1odRW+nx2R4w/lf3yWMpaz2dYz6Niz8bRDrgLWSEhEpR7t7KtoVJ8XWb9TfLxFRoX7tuV/DjB3Dy85/xS7zrGGCQ5hmyLcz5tMMFfg0PiUMEclO3s1Vs9LqYN6qbiXj5rJfePM1MxLECR85oem89ZcQUfFmLYvLlKIGFUmJVNgg3YjXrDipx/qNT//eVU3vs6svduqoOKtN8RcZFUmpt1oRyUZe9x5USYf3jlx68pc48+qTo1btq0MX6bWb8dLb6fE+EfVWKyLVUtSjSsvUpn7jE0u+NKOIqVWymHFNsTpV1FWbmFbUHfBtKGGISDaq0gV4nhrUFzzGMG9YvxIzOOurkQmi/iKl0c2E9SqQeJUwRCQbVekCPEd//J59eMPWVUwzynaMaUZ5C6tYw65FRS0TRL2YZNBpK60cKGGISDbqm6uOjMAee8Bpp1W7xVQLbzrgqzOKmFZtfO2OZq1z2M4hTO9IFlEJotnBvt1VWE6ttDqlSm8RyV4RHfzl4OS9r+fqX748at6OD52t9gnsOq1W8T06uvMZ4SU/olUJQ0Sy1yctpk7Y8z+4dutLo+bt+VDZbp9MhlZSGzYkVxy1JJHWSSut1Pqe5/7Eze7ze/wEKpISkRxk2WIqwzL7F8z7/owiplbJoqM6iBjt9knMneGxDQvqiq7mwW7dhp2mhCEi2cuqxVSPZfZHzrl9RoKYeuropvNmniDqZbFPYhsWxLS66oIShsggqUrXHFm1mOrwqXzp5GAGt20/vOmqc08Q9bLYJ7HPFs+pCa4ShsigKLkFzQyxB7Z22hTj1CeIVnJLEJ10RpjFPqkvuoJdt5/TvS9KGCKDosxnZDfSRW+tu2hy4Jv2xfkliE6u0spO0s22v2TJrlczWciiQ6qiBnU+KNJAreO5iOciFBJHp89/aLHcUlbv8sCgLQz7Ulbv8jEz+wyddKDYSWeEeXTO2Gr7qf16FDzuGRyDS08CnQxKGCJ1Gh2EuuxJNfM4Yg6GDZarTwj1DwyqTctFswPwyEjj+Tt52FIWT/LrcvvoiXsi0vLKIosz2F7jaHMwXEfj5dYx2vsVRDdXPM0OwNB4+U4+d1ZP8uti+1klDNVhiPSzVq1huq1UzTKOuvH1ldSLabzcYjbschTsSLd1C60qixvVBXXS8inLpsa1OpYtW2DevLjtZyGLrFPUoCsMkTp5FHNkGEejK4X00OwKo+f4u90vq1c3D7bZlUDslUwWdRiN1rHbbkmRWYvtMwhFUsD/Bm4DbgXWALu3ml8JQ6ROVZ5yF1EX0bCIqdP4Yw/OvRT/jIzkl4S7bRhQ02Ui7PuEARwIrAP2CO8/B5zRahklDJn1Gh1wej0I9SB9zGpWOd22DiKPM/RerryqkoQb6TIRDkrCuAfYD5gLfAX47VbLKGHIrFaBA1mr4qWGVxBZ6rUJq5n78uVx2yoxCbc0W68wks/AOcAWYBMw2WSeZcAUMLV48eJOd6/I4CihvqLUBFGvVQumRgf15ct3XaYqVwrd6vKkIauEUVorKTPbFzgZOAR4BrCnmZ1aP5+7r3L3cXcfX7RoUdFhilRHAc/M7rirjdWT+OgYbgX0XdWqNVGjVlDXXJMEmVaRZ2N3LavuRbpUZrPaVwHr3H2Tuz8JfAE4rsR4RMoR2xVFDs/M7qkvpqK7xWjUhDVt61Y455yd7wtIsKXIosuVLpWZMDYALzazYTMz4JXAHSXGU4yq9CYqncnre+vkoJtBb6eZdtZXdN9V6bPrZjZvTvbd5GTyXTWSU8d8s0IW5VrdDsBfAj8iaVZ7BTC/1fx9X+ldgUrLTFW1YjBreX5vndZLdLjPc62DyOPO5Vit7nAfGWneXUo//7/1gEGo9O506PuEUZWbrLIwaMmvlVYHp14TZcYH3UIrqSM7vsvlZKLVDXbNhjlzBvP3GaGwhAFcBzwvi431OvR9wijzjCxrg5T82mnVOqfXRNnjfqzN3u4eiEwSRL1mJw3LlxdzMtHsBrtmQz/+n2WkyIRxDHA98I/AAVlstNuh7xPGIB1kq5r8OjmzjZ23XQd/tWKQbrv27uDg2mjTzboAb7iOrM/8G62vqN95s323YEHz72iWKrxICvg94Gbg/YS7s4se+j5hDFIxThWTXyf7t9d52w2dfK8tDuIxm4ruj6mo31+RJxON9l2zKw8ljGISBmDAkcBZwIPARuC0LALoZOj7hOE+OBXFvRx88toHnSSxbiubO0kaXSTPTla/Q+wBuqgkX/bJRFWvfktUZJHUt4Cfh7qMDwAnAc8CPg6syiKI2GEgEsYg6ebAn+dZbicHim4PKp1cbUQcoLpKEPViD9BFHUib7aORkUo/m2OQFZkwjgCsybQ7sggidlDCGAB5/jPneYWRVp8oO+jdtF1SSFdeZ56EizyQNisaKqIIdpCKfjNSiWa1wKFZBBE7KGEMgDzPcvOqw+hhux1dQeRdzFf0gbTMM/1BKfrNSCUSRtGDEsYAyPsgkq5rmDNn57rzbjEU1tWqWWvbIqYiDrCdfuZe9pHqEipDCUP6U15nuekD28iI+7x52W+jgY6uINqp2gG21+9KdQmVkVXC0DO9pVh59LZZ3x/T5s3w5JMz58moj6NM+2Kql0Pngj3pta+oDPq+kmpRwpDiZd3bZqMDWyNd9FLac2+unXRYWLUDbK+9vZbcFbdkb27ZAYj0LPYAFnGm3i4ppLW8aqhd9dQSWa0XWmh+wKyNX7Ei+UyLFyfJoqwD7OLFSdyNxseamFCCGCC6wpD+F3sAa3CmnlsRU7fFOSU+62AXVbvikdIpYfSTfn2WRt5xt3uwDsDICExM5FsHkdauOKcfvksVKUm9LGrOixpmdSupfr0Zqai4081p61obbWG4u2auvWjX9Xc/fpf9QPdfNISa1c4yZTVR7PUfsOC4Ia6r78wTRL1WSUHNTfOhRNxUVgnDknX1h/HxcZ+amio7jHIMDTUuFzFLyrvzUF9xC0nRTyfFEjnHnVkldR4mJxtXYJfxXc4GY2ONK+lHR5P6oFnMzG5y9/Fe16M6jH5RRhv9LJ7ZnHHchdVB1OumzqFZBXbV7rcYFL02A5a2lDD6RRktVrL4B+wx7tISRFr9jYG1JrIxSaNRolHro3woEecvi3KtoobM6zD6rYKs2+7Eu/2MWfXoOjKSDBExZNrVRla63Q+tytT77bfXD1SH0RSDUOkN7ANcCfwIuAN4Sav5M00Ys+HH1etn7Hb5DpaLTQ7Gti52QEa67eNJldvFUyJuaFASxuXAm8Pr3YB9Ws2facKYDf/MWXzGbv4BW2w3NkEcwM+yiaVTnTyjes6c1rFUrTNBmbX6PmEAewPraPJwpkZDpgljNvwzl/UZm2x3G9Y0QRzCXa3XWcQVYbNtLF/e/il7ZT+wSKSFrBJGmZXehwCbgH80s++b2SVmtmf9TGa2zMymzGxq06ZN2W19NlSQ7bdfZ+MzYAbT3ngfbmDn+CO4dcZR9G4/tPWKs2ix1U6zbVxzzcw7nufM2XXZRrGoclsGTJkJYy5wDPAP7n408BjwnvqZ3H2Vu4+7+/iiRYuy27r+mTPRqBXTeazkMWbu2/9iPmOrV+5IELf6kZ1tqIgmk622kW4i2+xeifrl1bWGDJgyE8ZGYKO73xjeX0mSQIoxG/6ZH3qos/ERYpq5rmGCj/O2Gft29+VvSs7Au+07qYgrwthtdBJLlToTFOlRaQnD3e8D7jGzw8KoVwK3FxpE1f6Zs+6QLoODbOx9ECfb1TOKmN7jH9q5b1euhMsv7+4+hpoirghjt6GrU5mtsqgI6XYAng9MAbcAVwH7tpp/oPuSyqNSt4t1xrZiOn2Pz8bHkVXlb1mtpPKKRU1ApSD0eyupboaBThh5tahpc1CKTRBn7XVF9zFk2VprUA6ys+E+IKmMrBKGOh+sgslJOPXUxtMy7pAutrO+d+1/OR+67/RsNppVp3BZdIZYFeooTwqkzgcHRe0g2EyPlbqxdRB/ceClM07/M0sW0LjMf9482LKls/qaIprWFkUd5UkfUsIoW6ODYE0XFanRCeKAS2YkiAs2vqmj7XSkvkXayEjyd/PmzirBB+kgOxvuA5KBo4RRtlYHu4iiltgE8TfPWjUzQfz8zcmEoh4Vmm6RtmABPPHEzOkxVwqDdJBVSyvpQ0oYRWl2YG521/XoaMNkEZsgLj72kzMSxDt/0qDYq5duu3vR7ZXCIB1kZ8N9QDJ4sqg5L2ro21ZSrfoo2m23XVsPzZu3o7VMbCumS170ic7jKquvo6y6Te/nVlIiBWIA+pLKV1FFLTGaVdauWrVr0Qyw6cn/gZ060fIK4tO/ObOI6cy1LSrOa+r3SaNWOpB/nUAvVwpVu9lSZBYZzIRRVlFLM80OwNu2NRw9wq5dd/zzCTMrqU+7ISJBpDXaJ80yUt51AiqOEelLg5kw8mh+2csVS5MD8FM06PWUpFfXq06amSB+/1/e3HnMaY32ifuuSaOoOgFdKYj0ncFMGFk3v+zhisUM3rB+195bH2OYi1m2y3iGhxlbvZKTv9xjgqjX7LO760xfRKIMZsLIuvllB1csjVoxrWGCt7CKaUbZjjHNKPefvIy3+d+z5+qCimbaffYrrtCZvoi0NJgJI+vmly2uWGKbua748AhjPs2Qb2fMpzn0qouSCUUVzTTaJzVl1/GISF8YnISRrmNYsQJOPz27M/cmZ+fNniwHcPtFX5tRB3HEuSd2t+2spCuaG+nXLjZEpDCDkTAa1TFcfnlyVt3jmXurOojz2HnFcuffXTcjQfza20/o5RPlo3Y10+xSqEpdbFSpWbSIAIOSMDJsFRVbB7HljX/KZ3xiR4J4zltfncEHKUjVu9ioWrNoEQEGJWH00CqqUYJYyiTrGGMbQ6xjjKVM8qHJxTPqIPa/9K8z/AAFq3oXG72cAOjKRCQ3g5EwOjhjbldJvZRJPskyxljPEM4Y6/nM8DIO9goV1/Sq6jfOdXsCoCsTkVwNxgOUGj1YB2BkhDds/hhraH8g3PT1W1j4iqP0YJsq6PY70Hcn0pAeoJRWO2MeGWFG+tu8mU+yjKXseob5yI13zqikXviKo5IJg/TMhX7VbZGZvjuRXPV1wnCHP3/t95PipVMnmN68gPpSpj3Zyl+xgkdvXT8jQez9wsMar7TqFcKzQbdFZvruRHJVesIwszlm9n0z+0rbmR96CF88ynYbYtrGmBiaZOU1R++YvJjGZ5JjtoEFRzS5/6Be1SuEZ4tubmjUdyeSq9ITBnAOcEfUnOvXY/ds2FEZXStu+oN9r+PJh7cwNJrBGWbVK4SlOX13IrkqtdLbzA4CLgdWAu9w95NazT9u5rtUeacrNBtVfg8P66AhIrPaoFR6fxR4F7C92QxmtszMpsysQfMoZlZo6gxTRCQ3pSUMMzsJeMDdb2o1n7uvcvfxptmxvripUdl3pzdz6eYvEZFdlHmF8RvA75jZNPBZ4BVmtrrlEkN14cZUaHZ6M1dVb/5SEhORklXixj0zOx74s7Z1GIce6lPbtyfFUIsXJ8miXXFTpzdzVfHmL9XNiEgPsqrD6K+E0exO71aGhpIrhV03mhRb9Tp/EaqYxESkbwxKpTcA7v7Ndsmia53ezFXFm790B7OIVEAlEkauOr2Zq4o3f1UxiYnIrDP4CaPTprZVbJpbxSQmIrNOJeowYnVVhzEoJieT50F0UuEvIkJ2dRhzswhGCjAxoQQhIqUa/CIpERHJhBKGiIhEUcIQEZEoShgiIhJFCUNERKIoYYiISBQlDBERiaKEISIiUZQwREQkihKGiIhEUcIQEZEoShgiIhJFCUNERKIoYYiISBQlDBERiaKEISIiUUpLGGZ2sJldb2a3m9ltZnZOWbGIiEh7ZV5hPAWc6+6HAy8G3mpmh0ctOTkJY2MwNJT8nZzML0oREQFKfESru98L3BteP2pmdwAHAre3XHByEpYtg61bk/fr1yfvQY8wFRHJkbl72TFgZmPADcCR7v7LumnLgGUAixcvPna9WZIk6o2OwvR07rGKiPQbM7vJ3cd7XU/pld5mtgD4PPD2+mQB4O6r3H3c3ccXLVoEGzY0XlGz8SIikolSE4aZzSNJFpPu/oWohRYv7my8iIhkosxWUgZ8CrjD3T8SveDKlTA8PHPc8HAyXkREclPmFcZvAKcBrzCzH4RhSdulJiZg1aqkzsIs+btqlSq8RURyVolK71jj4+M+NTVVdhgiIn1lYCq9RUSkPyhhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REoihhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REoihhiIhIFCUMERGJooQhIiJRSk0YZnaimd1pZj81s/eUGYuIiLRWWsIwsznA/wNeAxwOLDWzw8uKR0REWivzCuOFwE/d/W53fwL4LHByifGIiEgLc0vc9oHAPan3G4EX1c9kZsuAZeHt42Z2awGx9Woh8GDZQURQnNnphxhBcWatX+I8LIuVlJkworj7KmAVgJlNuft4ySG1pTiz1Q9x9kOMoDiz1k9xZrGeMoukfgYcnHp/UBgnIiIVVGbC+E/g2WZ2iJntBpwCXF1iPCIi0kJpRVLu/pSZvQ34GjAHuNTdb2uz2Kr8I8uE4sxWP8TZDzGC4szarIrT3D2L9YiIyIDTnd4iIhJFCUNERKJUJmG06ybEzOab2T+F6Tea2Vhq2nvD+DvN7IQSY3yHmd1uZreY2dfNbDQ1bZuZ/SAMuVbuR8R5hpltSsXz5tS0083sJ2E4veQ4L0rF+GMzeyQ1rZD9aWaXmtkDze7/scT/DZ/hFjM7JjWtyH3ZLs6JEN8PzezbZva81LTpMP4HWTW/7CHO483sF6nv9n2paYV1JRQR5ztTMd4afo/7hWmF7E8zO9jMrg/HnNvM7JwG82T7+3T30geSSu+7gEOB3YCbgcPr5jkbuDi8PgX4p/D68DD/fOCQsJ45JcX4cmA4vF5eizG831KhfXkG8HcNlt0PuDv83Te83resOOvm/xOShhFF78+XAccAtzaZvgT4KmDAi4Ebi96XkXEeV9s+SXc8N6amTQMLK7I/jwe+0uvvJe846+Z9HfCNovcncABwTHi9F/DjBv/rmf4+q3KFEdNNyMnA5eH1lcArzczC+M+6++Puvg74aVhf4TG6+/XuvjW8XUtyb0nReuly5QTgOnd/yN0fBq4DTqxInEuBNTnF0pS73wA81GKWk4FPe2ItsI+ZHUCx+7JtnO7+7RAHlPfbjNmfzRTalVCHcZb127zX3b8XXj8K3EHSg0Zapr/PqiSMRt2E1H/wHfO4+1PAL4CRyGWLijHtTJLMXrO7mU2Z2Voze30O8dXExvl74RL1SjOr3UBZ1L7saFuhaO8Q4Bup0UXtz3aafY4i92Wn6n+bDlxrZjdZ0hVP2V5iZjeb2VfN7IgwrpL708yGSQ60n0+NLnx/WlJEfzRwY92kTH+fle8apB+Z2anAOPBbqdGj7v4zMzsU+IaZ/dDd7yonQr4MrHH3x83sj0mu3F5RUiwxTgGudPdtqXFV2p99w8xeTpIwXpoa/dKwL58GXGdmPwpn2GX4Hsl3u8XMlgBXAc8uKZYYrwO+5e7pq5FC96eZLSBJWG9391/mtR2ozhVGTDchO+Yxs7nA3sDmyGWLihEzexWwAvgdd3+8Nt7dfxb+3g18k+RsIA9t43T3zanYLgGOjV22yDhTTqHukr/A/dlOs89Rua5vzOwoku/7ZHffXBuf2pcPAF8knyLdKO7+S3ffEl5fA8wzs4VUcH8GrX6bue9PM5tHkiwm3f0LDWbJ9veZd8VMZOXNXJJKl0PYWaF1RN08b2VmpffnwusjmFnpfTf5VHrHxHg0ScXcs+vG7wvMD68XAj8hpwq7yDgPSL3+n8Ba31kRti7Eu294vV9ZcYb5nktSiWhl7M+wjTGaV9K+lpmVit8tel9GxrmYpH7vuLrxewJ7pV5/GzixxDifXvuuSQ60G8K+jfq9FBVnmL43ST3HnmXsz7BfPg18tMU8mf4+c9vZXXz4JSS1/HcBK8K4C0jO1AF2B/45/Oi/CxyaWnZFWO5O4DUlxvivwP3AD8JwdRh/HPDD8CP/IXBmyfvyr4HbQjzXA89NLfumsI9/CryxzDjD+/OBD9YtV9j+JDl7vBd4kqSc90zgLOCsMN1IHgR2V4hlvKR92S7OS4CHU7/NqTD+0LAfbw6/iRUlx/m21G9zLakE1+j3UlacYZ4zSBrcpJcrbH+SFCs6cEvqe12S5+9TXYOIiEiUqtRhiIhIxSlhiIhIFCUMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCI9CM8jeHV4faGZfbzsmETyos4HRXrzfuCC0NHc0cDvlByPSG50p7dIj8zs34AFwPGePJdAZCCpSEqkB2b26yRPPntCyUIGnRKGSJfCk8smSZ5qtsXMcnuinkgVKGGIdCE8ae0LwLnufgfwAZL6DJGBpToMERGJoisMERGJooQhIiJRlDBERCSKEoaIiERRwhARkShKGCIiEkUJQ0REovw3IJsTD8NS93IAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {
- "needs_background": "light"
- },
- "output_type": "display_data"
- }
- ],
+ "execution_count": 16,
+ "id": "a7a87cc2",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"from random import random, seed\n",
"import numpy as np\n",
@@ -2779,8 +3113,10 @@
},
{
"cell_type": "markdown",
- "id": "41ef6f59",
- "metadata": {},
+ "id": "ea03b76e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Using gradient descent methods, limitations\n",
"\n",
@@ -2799,8 +3135,10 @@
},
{
"cell_type": "markdown",
- "id": "14c608fa",
- "metadata": {},
+ "id": "9514dfc8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Challenge yourself this weekend\n",
"\n",
@@ -2808,25 +3146,7 @@
]
}
],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.14"
- }
- },
+ "metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
diff --git a/doc/src/week38/week38.do.txt b/doc/src/week38/week38.do.txt
index 5cea91e1f..552986014 100644
--- a/doc/src/week38/week38.do.txt
+++ b/doc/src/week38/week38.do.txt
@@ -11,9 +11,10 @@ DATE: September 22 and 23
* Lab Wednesday and Thursday: work on project 1
* Thursday: Summary of regression methods, cross-validation and discussion of project 1. Start Logistic Regression
- "Video of lecture":"https://youtu.be/sdt_BFla8uA"
- "Video on Bootstrapping":"https://www.youtube.com/watch?v=Xz0x-8-cgaQ"
+ * "Video of lecture":"https://youtu.be/sdt_BFla8uA"
+ * "Video on Bootstrapping":"https://www.youtube.com/watch?v=Xz0x-8-cgaQ"
* Friday: Classification problems and Logistic Regression, from binary cases to several categories. Start optimization methods
+ * "Video of lecture":"https://youtu.be/7OdqoLOphTA"
* Reading recommendations:
o See lecture notes for week 37 on cross-validation and week 38 at https://compphysics.github.io/MachineLearning/doc/web/course.html.