diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index 3bc363cd9..907ea81d3 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -1774,7 +1774,7 @@ $$

This implies that the Hessian matrix is positive definite, hence the stationary point is a minimum. -Note that the Ridge loss function is convex, as a sum of two convex +Note that the Ridge cost function is convex being a sum of two convex functions. Therefore, the stationary point is a global minimum of this function.

diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index db67d2a6f..811dab43e 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -1681,7 +1681,7 @@ $$

This implies that the Hessian matrix is positive definite, hence the stationary point is a minimum. -Note that the Ridge loss function is convex, as a sum of two convex +Note that the Ridge cost function is convex being a sum of two convex functions. Therefore, the stationary point is a global minimum of this function.

diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index 7464180a5..e8db8fc1e 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -1758,7 +1758,7 @@ $$

This implies that the Hessian matrix is positive definite, hence the stationary point is a minimum. -Note that the Ridge loss function is convex, as a sum of two convex +Note that the Ridge cost function is convex being a sum of two convex functions. Therefore, the stationary point is a global minimum of this function.

diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index d511de4b1..aa4cca30f 100644 Binary files a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz and b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 881d1a9b6..5bf0c2358 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "c4da4e52", - "metadata": {}, + "id": "27989178", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,8 +14,10 @@ }, { "cell_type": "markdown", - "id": "f29028c4", - "metadata": {}, + "id": "1deb5178", + "metadata": { + "editable": true + }, "source": [ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University\n", @@ -25,8 +29,10 @@ }, { "cell_type": "markdown", - "id": "7f7ee367", - "metadata": {}, + "id": "59210b98", + "metadata": { + "editable": true + }, "source": [ "## Plan for week 39\n", "\n", @@ -49,8 +55,10 @@ }, { "cell_type": "markdown", - "id": "fc2a836f", - "metadata": {}, + "id": "b70375c6", + "metadata": { + "editable": true + }, "source": [ "## Thursday September 30\n", "\n", @@ -59,8 +67,10 @@ }, { "cell_type": "markdown", - "id": "9617382d", - "metadata": {}, + "id": "83e12c9e", + "metadata": { + "editable": true + }, "source": [ "## Searching for Optimal Regularization Parameters $\\lambda$\n", "\n", @@ -76,8 +86,11 @@ { "cell_type": "code", "execution_count": 1, - "id": "d49166c2", - "metadata": {}, + "id": "8cc00952", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "%matplotlib inline\n", @@ -130,8 +143,10 @@ }, { "cell_type": "markdown", - "id": "19a9a5ab", - "metadata": {}, + "id": "c2e91e0f", + "metadata": { + "editable": true + }, "source": [ "Here we have performed a rather data greedy calculation as function of the regularization parameter $\\lambda$. There is no resampling here. The latter can easily be added by employing the function **RidgeCV** instead of just calling the **Ridge** function. For **RidgeCV** we need to pass the array of $\\lambda$ values.\n", "By inspecting the figure we can in turn determine which is the optimal regularization parameter.\n", @@ -140,8 +155,10 @@ }, { "cell_type": "markdown", - "id": "edeb088d", - "metadata": {}, + "id": "3469cb13", + "metadata": { + "editable": true + }, "source": [ "## Grid Search\n", "\n", @@ -153,8 +170,11 @@ { "cell_type": "code", "execution_count": 2, - "id": "71bdc0a2", - "metadata": {}, + "id": "183fbef5", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -203,8 +223,10 @@ }, { "cell_type": "markdown", - "id": "0ff3515b", - "metadata": {}, + "id": "fbd82e6a", + "metadata": { + "editable": true + }, "source": [ "By default the grid search function includes cross validation with\n", "five folds. The [Scikit-Learn\n", @@ -216,8 +238,10 @@ }, { "cell_type": "markdown", - "id": "fb36d7f9", - "metadata": {}, + "id": "d90912a2", + "metadata": { + "editable": true + }, "source": [ "## Randomized Grid Search\n", "\n", @@ -234,8 +258,11 @@ { "cell_type": "code", "execution_count": 3, - "id": "0dc5c1a6", - "metadata": {}, + "id": "031a0009", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -285,8 +312,10 @@ }, { "cell_type": "markdown", - "id": "c0482352", - "metadata": {}, + "id": "2dae05d8", + "metadata": { + "editable": true + }, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -302,8 +331,10 @@ }, { "cell_type": "markdown", - "id": "f13f9cf3", - "metadata": {}, + "id": "dde56423", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -317,8 +348,10 @@ }, { "cell_type": "markdown", - "id": "563d2108", - "metadata": {}, + "id": "287dd911", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -330,16 +363,20 @@ }, { "cell_type": "markdown", - "id": "cfebfdde", - "metadata": {}, + "id": "3e95f118", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] }, { "cell_type": "markdown", - "id": "f732663f", - "metadata": {}, + "id": "50bacdb9", + "metadata": { + "editable": true + }, "source": [ "## The equations to solve\n", "\n", @@ -352,8 +389,10 @@ }, { "cell_type": "markdown", - "id": "a4f74c15", - "metadata": {}, + "id": "331fc9ec", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -362,8 +401,10 @@ }, { "cell_type": "markdown", - "id": "ce57218f", - "metadata": {}, + "id": "dab31d07", + "metadata": { + "editable": true + }, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -371,8 +412,10 @@ }, { "cell_type": "markdown", - "id": "e40a4cac", - "metadata": {}, + "id": "d235b146", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -381,16 +424,20 @@ }, { "cell_type": "markdown", - "id": "2eafcd65", - "metadata": {}, + "id": "077d1780", + "metadata": { + "editable": true + }, "source": [ "This defines what is called the Hessian matrix." ] }, { "cell_type": "markdown", - "id": "bdb8d75d", - "metadata": {}, + "id": "2ef1828a", + "metadata": { + "editable": true + }, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -401,8 +448,10 @@ }, { "cell_type": "markdown", - "id": "7304cc0b", - "metadata": {}, + "id": "ab007df8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -411,16 +460,20 @@ }, { "cell_type": "markdown", - "id": "d04b9c03", - "metadata": {}, + "id": "a2232f70", + "metadata": { + "editable": true + }, "source": [ "or in matrix form as" ] }, { "cell_type": "markdown", - "id": "43c885d7", - "metadata": {}, + "id": "2550c2f0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -429,8 +482,10 @@ }, { "cell_type": "markdown", - "id": "46bde37c", - "metadata": {}, + "id": "0743917a", + "metadata": { + "editable": true + }, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -439,8 +494,10 @@ }, { "cell_type": "markdown", - "id": "a0b72eb7", - "metadata": {}, + "id": "8e14cb5e", + "metadata": { + "editable": true + }, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -457,8 +514,10 @@ }, { "cell_type": "markdown", - "id": "5b7485f1", - "metadata": {}, + "id": "74a0765f", + "metadata": { + "editable": true + }, "source": [ "## The equations\n", "\n", @@ -471,8 +530,10 @@ }, { "cell_type": "markdown", - "id": "4ba6228a", - "metadata": {}, + "id": "a322bd0e", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -485,8 +546,10 @@ }, { "cell_type": "markdown", - "id": "30380d0c", - "metadata": {}, + "id": "c2a5bad6", + "metadata": { + "editable": true + }, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -494,8 +557,10 @@ }, { "cell_type": "markdown", - "id": "23c99855", - "metadata": {}, + "id": "ad427199", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -504,16 +569,20 @@ }, { "cell_type": "markdown", - "id": "08273eaa", - "metadata": {}, + "id": "ee10e015", + "metadata": { + "editable": true + }, "source": [ "yielding" ] }, { "cell_type": "markdown", - "id": "e77fa937", - "metadata": {}, + "id": "724d2e8a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -522,16 +591,20 @@ }, { "cell_type": "markdown", - "id": "52d8ac79", - "metadata": {}, + "id": "d5873493", + "metadata": { + "editable": true + }, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] }, { "cell_type": "markdown", - "id": "c749910f", - "metadata": {}, + "id": "d880bf0b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -540,8 +613,10 @@ }, { "cell_type": "markdown", - "id": "ce3de0ff", - "metadata": {}, + "id": "15aece47", + "metadata": { + "editable": true + }, "source": [ "## Simple geometric interpretation\n", "\n", @@ -560,8 +635,10 @@ }, { "cell_type": "markdown", - "id": "7e046c1f", - "metadata": {}, + "id": "fc3d79af", + "metadata": { + "editable": true + }, "source": [ "## Extending to more than one variable\n", "\n", @@ -571,8 +648,10 @@ }, { "cell_type": "markdown", - "id": "6b1748dc", - "metadata": {}, + "id": "e357107f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -582,16 +661,20 @@ }, { "cell_type": "markdown", - "id": "81266388", - "metadata": {}, + "id": "34b27705", + "metadata": { + "editable": true + }, "source": [ "which we Taylor expand to obtain" ] }, { "cell_type": "markdown", - "id": "b8b28a01", - "metadata": {}, + "id": "7d0e5513", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -606,16 +689,20 @@ }, { "cell_type": "markdown", - "id": "a37caf24", - "metadata": {}, + "id": "b622ab6c", + "metadata": { + "editable": true + }, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] }, { "cell_type": "markdown", - "id": "4421201b", - "metadata": {}, + "id": "eff30163", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -627,16 +714,20 @@ }, { "cell_type": "markdown", - "id": "156aa5ac", - "metadata": {}, + "id": "25b8eae6", + "metadata": { + "editable": true + }, "source": [ "we can rephrase Newton's method as" ] }, { "cell_type": "markdown", - "id": "19c663a4", - "metadata": {}, + "id": "fb33d901", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -647,16 +738,20 @@ }, { "cell_type": "markdown", - "id": "fab6f264", - "metadata": {}, + "id": "56d2b104", + "metadata": { + "editable": true + }, "source": [ "where we have defined" ] }, { "cell_type": "markdown", - "id": "d3726f16", - "metadata": {}, + "id": "3d27ac93", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -667,8 +762,10 @@ }, { "cell_type": "markdown", - "id": "091973c9", - "metadata": {}, + "id": "21070210", + "metadata": { + "editable": true + }, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -680,8 +777,10 @@ }, { "cell_type": "markdown", - "id": "2042ba02", - "metadata": {}, + "id": "e9a8d5ab", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent\n", "\n", @@ -695,8 +794,10 @@ }, { "cell_type": "markdown", - "id": "66e3647e", - "metadata": {}, + "id": "301ef6ac", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -705,8 +806,10 @@ }, { "cell_type": "markdown", - "id": "038d29b8", - "metadata": {}, + "id": "9ac00842", + "metadata": { + "editable": true + }, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -717,8 +820,10 @@ }, { "cell_type": "markdown", - "id": "7b2aec41", - "metadata": {}, + "id": "f07f5084", + "metadata": { + "editable": true + }, "source": [ "## More on Steepest descent\n", "\n", @@ -730,8 +835,10 @@ }, { "cell_type": "markdown", - "id": "c5344c2b", - "metadata": {}, + "id": "194ec701", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -740,8 +847,10 @@ }, { "cell_type": "markdown", - "id": "e42471e7", - "metadata": {}, + "id": "33b8042a", + "metadata": { + "editable": true + }, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -749,8 +858,10 @@ }, { "cell_type": "markdown", - "id": "b38320ec", - "metadata": {}, + "id": "cfe687c2", + "metadata": { + "editable": true + }, "source": [ "## The ideal\n", "\n", @@ -775,8 +886,10 @@ }, { "cell_type": "markdown", - "id": "792e177b", - "metadata": {}, + "id": "8b49c45b", + "metadata": { + "editable": true + }, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -795,8 +908,10 @@ }, { "cell_type": "markdown", - "id": "760fca67", - "metadata": {}, + "id": "d0472c94", + "metadata": { + "editable": true + }, "source": [ "## Convex functions\n", "\n", @@ -815,8 +930,10 @@ }, { "cell_type": "markdown", - "id": "52452fa9", - "metadata": {}, + "id": "38efa238", + "metadata": { + "editable": true + }, "source": [ "## Convex function\n", "\n", @@ -825,8 +942,10 @@ }, { "cell_type": "markdown", - "id": "3dffb542", - "metadata": {}, + "id": "ba410ba0", + "metadata": { + "editable": true + }, "source": [ "## Conditions on convex functions\n", "\n", @@ -860,8 +979,10 @@ }, { "cell_type": "markdown", - "id": "24a9cadc", - "metadata": {}, + "id": "4f3b7299", + "metadata": { + "editable": true + }, "source": [ "## More on convex functions\n", "\n", @@ -886,8 +1007,10 @@ }, { "cell_type": "markdown", - "id": "b9748fe1", - "metadata": {}, + "id": "023159dc", + "metadata": { + "editable": true + }, "source": [ "## Some simple problems\n", "\n", @@ -914,8 +1037,10 @@ }, { "cell_type": "markdown", - "id": "4d93454f", - "metadata": {}, + "id": "e09d96ad", + "metadata": { + "editable": true + }, "source": [ "## Standard steepest descent\n", "\n", @@ -932,8 +1057,10 @@ }, { "cell_type": "markdown", - "id": "3b02576e", - "metadata": {}, + "id": "0c736f59", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -942,16 +1069,20 @@ }, { "cell_type": "markdown", - "id": "c34de3f5", - "metadata": {}, + "id": "bbc1eae2", + "metadata": { + "editable": true + }, "source": [ "In the iterative process we end up with a problem like" ] }, { "cell_type": "markdown", - "id": "cfc60396", - "metadata": {}, + "id": "b8e12b0e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -960,8 +1091,10 @@ }, { "cell_type": "markdown", - "id": "b6bc2825", - "metadata": {}, + "id": "7776d4c5", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", @@ -970,8 +1103,10 @@ }, { "cell_type": "markdown", - "id": "71cce93d", - "metadata": {}, + "id": "cc2b2591", + "metadata": { + "editable": true + }, "source": [ "## Gradient method\n", "\n", @@ -980,8 +1115,10 @@ }, { "cell_type": "markdown", - "id": "39b1c049", - "metadata": {}, + "id": "30813f54", + "metadata": { + "editable": true + }, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -990,8 +1127,10 @@ }, { "cell_type": "markdown", - "id": "ab1ecb22", - "metadata": {}, + "id": "b9f2a68a", + "metadata": { + "editable": true + }, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite." @@ -999,8 +1138,10 @@ }, { "cell_type": "markdown", - "id": "a3495238", - "metadata": {}, + "id": "dc4f825d", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent method\n", "\n", @@ -1010,8 +1151,10 @@ }, { "cell_type": "markdown", - "id": "44d95f33", - "metadata": {}, + "id": "9b6412ee", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1020,16 +1163,20 @@ }, { "cell_type": "markdown", - "id": "694edeb5", - "metadata": {}, + "id": "8034df75", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "4ec42882", - "metadata": {}, + "id": "ec6b3f52", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1038,16 +1185,20 @@ }, { "cell_type": "markdown", - "id": "5e3ada87", - "metadata": {}, + "id": "97eab53d", + "metadata": { + "editable": true + }, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "f996558a", - "metadata": {}, + "id": "294bafec", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1055,8 +1206,10 @@ }, { "cell_type": "markdown", - "id": "043db570", - "metadata": {}, + "id": "b614fd3a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1065,8 +1218,10 @@ }, { "cell_type": "markdown", - "id": "b2e31383", - "metadata": {}, + "id": "66e7d9f9", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1075,8 +1230,10 @@ }, { "cell_type": "markdown", - "id": "381b2b84", - "metadata": {}, + "id": "a4970262", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1085,8 +1242,10 @@ }, { "cell_type": "markdown", - "id": "0025ebc9", - "metadata": {}, + "id": "d1492859", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." @@ -1094,8 +1253,10 @@ }, { "cell_type": "markdown", - "id": "a39f2ec2", - "metadata": {}, + "id": "298fed8e", + "metadata": { + "editable": true + }, "source": [ "## Final expressions\n", "We can compute the residual iteratively as" @@ -1103,8 +1264,10 @@ }, { "cell_type": "markdown", - "id": "6374696c", - "metadata": {}, + "id": "e5ae1578", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1113,16 +1276,20 @@ }, { "cell_type": "markdown", - "id": "acb922cb", - "metadata": {}, + "id": "0311229d", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "05c95a7a", - "metadata": {}, + "id": "99eb2181", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -1131,16 +1298,20 @@ }, { "cell_type": "markdown", - "id": "9a1924d0", - "metadata": {}, + "id": "08b7b7f5", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "86c03b30", - "metadata": {}, + "id": "8a89c9f8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -1149,16 +1320,20 @@ }, { "cell_type": "markdown", - "id": "150d301d", - "metadata": {}, + "id": "e491710b", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "e7fab9f4", - "metadata": {}, + "id": "ce03e660", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -1167,16 +1342,20 @@ }, { "cell_type": "markdown", - "id": "5447c085", - "metadata": {}, + "id": "66cd9e49", + "metadata": { + "editable": true + }, "source": [ "leading to the iterative scheme" ] }, { "cell_type": "markdown", - "id": "dee1b865", - "metadata": {}, + "id": "34ad9715", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k-\\alpha_k\\boldsymbol{r}_{k},\n", @@ -1185,8 +1364,10 @@ }, { "cell_type": "markdown", - "id": "04f189e7", - "metadata": {}, + "id": "0d46875f", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent example" ] @@ -1194,8 +1375,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "fcb1d6ec", - "metadata": {}, + "id": "72331d76", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np\n", @@ -1222,8 +1406,10 @@ }, { "cell_type": "markdown", - "id": "78e14695", - "metadata": {}, + "id": "f9baba11", + "metadata": { + "editable": true + }, "source": [ "And then as countor plot" ] @@ -1231,8 +1417,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "4f10e658", - "metadata": {}, + "id": "b0aec5c4", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1242,8 +1431,10 @@ }, { "cell_type": "markdown", - "id": "10fb194e", - "metadata": {}, + "id": "aab04a85", + "metadata": { + "editable": true + }, "source": [ "Find guesses" ] @@ -1251,8 +1442,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "ee47201f", - "metadata": {}, + "id": "35ab2a27", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1261,8 +1455,10 @@ }, { "cell_type": "markdown", - "id": "844b4c66", - "metadata": {}, + "id": "38845ab4", + "metadata": { + "editable": true + }, "source": [ "Run it!" ] @@ -1270,8 +1466,11 @@ { "cell_type": "code", "execution_count": 7, - "id": "e68ec7b1", - "metadata": {}, + "id": "e40e952d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1285,8 +1484,10 @@ }, { "cell_type": "markdown", - "id": "8a3d5762", - "metadata": {}, + "id": "b17e7f44", + "metadata": { + "editable": true + }, "source": [ "What happened?" ] @@ -1294,8 +1495,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "29bd976c", - "metadata": {}, + "id": "d65ee839", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1306,16 +1510,20 @@ }, { "cell_type": "markdown", - "id": "0aee2856", - "metadata": {}, + "id": "18c7c22e", + "metadata": { + "editable": true + }, "source": [ "Note that we did only one iteration here. We can easily add more using our previous guesses." ] }, { "cell_type": "markdown", - "id": "8c7e6157", - "metadata": {}, + "id": "7f1456dc", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", @@ -1326,8 +1534,10 @@ }, { "cell_type": "markdown", - "id": "cb19fc55", - "metadata": {}, + "id": "dd173dfb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1336,8 +1546,10 @@ }, { "cell_type": "markdown", - "id": "fb0fa7bb", - "metadata": {}, + "id": "d701c9fb", + "metadata": { + "editable": true + }, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1345,8 +1557,10 @@ }, { "cell_type": "markdown", - "id": "b65328b9", - "metadata": {}, + "id": "f9a8b68b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1355,8 +1569,10 @@ }, { "cell_type": "markdown", - "id": "efd9856f", - "metadata": {}, + "id": "a2dce53d", + "metadata": { + "editable": true + }, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." @@ -1364,8 +1580,10 @@ }, { "cell_type": "markdown", - "id": "d0feaf6a", - "metadata": {}, + "id": "19ff9038", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" @@ -1373,8 +1591,10 @@ }, { "cell_type": "markdown", - "id": "e2994fb3", - "metadata": {}, + "id": "8cbcc7e2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1383,16 +1603,20 @@ }, { "cell_type": "markdown", - "id": "a09153f7", - "metadata": {}, + "id": "e9ea4968", + "metadata": { + "editable": true + }, "source": [ "which is zero unless $i=j$." ] }, { "cell_type": "markdown", - "id": "96825360", - "metadata": {}, + "id": "412198c3", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", @@ -1401,8 +1625,10 @@ }, { "cell_type": "markdown", - "id": "697f4b69", - "metadata": {}, + "id": "4e53e367", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1411,8 +1637,10 @@ }, { "cell_type": "markdown", - "id": "06baeae1", - "metadata": {}, + "id": "c5c23589", + "metadata": { + "editable": true + }, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1421,8 +1649,10 @@ }, { "cell_type": "markdown", - "id": "10331baf", - "metadata": {}, + "id": "cb47cf57", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1431,8 +1661,10 @@ }, { "cell_type": "markdown", - "id": "9c07ca94", - "metadata": {}, + "id": "5bdec91c", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1440,8 +1672,10 @@ }, { "cell_type": "markdown", - "id": "def0608f", - "metadata": {}, + "id": "ce3082b7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1450,16 +1684,20 @@ }, { "cell_type": "markdown", - "id": "cc38bf53", - "metadata": {}, + "id": "8b1139e9", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] }, { "cell_type": "markdown", - "id": "fb4d2066", - "metadata": {}, + "id": "58d65744", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1468,16 +1706,20 @@ }, { "cell_type": "markdown", - "id": "7c605c08", - "metadata": {}, + "id": "85fe84cb", + "metadata": { + "editable": true + }, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] }, { "cell_type": "markdown", - "id": "27d71b17", - "metadata": {}, + "id": "8e4ea650", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1486,8 +1728,10 @@ }, { "cell_type": "markdown", - "id": "33a2ec8f", - "metadata": {}, + "id": "7fc2b5db", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1504,8 +1748,10 @@ }, { "cell_type": "markdown", - "id": "874a4bf8", - "metadata": {}, + "id": "a653e167", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1514,16 +1760,20 @@ }, { "cell_type": "markdown", - "id": "38431600", - "metadata": {}, + "id": "140a6cf3", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "a0efad46", - "metadata": {}, + "id": "0e842bf0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1532,16 +1782,20 @@ }, { "cell_type": "markdown", - "id": "4c79cfc4", - "metadata": {}, + "id": "2d539424", + "metadata": { + "editable": true + }, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "e5be0110", - "metadata": {}, + "id": "4027b150", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1549,8 +1803,10 @@ }, { "cell_type": "markdown", - "id": "f9840d01", - "metadata": {}, + "id": "51bb612d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1559,8 +1815,10 @@ }, { "cell_type": "markdown", - "id": "2cd5295a", - "metadata": {}, + "id": "3eabe463", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1569,8 +1827,10 @@ }, { "cell_type": "markdown", - "id": "df1c1121", - "metadata": {}, + "id": "5680938b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1579,8 +1839,10 @@ }, { "cell_type": "markdown", - "id": "e0ee3be5", - "metadata": {}, + "id": "9342de43", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", @@ -1590,8 +1852,10 @@ }, { "cell_type": "markdown", - "id": "4c6749b3", - "metadata": {}, + "id": "dc6b0289", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" @@ -1599,8 +1863,10 @@ }, { "cell_type": "markdown", - "id": "93a6c073", - "metadata": {}, + "id": "d662d6cb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1609,8 +1875,10 @@ }, { "cell_type": "markdown", - "id": "64e5b77a", - "metadata": {}, + "id": "a1dfc17e", + "metadata": { + "editable": true + }, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1623,8 +1891,10 @@ }, { "cell_type": "markdown", - "id": "7948355f", - "metadata": {}, + "id": "e7217dd4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1633,8 +1903,10 @@ }, { "cell_type": "markdown", - "id": "b203f48c", - "metadata": {}, + "id": "fc97e4f1", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1642,8 +1914,10 @@ }, { "cell_type": "markdown", - "id": "7e4420a7", - "metadata": {}, + "id": "d68b2934", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1652,16 +1926,20 @@ }, { "cell_type": "markdown", - "id": "8e6faf5a", - "metadata": {}, + "id": "7be1c818", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "36a01503", - "metadata": {}, + "id": "71ce7281", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1670,16 +1948,20 @@ }, { "cell_type": "markdown", - "id": "7134e172", - "metadata": {}, + "id": "cbf9ce5a", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "b4a913e5", - "metadata": {}, + "id": "d7960c30", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1688,16 +1970,20 @@ }, { "cell_type": "markdown", - "id": "5a3c105a", - "metadata": {}, + "id": "cffbbb79", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "341d2ab4", - "metadata": {}, + "id": "718a0a6f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1706,8 +1992,10 @@ }, { "cell_type": "markdown", - "id": "617425bb", - "metadata": {}, + "id": "4db7f5dc", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our first homework\n", "\n", @@ -1728,8 +2016,11 @@ { "cell_type": "code", "execution_count": 9, - "id": "76c38d50", - "metadata": {}, + "id": "27af777d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1738,8 +2029,10 @@ }, { "cell_type": "markdown", - "id": "a4f55bbd", - "metadata": {}, + "id": "b5400386", + "metadata": { + "editable": true + }, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1747,8 +2040,10 @@ }, { "cell_type": "markdown", - "id": "c9072fa0", - "metadata": {}, + "id": "24ed3756", + "metadata": { + "editable": true + }, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1757,16 +2052,20 @@ }, { "cell_type": "markdown", - "id": "6e55a2c7", - "metadata": {}, + "id": "00d1f306", + "metadata": { + "editable": true + }, "source": [ "such that" ] }, { "cell_type": "markdown", - "id": "41bacc77", - "metadata": {}, + "id": "f30c3b64", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -1775,8 +2074,10 @@ }, { "cell_type": "markdown", - "id": "dc8223ad", - "metadata": {}, + "id": "165a54d8", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent example\n", "\n", @@ -1787,8 +2088,10 @@ }, { "cell_type": "markdown", - "id": "a8b03557", - "metadata": {}, + "id": "a9891168", + "metadata": { + "editable": true + }, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -1801,16 +2104,20 @@ }, { "cell_type": "markdown", - "id": "63fbae73", - "metadata": {}, + "id": "7d7d1698", + "metadata": { + "editable": true + }, "source": [ "The cost/loss/risk function is given by (" ] }, { "cell_type": "markdown", - "id": "4ad922b7", - "metadata": {}, + "id": "8f31115e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -1819,16 +2126,20 @@ }, { "cell_type": "markdown", - "id": "190ae31f", - "metadata": {}, + "id": "c0eb6ac5", + "metadata": { + "editable": true + }, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] }, { "cell_type": "markdown", - "id": "ffff543c", - "metadata": {}, + "id": "adff23ee", + "metadata": { + "editable": true + }, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -1837,8 +2148,10 @@ }, { "cell_type": "markdown", - "id": "92aa8f9d", - "metadata": {}, + "id": "071987fc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1849,16 +2162,20 @@ }, { "cell_type": "markdown", - "id": "7654f8fa", - "metadata": {}, + "id": "384bc1a2", + "metadata": { + "editable": true + }, "source": [ "where $X$ is the design matrix defined above." ] }, { "cell_type": "markdown", - "id": "703447b3", - "metadata": {}, + "id": "f2d21cdc", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -1866,8 +2183,10 @@ }, { "cell_type": "markdown", - "id": "856d4fd1", - "metadata": {}, + "id": "1e8bca50", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1879,16 +2198,20 @@ }, { "cell_type": "markdown", - "id": "a05f6979", - "metadata": {}, + "id": "f9fca584", + "metadata": { + "editable": true + }, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] }, { "cell_type": "markdown", - "id": "a6df9f0f", - "metadata": {}, + "id": "0860c223", + "metadata": { + "editable": true + }, "source": [ "## Simple program\n", "\n", @@ -1897,8 +2220,10 @@ }, { "cell_type": "markdown", - "id": "a643f0c7", - "metadata": {}, + "id": "130d7405", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -1907,8 +2232,10 @@ }, { "cell_type": "markdown", - "id": "86c2a1fb", - "metadata": {}, + "id": "122ab1a1", + "metadata": { + "editable": true + }, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -1920,8 +2247,10 @@ }, { "cell_type": "markdown", - "id": "f6eb295f", - "metadata": {}, + "id": "cd9cc831", + "metadata": { + "editable": true + }, "source": [ "## Gradient Descent Example\n", "\n", @@ -1931,8 +2260,11 @@ { "cell_type": "code", "execution_count": 10, - "id": "14bf489d", - "metadata": {}, + "id": "d927a289", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -1985,8 +2317,10 @@ }, { "cell_type": "markdown", - "id": "fa4674f0", - "metadata": {}, + "id": "6c19826d", + "metadata": { + "editable": true + }, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -1994,8 +2328,11 @@ { "cell_type": "code", "execution_count": 11, - "id": "1eef9dcb", - "metadata": {}, + "id": "f6b889bd", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -2018,8 +2355,10 @@ }, { "cell_type": "markdown", - "id": "92ad46a0", - "metadata": {}, + "id": "21739cb2", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -2028,8 +2367,10 @@ }, { "cell_type": "markdown", - "id": "0bb24dc6", - "metadata": {}, + "id": "c05f8428", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -2038,16 +2379,20 @@ }, { "cell_type": "markdown", - "id": "10a9fcb2", - "metadata": {}, + "id": "6c8ced5e", + "metadata": { + "editable": true + }, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] }, { "cell_type": "markdown", - "id": "495dbade", - "metadata": {}, + "id": "14452016", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2058,16 +2403,20 @@ }, { "cell_type": "markdown", - "id": "4b263196", - "metadata": {}, + "id": "542f52f5", + "metadata": { + "editable": true + }, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] }, { "cell_type": "markdown", - "id": "2a0b4fb5", - "metadata": {}, + "id": "f5c485ed", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -2076,8 +2425,10 @@ }, { "cell_type": "markdown", - "id": "a2b60831", - "metadata": {}, + "id": "1e4bbe6a", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -2085,8 +2436,10 @@ }, { "cell_type": "markdown", - "id": "a82f529c", - "metadata": {}, + "id": "8395f779", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2098,54 +2451,37 @@ }, { "cell_type": "markdown", - "id": "8fe1c41e", - "metadata": {}, + "id": "57327b35", + "metadata": { + "editable": true + }, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", - "Note that the Ridge loss function is convex, as a sum of two convex\n", + "Note that the Ridge cost function is convex being a sum of two convex\n", "functions. Therefore, the stationary point is a global\n", "minimum of this function." ] }, { "cell_type": "markdown", - "id": "a03ad61b", - "metadata": {}, + "id": "86d6e4b0", + "metadata": { + "editable": true + }, "source": [ "## Program example for gradient descent with Ridge Regression" ] }, { "cell_type": "code", - "execution_count": 13, - "id": "f0a079df", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Eigenvalues of Hessian Matrix:[0.29294218 4.47046976]\n", - "[[4.03410523]\n", - " [3.03026348]]\n", - "[[4.03408456]\n", - " [3.03028067]]\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnoklEQVR4nO3de5xdZX3v8c9vJhc6EMVMoiIwE/GO1AtE8NY2nlZFxNrTK3FApJYUPG2xB6sCp4hKqK22ij3H0ogUNSE9vvBS5WjFKqCoIAOCgsEbmUwiAiFckyBJJr/zx1qTrL1nrb2fvffa6zLzfb9e+5XZe91+e+2V9VvreZ71PObuiIiItDNQdgAiIlIPShgiIhJECUNERIIoYYiISBAlDBERCaKEISIiQZQw5jgzmzCz34n/PtfMLi0pjhVmtqWMbc8Vyd+6i2XPNLN7zWy7mQ3nHVvK9i4xs79tMd3N7Jn9jkMaKWFUmJmdZGY3mtkOM7sv/vttZmb92J67X+Tuf9bresxsWfwfel4ecZXNzC43swvLjqMsZjYf+CfgNe5+kLtvy2GdE2b2WJyA7on38UHT0939DHd/f6/bkXwpYVSUmZ0NXAx8EHgq8BTgDOAVwIKMZQYLC1DmkqcABwB3dLqgRbLOM29w94OAFwEvBs7pOkIphBJGBZnZE4H3AW9z9yvd/VGPfN/dx9z98Xi+y83sX8zsy2a2A3iVmb3ezL5vZo+Y2WYzu6Bp3aeY2SYz22Zm5zVNu8DM1ibev9TMvmNmD5nZbWa2IjHtWjN7v5l928weNbOrzWxJPPmb8b8PxVeQL0v5jr8Wx/+gmf0IeEnT9KeZ2WfNbKuZbTSzv0pMO9bMxuPveK+Z/VNi2isTMW82s7fEny80sw+Z2WS8zCVm9mvxtBVmtsXMzo7v5H5pZqfF01YBY8A74+/ypYzf7Llm9jUze8DMfmxmfxx//oz4s6MT3+v+6X1pZqeZ2YZ4H95lZn+eWOd0XO9MxPV7ZnaCmf0kXu+5Tb/flWb2f+P13WJmL8yId8DM3m1mP4+Phc+Y2eKU+Z4N/Djxe34j/vzlZnaTmT0c//vyxDLXmtlqM/s2sBM4Ii2Gae5+D/BVosQxvY6Guzoz+5v4+99tZn/aFOOwmX0pPh5uMrMLzez6dr+NdMHd9arYCzge2APMazPf5cDDRHcdA0RXgSuAX4/fvwC4F/i9eP4jge3AbwILiYoZ9gC/E0+/AFgb/30osA04IV7Xq+P3S+Pp1wI/B54N/Fr8/gPxtGWAt4of+ADwLWAxcDhwO7AlnjYA3AycT3Q3dQRwF/DaePp3gVPivw8CXhr/PQI8CqwE5gPDwIviaR8BvhhvbxHwJeDv4mkr4v3wvni5E4hOdE9K7OcLW3yXA4HNwGnAPOBo4H7g+fH004ENwBDRifFDiWVfDzwDMOC34u0e3RTX+XFcpwNbgSvi7/B84FfAEYnfbzfwh/H87wA2AvPj6ROJ3/rtwA3AYUTHwr8C6zO+X8PvGe/DB4FT4u+7Mn4/nDg2JuP45k1vv2mdyVgOA34IXNx0bF+Y+P9wL3BUvK+viON5Zjz93+PXENExvhm4PuS30avDc1PZAeiV8qPAycA9TZ99B3gIeAz4zfizy4FPtVnXR4APx3+fD/x7YtqBwC7SE8a7gE83reurwKnx39cC/ysx7W3Af8Z/N5xgMuK6Czg+8X4V+xPGccBk0/znAP8W//1N4L3AkpR5Pp+yLQN2AM9IfPYyYGP894p4v85LTL+P/Ylo38kr47v8CfCtps/+FXhP4v0X45PiD4CFLdb1BeCsprgG4/eL4v16XGL+m9l/QXABcENi2gDwS+A34vcTid96A/DbiXkPIUo2M36z5t+TKFF8r2me7wJvSRwb72tzXE4QXbw8Gq/768DBien79jlwGfHFSPz+2fEyzwQG47ifk5h+IfsTRtvfRq/wl4qkqmkbsMQSlcbu/nJ3PzielvzdNicXNLPjzOyauCjnYaJ6j+mioqcl53f3HfH60owCfxQX7TxkZg8BryQ6sUy7J/H3TqKr/VANsQCbmrb9tKZtn0tUlg7wVqKTxp1xEcSJ8eeHE931NFtKdPV5c2J9/xl/Pm2bu+/p8vuMAsc1xTtGVPc07eNEV8j/7HGRIoCZvc7MboiLSx4iurtZklhum7tPxX8/Fv97b2L6Y01xJn/fvcAWon2dFvPnE/FuAKbYv49beRqNvxfx+0PT4mjh99x9EVFifC6N37t5e1nHylKiO4fk9OTfIb+NBFLCqKbvAo8DbwyYt7m74SuIrmYPd/cnApcQXWFDdLV5+PSMZjZEVGyTZjPRHcbBideB7v6BLmJK0xALUXFSctsbm7a9yN1PAHD3n7r7SuDJwN8DV5rZdNHDM1K2dT/RifX5ifU90aMK1xDtvs9m4LqmeA9y9zMBLGr98xHgE8AF03UFZrYQ+CzwIeAp8QXBl9n/e3Uj+fsOEBX33J0R8+uaYj7A3X8RsI27iU7ESSNActngbrDd/TqiO4oPZczS6ljZSlRsd1jis+S8LX8b6YwSRgW5+0NERS4fM7M/NLOD4krKFxEVI7WyCHjA3X9lZscCb0pMuxI4Ma4YXkBUZp91DKwF3mBmrzWzQTM7IK6EPSxj/qStwF5aV3Z+BjjHzJ4Ur/MvE9O+BzxiZu+yqHJ80MyOMrOXAJjZyWa2NL6CfiheZgpYB/yOmf2xmc2LK0NfFM/3ceDDZvbkeB2HmtlrA74LRFf0rb7LVcCzLWpQMD9+vcTMnhdPvxi42aMmy/+PKIlDVD+zkPikZ2avA14TGFOWY8zs9+O707cTXXjckDLfJcBqMxsFMLOlZhZygQJRUnu2mb0p3s9/QlR3cFUPcX8EeHV8jDf7DPAWMzsyvsh5z/SE+O7rc0SJeMjMngu8ObFsu99GOqCEUVHu/g/A/wTeSVSefi9R2eu7iOozsrwNeJ+ZPUpUZ/GZxDrvAP4H0V3IL4kqKlMflnP3zUR3OOcSndA2A39DwDHj7juB1cC342KAl6bM9l6iooWNwNXApxPLTwFvIGo1s5HoDuFS4InxLMcDd5jZdqKT8Unu/it3nyQq0jkbeAC4FZhuJfQu4GfADWb2CPBfwHPafZfYJ4Aj4+/yhZTv+yjRif4koqvve4jufBbGJ+HjiYoGIfpNjzazsXi5vyL6jR4kSu5fDIwpy38QldtPV0r/vrvvTpnv4nhbV8fHyg1EdUdtefQcxolE+3kb0TF6orvf323Q7r4V+BQw42E9d/8KUUL5BtFv+I2mWf6C6Ni4h+g4Wk+UKFv+Nt3GOpeZuwZQEpkNLGpC/Ux3P7nsWMpkZn8PPNXdTy07ltlGdxgiUmvxcxYvsMixRI0iPl92XLPRrOi6QUTmtEVExVBPIyq+/UeiojnJmYqkREQkiIqkREQkSK2KpJYsWeLLli0rOwwRkVq5+eab73f3pe3nbK1WCWPZsmWMj4+XHYaISK2YWfOT+V1RkZSIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSB9TxhmdpmZ3Wdmt6dMe4eZuZllDf4uIiIVUcQdxuVEQ1Q2MLPDgVcDkwXEICIiPep7wnD3bxKNr9zsw0RjAWtADhGRGiilDsPMfhf4hbvfFjDvKjMbN7PxrVu3FhCdiIikKTxhmNkQcB5wfsj87r7G3Ze7+/KlS3vuzl1ERLpUxh3GM4CnA7eZ2QRwGHCLmT21hFhERCRQ4QMoufsPgSdPv4+TxnJ3v7/oWEREJFwRzWrXA98FnmNmW8zsrf3epoiI5K/vdxjuvrLN9GX9jkFERHqnJ71FRCSIEoaIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSBKGCIiEqTvCcPMLjOz+8zs9sRnHzSzO83sB2b2eTM7uN9xiIhIb4q4w7gcOL7ps68BR7n7C4CfAOcUEIeIiPSg7wnD3b8JPND02dXuvid+ewNwWL/jEBGR3lShDuNPga9kTTSzVWY2bmbjW7duLTAsERFJKjVhmNl5wB5gXdY87r7G3Ze7+/KlS5cWF5yIiDSYV9aGzexU4ETgt93dy4pDRETClJIwzOx44F3Ab7n7zjJiEBGRzhTRrHY98F3gOWa2xczeCvxvYBHwNTO71cwu6XccIiLSm77fYbj7ypSPP9Hv7YqISL6q0EpKRGS/detg2TIYGIj+XZfZJqY66hhzF0qr9BYRmWHdOli1CnbGVZubNkXvAcbGyourlTrG3CWrUwOl5cuX+/j4eNlhiEi/LFsWnXCbjY7CxETR0YSpQcxmdrO7L+91PSqSEpHqmJzs7PMqqGPMXVLCEJHqGBnp7PMqqGPMXVLCEJHqWL0ahoYaPxsaij6vqjrG3CUlDBGpjrExWLMmKv83i/5ds6balcedxNxpa6rp+c1g3rzo3xJbYanSW0SkCM2tqSC6E2mVXJrnn7ZgASxaBA88EBV9rV7dMqnmVemthCEiUoROW1NlzZ+mVeJBraREROql09ZUnbSy2rkTzjuv85g6pIQhIlKETltTddrKqoBmvEoYIiJF6LQ1Vdr8rSQTTFPl+hJY3HG8KZQwRESKENKaKnmiP+88OPXUaD6AwcHo3+FhmD+/cd3JxDNdWb5pE7jDpk2MwGgeX0GV3iIiVdBJK6p166KEMjk5s5VUSmX5cmDc3XoNUQlDRKQK8uqTamAgurNIyCthqEhKRKQK8uqTqo9dkihhiIhUQV6tpVIqyx32dhlVAyUMEZEqyKtPqpTK9UkIfAKwNSUMEZEqyLMfrbGxqN5j716YmOB+eCCPEJUwRGR2q9PwqU0n+qp1utj3hGFml5nZfWZ2e+KzxWb2NTP7afzvk/odh4jMQSnPJLBqVbWTRoUVcYdxOXB802fvBr7u7s8Cvh6/F5n96nS1m6Zu8Z933szeXovod6mf+6nFunfv3M3H3/xNlg7cjxmc87Jr89sugLv3/QUsA25PvP8xcEj89yHAj0PWc8wxx7hIba1d6z405B5d60avoaHo8zqoY/xmjfFOv8z6t81O9tPate6jo1E8o6Pt92XKunfPP8DfNe+DqV/znJdd4+7uwLjncS7PYyVtNzIzYTzUNP3BFsuuAsaB8ZGRkdY7U6TKRkfTT16jo2VHFqaO8ZcRc+g2u0nAGeveyOi+t8cv+Z6Pf/pHDYvllTAKedLbzJYBV7n7UfH7h9z94MT0B929bT2GnvSWWkt5AheIWsTszaWZfH/VMf5OBy3KQ+h+Cnyy+/VPvokvb30JAFMMMMDMdTsGU1PYQPrD3HUfD+NeMzsEIP73vpLiEClOXg9mlaWO8XfbVLWXOojQ/ZTxBLdvmsSMfa/pZAEwSfq6bXQkM1nkqayE8UXg1PjvU4H/KCkOkeLk9WBWWeoaf6dNVXttWRW6nzISy6aMpHDdR29j2dqSf4M8yrVavYD1wC+B3cAW4K3AMFHrqJ/G/y4OWZcqvaX2Oq3krJq6xx8ij3qPM890HxyMlhscjN7HfvXwrxzcV7LWt9NYh7GdIV/JWgf3Vc+9Ln3dXfwG1KkOIy+qwxCRvuu1rial3mQHQ5zOGtbTeHezknVcxHmMMMkkI9EdRB/qVvKqw1DCEBFJ6rWb8YzlJxjl6TQuX9Tpt+6V3iIi1dRFXc2xB92xr5J676b0yuwRJrnvR/fvL39aW7OHIFHCEBFpFNCy6sqzv9vQkummHc/fNy2rJdPA6AhLn7ckelPTLktUJCUi0sYDP3+Q4We27/JuZHALmz55XftnP/IaXS+QiqRERFrpsT+n5B1Eq2SRbOa0ac9hYc9+5DW6XsHmlR2AiEjumlsqTRf5QGYrJAt87i2oUGZsrHVrp5GR9DuMKj8Eie4wRGQ2CuilNnkH0SpZPLz5kYa7iFzU9CFIJQwRyU9Vuj/PKNrZm+h2I8un/vz6hgTxhMOekH98eY6uVyAlDJG5Ku+Te5Va/mQU7aS1YBpgqiFBnHLJK/sdXaTio+ulUcIQmYv6cXIva7CimO/1fXcPb9q0mh00FvnsYIhziYp8kgliygcLiW82UMIQmZbHFXdVimTa6cfJvZOWPzntp2QdxMDg/nKm9YxxOmuYYJS9REU+B65dwxU+VtjT1bNSHh1SFfVS54PSN3mMJlenEen6MRJdXgMHtehcL231aa/HH328++8xC1GnEffyeilhSN/k0UNpnUak60esoQmz1bZT1pHswTXr9YVzbug+7jkgr4ShIikRyOdBqjo9jNWPZp2hLX9a7KftZ5w9o6jsQHZyEY1FZYtobOr6xouO6z5uqE9RYsmUMEQgn9Hk6jQiXb+adYa0/MnYHxM+wtD29ME3R5hsSBCPeI5NXavUuqvilDBEIJ8r7ro9jFVSs85WLZhaddzXNyW37qoTJQwRyOeKu6YPY+Umo1in+Ynq5hZME4xyOmtYN/Wm8CFI8yxCqlNRYtnyqAgp6qVKb5GSpLVcmv4M3AcGZtREt6usvv5jt4Vvq3l6nq3R6tRYoUtoiFYRKUTKkKPMnx/dLuza1XLR5ChzT583yV27cyhayrtr8LTv19wdec2pe3MRKUZaGf/u3W2TBcAy219ZnUuygPyLkOZ6UWIHSk0YZvbXZnaHmd1uZuvN7IAy4+krNduTuuqlLL8fLcT60Rqthv06laG0hGFmhwJ/BSx396OAQeCksuLpq7o026tLUqtLnDWWrKSe8B5OxNu35/871a01Wq+qdLznURHSzQs4FNgMLCYayOkq4DWtlqltpXcdKtXq0q1FXeKsmVZPUa9krW+naZ/Pn+8+ONh6weZXnr9Tu4rx2SKn453Z0DUIcBawHdgKrMuYZxUwDoyPjIx0tJMqox/99uStDknNvT5xliXwRBp6jj9y4U/T13vmme4LFnSWMPQ7dS6n4732CQN4EvANYCkwH/gCcHKrZXSH0Ud1SGruxcRZhavXbmLIuho980zfefBTfQrzjYz6P3Omb2R03/vmpq9Bso7pdq+Q36kK+78qcjreZ0PC+CPgE4n3bwY+1mqZ2iaMOhSj1CGpufc/zir8Vt3GkLFvpmg86ezNo6go60Q2/VsMD3f3O1Vh/1eJ7jD2JYjjgDuAIcCATwJ/2WqZ2iYM9+pfNdXlP2q/46xC4uwyhubE0NGr0+/XLsack17lLlyKojqMhqTxXuBO4Hbg08DCVvPXOmHUQdWT2rR+xtmqCKCo/RNYDNE8eSOj3SeMTov0Qk5k3eyvuhSNFimH425WJIxOX0oY0ndZV7jDw8XdgWUU59zHcMtzfmprplZFR71ewfcjgeoOoy/yShhtn8Mws/8ysxd23mBXpMKy2rZntfGHSvdo6g5X+BgHrm16YvmMM2Z+n2bdPsPQj4fdTjihs8/7rUrPQFRBu4wCHE3UmunfgEPyyFLdvnSHIbnoZojQAopKdu3Y5ZBdFzGF7Xvb8fdtbhJb1aLHKt1h1KVeLwBFdz5oZn8AnA98DvgHd3+sPyksmzoflFx003ld3h3excxmfraRZSwj/231ZN266G5qcjLqgmP16v50nzEwEJ2am5lFdzJF6tNvXoZCOx80MwN+DPwL8JfAT83slF43LlKKbjqvy6k7iuaxIdKcy2q8Sl1fFNm1TZVGLdQ4GTOE1GFcD/wC+DBRdx5vAVYAx5rZmn4GJ9IX3ZyU0no0PfXU6Kq7Rfl2SIIAuOy0b+0r97jCx7B+9J7abXl8kSPSVaWfqHXrov2UpopD7halXZkVcBRERVcp0zbkUS4W+lIdhuQij7LpjHV8avS84JasherlOxfd1LXs5t1p+0p1GJ3VYaQxsyPc/a5cMlcA1WFIbnotk88o304OGNSsh/9qveulPH4WleUHyfq+g4PwyU/WsuvzvOowNOKeSIfMYIoBBpj5f8eBMdaynrFyE0SzXiqT58CIdA2qVPGeE424J1KQtHqISdLLsQ24YmgVvjauH+hXO/5O19tLZXIdRqTLcz9XqeK9avIo1yrqpToMKUJI/cNK1vqe+QdkzzBd9t6PdvzdrDdtmem6ieHh6FXF5zJC5L2fZ9HzF9NQ1yBSO2VVZrbZ7jPnbwxKEgewc+Z6s2ae3lY/HkLrdr3T+yGZLGZDxW4/9nPZFe85yythqA5DilFkOXiyQnvxYnjkEdi9e9/kvQsP4OTHL2U97bfb9r9Hqwrhycn+lIX3WsaeFXNSnSq0Z2GdQ95Uh1EE9SOTn6La8jc/ZLZtW0OyABh4/FdcRPp2my9T22r13EC/ysJ7XW/Ig2d1ejhNdQ7FyeM2pahXoUVSs7AcszTtim66WV9WcUHgSHDT/TLl9v3S4qlSHUZSyD7qR99N/Srm0f/VtlAdRp9VqRO0Omv1EFQ3+zPj5DA9zGhHgwgVUTadx0kybR29rLfdb9KPk22/T+qzrM4hb0oY/VblgVzq9J+j1dVsNyeMjPVtZNShi0GEqn4l2s+7lOljqIhWUroAK5USRr9V9QDvxwmkjBHsoKPtfOlvb2x5BzFdxLSSNlfPVfxNW+nkOKzyhUSVL8DmACWMfqtquWjeiayiY2RP7Z5KXSzzDiK5vmTz0ZBXlU9aoSfaqh6v06p6ATZHKGEUoYpXbHlfqfX7P3IHJ7KQc3vqHUTWibFdWX0dTlqhv0/VT8hVT2iznBLGXJX3iaGIooKMEd+mMN/I6L4K66zXzm0pD8wlx70eHs4+8TSX1c+fX6+TVuiJtg5FPlW8AJsjZkXCAA4GrgTuBDYAL2s1vxKG53+lVuCV6fQdwnYa499O1MppJWt9I6O+lzYnlF72QR1PWiExV/0OQ0o1WxLGJ4E/i/9eABzcan4ljFjoSS9kvj4WFVz8B9fOOH9l1kEMD4fHoZPjTCrykRZqnzCAJwAbyRicKe2lhNGBTk4gOV1179i6o211QUfPSWQlgToUv5ShjndPUoi8EkZpfUmZ2YuANcCPgBcCNwNnufuOpvlWAasARkZGjtnUrg8ciRQ06E2rYUeT9h1mIf0YNW+guT+guTagj0iPZkNfUvOAo4F/cfcXAzuAdzfP5O5r3H25uy9funRp0THWV58GsA8do3rvlDdc/u+T1ffS8HD6itL6A6rKuM8ic0yZCWMLsMXdb4zfX0mUQOqtKh0W5tQhW2iC+MY/fr8hQdhAxsxZg/FcfHF4EqjDgD4is1Ee5VrdvoBvAc+J/74A+GCr+Stfh1GliscuYznrRTMrqtNeT7IH+hOzyuBFckfd6zBgXz3GpUQtpO4CTnP3B7Pmr/x4GFUrW0+OCzEyEl2tN12Fb91wP08+cknQ6ko8VESkB3nVYWgApTzVZCCXjiuqyxSQ9ESktdlQ6T37VHQgl9B6iOaCp9I1D4a0aVP0XgNZiZRCCSNPFWm9E5ogNt94d3aCqELlfVGj9IlIECWMPJXUeue0Z10flCDesfzahgRx2LFPS5+xKlf2fWoaLCLdqV/CqMKVb8l+cvVEQ4K4/GevzJw3mSA+eNOKsA1U5cq+okV8InNVvSq9jzjCx++9t/FkNjRUnTb401fmOcfnHuXH0Hl7VpXK+z7tT5G5Zm62klq40Md37Zo5oSpdQuTYrLbUlkxVah6sVlIiPZubraTSkgWUV6bdXDyW1UfSpk1ti9FCK6p3bN3Z/5ZMFam8B6LkMDER3dlMTChZiJSoXgljwYL0zwcGiq/TSKsYzjrTm82oQL548QVBCeKq99zUkCCGlgxlz5wXdb0hIinqlTAOPXTmlS/A1FS+rXlCKtbTKobdZ579zWbeCuzcyRsfvDx10y884M6GBPH6C17S7bcIl/Z9dWUvIk3qlTAWL2688h0cnDlPr615QpuUZhWDucPoKG7GBKPszSg3GmGyYZHp162PPbf72LtRlSa0IlJ59ar0bu4apB+teUIrfDPmm2CUc1nNRZzHCJPsZYB5TLVfX1mqVMEtIn0xNyu9H3igsehk8eL0+Xpppx/6sNjq1eygsXhsB0NcxQl8nFUsYxMDOPOYYkZKq9LYDXo4TkQC1SthbNrUWHTy6KMwf37jPL2ejFs8LPb2F1+3v6L65DFOZ01U7ERU/HQ6aziRL3MgjXUbBlHxWbICGarxAGLW93Wfsw9GikiGPPpIL+p1TNrADMPD+Y6hkDKOxHaGfCVrM8eG+OgfXLN/+ZDxpqs+bkZz3GeeWXxcIpIbZsN4GJ1abuYzOjfP8enj3Tt3s+DA+axk3b46iElGOJfVrGd/K6HDB3/B5J5D01cSUidQtXqD6Yfjsp4jMYNPf1otpURqam4+6Z2WMHo8yeb+RHVIdxZV6XqjWVZcoEpwkRqbm5XezR0qdVFf8byFP+/v2BAhD71VtVO9VttXJbjInFevhDE62vHTx39/wrUNCeLOXc9Ine/hzY/k1+VGu4feqtT1RtLq1dlZdGBAFeAic1y9EsbixW2fPr5l3YaGBPHur6xIXdUNl97ekCCecNgT+hp6g6p2vTE2BmeckZ40pqb0QJ/IHFevhJFi1/ZdDQnimJOflzrfe1dc05AgjnvrUeEb6ccYHFXteuNjH4squPvxFL2I1FrpCcPMBs3s+2Z2Vegyf/sb+4uZFi5K75Dw8MFfNCSI8695VXcBltV1RpkDRY2NZVe+qy5DZM4qPWEAZwEbQmc2gwuvX5E6LZkgMpu9dqqM0edaJamiEklVK+ZFpDSlJgwzOwx4PXBp6DIXveZaAA4ZuIfdj+3Jr6I6SxldZ2QlqbPOKu5up6oV8yJSmrLvMD4CvBPIfPjAzFaZ2biZjW/dupVzvroCd7h76qnMO2Be/yMs40o7Kxlt21bc3U5VK+ZFpDSlJQwzOxG4z91vbjWfu69x9+Xuvnzp0qUFRZdQxpV2p8lo06bui6haFXG1q5gvs55FRIqXR/8i3byAvwO2ABPAPcBOYG2rZY455pio76M8+44KUfQ2s/qaGh7O7vOpmz6peunTqkr9YYlIS8ymvqTMbAXwDnc/sdV8y484wsfvvbd1txuzxXT/TpOT0R3H9B1Nc7cjaUK78eilT6uq9YclIplmVV9SwQlj4UIf37Vr5oS5dJJKJpKs3y60T6pe+rSqan9YIjLDrOpLyt2vbZcsAEhLFhCdPOdKeXqyXmF0NH2e0DqQXir01exWZM6pRMIItiD9IT0WL56b41L3WiHfy/Jqdisy59QrYRx66MyTlFmxzU2rpNemr70sr2a3InNOJeowQi1fvtzH//qv9w/2Y9b6iT2Vp4uIzK46jI5Ml+GPjrZ/vFvl6SIiualfwpjWrmsOlaeLiOSqvgmj1d2DytNFRHJX34SR1Upn7dpqjS8hIjJL1DdhqJWOiEihCujutY/GxpQgREQKUt87DBERKZQShoiIBFHCEBGRIEoYIiISRAlDRESCKGGIiEgQJQwREQmihCEiIkGUMEREJIgShoiIBFHCEBGRIKUlDDM73MyuMbMNZnaHmZ1VViwiItJemZ0P7gHOdvdbzGwRcLOZfc3df1RiTCIikqG0Owx3/6W73xL//SiwATi0rHhERKS1StRhmNky4MXAjSnTVpnZuJmNb926tfDYREQkUnrCMLODgM8Cb3f3R5qnu/sad1/u7suXLl1afIAiIgKUnDDMbD5Rsljn7p8rMxYREWmtzFZSBnwC2ODu/1RWHCIiEqbMO4xXAKcA/83Mbo1fJ5QYj4iItFBas1p3vx6wsrYvIiKdKb3SW0RE6kEJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiARRwhARkSBKGCIiEkQJQ0REgihhiIhIECUMEREJUmrCMLPjzezHZvYzM3t3mbGIiEhrpSUMMxsE/g/wOuBIYKWZHVlWPCIi0lqZdxjHAj9z97vcfRfw78AbS4xHRERamFfitg8FNifebwGOa57JzFYBq+K3j5vZ7QXE1qslwP1lBxFAceanDjGC4sxbXeJ8Th4rKTNhWMpnPuMD9zXAGgAzG3f35f0OrFeKM191iLMOMYLizFud4sxjPWUWSW0BDk+8Pwy4u6RYRESkjTITxk3As8zs6Wa2ADgJ+GKJ8YiISAulFUm5+x4z+wvgq8AgcJm739FmsTX9jywXijNfdYizDjGC4szbnIrT3GdUG4iIiMygJ71FRCSIEoaIiASpRMJo10WIRT4aT/+BmR0dumzBcY7F8f3AzL5jZi9MTJswsx+a2a15NXHrIc4VZvZwHMutZnZ+6LIFx/k3iRhvN7MpM1scTytkf5rZZWZ2X9bzPxU6NtvFWZVjs12cVTk228VZhWPzcDO7xsw2mNkdZnZWyjz5Hp/uXuqLqML758ARwALgNuDIpnlOAL5C9OzGS4EbQ5ctOM6XA0+K/37ddJzx+wlgSUX25wrgqm6WLTLOpvnfAHyjhP35m8DRwO0Z00s/NgPjLP3YDIyz9GMzJM6KHJuHAEfHfy8CftLvc2cV7jBCugh5I/Apj9wAHGxmhwQuW1ic7v4dd38wfnsD0bMlRetln1RqfzZZCazvUyyZ3P2bwAMtZqnCsdk2zoocmyH7M0ul9meTso7NX7r7LfHfjwIbiHrQSMr1+KxCwkjrIqT5S2fNE7JsXjrd1luJMvs0B642s5st6u6kX0LjfJmZ3WZmXzGz53e4bB6Ct2VmQ8DxwGcTHxe1P9upwrHZqbKOzVBlH5vBqnJsmtky4MXAjU2Tcj0+y+waZFpIFyFZ8wR1L5KT4G2Z2auI/lO+MvHxK9z9bjN7MvA1M7szvoopI85bgFF3325mJwBfAJ4VuGxeOtnWG4Bvu3vyiq+o/dlOFY7NYCUfmyGqcGx2ovRj08wOIkpYb3f3R5onpyzS9fFZhTuMkC5CsuYpsnuRoG2Z2QuAS4E3uvu26c/d/e743/uAzxPdEpYSp7s/4u7b47+/DMw3syUhyxYZZ8JJNN3yF7g/26nCsRmkAsdmWxU5NjtR6rFpZvOJksU6d/9cyiz5Hp/9rpgJqLiZB9wFPJ39lS/Pb5rn9TRW3HwvdNmC4xwBfga8vOnzA4FFib+/AxxfYpxPZf9Dm8cCk/G+rdT+jOd7IlFZ8oFl7M94G8vIrqQt/dgMjLP0YzMwztKPzZA4q3BsxvvlU8BHWsyT6/FZepGUZ3QRYmZnxNMvAb5MVNv/M2AncFqrZUuM83xgGPiYmQHs8agny6cAn48/mwdc4e7/WWKcfwicaWZ7gMeAkzw6iqq2PwH+O3C1u+9ILF7Y/jSz9UQtd5aY2RbgPcD8RIylH5uBcZZ+bAbGWfqxGRgnlHxsAq8ATgF+aGa3xp+dS3Rx0JfjU12DiIhIkCrUYYiISA0oYYiISBAlDBERCaKEISIiQZQwREQkiBKGiIgEUcIQEZEgShgiPYjHI3h1/PeFZvbRsmMS6ZfSn/QWqbn3AO+LO5p7MfC7Jccj0jd60lukR2Z2HXAQsMKjcQlEZiUVSYn0wMx+nWjks8eVLGS2U8IQ6VI8ctk6opHKdpjZa0sOSaSvlDBEuhCPtPY54Gx33wC8H7ig1KBE+kx1GCIiEkR3GCIiEkQJQ0REgihhiIhIECUMEREJooQhIiJBlDBERCSIEoaIiAT5/6qqV3GZBKQBAAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "execution_count": 12, + "id": "caf0e6e4", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "from random import random, seed\n", "import numpy as np\n", @@ -2201,8 +2537,10 @@ }, { "cell_type": "markdown", - "id": "fc6d73bc", - "metadata": {}, + "id": "993c191f", + "metadata": { + "editable": true + }, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -2221,8 +2559,10 @@ }, { "cell_type": "markdown", - "id": "40a565ce", - "metadata": {}, + "id": "cc175c37", + "metadata": { + "editable": true + }, "source": [ "## Challenge yourself\n", "\n", @@ -2231,16 +2571,20 @@ }, { "cell_type": "markdown", - "id": "13077723", - "metadata": {}, + "id": "6e8e9108", + "metadata": { + "editable": true + }, "source": [ "## Friday October 1" ] }, { "cell_type": "markdown", - "id": "3227c51a", - "metadata": {}, + "id": "f394fbb7", + "metadata": { + "editable": true + }, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2254,8 +2598,10 @@ }, { "cell_type": "markdown", - "id": "d414c535", - "metadata": {}, + "id": "55224423", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2265,8 +2611,10 @@ }, { "cell_type": "markdown", - "id": "0c25b0fc", - "metadata": {}, + "id": "b2e74eb7", + "metadata": { + "editable": true + }, "source": [ "## Computation of gradients\n", "\n", @@ -2276,8 +2624,10 @@ }, { "cell_type": "markdown", - "id": "23dca3ab", - "metadata": {}, + "id": "07f89fcc", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2287,8 +2637,10 @@ }, { "cell_type": "markdown", - "id": "3b169c06", - "metadata": {}, + "id": "6e739ebb", + "metadata": { + "editable": true + }, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2299,8 +2651,10 @@ }, { "cell_type": "markdown", - "id": "d7ce6406", - "metadata": {}, + "id": "abc794f3", + "metadata": { + "editable": true + }, "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", @@ -2319,8 +2673,10 @@ }, { "cell_type": "markdown", - "id": "0a528877", - "metadata": {}, + "id": "ec3cd60b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2332,8 +2688,10 @@ }, { "cell_type": "markdown", - "id": "0f93ca5e", - "metadata": {}, + "id": "f1ba24d4", + "metadata": { + "editable": true + }, "source": [ "## The gradient step\n", "\n", @@ -2342,8 +2700,10 @@ }, { "cell_type": "markdown", - "id": "1d87bd14", - "metadata": {}, + "id": "7d8ddcd7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2353,8 +2713,10 @@ }, { "cell_type": "markdown", - "id": "4181f46b", - "metadata": {}, + "id": "d7ec3248", + "metadata": { + "editable": true + }, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2365,8 +2727,10 @@ }, { "cell_type": "markdown", - "id": "af66da80", - "metadata": {}, + "id": "b1f5fefb", + "metadata": { + "editable": true + }, "source": [ "## Simple example code" ] @@ -2374,8 +2738,11 @@ { "cell_type": "code", "execution_count": 13, - "id": "83e51ad8", - "metadata": {}, + "id": "5f016a5c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2396,8 +2763,10 @@ }, { "cell_type": "markdown", - "id": "e73bde3d", - "metadata": {}, + "id": "d027c5cf", + "metadata": { + "editable": true + }, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2410,8 +2779,10 @@ }, { "cell_type": "markdown", - "id": "97d463c4", - "metadata": {}, + "id": "de22c064", + "metadata": { + "editable": true + }, "source": [ "## When do we stop?\n", "\n", @@ -2429,8 +2800,10 @@ }, { "cell_type": "markdown", - "id": "cc362c51", - "metadata": {}, + "id": "dc00cb26", + "metadata": { + "editable": true + }, "source": [ "## Slightly different approach\n", "\n", @@ -2450,8 +2823,11 @@ { "cell_type": "code", "execution_count": 14, - "id": "0716d374", - "metadata": {}, + "id": "b001ca3d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2482,48 +2858,23 @@ }, { "cell_type": "markdown", - "id": "f7480cca", - "metadata": {}, + "id": "f6733633", + "metadata": { + "editable": true + }, "source": [ "## Program for stochastic gradient" ] }, { "cell_type": "code", - "execution_count": 14, - "id": "440f74c2", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Own inversion\n", - "[[3.97284805]\n", - " [3.13660672]]\n", - "sgdreg from scikit\n", - "[3.99061711] [3.20487971]\n", - "theta from own gd\n", - "[[3.97284805]\n", - " [3.13660672]]\n", - "theta from own sdg\n", - "[[3.94799953]\n", - " [3.10740003]]\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAArHElEQVR4nO3de5xVdb3/8ddnhpsDKlcBgRlQvGGWKUfzlhwNstKsPFmGpYUHxV+nOvmrNDppGZ0udupnZUaWl5iyrGN1jlZgDndRUEEEvMNwE7nLHWZmf35/rD04bPZl7b3X3mvPzPv5eOwH+7Iun7Vm8f2s7/e71neZuyMiIpJLVdwBiIhI+6CEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGFIh2Vmt5nZtLjjiJKZjTGztXHHIZ2TEoaUlZmtMrO9ZrbLzDaY2X1m1ivuuEQkNyUMicNl7t4LOB14J3BLvOF0TmbWJe4YpH1RwpDYuPsG4O8EiQMAM7vZzF41s51mttzMPtzmt2vNbK6Z3WFm28xspZm9r83vI8xsVnLeGUD/tuszsw+a2TIz225mM83slDa/rTKzL5nZc2a228x+aWYDzeyvyeU9ZmZ90m1HazORmd1kZhvN7HUz+3Sb32ea2XWp29Hms5vZjWb2cnJdt5vZ8Wb2hJntMLPfm1m3lHV+1cw2J+Me3+b77sn9s9rM3jCzu83siJQ4v2JmG4B7zay/mf1vcp9sNbM5ZqZyQdLSgSGxMbOhwPuAV9p8/SpwAXA08A1gmpkNbvP72cCLBMnge8AvzcySv/0GeDr52+3ANW3WdSLwW+ALwADgUeB/UgriK4CxwInAZcBfga8ml1cFfC7L5gxKxjwEmAD8NFOCyeAS4EzgXcCXganAeGAY8DbgqpR19U+u6xpgqpmdlPztu8n4TwdGJqf5esq8fYE6YCJwE7CWYJ8MTG6vxguStJQwJA5/MrOdwBpgI3Br6w/u/pC7r3f3hLv/DngZOKvNvI3u/gt3bwHuBwYDA82sFvgn4D/cfb+7zwb+p818HwMecfcZ7t4E3AEcAZzbZpofu/sb7r4OmAM86e7Puvt+4GGC5rNMmoBvunuTuz8K7AJOyjJ9qu+6+w53XwY8D0x399fc/U2CxJW67tbtnAU8AlyZTJz/Cvy7u291953At4GPt5kvAdyanHdvMu7BQF0y9jmuAeYkAyUMicOH3P1IYAxwMm2ajszsU2a2ONlEsp3g7Lpt09KG1jfuvif5thdwLLDN3Xe3mbaxzftj23529wRBwhrSZpo32rzfm+Zzts75Le7e3ObznhzTp8pn3em281iCWkIN8HSb/fe35PetNrn7vjafv09Qw5tuZq+Z2c15xCydjBKGxCZ5dnwfwdk+ZlYH/AL4LNDP3XsTnG1bhkW09TrQx8x6tvmuts379QTNMCTXZQTNPesK34LQdhMU5K0GFbm8dNu5HthMkFxOdffeydfRyQsMWh1Se3D3ne5+k7sfR9AM90Uzu7jI+KSDUsKQuP0IGGtmpwM9CQq0TQDJjuO3hVmIuzcCi4BvmFk3MzufoABs9XvgA2Z2sZl1JWi73w/Mj2g7slkMfMTMasxsJEEfR7Fat/MC4FLgoWSt6RfAD83sGAAzG2Jm7820EDO71MxGJhPoDqAl+RI5jBKGxMrdNwEPELTJLwd+ADxB0CRzGjAvj8V9gqBTfCtBv8gDbdbzInA18GOCM/HLCC7vPRDBZuTyQ+AAwTbdD9QXubwNwDaCWkU9cIO7v5D87SsETUwLzGwH8BjZ+1JOSE6zi2C/3+XuM4uMTzooU/+WiIiEoRqGiIiEooQhIiKhKGGIiEgoShgiIhJKuxp8rH///j58+PC4wxARaVeefvrpze4+IPeU2bWrhDF8+HAWLVoUdxgiIu2KmTXmnio3NUmJiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISSskThpn9ysw2mtnzaX77v2bmZta/1HGIiEhxylHDuA+4JPVLMxsGjAVWlyEGEREpUskThrvPJnjGcqofAl8G9IxYEZF2IJY+DDP7ILDO3ZeEmHaimS0ys0WbNm0qQ3QiIpJO2ROGmdUAk4Gvh5ne3ae6+2h3Hz1gQNHDuYuISIHiqGEcD4wAlpjZKmAo8IyZDYohFhERCansD1By96XAMa2fk0ljtLtvLncsIiISXjkuq/0t8ARwkpmtNbMJpV6niIhEr+Q1DHe/Ksfvw0sdg4iIFE93eouISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhlDxhmNmvzGyjmT3f5rvvm9kLZvacmT1sZr1LHYeIiBSnHDWM+4BLUr6bAbzN3d8OvATcUoY4RESkCCVPGO4+G9ia8t10d29OflwADC11HCIi7Vp9PQwfDlVVwb/19WUPoUvZ13i4zwC/y/SjmU0EJgLU1taWKyYRkcpRXw8TJ8KePcHnxsbgM8D48WULI9ZObzObDDQDGVOlu09199HuPnrAgAHlC05EpFJMnvxWsmi1Z0/wfRnFVsMws2uAS4GL3d3jikNEpOKtXp3f9yUSSw3DzC4BvgJ80N335JpeRKRDCtsvkak5vszN9OW4rPa3wBPASWa21swmAD8BjgRmmNliM7u71HGIiGQUR4dya79EYyO4v9UvkW7dU6ZATc2h39XUBN+XkbWn1qDRo0f7okWL4g5DRDqS1A5lCArjqVNL26E8fHiQJFLV1cGqVYd/X18f9FmsXh3ULKZMCR2fmT3t7qOLihclDBHp7PItuKNSVRXULFKZQSIR6aqiShgaGkREOre4OpQrpF8iH0oYItK5xVVwV0i/RD6UMESkc4ur4B4/PugnqasLmqHq6krfb1KkSrjTW0QkPq0FdIEdykWvu4ITRColDBGRdlZwx0VNUiIi7VWZ7x9RDUNEpD2KYUBC1TBEpOOqgCHBSyaGAQlVwxCRjqlChgQvmRjuH1ENQ0Q6pnKegcdRk4nh/hElDBHpmMp1Bp5uEMGrr4b+/UubOGK4f0QJQ0Q6pnKdgaeryQBs2ZJ59NkoxHDjnxKGiHRM5ToDz1ZjKfVT8caPDwZITCSCf0vcN6OEISIdU7nOwHPVWMr8VLxSUsIQkY6rHGfg6WoybVXw6LP5UsIQESlGa02mX7/Df6vw0WfzpYQhIlKs8eNh82aYNi1zE1iUl97GdEOiEoaISBhhCulMTWD5PL87TBwhluUJZ9mfX+FHH56V/zoy0CNaRURyKfa531E+BjbLsjY+spDH7nqJ6X93Zqw8nvWJwckf28kzvc3sV8ClwEZ3f1vyu77A74DhwCrgSnfflmtZShgiEotiC/won9+dYVkJjGqCZfW1rYwd+gJjL2ph7PXHUXfu0HbzTO/7gEtSvrsZ+Ie7nwD8I/lZRKQyFXvXeLE3ESabw7yqCrf0xfZGG8S3x81k4f3L2bjvaB5cfS4T7ruA2nOGhFtHCCVPGO4+G9ia8vXlwP3J9/cDHyp1HCIiBSu2wC/iJsJt376L5msnQGMj5o4lWkitX7gZg274ELf8fQyjPzWK6m7V4eLKU1yd3gPd/XWA5L/HZJrQzCaa2SIzW7Rp06ayBSgiclCxd43nuomwTYd6YlgtSz56OzeNnsnbj3iJNyd/jy7N+w9ZnKUs3tzh/vtLfrVUxV8l5e5T3X20u48eMGBA3OGISGdUyF3jqVdVQdorqBK/nkbLZ647eNVT1do1jPzDd9j89CqOOWIXdYRs9ir1MCTE9zyMN8xssLu/bmaDgY0xxSEiEk4+z/3O8SyOdYteZ8bdrzL9MeO7jbcwjH2HzN6TPdw79OtUrVkN/fsGAxmGUeJhSOJKGH8BrgG+k/z3zzHFISISvQzP4tj66Zu4YMLZLN8/EhjMwKqNDGVd2kVUrVub/3pLPAxJyZukzOy3wBPASWa21swmECSKsWb2MjA2+VlEpDAV9ihWz3Cm37tpI8N6beeOS2fy3B9e4vWmAVhdjg71ranXDGVQhmFISl7DcPerMvx0canXLSKdQIU8inX1E+uY8fPXmP54NXf4EIaRpoYwdCh/W5NyO8SUKelvCmwt/Gtr098D0q8f9OoVNEPV1gbTa3hzEZEsSvko1iw1l53rd/I///EU//b2WZzc/TXqzh3CdfdfwNx1I3j0mE/T3KX7ocuqqaHqO/95+Dpydai///3pY7vyyrI+CwM0NIiItHdR3kXdVprhQFq69eDPI2/iR2s/yhM7RtFMV2rYzZgByxl73h7GfXoIp1x6PFZlwfyTJxdfA4hgWBGzdjI0SJSUMETkMFGO0xRiuauo5aM1f2XsOzYy7srenPOZU+h+VPfD549KBAkxqoQR11VSIiLRyNUHkIc3V7/J4z9dwYxHDvCTxtVp2+zrbA0Ld48CRhUccl4y9WHE8GAm9WGISPtWxKNYm/c1M//nS7ltzEzOO+o5+tX15CPfexe/XvZONlcNTDuPlbugLtezyUNQwhDp6CrsktOSCPkoVk84r/yjkbs+PosPH7uAfkfs5rwbTuP2WRfQkqjilvPnMvvHS9i6uwfHPHBHZRTU5Xo2eQjqwxDpyIp9jkMHsG3ldv7xkxXMeLSJ6a+MYFXzMACGd1nDuJErGXdpVy668RT6jOh9+MxRdVzHTJ3eIpJbKTqEK7wQPbDrAAvuXcGM329j+rP9WbT7FBJUcxRvctHgFYy78ABjr6vj+H+uDa5m6gTU6S0iuRX7HIdUFXKTXFuecF76+0qm/3INM+b2oOGNUeziHVTTzNlHLuc/LpzDuKv6cdY1p9Clx7tiibGjUB+GSKUoRV9Dsc9xSFXKm+TysPnFLfzu8/O57qQ51HVbz8nvP47P/fFCVmwdyCdPfZaHb36SLY27mbfj7dw2cwznXn8aXXpU0PlxVH/rcvdPuXu7eZ155pku0iFNm+ZeU+MeXHEfvGpqgu+jXq6Z+8UXu9fVBe/r6sKvx+zQZbV9FRtrFvve3OeP/+AZv+WcBj+zZpkbLQ7uvW2bXzFkvv98/Cx/bdbqIIZCtqsY+a4zqr91HssBFnkEZXDsSSCflxKGdFh1dekL4bq64pc9aVL2gj6fAqtfv+KXEUKiJeHP/+ll/+GHZvr7BjzlNexycO/CAb/gqMV++8UNvuCepd68v/mtmUqVdLMpZJ1R/a3zWE5UCUOd3iKVoFTDW0Dmju9U/frB5s3Zp+nfP/uzGYroTN+4bBOP3fUSM6YnmP7aSNYnBgNwUrfXGHvSGsZdfgRjJp3CkccemX4BpbrjO5tC1hnV3zqP5ajTW6QjKeXdvGE7uLdsCdrAs3Ve5xpqO4/O9H3b9zF36nJm/HEH05cOYvHek4EB9LWtvGfoC4y76BXGXn8cteccBxyXe4FRd/CHUcg6o/pbx3AHuDq9RSpBKe/mzacAydV5nWtZWX73hPPcH17iB5fN5L39F9GnjzP2K2fww6fOpXf3vXx73EwW3r+cjfuO5nerz2XCfRdQe86Q8LEX2sFfTMdxIeuM6m8dxx3gUbRrleulPgzp0ErVYZuunT3Ty6zwZaVpu1//7AZ/4Po5fvWIOT6oasPBSUd1f9m/8M6Z/shtT/nO13eWbjtz9ScU2+9R6PxR/a1DLgd1eotIaKkFS69e6Qv9MB2vrcsC9+rqt+abNs13b9rtf/vWQv/imQ1+Wo8XDy52gG30q+rm+r0T5viap9aXbztzFcRRdEDHcWVWnqJKGOr0FunoWu/MbmyE6mpoaQk6uHfuhAMH3pqugCFDEs0Jljz0EjN+vYHpC45i7rZR7KcH3dnH+X2WM+5dOxj7yUG846MnUtWlAlvAS3mxQQVRp7eI5JZ6Z3ZLS/Dvli3QtWuQOLZuzWuIj3WLXmfG3a8y/THjsdUnsslPBk7mtB4v8dkzFzD2w7244PpR1PQ/o3TbFZUKGjq8PVDCEOnI0t2Z3aqpKXgmdI5LaXdv3M2sny1n+p/2MGPFEJbvHwkMZmDVRt47/EXGjXuR99xwAoNPPxE4MfJNKKkIn6XRGcSaMMzs34HrAAeWAp92931xxiTSoeS6pDTN7y0HWnj2wReZPm0jMxYezbztp9LEP9GDvby73zI+M3YmY685ltM+cgJWdUyJAi+T1hpVBQ+mWEli68MwsyHAXGCUu+81s98Dj7r7fZnmUR+GSJ5y3bSXvMFs9RPrmPHz15j+eDWPrT2Zrd4XgNOPeIFxp21g7BVHcf7EUfTo3aM8cUtxUkYUHtDYuHKTe4ibWbKLu0mqC3CEmTUBNcD6mOMR6VjSNbkkNXftwbSWq/lO99d48cBxwBCOrXqdDx6/jLHjqnjPjSdyzKlB/4S0I2lGFK6FuigWHdtlC+6+DrgDWA28Drzp7tNTpzOziWa2yMwWbdq0qdxhirRvyae1eW0tDrRQRQJopJZPNd3D/1l7C8cdvYUffmgWy/78CmubBnHvyxfwiZ+exzGnDsi+7M7wJL/2KE2/lUVU1seWMMysD3A5MAI4FuhpZlenTufuU919tLuPHjAgxwEsUkrtrIBcOXsNPx8/m3/5ynH0XbOEKpyuNHFWzTJ+fs79/OsPRrH1zS48uvGf+MLDFzLqgyPDP1Co9Sy2sTG4LLX1uRip+6Sd7bMOoZRDoURxM0chL+CjwC/bfP4UcFe2eXTjnsQm7jt6Q9jeuN0fvnmBTzp1lo/suvJgmMOq1/qEE2f7g5+b55te2BzNysLc8BbH6LFxqaSb99L8bc4E9yjK7SgWUtCK4WxgGUHfhQH3A/+WbR4lDIlNIXcEl7jAbNrb5PPufs5vvbDBzz1yiVfT5ODeix1+2cAFfucVM/2FR1/1REsikvUdItNw6W2HFinlkO2VpNISY5p4zoAWj6DcjvVObzP7BvAxoBl4FrjO3fdnml5XSUls8r0juL4errnmrRvl2ipwuG1POK82rGb6L1YxY3Z3Hn/9FHZwNFW0MLrnCsa9czPjPtaHs689hW69uuW9/LyEGda7k9xFHcuw6rmU6CopDQ0iEkY+hULqVSqp8igwt63czj9+soIZjzYx/ZURrGoeFoTTZQ3jRq5k7Pu7ctGNJ9P3+D7htyUK6bYxdWiRSixIS6EdJMaohgYJ03T0GPCOKKozxb7UJCWxyafZIVNTTIgmmf079/usOxf7185v8LN6LvUqmh3cj2K7f2jwE/7Tj830lx9bVZpmpnzlarevtKaaUmkHTW+Ua7Ra4AzgceBeYHAUKy30pYTRSVRSB2JbYePK9jjUlAIz0ZLwFx591e+8YqZfNnCB92JHMAgsTX7ukUv81gsbfN7dz3nT3qaybGLkKvVvGaV2kBjLljAOTghXAEuAW4Ejolh5vi8ljE4g3X++bt2CZ0m3l0In0xlndbX7tGm+6YXN/uDn5vmEE2f7sOq1B38+vssqn3TqLH/45gW+vXF73Fsh+ajwxFjWhEFwFdPbgBuAzcBa4JNRBJDPSwmjE8jVnBPF2Vup/3OnSXrNXXv4gyMn+5k1y9xocXDvbdv8iiHz/e5PzPJXGxqjjaE1jrgLsUqIQcraJDWXYMiOGcDtwKXASODHwNQoggj7UsLoBLI150TRPhxF80GOQjDRkvDV//4Df/OIgd6CeSPD/CqmeRcO+AVHLfZvXtTgC+5Z6s37mwvbhrAxxt1MUgkxiLuXN2G8jeTVVGl+WxFFEGFfShglMmnSW09Oq64OPsclTA0j9Xr/VNkK9GI7KDMUgtu/8zOvv3GuXztyth9btf7gTyd1e9U/e9pM/8vXnvQd63YUvFvyVgkdsZUQg7h7GRNG1pnhuCiCCPtSwiiBSZPS/6eOK2mEff50pkIn11ltmBvOsslQCK6kzsG9r23xK4fN83uume2N89dGsUcKU+x2lisGNVmVRUUkjHK/lDBKoLVmka6DNi6TJuV1pdEhcp3VFnjWm2hJ+JKHXvQE6eNKYL7w/mXRNjMVU5hWwtl9rhjUZFU2ShgSjWxn8XHJ1iyVq+DMdVabRyG1/tkN/sD1c/zqEXN8UNUGB/eVZIgt6oK42MK0EgrjXDFUQlLrJJQwJBqVWMMopjkl7KB4ac7cd2/a7X/71kK/6cwGP63HiwdnHWAb/aq6uX7vhDm++Zs/iaYgzlV7iKIwrYTmnmwxVEKzWSehhCHRqLQ+DPfiCss8zqxbmlr8md+s8O++r8Ev7vO0f4p7fSV13oL5hqpB/pe3T/ZnfrPCW5paDl9HMQVxmBg7Q2GqGkbZKGFIdCrpKin3aJpjMhToaxeu93snzPGr6ub6ANt4cPFf7nKH77fuha8zH2EKys5QmFZCs1knoYQh8SrHzW8RLH/XG7v8kdue8s+fPtNHdX/5YLk0sOoNv3rEHH/g+jm+/tkN5S2gw1491BkK00poNusEokoYGq1W8hdmpNKYtBxo4dkHX2T6tI3MWHg087afShPd6MFe3t1vGePO2cXYa47ltI+ccOjT5co54mjYUVxThqhmypTY96+0T2UbrbaSXqphVIhSn43nedbZOH+t33PNbL9y2Dzva1sOhnP6ESv8y2c1+IzvPu17t+2Nd5va6iy1B6kYqElKYlPKDtkQhemOdTv8L1970j972kw/qdurByc7tmq9X3P8HK+/ca5vWLox8vUWvD3pkp+aYqSMokoYapKS/JXywTgZlr2/72C+d/pvmLGoD0/sGEUzXalhNxcOWM648/Yw9tohjLrs+EObmfJ1441Bs1pLC1RXB81ud91V+PIquOlOOhc1SUl4UZ/NlrJJJUPtpQVzo8XPrFnmt5zT4I//4Bnf9+a+4tfXqhTb1BmudJJ2ATVJSSjlbmop0PbG7f7wzQt8c9eBaQvZXUcO9E0vbC4u5mxKUbh3hnsppF2IKmFUFV1FkdKqrw+aaaqqgn/r6/Obf/Lkw58tvWdP8H0xxo8Pmp8SieDfPJtYmvc1M//nS7ltzEzOO+o5+tX15MPfOZsvNU1hHz0Onbimhp4/+wH9T+pXXMzZrF4d/vuwf5Pa2vy+F6l0UWSdQl9Ab+APwAvACuCcbNN3uhpGFLWDCjnLTbQk/OXHVvlPPzbTPzT4CT+K7Q7uVTT7WT2X+tfOb/BZdy72/Tv3x9MhHLaGkc/fRFdDSYWgIzRJAfcD1yXfdwN6Z5u+ohJGOQq1KJpJYmxH3/raNn/oi/N94smzfHiX1QdXPbzLav/F4K/5riMHeqJSrhLKNKx6v37FPU9DV0NJBWj3CQM4ClhJhoczpXuVLGHk+5+6XGeOUdQOyhFrcv8lzHxvn8H++xMn+9k9l3oVzQ7uR7HdLx+0wH/6sZn+8mOrPPHrCGOKskCeNi1IEKn7O8rnaYjEoCMkjNOBp4D7gGeBe4CeaaabCCwCFtXW1ka8G72wArVcZ+1RradEZ7mJloSv+9J/+YHqQ8dg2kWNf737f/qtFzb43LuW+IHdB0q3XeW+sklXPkk71BESxmigGTg7+fn/Abdnm6ckNYxCCoBynWVWYBv4phc2+4Ofm+cTTpztw6rXFvZ8iHz3X6aEF8eVTRX4NxHJpSMkjEHAqjafLwAeyTZPSRJGIYV/uYeRiKsNfNo0Twyr9QTmW7sN9K92/a4bLQ7uvW2bXzFkfsYn0EW2/7IV0KVI3EU8T0OkUrX7hBFsA3OAk5LvbwO+n236iqlhdOCzzERLwp//08v+6JmTfS89DtnGPfTwP46a7AvuWepNe5uCGUq9/7ItvxSJuwP/baXz6igJ4/Rk/8RzwJ+APtmmL0nCKLSA6EBnmW88v9Hrb5zr146c7cdWrXfI41Gkpd5/2WoR7eSmRJG4dYiEke+rYq6SimoZbefp1y94laGQ2rttr8/47tP+5bMa/PQjVhwsa/vaFr9y2Dz/xadmB5e7hm3uKWUBm6sWocJdJCcljEpSyJlupuv+S9AMkmhJ+JKHXvQ7Lm3wcf0Weg/2OLh3Zb+P6f2MTxnb4AvvX+bN+5vfmqlSrgZSE5FI0ZQwClGqs9FCCtdM8xRSOKfZrvXPbvAHrp/jV4+Y44OqNhxc5CndXvHPnz7TH7ntKd/5+s7sy6yUglq1CJGiKGHkK4YRVrNerZNpnnyv9kmzXXvo4VcxzcG9v23yq+rm+q8+PdvXPLU+v+1SQS3SIShh5KuQIR3a3vWbOkREMct2T39HcR7ztzS1+DO/WeHbe6Qf3XV7j4H+zG9WeEtTS7j90xkpIUonoYSRr3w7cbt2PXzabt2iGWRu2rRgWdmSRZr51y5c7/dOmONX1c31AbbRwb2lkPsgChXXxQGlUElNbiIlpoSRr3xqAdn6F6IYZC7T8quqDpl/1xu7/JHbnvLPnz7TR3V/+eBkA6ve8KtHzPEHrp/jTYOH5hdnNtm2IYoCtpIK6Urp1BcpAyWMfOVTWGXrX4jizD3DshPgC+9f5t8e1+Bjej/jXdnv4N6DPT6u30L//gcafMlDL3qiJZF9u3I1oRWyf9r5yLmH0SCC0okoYRQibC0gWw2jX7/iYnB3r65Ou+wmqg9+vLnr93xrt4GewLxl6LDcl+jmGmU1l0zb3K9f9v2RTwFbaCFdimasSkpeIiWmhFFKmfowIPi+iAJrx7odnshSw6i/ca5v/87Pyj+CbpirtnItP1fBXknDsFRS85hIiSlhlFq2we3yOAtt3t/sC+5Z6t+8qMEvOGqxd+FA7mE34hhBN8x9IdlqMGEK4EobSr5SOuBFSkwJI1+FFA4FFsKvzVrtd39ill8xZL73tm3BLLT4mTXL/OZ3Nfjzn/iWJ47IUnDGMYJurjvP0y237T4Mu/58/w7qaxApmhJGPgptfghZCG5v3O4P37zAJ506y0d2XXlwsqHV6/wzJ8z2Bz83zze9sPnwmDIVnIUU/pMmHV64FnIVU9uYMt0rki6OUhXs6msQKZoSRj4KLXQmTUo7X8vEG3ze3c/5rRc2+LlHLvFqmhzce7HDLz3mSb/zipm+4pFXD72aKR+F3NeROr1ZEH8xohqGvFwxiEhaShj5KPTsN0Mh2Miwg81MZ/Vc6pPPa/BZdy72/Tv3FxZfOlHc1xHFPSNhpy1lwa6+BpGiKGHk0raQyXAZa7az362vbcv4NLkE5g99cb5veWVr+HgyxRZFAZjvXezZCvZiYlPBLlKRlDCyCdOBm3L2e2D3AZ/948X+tfMb/OyeS72K5sKeV11IbMWeiUdxF3trAa/mH5EOJ6qEYcGy2ofRo0f7okWLck84fDg0Nh7+fXU1JBJQW4t/awov9TuH6b9cw4y5PWh4YxS7OJIqWji713LGnbmFq0Y8wYm/+xa2d89by6ipgalTYfz4wjYiU2x1dbBqVWHLrK+HiRNhT4g4q6qCVJDKDGpro49NRGJnZk+7++hil9MlimAqzurVab/2RILf/9tcZvythenXHsealiHAcRzfpZGrRz3LuEu78c83nkLvutOSc4yB99TC5MnBMmtrYcqUwpNFltgyfh9Gazxh4syUFGprSxObiHQcUVRTyvUK3SSVodllJbUO7r1tm3/k2Cf87k/M8lcbGsMtM1Wh7fVxXyaardkp7thEpCSIqEmqKu6EFSVPOMv/8gp/7X81++hxyG976cEzoz7JE794nk17evHHde/i+vp3c9yY2vxX1NoE1NgYFKmNjcHn+vrc806ZEjQXtVVTE3xfDuPHB01VdXVBM1Rd3VtNV3HHJiKVLYqsU8wLqAaeBf4317TpahhvPL/R62+c69eOnO3HVq0/eFL8xaof+pauycH7huQYvC9fUdxVXalXE1VybCJSEDpKp7eZfREYDRzl7pdmm3b06NE+97G5zJ26nBl/3MH0pYNYvPdkAPraVi4e8iLjLmpm7MQR1J03tHRBZ+s4TiRKt14RkQJ0iE5vMxsKfACYAnwx1/QvL95Dnz7OPs6gKwc49+jlTDl/JmPHH8MZV51EdbdzSh4zkL3jWESkg4r7KqkfAV8Gjsw0gZlNBCYCdOdUbjj9KcZeXsOFN5xCr0GnlyXIw0yZkv4yVrX1i0gHFlvCMLNLgY3u/rSZjck0nbtPBaZC0CT1o0UXlifAbPK5jFVEpIOI8yqp84APmtkq4EHgIjObFuka6uuDG+WqqoJ/w1zFFNb48cHNbIlE8K+ShYh0cLElDHe/xd2Huvtw4OPA4+5+dWQrKObSVxEROUyHug/jEJMnH9rHAMHnyZPLG0cpazkiImUUd6c3AO4+E5gZ6UIrYZiL1DGeWms5oCYsEWl32l8NI+wZe6ZLXMt56Wul1HJERCLQvhLG1q3h+yUqYZiLSqjliIhEpH0ljHXrwp+xZxszqVwqoZYjIhKR2IcGycdoM0/7NIxKHZIjn+dUiIiUSFRDg7SvGka3bum/r9Qz9kqo5YiIRKQirpIK7eijYfPmQwf+q/QhOcaPV4IQkQ6hfdUwtmw5NFmYwTXXqEAWESmD9pUwUvsp3OHRR+OJRUSkk2lfCSMdXaIqIlIW7T9hVGqHt4hIB9O+EkZVSriV3uEtItKBtK+EUVenS1RFRGLSvi6r7dsXFqW9dU9EREqsfdUwREQkNkoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiIQSW8Iws2Fm1mBmK8xsmZl9Pq5YREQktzhv3GsGbnL3Z8zsSOBpM5vh7stjjElERDKIrYbh7q+7+zPJ9zuBFcCQuOIREZHsKqIPw8yGA+8Enkzz20QzW2RmizZt2lT22EREJBB7wjCzXsAfgS+4+47U3919qruPdvfRAwYMKH+AIiICxJwwzKwrQbKod/f/jjMWERHJLs6rpAz4JbDC3f8rrjhERCScOGsY5wGfBC4ys8XJ1/tjjEdERLKI7bJad58LWFzrFxGR/MTe6S0iIu2DEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKLEmDDO7xMxeNLNXzOzmOGMREZHsYksYZlYN/BR4HzAKuMrMRsUVj4iIZBdnDeMs4BV3f83dDwAPApfHGI+IiGTRJcZ1DwHWtPm8Fjg7dSIzmwhMTH7cb2bPlyG2YvUHNscdRAiKMzrtIUZQnFFrL3GeFMVC4kwYluY7P+wL96nAVAAzW+Tuo0sdWLEUZ7TaQ5ztIUZQnFFrT3FGsZw4m6TWAsPafB4KrI8pFhERySHOhLEQOMHMRphZN+DjwF9ijEdERLKIrUnK3ZvN7LPA34Fq4FfuvizHbFNLH1kkFGe02kOc7SFGUJxR61Rxmvth3QYiIiKH0Z3eIiISihKGiIiEUhEJI9cQIRa4M/n7c2Z2Rth5yxzn+GR8z5nZfDN7R5vfVpnZUjNbHNUlbkXEOcbM3kzGstjMvh523jLH+aU2MT5vZi1m1jf5W1n2p5n9ysw2Zrr/p4KOzVxxVsqxmSvOSjk2c8VZCcfmMDNrMLMVZrbMzD6fZppoj093j/VF0OH9KnAc0A1YAoxKmeb9wF8J7t14F/Bk2HnLHOe5QJ/k+/e1xpn8vAroXyH7cwzwv4XMW844U6a/DHg8hv35buAM4PkMv8d+bIaMM/ZjM2ScsR+bYeKskGNzMHBG8v2RwEulLjsroYYRZoiQy4EHPLAA6G1mg0POW7Y43X2+u29LflxAcG9JuRWzTypqf6a4CvhtiWLJyN1nA1uzTFIJx2bOOCvk2AyzPzOpqP2ZIq5j83V3fyb5fiewgmAEjbYiPT4rIWGkGyIkdaMzTRNm3qjku64JBJm9lQPTzexpC4Y7KZWwcZ5jZkvM7K9mdmqe80Yh9LrMrAa4BPhjm6/LtT9zqYRjM19xHZthxX1shlYpx6aZDQfeCTyZ8lOkx2ecQ4O0CjNESKZpQg0vEpHQ6zKzfyb4T3l+m6/Pc/f1ZnYMMMPMXkiexcQR5zNAnbvvMrP3A38CTgg5b1TyWddlwDx3b3vGV679mUslHJuhxXxshlEJx2Y+Yj82zawXQcL6grvvSP05zSwFH5+VUMMIM0RIpmnKObxIqHWZ2duBe4DL3X1L6/fuvj7570bgYYIqYSxxuvsOd9+VfP8o0NXM+oeZt5xxtvFxUqr8ZdyfuVTCsRlKBRybOVXIsZmPWI9NM+tKkCzq3f2/00wS7fFZ6o6ZEB03XYDXgBG81flyaso0H+DQjpunws5b5jhrgVeAc1O+7wkc2eb9fOCSGOMcxFs3bZ4FrE7u24ran8npjiZoS+4Zx/5MrmM4mTtpYz82Q8YZ+7EZMs7Yj80wcVbCsZncLw8AP8oyTaTHZ+xNUp5hiBAzuyH5+93AowS9/a8Ae4BPZ5s3xji/DvQD7jIzgGYPRrIcCDyc/K4L8Bt3/1uMcf4LMMnMmoG9wMc9OIoqbX8CfBiY7u6728xetv1pZr8luHKnv5mtBW4FuraJMfZjM2ScsR+bIeOM/dgMGSfEfGwC5wGfBJaa2eLkd18lODkoyfGpoUFERCSUSujDEBGRdkAJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBEipB8HsHY5PtvmdmdccckUiqx3+kt0s7dCnwzOdDcO4EPxhyPSMnoTm+RIpnZLKAXMMaD5xKIdEhqkhIpgpmdRvDks/1KFtLRKWGIFCj55LJ6gieV7Taz98YckkhJKWGIFCD5pLX/Bm5y9xXA7cBtsQYlUmLqwxARkVBUwxARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJ5f8Dan8nF6iTRYwAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "execution_count": 15, + "id": "cc6e4463", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -2594,16 +2945,20 @@ }, { "cell_type": "markdown", - "id": "c3d4f269", - "metadata": {}, + "id": "aab422e4", + "metadata": { + "editable": true + }, "source": [ "**Challenge**: try to write a similar code for a Logistic Regression case." ] }, { "cell_type": "markdown", - "id": "21369b43", - "metadata": {}, + "id": "017e3c2e", + "metadata": { + "editable": true + }, "source": [ "## Code with a Number of Minibatches which varies\n", "\n", @@ -2613,1038 +2968,12 @@ { "cell_type": "code", "execution_count": 16, - "id": "00fc4272", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Own inversion\n", - "[[3.96113114]\n", - " [3.11062375]]\n", - "Eigenvalues of Hessian Matrix:[0.26820372 4.51917517]\n", - "theta from own gd\n", - "[[3.96113114]\n", - " [3.11062375]]\n", - "65\n", - "25\n", - "55\n", - "0\n", - "50\n", - "15\n", - "20\n", - "40\n", - "45\n", - "90\n", - "35\n", - "40\n", - "20\n", - "75\n", - "85\n", - "80\n", - "0\n", - "0\n", - "20\n", - "30\n", - "30\n", - "95\n", - "95\n", - "75\n", - "95\n", - "15\n", - "10\n", - "80\n", - "75\n", - "25\n", - "35\n", - "60\n", - "10\n", - "20\n", - "40\n", - "40\n", - "25\n", - "90\n", - "50\n", - "70\n", - "70\n", - "85\n", - "60\n", - "75\n", - "0\n", - "20\n", - "15\n", - "90\n", - "75\n", - "25\n", - "80\n", - "15\n", - "65\n", - "70\n", - "90\n", - "70\n", - "90\n", - "30\n", - "80\n", - "15\n", - "15\n", - "30\n", - "5\n", - "15\n", - "40\n", - "50\n", - "20\n", - "20\n", - "5\n", - "85\n", - "15\n", - "70\n", - "80\n", - "85\n", - "40\n", - "70\n", - "75\n", - "30\n", - "60\n", - "80\n", - "50\n", - "25\n", - "75\n", - "80\n", - "5\n", - "70\n", - "55\n", - "85\n", - "90\n", - "0\n", - "75\n", - "30\n", - "45\n", - "35\n", - "20\n", - "15\n", - "55\n", - "50\n", - "0\n", - "80\n", - "10\n", - "10\n", - "95\n", - "25\n", - "20\n", - "20\n", - "80\n", - "20\n", - "10\n", - "85\n", - "65\n", - "90\n", - "25\n", - "70\n", - "95\n", - "35\n", - "65\n", - "10\n", - "85\n", - "15\n", - "85\n", - "65\n", - "50\n", - "70\n", - "5\n", - "10\n", - "75\n", - "5\n", - "15\n", - "20\n", - "75\n", - "50\n", - "70\n", - "20\n", - "50\n", - "65\n", - "50\n", - "90\n", - "40\n", - "90\n", - "75\n", - "55\n", - "40\n", - "15\n", - "0\n", - "0\n", - "90\n", - "5\n", - "95\n", - "95\n", - "95\n", - "80\n", - "60\n", - "35\n", - "50\n", - "40\n", - "70\n", - "75\n", - "75\n", - "60\n", - "80\n", - "5\n", - "70\n", - "70\n", - "80\n", - "5\n", - "85\n", - "5\n", - "25\n", - "50\n", - "25\n", - "40\n", - "70\n", - "70\n", - "95\n", - "90\n", - "55\n", - "45\n", - "45\n", - "15\n", - "0\n", - "0\n", - "10\n", - "70\n", - "70\n", - "95\n", - "15\n", - "85\n", - "85\n", - "70\n", - "50\n", - "45\n", - "5\n", - "70\n", - "30\n", - "75\n", - "35\n", - "50\n", - "35\n", - "45\n", - "75\n", - "65\n", - "35\n", - "65\n", - "55\n", - "75\n", - "0\n", - "95\n", - "65\n", - "55\n", - "0\n", - "35\n", - "50\n", - "45\n", - "85\n", - "15\n", - "30\n", - "25\n", - "60\n", - "50\n", - "60\n", - "10\n", - "85\n", - "5\n", - "35\n", - "30\n", - "75\n", - "55\n", - "55\n", - "60\n", - "90\n", - "15\n", - "15\n", - "85\n", - "15\n", - "15\n", - "85\n", - "25\n", - "95\n", - "15\n", - "60\n", - "10\n", - "5\n", - "95\n", - "20\n", - "70\n", - "40\n", - "55\n", - "95\n", - "50\n", - "45\n", - "80\n", - "80\n", - "45\n", - "75\n", - "40\n", - "65\n", - "25\n", - "55\n", - "90\n", - "5\n", - "0\n", - "75\n", - "75\n", - "60\n", - "30\n", - "70\n", - "5\n", - "20\n", - "80\n", - "90\n", - "0\n", - "10\n", - "55\n", - "25\n", - "45\n", - "80\n", - "25\n", - "10\n", - "45\n", - "40\n", - "35\n", - "70\n", - "55\n", - "10\n", - "35\n", - "60\n", - "5\n", - "20\n", - "80\n", - "60\n", - "90\n", - "75\n", - "0\n", - "85\n", - "90\n", - "60\n", - "65\n", - "20\n", - "95\n", - "30\n", - "65\n", - "0\n", - "45\n", - "45\n", - "35\n", - "65\n", - "80\n", - "35\n", - "5\n", - "15\n", - "70\n", - "25\n", - "40\n", - "55\n", - "15\n", - "80\n", - "25\n", - "85\n", - "45\n", - "45\n", - "65\n", - "65\n", - "85\n", - "85\n", - "15\n", - "60\n", - "0\n", - "40\n", - "60\n", - "45\n", - "40\n", - "30\n", - "30\n", - "25\n", - "50\n", - "75\n", - "95\n", - "35\n", - "50\n", - "85\n", - "55\n", - "90\n", - "10\n", - "70\n", - "10\n", - "25\n", - "10\n", - "75\n", - "80\n", - "35\n", - "75\n", - "60\n", - "60\n", - "20\n", - "5\n", - "85\n", - "95\n", - "50\n", - "80\n", - "30\n", - "60\n", - "90\n", - "95\n", - "15\n", - "90\n", - "25\n", - "20\n", - "20\n", - "75\n", - "20\n", - "80\n", - "40\n", - "45\n", - "65\n", - "55\n", - "80\n", - "90\n", - "75\n", - "40\n", - "40\n", - "25\n", - "75\n", - "50\n", - "85\n", - "60\n", - "0\n", - "5\n", - "80\n", - "90\n", - "60\n", - "25\n", - "55\n", - "40\n", - "60\n", - "65\n", - "65\n", - "75\n", - "45\n", - "0\n", - "75\n", - "30\n", - "10\n", - "75\n", - "85\n", - "35\n", - "85\n", - "5\n", - "85\n", - "65\n", - "50\n", - "95\n", - "65\n", - "15\n", - "75\n", - "60\n", - "70\n", - "50\n", - "60\n", - "25\n", - "40\n", - "85\n", - "70\n", - "40\n", - "35\n", - "10\n", - "20\n", - "90\n", - "65\n", - "50\n", - "35\n", - "60\n", - "60\n", - "30\n", - "75\n", - "35\n", - "10\n", - "0\n", - "65\n", - "10\n", - "25\n", - "75\n", - "25\n", - "95\n", - "0\n", - "65\n", - "60\n", - "30\n", - "85\n", - "0\n", - "50\n", - "5\n", - "20\n", - "5\n", - "35\n", - "0\n", - "70\n", - "65\n", - "95\n", - "0\n", - "10\n", - "15\n", - "85\n", - "35\n", - "25\n", - "10\n", - "70\n", - "75\n", - "80\n", - "45\n", - "25\n", - "0\n", - "90\n", - "40\n", - "45\n", - "40\n", - "85\n", - "90\n", - "40\n", - "65\n", - "85\n", - "15\n", - "70\n", - "40\n", - "10\n", - "15\n", - "15\n", - "35\n", - "95\n", - "55\n", - "75\n", - "0\n", - "5\n", - "60\n", - "50\n", - "75\n", - "35\n", - "20\n", - "30\n", - "80\n", - "30\n", - "20\n", - "55\n", - "85\n", - "90\n", - "80\n", - "70\n", - "60\n", - "80\n", - "85\n", - "90\n", - "15\n", - "0\n", - "65\n", - "0\n", - "75\n", - "80\n", - "0\n", - "35\n", - "5\n", - "20\n", - "45\n", - "90\n", - "75\n", - "35\n", - "35\n", - "10\n", - "25\n", - "85\n", - "30\n", - "90\n", - "30\n", - "20\n", - "90\n", - "75\n", - "0\n", - "35\n", - "15\n", - "5\n", - "90\n", - "80\n", - "20\n", - "40\n", - "80\n", - "75\n", - "5\n", - "15\n", - "80\n", - "25\n", - "25\n", - "65\n", - "80\n", - "20\n", - "5\n", - "75\n", - "15\n", - "50\n", - "20\n", - "65\n", - "5\n", - "95\n", - "95\n", - "55\n", - "15\n", - "10\n", - "10\n", - "95\n", - "70\n", - "70\n", - "0\n", - "65\n", - "35\n", - "80\n", - "0\n", - "50\n", - "75\n", - "20\n", - "30\n", - "15\n", - "25\n", - "55\n", - "70\n", - "25\n", - "70\n", - "35\n", - "15\n", - "40\n", - "45\n", - "55\n", - "15\n", - "55\n", - "45\n", - "0\n", - "10\n", - "75\n", - "50\n", - "65\n", - "60\n", - "20\n", - "75\n", - "40\n", - "90\n", - "55\n", - "35\n", - "25\n", - "50\n", - "95\n", - "10\n", - "20\n", - "0\n", - "10\n", - "25\n", - "45\n", - "0\n", - "90\n", - "30\n", - "50\n", - "5\n", - "20\n", - "25\n", - "40\n", - "5\n", - "70\n", - "95\n", - "35\n", - "15\n", - "20\n", - "90\n", - "50\n", - "15\n", - "30\n", - "0\n", - "75\n", - "30\n", - "75\n", - "90\n", - "0\n", - "55\n", - "95\n", - "45\n", - "20\n", - "50\n", - "30\n", - "95\n", - "60\n", - "75\n", - "30\n", - "50\n", - "85\n", - "50\n", - "15\n", - "80\n", - "25\n", - "20\n", - "95\n", - "30\n", - "55\n", - "50\n", - "95\n", - "55\n", - "50\n", - "20\n", - "30\n", - "40\n", - "85\n", - "90\n", - "85\n", - "60\n", - "55\n", - "85\n", - "75\n", - "15\n", - "75\n", - "60\n", - "70\n", - "20\n", - "60\n", - "90\n", - "0\n", - "30\n", - "85\n", - "80\n", - "5\n", - "30\n", - "85\n", - "65\n", - "35\n", - "95\n", - "80\n", - "55\n", - "50\n", - "45\n", - "75\n", - "25\n", - "10\n", - "55\n", - "95\n", - "0\n", - "30\n", - "75\n", - "80\n", - "85\n", - "90\n", - "95\n", - "50\n", - "15\n", - "90\n", - "35\n", - "15\n", - "50\n", - "45\n", - "25\n", - "30\n", - "0\n", - "45\n", - "10\n", - "45\n", - "15\n", - "0\n", - "50\n", - "75\n", - "40\n", - "50\n", - "0\n", - "10\n", - "5\n", - "15\n", - "55\n", - "50\n", - "30\n", - "0\n", - "75\n", - "85\n", - "10\n", - "80\n", - "70\n", - "75\n", - "5\n", - "95\n", - "10\n", - "5\n", - "40\n", - "20\n", - "45\n", - "60\n", - "95\n", - "95\n", - "30\n", - "60\n", - "25\n", - "40\n", - "5\n", - "0\n", - "30\n", - "80\n", - "35\n", - "75\n", - "60\n", - "45\n", - "25\n", - "80\n", - "70\n", - "25\n", - "55\n", - "15\n", - "75\n", - "70\n", - "0\n", - "20\n", - "25\n", - "25\n", - "60\n", - "70\n", - "25\n", - "70\n", - "25\n", - "20\n", - "90\n", - "15\n", - "30\n", - "55\n", - "65\n", - "15\n", - "95\n", - "0\n", - "50\n", - "95\n", - "25\n", - "15\n", - "50\n", - "20\n", - "80\n", - "45\n", - "25\n", - "65\n", - "35\n", - "15\n", - "75\n", - "15\n", - "85\n", - "10\n", - "55\n", - "25\n", - "70\n", - "15\n", - "15\n", - "65\n", - "75\n", - "95\n", - "10\n", - "70\n", - "55\n", - "0\n", - "90\n", - "60\n", - "0\n", - "10\n", - "55\n", - "10\n", - "65\n", - "95\n", - "45\n", - "5\n", - "55\n", - "25\n", - "15\n", - "50\n", - "20\n", - "95\n", - "15\n", - "80\n", - "25\n", - "70\n", - "60\n", - "30\n", - "70\n", - "75\n", - "55\n", - "80\n", - "45\n", - "10\n", - "80\n", - "80\n", - "15\n", - "20\n", - "25\n", - "0\n", - "35\n", - "5\n", - "60\n", - "0\n", - "40\n", - "25\n", - "55\n", - "55\n", - "45\n", - "30\n", - "35\n", - "30\n", - "65\n", - "80\n", - "15\n", - "95\n", - "55\n", - "0\n", - "75\n", - "75\n", - "80\n", - "15\n", - "70\n", - "85\n", - "40\n", - "85\n", - "45\n", - "45\n", - "40\n", - "40\n", - "85\n", - "65\n", - "80\n", - "75\n", - "50\n", - "25\n", - "95\n", - "55\n", - "65\n", - "60\n", - "30\n", - "30\n", - "5\n", - "95\n", - "40\n", - "30\n", - "50\n", - "45\n", - "55\n", - "70\n", - "60\n", - "45\n", - "80\n", - "95\n", - "95\n", - "65\n", - "75\n", - "85\n", - "5\n", - "80\n", - "30\n", - "60\n", - "70\n", - "85\n", - "85\n", - "15\n", - "75\n", - "45\n", - "70\n", - "55\n", - "15\n", - "5\n", - "75\n", - "0\n", - "55\n", - "10\n", - "40\n", - "85\n", - "45\n", - "45\n", - "50\n", - "70\n", - "40\n", - "70\n", - "15\n", - "60\n", - "15\n", - "65\n", - "95\n", - "15\n", - "45\n", - "30\n", - "5\n", - "55\n", - "85\n", - "40\n", - "85\n", - "65\n", - "20\n", - "25\n", - "45\n", - "70\n", - "75\n", - "45\n", - "45\n", - "75\n", - "30\n", - "45\n", - "20\n", - "40\n", - "95\n", - "15\n", - "20\n", - "85\n", - "20\n", - "15\n", - "80\n", - "5\n", - "45\n", - "85\n", - "10\n", - "5\n", - "45\n", - "0\n", - "80\n", - "40\n", - "45\n", - "65\n", - "25\n", - "70\n", - "70\n", - "15\n", - "40\n", - "25\n", - "15\n", - "5\n", - "20\n", - "5\n", - "45\n", - "65\n", - "35\n", - "20\n", - "50\n", - "65\n", - "55\n", - "25\n", - "10\n", - "60\n", - "15\n", - "25\n", - "85\n", - "70\n", - "30\n", - "theta from own sdg\n", - "[[3.97500896]\n", - " [3.09893656]]\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAqW0lEQVR4nO3dd3wc9Z3/8ddHtlyECdgyLtiWjIkxJrSAKQkmMSHiCCX1LoGYUELOiXnkUkgjcQLJcU69+6VdCGcgoViBS3IpF5JcEEEu2DSZ6kIzLhj3BrZcJX1+f8zKXq23zGpnd3al9/PxmAfa3SmfGQ/zmW+Z75i7IyIikktV3AGIiEhlUMIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMKTHMrNvmtnsuOOIkplNMbM1ccchvZMShpSUma00s91mttPM1pvZnWY2KO64RCQ3JQyJw6XuPgg4FXgr8NV4w+mdzKxv3DFIZVHCkNi4+3rgbwSJAwAzu8HMlpvZDjNbamYfSPrtajN72Mz+3cy2mdkKM3tP0u/HmNncxLJNwNDk7ZnZe81siZltN7M5ZjYx6beVZvYlM3vWzFrN7A4zG25mf02s70EzG5xuPzqriczsC2a20czWmdk1Sb/PMbNPpO5H0mc3s+vM7KXEtm42s2PN7BEze8PMfm1m/VK2+TUz25yIe2rS9/0Tx2e1mW0ws1vNbGBKnF8xs/XAL81sqJndnzgmW81svpnpuiBp6cSQ2JjZaOA9wMtJXy8HzgWOAL4FzDazkUm/nwW8QJAMvg/cYWaW+O1XwKLEbzcDVyVt6zjgXuBzwFHAX4A/pVyIPwQ0AMcBlwJ/Bb6WWF8V8JksuzMiEfMo4FrgZ5kSTAYXAqcDZwNfBmYBU4ExwInA5SnbGprY1lXALDObkPjte4n4TwXenJjnxpRlhwD1wDTgC8AagmMyPLG/Gi9I0lLCkDj8wcx2AK8CG4GbOn9w99+4+1p373D3/wZeAs5MWnaVu9/m7u3AXcBIYLiZ1QFnAN9w973uPg/4U9JyHwH+7O5N7r4f+HdgIPD2pHl+6u4b3P01YD7wmLs/5e57gd8TVJ9lsh/4V3ff7+5/AXYCE7LMn+p77v6Guy8BFgMPuPsr7v46QeJK3Xbnfs4F/gx8OJE4/xn4vLtvdfcdwLeBy5KW6wBuSiy7OxH3SKA+Eft81wBzkoEShsTh/e5+ODAFOJ6kqiMzu9LMnk5UkWwnuLtOrlpa3/mHu+9K/DkIOBrY5u6tSfOuSvr76OTP7t5BkLBGJc2zIenv3Wk+Z2uc3+LubUmfd+WYP1U+2063n0cTlBJqgEVJx+//Et932uTue5I+/4CghPeAmb1iZjfkEbP0MkoYEpvE3fGdBHf7mFk9cBvwaaDW3Y8kuNu2DKtItg4YbGaHJX1Xl/T3WoJqGBLbMoLqnte6vwehtRJcyDuNKHB96fZzLbCZILm8xd2PTExHJDoYdOpSenD3He7+BXcfR1ANd72ZnV9gfNJDKWFI3H4ENJjZqcBhBBe0TQCJhuMTw6zE3VcBLcC3zKyfmU0muAB2+jVwsZmdb2bVBHX3e4GFEe1HNk8DHzSzGjN7M0EbR6E69/Nc4BLgN4lS023AD81sGICZjTKzf8i0EjO7xMzenEigbwDtiUnkEEoYEit33wTcTVAnvxT4D+ARgiqZk4AFeazuowSN4lsJ2kXuTtrOC8AVwE8J7sQvJejeuy+C3cjlh8A+gn26C2gscH3rgW0EpYpG4FPu/nzit68QVDE9amZvAA+SvS1lfGKenQTH/RZ3n1NgfNJDmdq3REQkDJUwREQkFCUMEREJRQlDRERCUcIQEZFQKmrwsaFDh/rYsWPjDkNEpKIsWrRos7sflXvO7CoqYYwdO5aWlpa4wxARqShmtir3XLmpSkpEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQip4wzOwXZrbRzBan+e2LZuZmNrTYcYiISGFKUcK4E7gw9UszGwM0AKtLEIOIiBSo6AnD3ecRvGM51Q+BLwN6R6yISAWIpQ3DzN4LvObuz4SYd5qZtZhZy6ZNm0oQnYiIpFPyhGFmNcAM4MYw87v7LHef5O6Tjjqq4OHcRUSkm+IoYRwLHAM8Y2YrgdHAk2Y2IoZYREQkpJK/QMndnwOGdX5OJI1J7r651LGIiEh4pehWey/wCDDBzNaY2bXF3qaIiESv6CUMd788x+9jix2DiIgUTk96i4hIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEUPWGY2S/MbKOZLU767gdm9ryZPWtmvzezI4sdh4iIFKYUJYw7gQtTvmsCTnT3k4EXga+WIA4RESlA0ROGu88DtqZ894C7tyU+PgqMLnYcIiJSmHJow/g48NdMP5rZNDNrMbOWTZs2lTAsERFJFmvCMLMZQBvQmGked5/l7pPcfdJRRx1VuuBERKSLvnFt2MyuAi4Bznd3jysOEREJJ5aEYWYXAl8B3unuu+KIQURE8lOKbrX3Ao8AE8xsjZldC/wncDjQZGZPm9mtxY5DREQKU/QShrtfnubrO4q9XRERiVY59JISEZEoNDbC2LFQVRX8tzFjf6Juia3RW0REItTYCNOmwa5Es/CqVcHnCKmEISLSE8yYcTBZdNq1K/g+IkoYIiJhFLm6p2CrV+f3fTcoYYiI5NJZ3bNqFbgfrO4pp6RRV5ff992ghCEikksJqnsKNnMm1NR0/a6mJvg+IkoYIiK5lKC6J5Rs1WJTp8KsWVBfD2bBf2fNCr6PiHpJiYjkUlcXVEOl+75UsvWC6kwKU6dGmiBSqYQhIpJLCap7ciqDajElDBGRXEpQ3ZNTGVSLqUpKRCSMIlf35FQG1WIqYYiIVIIyqBZTwhARKUepPaIgfLVYyrJDYUgUIalKSkSk3GTqETVrFqxcmfeydVAfRVgqYYiIlJtCekSlWdYiutYrYYhI5Sv3cZ7yVUiPqCL2mlLCEJHKVgnjPOWrkHGhithrSglDRCpbHA+0JZdohg4NpihLN4X0iEqzrENH4UEpYYhI1EpdPVTqB9pSSzRbtgRTlKWbQh4UTLPsakjzAEf+zN2jWE9JTJo0yVtaWuIOQ0QySe2hA8HdbjGfih47Nv0DbfX1uXsURbm9Umw7D/t27mPh7Ut58LfbmbngvEXuPqnQdRa9hGFmvzCzjWa2OOm7IWbWZGYvJf47uNhxiEgJxFE9VOoH2mJueM7EO5zFv3+JH75/DhcNe4LBh+/nvM+fyncXTI5sG6WokroTuDDluxuAv7v7eODvic8iUuniGO+o1OM8xdzwnGzd0xu451MPc+WxDzOqegMnfXA81/9xCsu3D+Wak1r4w1cfY8uq1si2V5IqKTMbC9zv7icmPr8ATHH3dWY2Epjj7hNyrUdVUiJlrtTVQ3FIV+2WavbsoiSs1o2tzLt1KU1/bKVpySgW7x0PQK1t4d2jX6DhXe00fHIcdW8b1WU5M4ukSgp3L/oEjAUWJ33envL7tizLTgNagJa6ujoXKRuzZ7vX17ubBf+dPTvuiOI3e7Z7TY170AQcTDU1Pe/YzJ7t3qdP1/3snGprI9tM2942f/zOJT6zodmnHPmkf4w7fQX13o75+qoR/seTZ/ii2Uu9fX971vUALR7FtTyKleTcSAEJI3k6/fTT8zvaIsXSWy6M3dFbEmmRzoHlzav8v6bO9X8ctdAH29YDq76h7w98b1X/bm0vqoShKimR7ugNVS+SW2Nj0KC/enXQbjFzZt5VUdtWbOehW5bR9Of9NL1UzyttwbBPo6rW0TDuZRouqOL86ccx/JIzun3ORVUlFVfC+AGwxd2/a2Y3AEPc/cu51qOEIWWjqiq4x0tlBh2RPCMlPdS+nft45I6lNP1mO01PD6WldSId9GEQOzhv+FLefc4eGq4ZzfEXjcOq7OCC+Z5zScnsFPd9z7j3LzT2oo9Wa2b3AlOAoWa2BrgJ+C7wazO7FlgN/FOx4xCJVBm8zEYqg3c4S/+0nKY7X6NpQQ1zN02klVPpQxtnDlrG198xn4aPDOGsqydSXXNW5hXlc86lNMxXQ79odqYEbRhRTWrDkLKhNoxAb2mvyNO6Zzb4PZ962K8cN9+Prlp74BQZX/2KX3fiHP/9DY/69lXb81tpPudcfX2X+U4H9yjao6NYSakmJQwpK/lcLHvihVVJ84DWTa3+15uf8OtPb/aTBrxwsMOUbfYPj1ngt105z1c+/Gr4FWY6X8KeR2ZFSRgaGkSk2OIYLqMUenHDf/u+dp667wWaGjfS9PgRLNh+AvvoTz/2MnnwEhrOfIOGK4bz1ssmUNU3z+ejozhfhg4NxrdKmAS0uFvmBcJRwhAptp56Ye1uw38EPYvisGLeqzTNWsGDc6v5+2sT2OrBW09PHvACDW9ZR8MHBnHuJ0+gZmhNjjXlUOj50tgI11wD+/cf+CqqhKFXtIoUWxzDZZRCdxr+M716FMouaWxf9ToP/WwZTX/eR9OL9SxvqwfGcHTVOi4dt5SGC4x3X3ccw0+cAOR8KiC8Qs+XGTO6JIsoqYQhUmw9tYTRnaqTMj4W+3bu49FfLqPp19toemooTyR1d50ybBkNk3en7+4atUKPUZqSn0oYIpVi5sz0F9ZijaZaKp1JIZ/qpTIqbXmHs+z+zu6uA5mz8QRaOYUq2jlz0FJmnDufhg8P5qyrJ9Jv0JmlC6zQ8yVTyS8KUbScl2pSLympWMXoJVWJPa9SunsemOrrS7L59c9t9NnTH/arjp3vo5K6u765eoVPf8tc/92XH/FtK7eXJJasCvm3TdN7LapeUiphiJTC1Km56+jzaQyuoLaALkpc2tq1eRfz/2spTb/fSdOSkTy7ZwJwFENsK+ePeoGG816iYdo4xk4eSzDkXQ/Q+e//2c926SkViSiyTqkmlTCkx8r3mYaY79RDS3ennOvuuYC76/b97d5yz1L/zj80+7sGL/J+7HFw78ceP+/IJ/3bFzT7E3ct8ba9bZHtYuSier4l6RzRg3siPUm+CSDlwawDk1kpoz4oU2LI98LXjWVWzH/VZ31srn94zAKvtc0HFjtpwAt+/enN/tebn/DWTa2R73LRRHUzkHSO6ME9kZ4k32caouptFMUzEZl6Sw0cmL5KJFuMIfZr+6rXab6ls7trHS/vHwvA0VXraDhmOQ0XGOd/ajwjTh6W336Ui6gGtkw6llH1koq91JDPpBKG9Fj53lVGUW1RhKqPUFO2UlCGklMH5t84t9nPHvSsV9Hm4H4YO/ziYY/5jz4wx5f88SXvaO/IL+5yFVUJI+nfV1VSIj1Juot3v37B29uKUNfv7kWp+gg1ZVt/hphWUO9VtPlZhz3nX5/c7HN/8rTv3bE3vzgrpVdZlGN0JfZZCUOkHER5EUpeV22te3V1NBeNTKJqB8mUeGprQ1/4Nize6I3XPey3DvuGtzKwyzJ7rL8/dvE3fesr27q/r5U2UGLEyY1KekVrVJMShpSVYl6EStELqghVH4cchwwXvl1bdvnfvt3iX5zU7KcMeP7AYoNtq//oyJv8jZrh3hFlSSCOXmVlVKJRwhCJWzEvQqXoBVWEqo9MF8fU7q792e3gXs1en3Lkkz6zodkfv7OI3V1L3auszEo0USUM9ZIS6a5ivqa1VGMuFXHk2FUL1tA0awVNzX34+5oJbPFaAE4a8CINb1lLw/sHce60iRw27LBItpdVqcewKrMxs6J6p7ee9BbprmK+prVUT0SHeQI9pNdXv07zz5+n6f49NL1Qx0v7jwFGM7JqPRcfsywY3XX6eEacfBxwXCTbDO2ii+DWW7sm+GKO51VGY2ZFKopiSqkmVUlJWSl2tUOYOvAY68n3te7z+T97xm98R7O/bdCz3of9B7q7XnTU4/7D98/xxX8og+6u6f6dzNynTy/eNovdZpLnvztqwxApA3E2bBaSsLoRd0d7hy/783L/yYfm+KXDH/XDed3BvYo2P/Ow53zGOc0+50dP5d/dtdjiavAu1s1EN9bdIxIG8HlgCbAYuBcYkG1+JQyRJPlcCLvZZXfD4o3+q08v8GvGz/PRfV47MPu4viv9kxPn+m+/uLCw7q7p4os68cY1jEqx9qkbCbDiEwYwClgBDEx8/jVwdbZllDBEkoS9EKa7I81wwdm1ZZc/8J0W/9IZzX7qwGVdurt+aNRCv/Wjc31586rwMYatVivG3XjntrPsb0XqRgLsKQnjVWAIQeP7/cAF2ZZRwhBJEvZOM+TQHe1Yl+6u7zziKf+3dzf7Y79Y3L3urmETQTGqjHIlyXJ+aC+X3ljCCPaBzwI7gU1AY4Z5pgEtQEtdXV13Dq9Iz5J855x6t5nuQhhy6I7XONo/f1qz/+Vbj/uOdTsKjzPsha0YVUa5ShaVmizce2cbBjAYeAg4CqgG/gBckW0ZlTAkozJ6qraoMvX4yXIhbB81JmeyaB8wMPpjFjYRFKOEUW7Dv0ett/WSAv4JuCPp85XALdmWUcKQtMrsqdqiCnFx3de6zx++5Rm/6Z3N/vbDn/Gp3O076Xp82vtWe8eQIcVNsGETQTH+/SrlBVMl0hMSxlmJHlI1gAF3Af+SbRklDEmrHC4OpSrhZBr+28x/+o9z/L0jDnZ3Ndr9jMMW+9fe3uxLps70jjF1+ccX8bulMyaCqI9fb7qJCKHiE0awD3wLeD7RrfYeoH+2+ZUwJK24qx8yXZymT48+iWQZ/hvcj+m7yqcdP9d/c/1C3/Ly1vj3K+7nVHpDNWUIPSJh5DspYUhacZcwMm0/TIN0siwXuM7uro3HzPBdDOiy3t02wJve9nV/+e8ry3O/CqGLfiRKljCAB4FTothYoZMShhxQ6LsjorwQ5fMCoTzeoNfWb4D/78kz/N1DWnwAuxyC7q7/OnCmbxsw3Dsw76irK3n1V8mSs6qVIlPKhHFaojfTL4GRUWy0u5MShrh7995Ol2v5Qi5E+byiNFM1WZaqprf0f9E/99Y5/udvJnV3LcWddxT7VYztJycnlUBCKXmVFPAh4Bngps6ns0s9KWGIuxdeBRV1FVa2rq5ZtvH6q6/7H7/2qH/6pDneTubG7FDbK8addzf3KzK52qZUAgmtpAkj0YvpROBTwGZgDfCxKALIZ1LCEHcvvJE72/LdvWNNXW769EMuZh01Nf7CNd/xb05p9nMOf+bA6K4DafX1VSPSx9Snz6GxlLLNJsR+Fe0iXVubfj9ra4Pf4267qiClrJJ6GFgLNAE3A5cAbwZ+CsyKIoiwkxKGuHvxShh5vIM6jI57Zvu+YUd7B+Ybqkb4x7ktyEu0+6SaJf7VtzX7Q//xpO95fU+48Z46YymHXmGlqAbKlTDiPg4VpJQJ40QI3syX5rdlUQQRdlLCEHcvvCoi0/KZLlB53LFuen6z3/eZBX7tcfO8rs+rB1Yxtu9q/+fj5/qvP7/QN7+4JXNcnRfiPn0yx9Jb7qxzJYTechwiUBbdaoFxUQQRdlLCkANmz+56ga+tDb4Le/ebbr5svYIyrGv3tt3e9L1F/uUzm/2tA5cemP0ItvsHRj7it1w21196cGX+LxHKVW3Wr1/X7/v163l197kSgtowQiuLhFHqSQlDDkh3saiuPvRCms8FJFevoJoab7/7Hn/qvuf9+xc1e0NSd9e+7PN3HPGU33x+sz9y23O+f/f+wvYvUyx9+gTtCKndiKure96FMkxCUC+pUKJKGBasqzJMmjTJW1pa4g5DysHYsenfp51OfT2sXJl7vsbGQ9+jneJVRlPHqwCc0P9lGia+RsP7anjnpyYyaMSgcPGEkS0Ws+DymSrsflaSxkaYMSN4F3ZdXfAO7ojeQd6bmNkid59U8HqUMKQiVVWlv2hm06dPcBG+5Zau3ydflIYMoaPDsW1bsTSr6MC45xPzePcnj2XUpJHdDj+Uxka46ipobw83vxl0dBQ3JqlIUSWMqiiCESmJxsagZFFVFUz5am+Hn/8crruuyzp92rSgtOIOW7awe9seNlObdhVV9XVcddvk4icLCO6k80kAdXXFi0UEJYzoJV/Uxo4NPkthGhth6FC44oqDF/Z0d93V1dCvX87V+axZvNS0klsum8vGK7+IpVT7HMYujqhpwwfWdF2wpiaoEimlTEnAUso/ccQmvU8UDSGlmsq+0Vu9NqKX6/mE1AfbkhtBMyzTAQc+ZnrCuqCH+Iq9/7lGjC2HuKWsoF5SZUj9wsPJ54KWq+dShoe0dm/b7e1WlXaZNqr8lsvm+osPrAgG7yv3f7N8jpduWiQNJYxyVKlPnpbyjjTfC1quEVMTF/b2/e3+9H8/7z+4uNkvqH3CB9LqP2W6d6QpXfj06d2Pp9x156ZFJZIeTwmjHFViCaPUF8x8j1GWEkb7gIE+553f8I/WP+zDbOOBnyb2e9k/c8oc/9M3HvP9k6d0Xe7889Mfg55ywcz3pqWnJUxJSwmjHFXi/3ylTnIRXNA6wLdwpF/ObAf3YbbRP1r/sP/y2vn+6uNrsy5b9v8ehYoqIZfzTY7kLaqEoV5SUZo6FWbNCh6gMgv+O2tWeT9otHp1ft8XKlOvnzTft+1p49Hdp/C7sZ9nvY2kA2Ml9VzNL7i8tonTLh7F0//9Auv21dK48hyuvn0yo89I6u46Y8ahD77t2hU829BTe7HNnBn0mEqWrQdVqf/9pbJFkXVKNZV9CaMSlfoOM8tdf0d7h7/4wAr/2Ufm+PtHPuJHsD0ofNDupw1c6l85q9kf/P4i371td7hthXljXE8scUTRqUAljB4FVUlJJOKotkm6oLUdPdoXXniTf2LCXK9PGt21rs+rfu1x8/y+zyzwTc9v7t52wr4xrjdfHHtjtV0v1CMSBnAk8FvgeWAZ8LZs8ythFEkhjb55Lrvn9T3+939/0m84u9lPr1niRruD+5vY7u8f+Yj/7CNzgu6u+Y7umim2XO+YyNZ+0lv0pEZ/SSuqhBHrWFJmdhcw391vN7N+QI27b880v8aSKjPpBsirqenSbuMdznO/e4mmu9bS9Mgg5m05gd3U0Jf9nP2mpTScvo2Gy2o548qJ9B3Qtzgxdo4TVVWV/gnxnjhon0iSih980MzeRPCO8HEeMggljDKTYcTYthGjabzkXpoeNB5cPZ4NHcMAOL7fchqOX0PDewcyZfpEDj/68NLGGyLBifREPWHwwXHAJuCXZvaUmd1uZoelzmRm08ysxcxaNm3aVPooe7pCxr7K0JOmav1rXH37ZB5YdRzvGvMSze+4kf0jRrNs/3h+8vpVXHr8S6VPFlCZvdhEykkU9VrdmYBJQBtwVuLzj4Gbsy2jNoyIdNZZd9bf59ng2ba3zR+9/TnfNmB42jaB7QOG+1P3Pe/t+9vVqCpSBugBz2GsAda4+2OJz78FTosxnt6hs1qmsyoptTZw166gzj+Jdzgv/30VP798Hh88+lGGDtjB2Z84kdl7Pki6usQjrvkgp35kAlV9qzI/C5GyDREpf7ElDHdfD7xqZhMSX50PLI0rnlB6wtDl6S7gqVavZuvybfzm+keYNnEe4/qvYfy767nuvnfQsnEMH3zzc9z7LwuZPvJPaV8yxF/+0mVdmbYhIhUmimJKdyfgVKAFeBb4AzA42/yxVklVctVKiCG/k6e1NrJLd9f3jXjU//Of5vgL//dK1+6uYYb5yPfBsFIPhKjupNIL0BOew8h3ijVhVOoTsWGfRUhMO6nxm/p/2785pdkX3Pqs79+9P/O6wxyTfBJtKZNyJd8AiORJCaPUKnXo8hBPO7dj3gHe+qYR3vqT28KvO+xFN+ydfClLI5V6AyDSDUoYpVaBF5idG3Z6R4Y3ynWAd2C+f8Sowu6qp08P3noHwX+T3zWRr3yScqElhEq9ARDphqgShkarDSvfUUBj0L6vncfuWMzMhjlMOfJpBg+vZhXpR4e1Pn2w2ffQd92a7j+H0NgId9118Onp9vbgc3c7A+Qxkm3Bva/y2ZaIAPE+uFdZyvShrw1f+zE7DhtBh1XxWv+x/PgTz/D1B6fwxr7+fO6Mhez48CfwgQMPXbC9PeheW0hPr6i7zOaTlAvtfVUBNwAiZSeKYkqppl7x4F6OevktL2/131y/0KcdP9c/az/2nXStltnft7+//v2fH7rOTFUwhVSpFaNap1jtHYVsS6TC0RMGH8xXjx9LKs1YRz6whmUfnEHjirfT9MxRtLROxKnicN7g5aoJDOtYf+h6UgfTa2yEK65Iv00z6OjoXrwZxpIqyWB+GhdKJLSeMJaUpEpTxWO7d1HTOIvvLZxMdVU7N75zHg/f8ixbWgcyzDekX09qtUy2KqJC6uzjrNYp0ypCkR4timJKqabQVVIVVtWw9qn1fvcn52fp0WS+fdX2QxcMWy2T7YG9Qo9NhR1rkd4IVUllUAFVFa0bW5l361Ka/thK05JRLN47HoDVjGEMaw5dIFMVT9h9zVR1VFsLmzcXtC8iUv5UJZVJpp47V10VzRhQ3RhPqn1fO4//ckmX7q4X3XQGtzx5NsNrdvDdC+ewaPYyRt39nfyqeDqrZWprD36XrkdUpqqjH/84Z+wiIgdEUUwp1RSqSirMeEndHQIij4fFljev8ls/Otc/NGqhD7atB2Y/deAy/+KkZv/bt1t815Zd6bfRWdXU+UBctqqeqJ+2LpSqqETKDnrSO4MQQ2F0uztpljaDra9s899+caF/cuJcH9d35YGfRvd5za8ZP89/9ekFvmHxxnDbyecp5nJ6Al3jM4mUpagSRu9ow0gntTtp8ruf6+qCapzUNo+qquAymKIDo5r9dNCHQezgvOFLaZi8h4aPj2HChcdgVWkHAc8sn+6qGWIqqLtsd8XZzVZEMoqqDaNvFMGUlc6LfOfFv6rq4NAVyZK7k6YmmVWrgs9J6/MOZ//QkfTbtPaQVW2wEXz93Pk0fGQIZ109keqaswrbh3yeYq6rS3+RjmOIC737QqRH63mN3hBc5FeuDO6w77ord0Nyhobyti/dwOzpC7jq2IcZXb2eqzd9n1a6rssHDmTkPT/gW3OnMPm6k6muqS48/nzGOSrlsxC5Gvw1PpNIzxZFvVappm4PDZKrITZDQ3k75uBea5v9w2MW+G1XzvONN/7Evbb24Hy1tbnr6PNtCM63LaAUDc1hYlIbhkhZQo3ehWvb2+aP37nEtw8YnjZhtPZ9k++pHekdyRfi7lzMu3MRLbfeRmEb18stbhFRo3d3rZj3Kk2zVtA0p5qH1k5gqw/hchq5g08wkD0HZ6yuDhqO9+07+F1NTfCcw5Yth644U8NuT2kILqfGdRHJixq9Q9q+6nUe+tkymv68j6YX61neVg+MYVTVOt577BIaLqji/OkXMPCZ27v2ktq589DEsGtX5t5X+Tb4VlpDcDk1rotILHpcwti3cx+P/nIZTb/eRtNTQ3midSIdnM0gdjBl2DI+M3klDdeM5viLxmFVIw8ueOLUrt1oq/LsD5CtwbcnXGhnzkw/DIneHyHSa8SeMMysD9ACvObul+S7vHc4y+5fTtOdr9G0YCBzNp5AK6dQRTtnDlrKjHPn0/DhwZz98ROorjkz/IozXehra2H37vAXzp5yoU3trpzpWRUR6bFib8Mws+uBScCbciWMSWbeUl/P9k9+hftXn0zTA86DK49lbUdQUnhz9UoajltNw8X9OO+6iRxZf0T3A8s2sB/kd+EM81CgiEiRRNWGEWvCMLPRwF3ATOD6UAkDaKWGf2YWf7P3cP6oF2g4bz8N08YxdvLoaAPUhV5EeoCekjB+C3wHOBz4YrqEYWbTgGkAp8PpnX2k9taOpHr9Gqr69sxnD0VEolLxw5ub2SXARndflG0+d5/l7pNSd7b/1vVKFiIiJRTnFfcc4L1mthK4D3iXmc0OvXSl9TIqpW68s0NEJJfYEoa7f9XdR7v7WOAy4CF3vyLUwpXYy6hUOhvrV60KHrTrHEhRSUNEClR5dTr19WX1utWyk+mNgzNmxBOPiPQYsXerzUcUQ4P0eBrCQ0RSVHyjtxSJhhgXkSJRwuhpSvl+DBHpVZQwepqpU4M2nvr6oBpKbT4iEpHYx5KSIpg6VQlCRCKnEoaIiISihCEiIqEoYYiISChKGJVAQ32ISBlQo3e5S30vR+dQH6CGbREpKZUwyp2G+hCRMtF7EkalVuusXp3f9yIiRdI7EkYlj+CqoT5EpEz0joRRydU6GupDRMpE70gYlVyto6E+RKRM9I5eUnV1QTVUuu8rgYb6EJEy0DtKGKrWEREpWO9IGKrWEREpWO+okgJV64iIFKh3lDBERKRgShgiIhJKbAnDzMaYWbOZLTOzJWb22bhiERGR3OJsw2gDvuDuT5rZ4cAiM2ty96UxxiQiIhnEVsJw93Xu/mTi7x3AMmBUXPGIiEh2ZdGGYWZjgbcCj6X5bZqZtZhZy6ZNm0oem4iIBGJPGGY2CPgf4HPu/kbq7+4+y90nufuko446qvQBiogIEHPCMLNqgmTR6O6/izMWERHJLs5eUgbcASxz9/8XVxwiIhJOnCWMc4CPAe8ys6cT00UxxiMiIlnE1q3W3R8GLK7ti4hIfmJv9BYRkcqghCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISSqwJw8wuNLMXzOxlM7shzlhERCS72BKGmfUBfga8BzgBuNzMTogrHhERyS7OEsaZwMvu/oq77wPuA94XYzwiIpJF3xi3PQp4NenzGuCs1JnMbBowLfFxr5ktLkFshRoKbI47iBAUZ3QqIUZQnFGrlDgnRLGSOBOGpfnOD/nCfRYwC8DMWtx9UrEDK5TijFYlxFkJMYLijFolxRnFeuKskloDjEn6PBpYG1MsIiKSQ5wJ4wlgvJkdY2b9gMuA/40xHhERySK2Kil3bzOzTwN/A/oAv3D3JTkWm1X8yCKhOKNVCXFWQoygOKPWq+I090OaDURERA6hJ71FRCQUJQwREQmlLBJGriFCLPCTxO/PmtlpYZctcZxTE/E9a2YLzeyUpN9WmtlzZvZ0VF3cCohzipm9nojlaTO7MeyyJY7zS0kxLjazdjMbkvitJMfTzH5hZhszPf9TRudmrjjL5dzMFWe5nJu54iyHc3OMmTWb2TIzW2Jmn00zT7Tnp7vHOhE0eC8HxgH9gGeAE1LmuQj4K8GzG2cDj4VdtsRxvh0YnPj7PZ1xJj6vBIaWyfGcAtzfnWVLGWfK/JcCD8VwPN8BnAYszvB77OdmyDhjPzdDxhn7uRkmzjI5N0cCpyX+Phx4sdjXznIoYYQZIuR9wN0eeBQ40sxGhly2ZHG6+0J335b4+CjBsyWlVsgxKavjmeJy4N4ixZKRu88DtmaZpRzOzZxxlsm5GeZ4ZlJWxzNFXOfmOnd/MvH3DmAZwQgaySI9P8shYaQbIiR1pzPNE2bZqOS7rWsJMnsnBx4ws0UWDHdSLGHjfJuZPWNmfzWzt+S5bBRCb8vMaoALgf9J+rpUxzOXcjg38xXXuRlW3OdmaOVybprZWOCtwGMpP0V6fsY5NEinMEOEZJon1PAiEQm9LTM7j+B/yslJX5/j7mvNbBjQZGbPJ+5i4ojzSaDe3Xea2UXAH4DxIZeNSj7buhRY4O7Jd3ylOp65lMO5GVrM52YY5XBu5iP2c9PMBhEkrM+5+xupP6dZpNvnZzmUMMIMEZJpnlIOLxJqW2Z2MnA78D5339L5vbuvTfx3I/B7giJhLHG6+xvuvjPx91+AajMbGmbZUsaZ5DJSivwlPJ65lMO5GUoZnJs5lcm5mY9Yz00zqyZIFo3u/rs0s0R7fha7YSZEw01f4BXgGA42vrwlZZ6L6dpw83jYZUscZx3wMvD2lO8PAw5P+nshcGGMcY7g4EObZwKrE8e2rI5nYr4jCOqSD4vjeCa2MZbMjbSxn5sh44z93AwZZ+znZpg4y+HcTByXu4EfZZkn0vMz9iopzzBEiJl9KvH7rcBfCFr7XwZ2AddkWzbGOG8EaoFbzAygzYORLIcDv0981xf4lbv/X4xx/iMw3czagN3AZR6cReV2PAE+ADzg7q1Ji5fseJrZvQQ9d4aa2RrgJqA6KcbYz82QccZ+boaMM/ZzM2ScEPO5CZwDfAx4zsyeTnz3NYKbg6KcnxoaREREQimHNgwREakAShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEIVKAxPsIGhJ//5uZ/STumESKJfYnvUUq3E3AvyYGmnsr8N6Y4xEpGj3pLVIgM5sLDAKmePBeApEeSVVSIgUws5MI3ny2V8lCejolDJFuSry5rJHgTWWtZvYPMYckUlRKGCLdkHjT2u+AL7j7MuBm4JuxBiVSZGrDEBGRUFTCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQnl/wNIgRvRIRIfLQAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "id": "6dd6f473", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -3712,35 +3041,9 @@ "plt.title(r'Random numbers ')\n", "plt.show()" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "ffd93055", - "metadata": {}, - "outputs": [], - "source": [] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.8" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index 96e0eb799..f17d5ea21 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -1177,7 +1177,7 @@ The Hessian matrix of Ridge Regression for our simple example is given by !et This implies that the Hessian matrix is positive definite, hence the stationary point is a minimum. -Note that the Ridge loss function is convex, as a sum of two convex +Note that the Ridge cost function is convex being a sum of two convex functions. Therefore, the stationary point is a global minimum of this function.