diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html index 22236b38e..620723633 100644 --- a/doc/pub/week39/html/week39-reveal.html +++ b/doc/pub/week39/html/week39-reveal.html @@ -199,6 +199,11 @@ MathJax.Hub.Config({ diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html index a503ae23e..6914a3f5e 100644 --- a/doc/pub/week39/html/week39-solarized.html +++ b/doc/pub/week39/html/week39-solarized.html @@ -314,6 +314,9 @@ MathJax.Hub.Config({ diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html index 6da4ddd96..85922cc9a 100644 --- a/doc/pub/week39/html/week39.html +++ b/doc/pub/week39/html/week39.html @@ -391,6 +391,9 @@ MathJax.Hub.Config({ diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz index 77def0462..75e5c1cb8 100644 Binary files a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz and b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 6cb26a370..be6bd4b83 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "4c73ece8", - "metadata": {}, + "id": "266a989e", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,8 +14,10 @@ }, { "cell_type": "markdown", - "id": "22602ad0", - "metadata": {}, + "id": "0556a7de", + "metadata": { + "editable": true + }, "source": [ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -25,13 +29,17 @@ }, { "cell_type": "markdown", - "id": "f2fcd724", - "metadata": {}, + "id": "fba6c7a0", + "metadata": { + "editable": true + }, "source": [ "## Plan for week 39\n", "\n", "* Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent\n", "\n", + " * [Video of lecture](https://youtu.be/rDBj50Lv3Go)\n", + "\n", "* Friday: Stochastic Gradient descent with examples and automatic differentiation\n", "\n", "* Reading recommendations:\n", @@ -47,8 +55,10 @@ }, { "cell_type": "markdown", - "id": "67125149", - "metadata": {}, + "id": "b647f340", + "metadata": { + "editable": true + }, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -66,8 +76,10 @@ }, { "cell_type": "markdown", - "id": "0d07977c", - "metadata": {}, + "id": "579fc60e", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -81,8 +93,10 @@ }, { "cell_type": "markdown", - "id": "973763d7", - "metadata": {}, + "id": "ab60fa1c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -94,16 +108,20 @@ }, { "cell_type": "markdown", - "id": "d0a82293", - "metadata": {}, + "id": "86e72472", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] }, { "cell_type": "markdown", - "id": "de856843", - "metadata": {}, + "id": "d313a983", + "metadata": { + "editable": true + }, "source": [ "## The equations to solve\n", "\n", @@ -116,8 +134,10 @@ }, { "cell_type": "markdown", - "id": "a60566a0", - "metadata": {}, + "id": "a81835f9", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -126,8 +146,10 @@ }, { "cell_type": "markdown", - "id": "ef8826c0", - "metadata": {}, + "id": "a6eb84ee", + "metadata": { + "editable": true + }, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -135,8 +157,10 @@ }, { "cell_type": "markdown", - "id": "566fdac8", - "metadata": {}, + "id": "63610cb8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -145,16 +169,20 @@ }, { "cell_type": "markdown", - "id": "69a2105f", - "metadata": {}, + "id": "b1049f68", + "metadata": { + "editable": true + }, "source": [ "This defines what is called the Hessian matrix." ] }, { "cell_type": "markdown", - "id": "0e766c2c", - "metadata": {}, + "id": "61782910", + "metadata": { + "editable": true + }, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -165,8 +193,10 @@ }, { "cell_type": "markdown", - "id": "900e0a5d", - "metadata": {}, + "id": "889f7cf7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -175,16 +205,20 @@ }, { "cell_type": "markdown", - "id": "fb485e9e", - "metadata": {}, + "id": "cf341d16", + "metadata": { + "editable": true + }, "source": [ "or in matrix form as" ] }, { "cell_type": "markdown", - "id": "c7110832", - "metadata": {}, + "id": "99617477", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -193,8 +227,10 @@ }, { "cell_type": "markdown", - "id": "db0fd8b9", - "metadata": {}, + "id": "048efeba", + "metadata": { + "editable": true + }, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -203,8 +239,10 @@ }, { "cell_type": "markdown", - "id": "73cc9db9", - "metadata": {}, + "id": "7eeb2a3f", + "metadata": { + "editable": true + }, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -221,8 +259,10 @@ }, { "cell_type": "markdown", - "id": "0483698c", - "metadata": {}, + "id": "4b243b9f", + "metadata": { + "editable": true + }, "source": [ "## The equations\n", "\n", @@ -235,8 +275,10 @@ }, { "cell_type": "markdown", - "id": "0b41326e", - "metadata": {}, + "id": "232e81e8", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -249,8 +291,10 @@ }, { "cell_type": "markdown", - "id": "bf22ca78", - "metadata": {}, + "id": "5048b0ce", + "metadata": { + "editable": true + }, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -258,8 +302,10 @@ }, { "cell_type": "markdown", - "id": "e70a4b40", - "metadata": {}, + "id": "26fc11aa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -268,16 +314,20 @@ }, { "cell_type": "markdown", - "id": "e6ab42db", - "metadata": {}, + "id": "268b40ad", + "metadata": { + "editable": true + }, "source": [ "yielding" ] }, { "cell_type": "markdown", - "id": "60f49110", - "metadata": {}, + "id": "1e8df82c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -286,16 +336,20 @@ }, { "cell_type": "markdown", - "id": "4a5daac5", - "metadata": {}, + "id": "9dad7b22", + "metadata": { + "editable": true + }, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] }, { "cell_type": "markdown", - "id": "dc406fad", - "metadata": {}, + "id": "40fb9be6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -304,8 +358,10 @@ }, { "cell_type": "markdown", - "id": "4a01cef1", - "metadata": {}, + "id": "3d5b5285", + "metadata": { + "editable": true + }, "source": [ "## Simple geometric interpretation\n", "\n", @@ -324,8 +380,10 @@ }, { "cell_type": "markdown", - "id": "b9ab351f", - "metadata": {}, + "id": "92ce5653", + "metadata": { + "editable": true + }, "source": [ "## Extending to more than one variable\n", "\n", @@ -335,8 +393,10 @@ }, { "cell_type": "markdown", - "id": "c4a5c175", - "metadata": {}, + "id": "0defe140", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -346,16 +406,20 @@ }, { "cell_type": "markdown", - "id": "5a0ec73f", - "metadata": {}, + "id": "9c81d7b3", + "metadata": { + "editable": true + }, "source": [ "which we Taylor expand to obtain" ] }, { "cell_type": "markdown", - "id": "43a0b666", - "metadata": {}, + "id": "5ec8b48f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -370,16 +434,20 @@ }, { "cell_type": "markdown", - "id": "9378b391", - "metadata": {}, + "id": "39d6e38f", + "metadata": { + "editable": true + }, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] }, { "cell_type": "markdown", - "id": "742ab267", - "metadata": {}, + "id": "ae2035ba", + "metadata": { + "editable": true + }, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -391,16 +459,20 @@ }, { "cell_type": "markdown", - "id": "974e20a1", - "metadata": {}, + "id": "96aca494", + "metadata": { + "editable": true + }, "source": [ "we can rephrase Newton's method as" ] }, { "cell_type": "markdown", - "id": "65662f4e", - "metadata": {}, + "id": "a8df5a1b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -411,16 +483,20 @@ }, { "cell_type": "markdown", - "id": "9363cc25", - "metadata": {}, + "id": "1b245bd4", + "metadata": { + "editable": true + }, "source": [ "where we have defined" ] }, { "cell_type": "markdown", - "id": "6e0064bd", - "metadata": {}, + "id": "2e07b2d6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -431,8 +507,10 @@ }, { "cell_type": "markdown", - "id": "94fd2ee4", - "metadata": {}, + "id": "1996803f", + "metadata": { + "editable": true + }, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -444,8 +522,10 @@ }, { "cell_type": "markdown", - "id": "36b663f2", - "metadata": {}, + "id": "a0a6d7cd", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent\n", "\n", @@ -459,8 +539,10 @@ }, { "cell_type": "markdown", - "id": "9d0c9ad9", - "metadata": {}, + "id": "be745e5e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -469,8 +551,10 @@ }, { "cell_type": "markdown", - "id": "47f4fafb", - "metadata": {}, + "id": "1fb290b3", + "metadata": { + "editable": true + }, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -481,8 +565,10 @@ }, { "cell_type": "markdown", - "id": "e380cfcc", - "metadata": {}, + "id": "5daf5176", + "metadata": { + "editable": true + }, "source": [ "## More on Steepest descent\n", "\n", @@ -494,8 +580,10 @@ }, { "cell_type": "markdown", - "id": "db59d2b8", - "metadata": {}, + "id": "01556473", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -504,8 +592,10 @@ }, { "cell_type": "markdown", - "id": "21c9aac9", - "metadata": {}, + "id": "bb911199", + "metadata": { + "editable": true + }, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -513,8 +603,10 @@ }, { "cell_type": "markdown", - "id": "327ae752", - "metadata": {}, + "id": "8c872bd4", + "metadata": { + "editable": true + }, "source": [ "## The ideal\n", "\n", @@ -539,8 +631,10 @@ }, { "cell_type": "markdown", - "id": "f1c424b9", - "metadata": {}, + "id": "e89e8886", + "metadata": { + "editable": true + }, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -559,8 +653,10 @@ }, { "cell_type": "markdown", - "id": "c30dc8df", - "metadata": {}, + "id": "b975f76b", + "metadata": { + "editable": true + }, "source": [ "## Convex functions\n", "\n", @@ -579,8 +675,10 @@ }, { "cell_type": "markdown", - "id": "6298f287", - "metadata": {}, + "id": "5c99441c", + "metadata": { + "editable": true + }, "source": [ "## Convex function\n", "\n", @@ -589,8 +687,10 @@ }, { "cell_type": "markdown", - "id": "ea9513ed", - "metadata": {}, + "id": "04eb49ef", + "metadata": { + "editable": true + }, "source": [ "## Conditions on convex functions\n", "\n", @@ -624,8 +724,10 @@ }, { "cell_type": "markdown", - "id": "2f218b22", - "metadata": {}, + "id": "5fdf3577", + "metadata": { + "editable": true + }, "source": [ "## More on convex functions\n", "\n", @@ -650,8 +752,10 @@ }, { "cell_type": "markdown", - "id": "67110e2a", - "metadata": {}, + "id": "49b00751", + "metadata": { + "editable": true + }, "source": [ "## Some simple problems\n", "\n", @@ -678,8 +782,10 @@ }, { "cell_type": "markdown", - "id": "c7ec5012", - "metadata": {}, + "id": "eda76339", + "metadata": { + "editable": true + }, "source": [ "## Standard steepest descent\n", "\n", @@ -696,8 +802,10 @@ }, { "cell_type": "markdown", - "id": "54f15818", - "metadata": {}, + "id": "853dc57b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -706,16 +814,20 @@ }, { "cell_type": "markdown", - "id": "bb846136", - "metadata": {}, + "id": "5379fa5a", + "metadata": { + "editable": true + }, "source": [ "In the iterative process we end up with a problem like" ] }, { "cell_type": "markdown", - "id": "14a050e6", - "metadata": {}, + "id": "a49c9a24", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -724,8 +836,10 @@ }, { "cell_type": "markdown", - "id": "b5be57be", - "metadata": {}, + "id": "42e20b26", + "metadata": { + "editable": true + }, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", @@ -734,8 +848,10 @@ }, { "cell_type": "markdown", - "id": "6a8ce76f", - "metadata": {}, + "id": "c0af34f2", + "metadata": { + "editable": true + }, "source": [ "## Gradient method\n", "\n", @@ -744,8 +860,10 @@ }, { "cell_type": "markdown", - "id": "594769d8", - "metadata": {}, + "id": "93e49cad", + "metadata": { + "editable": true + }, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -754,8 +872,10 @@ }, { "cell_type": "markdown", - "id": "05f1fe97", - "metadata": {}, + "id": "991756d0", + "metadata": { + "editable": true + }, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite." @@ -763,8 +883,10 @@ }, { "cell_type": "markdown", - "id": "6efaa986", - "metadata": {}, + "id": "9e0bab1f", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent method\n", "\n", @@ -774,8 +896,10 @@ }, { "cell_type": "markdown", - "id": "99e242c6", - "metadata": {}, + "id": "b7e002d6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -784,16 +908,20 @@ }, { "cell_type": "markdown", - "id": "07d95ef1", - "metadata": {}, + "id": "41a88a3a", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "b81f2f36", - "metadata": {}, + "id": "bf7d650e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -802,16 +930,20 @@ }, { "cell_type": "markdown", - "id": "83c5e15f", - "metadata": {}, + "id": "21b306ec", + "metadata": { + "editable": true + }, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "0d5239fd", - "metadata": {}, + "id": "c2595c2c", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -819,8 +951,10 @@ }, { "cell_type": "markdown", - "id": "95cf152f", - "metadata": {}, + "id": "d9631d23", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -829,8 +963,10 @@ }, { "cell_type": "markdown", - "id": "18692235", - "metadata": {}, + "id": "e9520154", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -839,8 +975,10 @@ }, { "cell_type": "markdown", - "id": "a5076cdd", - "metadata": {}, + "id": "fe8ab6ff", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -849,8 +987,10 @@ }, { "cell_type": "markdown", - "id": "daa366ec", - "metadata": {}, + "id": "48b2df0a", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." @@ -858,8 +998,10 @@ }, { "cell_type": "markdown", - "id": "75a681f3", - "metadata": {}, + "id": "3120af28", + "metadata": { + "editable": true + }, "source": [ "## Final expressions\n", "We can compute the residual iteratively as" @@ -867,8 +1009,10 @@ }, { "cell_type": "markdown", - "id": "98536b08", - "metadata": {}, + "id": "349f81a7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -877,16 +1021,20 @@ }, { "cell_type": "markdown", - "id": "aff9b796", - "metadata": {}, + "id": "2fa0028e", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "4c01226c", - "metadata": {}, + "id": "f9ec53a2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -895,16 +1043,20 @@ }, { "cell_type": "markdown", - "id": "62d91969", - "metadata": {}, + "id": "e881ad44", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "a5d9f79a", - "metadata": {}, + "id": "9e652153", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -913,16 +1065,20 @@ }, { "cell_type": "markdown", - "id": "04fb0972", - "metadata": {}, + "id": "235b588a", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "69f177ee", - "metadata": {}, + "id": "1b1cf967", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -931,16 +1087,20 @@ }, { "cell_type": "markdown", - "id": "826c4c1a", - "metadata": {}, + "id": "998bb78c", + "metadata": { + "editable": true + }, "source": [ "leading to the iterative scheme" ] }, { "cell_type": "markdown", - "id": "5b65620e", - "metadata": {}, + "id": "59cccb52", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_{k},\n", @@ -949,8 +1109,10 @@ }, { "cell_type": "markdown", - "id": "5ff7d0ef", - "metadata": {}, + "id": "8e031fcc", + "metadata": { + "editable": true + }, "source": [ "## Steepest descent example" ] @@ -958,8 +1120,11 @@ { "cell_type": "code", "execution_count": 1, - "id": "aeb6d5c9", - "metadata": {}, + "id": "42d5c457", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "%matplotlib inline\n", @@ -988,8 +1153,10 @@ }, { "cell_type": "markdown", - "id": "3ae5899d", - "metadata": {}, + "id": "3d0317f0", + "metadata": { + "editable": true + }, "source": [ "And then as countor plot" ] @@ -997,8 +1164,11 @@ { "cell_type": "code", "execution_count": 2, - "id": "c6c3a008", - "metadata": {}, + "id": "23fce077", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1008,8 +1178,10 @@ }, { "cell_type": "markdown", - "id": "0c47c360", - "metadata": {}, + "id": "19c01aa8", + "metadata": { + "editable": true + }, "source": [ "Find guesses" ] @@ -1017,8 +1189,11 @@ { "cell_type": "code", "execution_count": 3, - "id": "db86ff05", - "metadata": {}, + "id": "3c018860", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1027,8 +1202,10 @@ }, { "cell_type": "markdown", - "id": "9e873223", - "metadata": {}, + "id": "07d57e1b", + "metadata": { + "editable": true + }, "source": [ "Run it!" ] @@ -1036,8 +1213,11 @@ { "cell_type": "code", "execution_count": 4, - "id": "71c9d8c1", - "metadata": {}, + "id": "5f77c7ff", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1051,8 +1231,10 @@ }, { "cell_type": "markdown", - "id": "8ffc7e06", - "metadata": {}, + "id": "31dfadfb", + "metadata": { + "editable": true + }, "source": [ "What happened?" ] @@ -1060,8 +1242,11 @@ { "cell_type": "code", "execution_count": 5, - "id": "857c0d1d", - "metadata": {}, + "id": "0cedd6a4", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1072,16 +1257,20 @@ }, { "cell_type": "markdown", - "id": "78a9e67c", - "metadata": {}, + "id": "c9a57db7", + "metadata": { + "editable": true + }, "source": [ "Note that we did only one iteration here. We can easily add more using our previous guesses." ] }, { "cell_type": "markdown", - "id": "a759a930", - "metadata": {}, + "id": "720e9388", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", @@ -1092,8 +1281,10 @@ }, { "cell_type": "markdown", - "id": "f185c679", - "metadata": {}, + "id": "ccdd73b8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1102,8 +1293,10 @@ }, { "cell_type": "markdown", - "id": "bbef395d", - "metadata": {}, + "id": "76cd5686", + "metadata": { + "editable": true + }, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1111,8 +1304,10 @@ }, { "cell_type": "markdown", - "id": "1f1417d3", - "metadata": {}, + "id": "001483f6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1121,8 +1316,10 @@ }, { "cell_type": "markdown", - "id": "8482b2fb", - "metadata": {}, + "id": "29da6d30", + "metadata": { + "editable": true + }, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." @@ -1130,8 +1327,10 @@ }, { "cell_type": "markdown", - "id": "12aa0a01", - "metadata": {}, + "id": "1cccf0ea", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" @@ -1139,8 +1338,10 @@ }, { "cell_type": "markdown", - "id": "da7bd359", - "metadata": {}, + "id": "ef9f4a74", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1149,16 +1350,20 @@ }, { "cell_type": "markdown", - "id": "85c05b4f", - "metadata": {}, + "id": "cf59c588", + "metadata": { + "editable": true + }, "source": [ "which is zero unless $i=j$." ] }, { "cell_type": "markdown", - "id": "57b05592", - "metadata": {}, + "id": "a2106bb7", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", @@ -1167,8 +1372,10 @@ }, { "cell_type": "markdown", - "id": "25edaca6", - "metadata": {}, + "id": "f3cd45cd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1177,8 +1384,10 @@ }, { "cell_type": "markdown", - "id": "36dbf570", - "metadata": {}, + "id": "ae979921", + "metadata": { + "editable": true + }, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1187,8 +1396,10 @@ }, { "cell_type": "markdown", - "id": "a8adfc31", - "metadata": {}, + "id": "ad4149ba", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1197,8 +1408,10 @@ }, { "cell_type": "markdown", - "id": "8210151d", - "metadata": {}, + "id": "9d4a2bb5", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1206,8 +1419,10 @@ }, { "cell_type": "markdown", - "id": "06d06096", - "metadata": {}, + "id": "876225f8", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1216,16 +1431,20 @@ }, { "cell_type": "markdown", - "id": "d98b2f5d", - "metadata": {}, + "id": "fd69b613", + "metadata": { + "editable": true + }, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] }, { "cell_type": "markdown", - "id": "1b3d21d4", - "metadata": {}, + "id": "f2de9fd2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1234,16 +1453,20 @@ }, { "cell_type": "markdown", - "id": "0fa5c281", - "metadata": {}, + "id": "31e2d00e", + "metadata": { + "editable": true + }, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] }, { "cell_type": "markdown", - "id": "36a79c2b", - "metadata": {}, + "id": "88c8b3ca", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1252,8 +1475,10 @@ }, { "cell_type": "markdown", - "id": "7400bf9b", - "metadata": {}, + "id": "af1050c2", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1270,8 +1495,10 @@ }, { "cell_type": "markdown", - "id": "05dcda06", - "metadata": {}, + "id": "8f3d0c5e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1280,16 +1507,20 @@ }, { "cell_type": "markdown", - "id": "f59f43f1", - "metadata": {}, + "id": "e8d03d21", + "metadata": { + "editable": true + }, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "79375e76", - "metadata": {}, + "id": "0c4d6c4b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1298,16 +1529,20 @@ }, { "cell_type": "markdown", - "id": "d8104364", - "metadata": {}, + "id": "22c9d676", + "metadata": { + "editable": true + }, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "c014b000", - "metadata": {}, + "id": "33f135cc", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1315,8 +1550,10 @@ }, { "cell_type": "markdown", - "id": "ada74dc2", - "metadata": {}, + "id": "4638cec5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1325,8 +1562,10 @@ }, { "cell_type": "markdown", - "id": "183da2ae", - "metadata": {}, + "id": "78176236", + "metadata": { + "editable": true + }, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1335,8 +1574,10 @@ }, { "cell_type": "markdown", - "id": "edab709c", - "metadata": {}, + "id": "c71e2985", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1345,8 +1586,10 @@ }, { "cell_type": "markdown", - "id": "ed04ea87", - "metadata": {}, + "id": "a82f2c13", + "metadata": { + "editable": true + }, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", @@ -1356,8 +1599,10 @@ }, { "cell_type": "markdown", - "id": "8ca6c92a", - "metadata": {}, + "id": "f1726a68", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" @@ -1365,8 +1610,10 @@ }, { "cell_type": "markdown", - "id": "e8186d4e", - "metadata": {}, + "id": "d10d83de", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1375,8 +1622,10 @@ }, { "cell_type": "markdown", - "id": "c0709a75", - "metadata": {}, + "id": "0785af64", + "metadata": { + "editable": true + }, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1389,8 +1638,10 @@ }, { "cell_type": "markdown", - "id": "e1d261da", - "metadata": {}, + "id": "bcbabbe5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1399,8 +1650,10 @@ }, { "cell_type": "markdown", - "id": "dd0a7fcb", - "metadata": {}, + "id": "e177513d", + "metadata": { + "editable": true + }, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1408,8 +1661,10 @@ }, { "cell_type": "markdown", - "id": "a9a717b8", - "metadata": {}, + "id": "f62ef65f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1418,16 +1673,20 @@ }, { "cell_type": "markdown", - "id": "b549f75f", - "metadata": {}, + "id": "51c56e08", + "metadata": { + "editable": true + }, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "c4aecd92", - "metadata": {}, + "id": "4b4b7571", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1436,16 +1695,20 @@ }, { "cell_type": "markdown", - "id": "8df935ba", - "metadata": {}, + "id": "f1d91da4", + "metadata": { + "editable": true + }, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "3e641382", - "metadata": {}, + "id": "ca59c2b0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1454,16 +1717,20 @@ }, { "cell_type": "markdown", - "id": "4478daea", - "metadata": {}, + "id": "35b934d5", + "metadata": { + "editable": true + }, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "a08581d0", - "metadata": {}, + "id": "7b81ff02", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1472,8 +1739,10 @@ }, { "cell_type": "markdown", - "id": "29a102a4", - "metadata": {}, + "id": "cc532b40", + "metadata": { + "editable": true + }, "source": [ "## Revisiting our first homework\n", "\n", @@ -1494,8 +1763,11 @@ { "cell_type": "code", "execution_count": 6, - "id": "16f91d0d", - "metadata": {}, + "id": "fd40460a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1504,8 +1776,10 @@ }, { "cell_type": "markdown", - "id": "f165490e", - "metadata": {}, + "id": "e82a0302", + "metadata": { + "editable": true + }, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1513,8 +1787,10 @@ }, { "cell_type": "markdown", - "id": "84498422", - "metadata": {}, + "id": "68a4daa0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1523,16 +1799,20 @@ }, { "cell_type": "markdown", - "id": "f98e851e", - "metadata": {}, + "id": "1e3644b8", + "metadata": { + "editable": true + }, "source": [ "such that" ] }, { "cell_type": "markdown", - "id": "ab911929", - "metadata": {}, + "id": "b52d0e59", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -1541,8 +1821,10 @@ }, { "cell_type": "markdown", - "id": "170e4189", - "metadata": {}, + "id": "78ad760b", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent example\n", "\n", @@ -1553,8 +1835,10 @@ }, { "cell_type": "markdown", - "id": "1a2651d2", - "metadata": {}, + "id": "d71f7c0d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -1567,16 +1851,20 @@ }, { "cell_type": "markdown", - "id": "310ad769", - "metadata": {}, + "id": "ce9d921f", + "metadata": { + "editable": true + }, "source": [ "The cost/loss/risk function is given by (" ] }, { "cell_type": "markdown", - "id": "a5b0496c", - "metadata": {}, + "id": "27aa96d7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -1585,16 +1873,20 @@ }, { "cell_type": "markdown", - "id": "c0759118", - "metadata": {}, + "id": "9a8d11f0", + "metadata": { + "editable": true + }, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] }, { "cell_type": "markdown", - "id": "c65d80ad", - "metadata": {}, + "id": "f3e390cf", + "metadata": { + "editable": true + }, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -1603,8 +1895,10 @@ }, { "cell_type": "markdown", - "id": "30544a22", - "metadata": {}, + "id": "b7228342", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1615,16 +1909,20 @@ }, { "cell_type": "markdown", - "id": "f7e27f43", - "metadata": {}, + "id": "62c62291", + "metadata": { + "editable": true + }, "source": [ "where $X$ is the design matrix defined above." ] }, { "cell_type": "markdown", - "id": "0c7d8c1d", - "metadata": {}, + "id": "1564d6a0", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -1632,8 +1930,10 @@ }, { "cell_type": "markdown", - "id": "afa82cb3", - "metadata": {}, + "id": "8cc61784", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1645,16 +1945,20 @@ }, { "cell_type": "markdown", - "id": "837c1e3f", - "metadata": {}, + "id": "54ba3f55", + "metadata": { + "editable": true + }, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] }, { "cell_type": "markdown", - "id": "89fc16ec", - "metadata": {}, + "id": "c150ee31", + "metadata": { + "editable": true + }, "source": [ "## Simple program\n", "\n", @@ -1663,8 +1967,10 @@ }, { "cell_type": "markdown", - "id": "983001de", - "metadata": {}, + "id": "5bec697f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -1673,8 +1979,10 @@ }, { "cell_type": "markdown", - "id": "915c0c75", - "metadata": {}, + "id": "639805a1", + "metadata": { + "editable": true + }, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -1686,8 +1994,10 @@ }, { "cell_type": "markdown", - "id": "30d20294", - "metadata": {}, + "id": "734657b8", + "metadata": { + "editable": true + }, "source": [ "## Gradient Descent Example\n", "\n", @@ -1697,8 +2007,11 @@ { "cell_type": "code", "execution_count": 7, - "id": "c6017dfa", - "metadata": {}, + "id": "0aec41a8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -1751,8 +2064,10 @@ }, { "cell_type": "markdown", - "id": "dcc6889f", - "metadata": {}, + "id": "a6d8295f", + "metadata": { + "editable": true + }, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -1760,8 +2075,11 @@ { "cell_type": "code", "execution_count": 8, - "id": "8acb6aad", - "metadata": {}, + "id": "a903b53b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Importing various packages\n", @@ -1784,8 +2102,10 @@ }, { "cell_type": "markdown", - "id": "fcc08db2", - "metadata": {}, + "id": "c6eb1b40", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -1794,8 +2114,10 @@ }, { "cell_type": "markdown", - "id": "ad450622", - "metadata": {}, + "id": "8a8cd057", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -1804,16 +2126,20 @@ }, { "cell_type": "markdown", - "id": "d86008e8", - "metadata": {}, + "id": "f2983a73", + "metadata": { + "editable": true + }, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] }, { "cell_type": "markdown", - "id": "8074070e", - "metadata": {}, + "id": "e352badb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1824,16 +2150,20 @@ }, { "cell_type": "markdown", - "id": "05356160", - "metadata": {}, + "id": "aa884fde", + "metadata": { + "editable": true + }, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] }, { "cell_type": "markdown", - "id": "58c8ba7c", - "metadata": {}, + "id": "f91ba4e0", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -1842,8 +2172,10 @@ }, { "cell_type": "markdown", - "id": "abbd7c72", - "metadata": {}, + "id": "9a18134e", + "metadata": { + "editable": true + }, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -1851,8 +2183,10 @@ }, { "cell_type": "markdown", - "id": "134a8134", - "metadata": {}, + "id": "d3ec0891", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1864,8 +2198,10 @@ }, { "cell_type": "markdown", - "id": "b5b89343", - "metadata": {}, + "id": "8a042993", + "metadata": { + "editable": true + }, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -1876,8 +2212,10 @@ }, { "cell_type": "markdown", - "id": "f79ceb7a", - "metadata": {}, + "id": "148f61dd", + "metadata": { + "editable": true + }, "source": [ "## Program example for gradient descent with Ridge Regression" ] @@ -1885,8 +2223,11 @@ { "cell_type": "code", "execution_count": 9, - "id": "a9c7f630", - "metadata": {}, + "id": "707deca6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from random import random, seed\n", @@ -1943,8 +2284,10 @@ }, { "cell_type": "markdown", - "id": "8c6aece2", - "metadata": {}, + "id": "f39bd794", + "metadata": { + "editable": true + }, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -1963,8 +2306,10 @@ }, { "cell_type": "markdown", - "id": "ea1e6f53", - "metadata": {}, + "id": "41eae562", + "metadata": { + "editable": true + }, "source": [ "## Improving gradient descent with momentum\n", "\n", @@ -1973,59 +2318,13 @@ }, { "cell_type": "code", - "execution_count": 1, - "id": "d1f044c9", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - ">0 f([0.74724774]) = 0.55838\n", - ">1 f([0.59779819]) = 0.35736\n", - ">2 f([0.47823856]) = 0.22871\n", - ">3 f([0.38259084]) = 0.14638\n", - ">4 f([0.30607268]) = 0.09368\n", - ">5 f([0.24485814]) = 0.05996\n", - ">6 f([0.19588651]) = 0.03837\n", - ">7 f([0.15670921]) = 0.02456\n", - ">8 f([0.12536737]) = 0.01572\n", - ">9 f([0.10029389]) = 0.01006\n", - ">10 f([0.08023512]) = 0.00644\n", - ">11 f([0.06418809]) = 0.00412\n", - ">12 f([0.05135047]) = 0.00264\n", - ">13 f([0.04108038]) = 0.00169\n", - ">14 f([0.0328643]) = 0.00108\n", - ">15 f([0.02629144]) = 0.00069\n", - ">16 f([0.02103315]) = 0.00044\n", - ">17 f([0.01682652]) = 0.00028\n", - ">18 f([0.01346122]) = 0.00018\n", - ">19 f([0.01076897]) = 0.00012\n", - ">20 f([0.00861518]) = 0.00007\n", - ">21 f([0.00689214]) = 0.00005\n", - ">22 f([0.00551372]) = 0.00003\n", - ">23 f([0.00441097]) = 0.00002\n", - ">24 f([0.00352878]) = 0.00001\n", - ">25 f([0.00282302]) = 0.00001\n", - ">26 f([0.00225842]) = 0.00001\n", - ">27 f([0.00180673]) = 0.00000\n", - ">28 f([0.00144539]) = 0.00000\n", - ">29 f([0.00115631]) = 0.00000\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtNUlEQVR4nO3dd3wUdf7H8ddnNz0hhJDQEtKooqBAaCqIHcuBFREbpyfq2fXuxDvP7u/0zn7qKWA5PQW7ouKhNAENJYDSISFACiWBhAAhPd/fH7vxVkzIArs7u5vP8/HgQXZmsvNmEt6ZzMx3RowxKKWUCnw2qwMopZTyDC10pZQKElroSikVJLTQlVIqSGihK6VUkAixasUJCQkmLS3NqtUrpVRAWr58+W5jTGJT8ywr9LS0NLKzs61avVJKBSQR2dbcPD3kopRSQUILXSmlgoQWulJKBQktdKWUChJa6EopFSRaLHQReUNEikVkTTPzRUReFJFcEVklIgM8H1MppVRL3NlDfwsYdZj55wE9nH8mAv869lhKKaWOVIuFboxZAJQeZpExwNvGYTEQJyKdPRXwUD8V7OXJrzegt/1VSgWahgbDE1+tY932fV55f08cQ08CClxeFzqn/YqITBSRbBHJLikpOaqVrSrcy6vfbWZVYflRfb5SSlklK28PUxZuYeMu/y10txljJhtjMo0xmYmJTY5cbdGY/klEhtp5b0m+h9MppZR3vbckn7ioUM47wTsHMTxR6EVAV5fXyc5pXhEbEcqYk7ow46ftlFfWems1SinlUcX7q5i1dieXDUgmItTulXV4otBnANc6r3YZCpQbY3Z44H2bddWQVCpr6/lspdd+biillEd9mF1IXYPhyiEpXluHO5ctTgOygF4iUigiN4jIzSJys3ORmUAekAtMAX7vtbROfZPb0i+5Le8u2aYnR5VSfq++wfDeknxO7taebokxXltPi3dbNMZc2cJ8A9zqsURuGj84hUmfrGb5tjIy0+J9vXqllHLbgpwSivZWcv/5vb26noAdKfqbE7vQJjyEd/XkqFLKz727OJ+EmDDO6dPJq+sJ2EKPDg/h4gFJfLV6B2UVNVbHUUqpJm3fW8ncDbsYm9mVsBDvVm7AFjrA+CEp1NQ18PGKQqujKKVUk95fVoABrhzsvZOhjQK60Ht3imVgajveXZKvJ0eVUn6nrr6B6cvyGdEjka7xUV5fX0AXOsBVQ1LYsruCrM17rI6ilFK/MGdDMbv2VXOVFy9VdBXwhX5+387ERYXqyVGllN95d0k+nWIjOKN3B5+sL+ALPSLUzqUDkpm1dicl+6utjqOUUgDk7znIwpwSrhjUlRC7b6o24AsdHCdH6xoMH2QXtLywUkr5wLRl+QgwbnDXFpf1lKAo9G6JMQzLaM+0pfk0NOjJUaWUtWrqGvgwu4Azj+tI57aRPltvUBQ6OPbSC8sqWZBzdLflVUopT/lm3U52H6hhvI9OhjYKmkI/9/hOtI8O05OjSinLvbs4n+R2kYzocXS3CT9aQVPoYSE2xg7qypz1u9hRXml1HKVUK5VbfICsvD1cOTgFu018uu6gKXSAKwelYIDpS/XkqFLKGtOW5hNiE8Zm+u5kaKOgKvSU9lEM75HI+8sKqKtvsDqOUqqVqaqt5+MVhZx7fCcS24T7fP1BVejgGDm6c18VczcUWx1FKdXKzFy9g70Ha302MvRQQVfoZ/buQMfYcD05qpTyuXeX5JOREM2wbu0tWX/QFXqI3ca4QSksyCmhoPSg1XGUUq3Ehp37WL6tjPFDUhDx7cnQRkFX6OAYmSU4Tk4opZQvvLckn7AQG5cOSLYsQ1AWeue2kZzRuyMfZBdQU6cnR5VS3lVRXccnK4q4oG9n2kWHWZYjKAsd4KqhKew+UMM363ZaHUUpFeS++Gk7B6rrLDsZ2ihoC31Ej0SS4iJ5T0+OKqW87L2l+fTsGMPA1HaW5gjaQrfbhPFDUvhh8x42lxywOo5SKkitKtzLqsJyrhqSatnJ0EZBW+gAl2cmE2ITpuleulLKS95bkk9kqJ2LByRZHSW4C71DmwjOPb4TH60opKq23uo4Sqkgs6+qlhk/bWf0iV2IjQi1Ok5wFzo4bqu792AtX6/ZYXUUpVSQ+XxlEQdr6n1+m9zmBH2hD8toT3pCNO8u1sMuSinPMcbw7pJ8TkiKpV9yW6vjAK2g0G02YfzgFLK3lbFx536r4yilgsSK/DI27NzvFydDGwV9oQNcOjCZMLuN95ZsszqKUipIvLskn5jwEEaf2MXqKD9rFYUeHx3G+X078cmKIg7W1FkdRykV4PYerOHLVTu4qH8XosNDrI7zs1ZR6ABXDU1lf3UdX/y03eooSqkA99HyQmrqGhg/ONXqKL/Qago9M7UdPTvG6G11lVLHxBjDe0vzGZASR58usVbH+QW3Cl1ERonIRhHJFZFJTcxPEZF5IrJSRFaJyPmej3psRBwnR1cVlrOqcK/VcZRSASpr8x7ySioYP8S/9s7BjUIXETvwMnAe0Ae4UkT6HLLYA8AHxpj+wDjgFU8H9YRLBiYTEx7C1IVbrI6ilApQUxdtoX10GBf262x1lF9xZw99MJBrjMkzxtQA04ExhyxjgMbfPdoCfnmgOjYilCsHd+Wr1TsoLNOHXyiljkzOrv3M3VDMtcPSiAi1Wx3nV9wp9CSgwOV1oXOaq4eBq0WkEJgJ3N7UG4nIRBHJFpHskpKSo4h77H57SjoCvPn9VkvWr5QKXFMXbiE8xMY1w/zvcAt47qTolcBbxphk4HzgHRH51XsbYyYbYzKNMZmJiYkeWvWR6RIXyYX9OjN9aT7llbWWZFBKBZ7i/VV8urKIyzOTibfwIRaH406hFwFdXV4nO6e5ugH4AMAYkwVEAAmeCOgNvxueQUVNvT6iTinltrd/2EZtQwM3nJphdZRmuVPoy4AeIpIuImE4TnrOOGSZfOBMABE5DkehW3NMxQ0nJLXllO7tefP7LfqIOqVUiw7W1PHO4m2c06cj6QnRVsdpVouFboypA24DZgHrcVzNslZEHhWR0c7F7gVuFJGfgGnABGOM8VZoT7hxeAa79lXrQCOlVIs+zC6kvLKWiSP8d+8cwK0xq8aYmThOdrpOe9Dl43XAKZ6N5l2n9UykV8c2TFmYxyUDkvzm5jpKKf9S32CYuiiPASlxDEyNtzrOYbWakaKHEhF+NzydDTv3szBnt9VxlFJ+atbanRSUVvr93jm04kIHGH1SFzq0CWfKwjyroyil/JAxhtcW5JHaPoqz+3SyOk6LWnWhh4fYmXBKGgtzdrNu+z6r4yil/Ez2tjJ+KtjL705Nx27z/8OyrbrQAa4anEpUmJ2pi3QvXSn1S1MW5NEuKpTLBnZteWE/0OoLvW1UKFcM6sqMH7ezo7zS6jhKKT+RV3KAb9fv4pqhqUSG+d8w/6a0+kIHuP6UdBqM4a0ftlodRSnlJ15ftIVQu41rhqVZHcVtWuhA1/gozu/bmfcW57O/Sm8HoFRrt+dANR8tL+TSAUkktgm3Oo7btNCdJo7IYH91He8vK2h5YaVUUHtn8Taq6/x7mH9TtNCd+iXHMTg9nje/30ptvd4OQKnWqqq2nreztnFm7w507xBjdZwjooXuYuLwDIr2VjJz9Q6royilLPLxikJKK2q4MQAGEh1KC93FGb07kJEYzZSFefj5rWiUUl7Q0GCYunAL/ZLbMiTdv4f5N0UL3YXNJtw4PIM1RfvIyttjdRyllI/NXr+LLbsruHF4RkDe30kL/RAX908iISaMKQt0oJFSrc2UhXkkxUVy3gn+P8y/KVroh4gItXPtsDTmbSxh0679VsdRSvnIivwylm0t44ZT0wmxB2Y1BmZqL7t6aCoRoTam6k27lGo1pi7MIzYihLGDAmOYf1O00JsQHx3G5QO78tnK7RTvq7I6jlLKy7btqeC/a3Zy1dBUYsLdekyEX9JCb8YNp6ZT29DAv7O2Wh1FKeVlbyzagt0mTDg5zeoox0QLvRlpCdGc26cT/1mcT0V1ndVxlFJeUlZRwwfZhYw5KYmOsRFWxzkmWuiHceOIDMora/kwW28HoFSwenfJNipr67lxeOANJDqUFvphDExtx8DUdrz+/Rbq9HYASgWdqtp63vphm+MZw53aWB3nmGmht+DG4RkUlFYya+0uq6MopTzs8x+L2H2gOiCeF+oOLfQWnN2nI2nto5i8YLPeDkCpINLQYJiycAt9Osdycrf2VsfxCC30Fthtwg3DM/ipsJxlW8usjqOU8pD5m4rJLT7AxBGBOcy/KVrobrhsQDLtokKZrLcDUCpoTF6QR+e2EVzQr7PVUTxGC90NkWF2rhmWxpwNu9hccsDqOEqpY7S6sJzFeaVcf0o6oQE6zL8pwfMv8bJrh6USZrfxyrzNVkdRSh2jl+fl0iY8hCsGB+4w/6ZoobspISacq4em8unKQt1LVyqArSkq579rd3L9qenERoRaHcejtNCPwC0juxEeYueF2TlWR1FKHaVnv91E28hQbhiebnUUj9NCPwIJMeFMOCWNL1ZtZ+NOvbWuUoFmRX4ZczcUM3FERtDtnYMW+hGbODyD6LAQnvt2k9VRlFJH6NlvNtE+Oizgb8LVHLcKXURGichGEckVkUnNLDNWRNaJyFoRec+zMf1Hu+gwbjg1nf+u3cmaonKr4yil3LQkbw+Lcndzy8huRAfwLXIPp8VCFxE78DJwHtAHuFJE+hyyTA/gfuAUY8zxwF2ej+o/bhieTtvIUN1LVypAGGN45ptNdGjjuLghWLmzhz4YyDXG5BljaoDpwJhDlrkReNkYUwZgjCn2bEz/EhsRysQRGczZUMyKfB09qpS/W5S7m6VbS7n19O5EhNqtjuM17hR6EuB6/9hC5zRXPYGeIvK9iCwWkVFNvZGITBSRbBHJLikpObrEfmLCyWnER4fpXrpSfq5x77xL2wjGBdl154fy1EnREKAHMBK4EpgiInGHLmSMmWyMyTTGZCYmJnpo1daIDg/hltO6sTBnN0vy9lgdRynVjLkbivmxYC+3n9mD8JDg3TsH9wq9CHD9sZbsnOaqEJhhjKk1xmwBNuEo+KB29dBUOrQJ55lvN+mdGJXyQ8YYnv12EynxUVw2MNnqOF7nTqEvA3qISLqIhAHjgBmHLPMZjr1zRCQBxyGYoL+TVWSYnVtP787SLaV8n6t76Ur5m1lrd7J2+z7uPLNHUN2zpTkt/guNMXXAbcAsYD3wgTFmrYg8KiKjnYvNAvaIyDpgHvBHY0yraLhxg7vSpW0ET3+zUffSlfIj9Q2OvfOMxGgu6n/oab/g5NaPLGPMTGNMT2NMN2PME85pDxpjZjg/NsaYe4wxfYwxfY0x070Z2p+Eh9i5/cwe/Fiwl3kbg/riHqUCypertrNp1wHuPqsndltw3O+8JcH/O4gPXDYwmZT4KJ75Ro+lK+UP6uobeH52Dr07teGCvsFzv/OWaKF7QKjdxh1n9mDt9n3MWrvT6jhKtXqLn3uTq6c/x8MJ+7C1kr1z0EL3mItO6kJGYjTPfruJ+gbdS1fKKrULv2fYnyZy/fIZDJk4FrKyrI7kM1roHhJit3HXWT3ZtOsAX67abnUcpVqtrc++gt00IIDU1MD8+VZH8hktdA+6sG9nenVswwuzc6irb7A6jlKtTlV1LSGLFmIAY7dDWBiMHGl1LJ/RQvcgm024++ye5O2u4LMfdS9dKV9b9tgLpO8uoOC2PyCPPQZz5sCwYVbH8hmx6qqMzMxMk52dbcm6vckYw29eWkR5ZS1z7x3ZKgYzKOUPKkvLOZCWwd72neieuwqxB+cwfxFZbozJbGqeto2HiQj3nt2LgtJKPswutDqOUq3G+rv+TOL+UmqeeTZoy7wlWuheMLJXIv1T4vjn3ByqauutjqNU0KvYtJnjp03hhyHncPwl51odxzJa6F7QuJe+o7yK6UvzrY6jVNDbftPtNGCj7QvPWB3FUlroXnJK9/YMSY/n5fmbqazRvXSlvOXAnPn0mP81315wDccPOcHqOJbSQvcSEeHec3pRsr+adxZvtTqOUsGpoYEDt9zG9jYJdH/mUavTWE4L3YsGp8czvEcCr36Xx4HqOqvjKBV0Kqa8Qaectfz3qjvp06313LOlOVroXnbP2T0prajh3z9stTqKUsFl/37M/fezoksvTn3oDqvT+AUtdC/rn9KOM3t34LXvNlNeWWt1HKWCRsWjjxNTtpv5t/yZnp1irY7jF7TQfeDus3uyr6qOqQuD/iFOSvnGli2Ev/A8nx5/OhfddInVafyGFroPnJDUlgv7dWbygjwKyw5aHUepgLf/rnupQVh/x/1kJMZYHcdvaKH7yP3nH4cIPPHVequjKBXQzHff0WbGp7x5ylgmjh9hdRy/ooXuI0lxkdx2ene+XrOThTklVsdRKjDV17PvltsoapNI7AOTSIgJtzqRX9FC96HfDc8gtX0UD89YS02d3l5XqSNV8/qbtF2/hncuupkrR/a2Oo7f0UL3oYhQOw9e2IfNJRV6GaNSR2r/fmon3c/yLr0Z+dAdhOidTH9Ft4iPnXlcR87o3YHnZ2+ieF+V1XGUChjlf32E6LLdzLv5zwztlmB1HL+khW6BBy/sQ2294cmvN1gdRanAsGULUS+/yOd9z+TqOy63Oo3f0kK3QFpCNDeOSOeTlUUs21pqdRyl/F7xzXdQKzbK//oIndpGWB3Hb2mhW+TW07vTuW0ED32+lvoGa54apVQgqJk7lw7ffMn008dzxcVDrY7j17TQLRIVFsJfLjiOdTv28Z7eM12pptXXU36T4zLF7n9/iPCQ1vkkIndpoVvogr6dGZbRnqdnbaS0osbqOEr5nfJXJpOYu56vrr6LESemWh3H72mhW0hEeGTM8RyoruPpbzZaHUcp/7JvH/LAAyxPPo7zHrvT6jQBQQvdYj07tuG6YWlMW5rP6sJyq+Mo5Te2//EBYveVsuG+R+naPtrqOAFBC90P3HV2D9pHh/HgjDU06AlSpajblEPC668yc8A5XDLxYqvjBAwtdD8QGxHKfaN6szJ/L5+sLLI6jlKWK7rxNmptdiL/8SSRYXoi1F1uFbqIjBKRjSKSKyKTDrPcpSJiRCTTcxFbh0sHJNM/JY4nv17Pvip9EIZqvcpnziJ1wTfMvOA6Rp5+ktVxAkqLhS4iduBl4DygD3CliPRpYrk2wJ3AEk+HbA1sNuHR0Sewp6KGF2bnWB1HKWvU11Px+zspbNuBAc8/hohYnSiguLOHPhjINcbkGWNqgOnAmCaWewx4CtAblBylvsltGTcohbd+2MqmXfutjqOUz+U//RJdtm1k2S330S010eo4AcedQk8CClxeFzqn/UxEBgBdjTFfHe6NRGSiiGSLSHZJid4TvCl/PLcXMeEhPPT5WozRE6Sq9Wgo20vs4w/zY+oJnPXgbVbHCUjHfFJURGzAs8C9LS1rjJlsjMk0xmQmJupP36bER4fxh3N7kZW3h5mrd1odRymf2XjHJGIPlFP2xN9pExlmdZyA5E6hFwFdXV4nO6c1agOcAMwXka3AUGCGnhg9euMHp9CncyyPf7WOgzV1VsdRyuv2r15P92mv892w8xg5fpTVcQKWO4W+DOghIukiEgaMA2Y0zjTGlBtjEowxacaYNGAxMNoYk+2VxK2A3SY8OuZ4dpRX8cq8zVbHUcrrCm+8jRpbCF1eflZPhB6DFgvdGFMH3AbMAtYDHxhj1orIoyIy2tsBW6vMtHgu7p/E5AV5bN1dYXUcpbxm24dfcNySufwwdiK9+veyOk5AE6tOvGVmZprsbN2JP5zifVWc/vR8hmS0540Jg6yOo5THmbo68tN6E1pxgKjNm4iLj7U6kt8TkeXGmCYPaetIUT/WITaCu87qydwNxcxZv8vqOEp5VlYWO86+kNSizeT96SEtcw/QQvdz152cRrfEaB7+Yi0V1XqCVAWJrCzMGWfQef4sGkQYNryf1YmCgha6nwsLsfG3S/pRWFbJ41+tszqOUh5h5s2DqioEx22k7QsXWB0pKGihB4DB6fHcNKIb05YW8O06PfSiAt/6/N0IYESQ8HAYOdLqSEFBCz1A3HN2T/p0jmXSx6so2V9tdRyljtqOH9fR9a3X2JrUDfPoozBnDgwbZnWsoKCFHiDCQmw8P+4k9lfXcd/Hq/S2ACog1dfUsu/SKwAI+/ILbA88oGXuQVroAaRnxzbcf15v5m4o1gdLq4C0cuK99Mpbw9q/PkWXk46zOk7Q0UIPMNcNS2N4jwQe/3I9eSUHrI6jlNu2fP4N/d9+mcWnnM+QP99qdZygpIUeYGw24R+XnUhYiI27P/iJ2voGqyMp1aKq3aVE/vZadsR1pOeH/9bh/V6ihR6AOrWN4P8u7stPBXt5aW6u1XGUalHOZdeSsLeE4n9NJb5zgtVxgpYWeoC6oF9nLumfxEvzclmRX2Z1HKWatfHpf9H3u6/47srfM+CK862OE9S00APYw2OOp1NsBHe//6OOIlV+ad/aDSQ/8AdWpffj5KnPWB0n6GmhB7DYiFCeHXsi+aUHdRSp8jumpoY9F42lHiFs2rtE6kMrvE4LPcANyWivo0iVX9p4259Iz11N1p+eoPeQE6yO0ypooQcBHUWq/E3JzNn0mPpP5g09j7MevsPqOK2GFnoQcB1FOklHkSqLNZSWwTVXUxTXkR7T38Ru00sUfUULPUj07NiGSaN6M2dDMdOWFlgdR7VWxpB3+bW0Kyth07Ovkpza0epErYoWehCZcHIap3ZP4LEv1+koUmWJohcn033ul3x56c2ced1vrI7T6mihBxGbTXj6ch1FqqxRvX4j7e67m+Vp/Rjx+tM6GtQCWuhBplPbCJ64+AQdRap8q7aW3RddTi02qt98i/jYSKsTtUpa6EHown5dfh5FulJHkSofKLjzPpI2rea/dzzCySP7Wx2n1dJCD1I6ilT5yoFv5pL06vN8PWgUox+/y+o4rZoWepBqHEW6rfQgj3+13uo4KkiZ0lJqxo8nP64zKe9MJTLMbnWkVk0LPYgNyWjPxBEZTFuaz4fZeimj8jBj2Hr5tbQp283Sx1/k+F5JVidq9bTQg9wfzunFqd0T+POnq8navMfqOCqIrHvyRdLnfsWMS2/m0psvsTqOQgs96IXabbx81QDS2kdz83+Ws1mvT1cekLNoBWkPTWJV9/6c9+9ndDSon9BCbwXaRobyxoRBhNiE699aRmlFjdWRVADb8/VsYi84FyM2Os/4gKjIcKsjKSct9Faia3wUU67LZGd5FRPfzqaqtt7qSCoAVc5fSNyFo+i4bzeRpo7EvSVWR1IutNBbkQEp7Xh27ElkbyvjPr2JlzpC9Q2G3Nv/hL3BsTNga6iH+fOtDaV+QQu9lbmgX2f+eG4vPv9xO8/NzrE6jgogM/7wFH3XLKbBZgO7HcLCYORIq2MpF24VuoiMEpGNIpIrIpOamH+PiKwTkVUiMkdEUj0fVXnK70d2Y2xmMi/OyeGTFYVWx1EBYPZz/+bCFx4gr98QbHPnwmOPwZw5MGyY1dGUi5CWFhARO/AycDZQCCwTkRnGGNdnnq0EMo0xB0XkFuDvwBXeCKyOnYjw+EV9KSit5L6PV5EUF8mQjPZWx1J+atnH3zLsvpvZkZRB6nezIK4tnHaa1bFUE9zZQx8M5Bpj8owxNcB0YIzrAsaYecaYg86Xi4Fkz8ZUnhYWYuPVqweSEh/FTf9ZzpbdFVZHUn4oN+tH0q8by4E2cSQsmIM9rq3VkdRhuFPoSYDrMMNC57Tm3AB83dQMEZkoItkikl1SomfHrdY2KpQ3JwzGJsJv31xKmV7OqFyUbM4n/DcXEtpQj23WLKLSulodSbXAoydFReRqIBP4R1PzjTGTjTGZxpjMxMRET65aHaWU9lFMuXYg28uruOmd5VTX6eWMCg6W7qXsjFEklO9mz/SPSczsZ3Uk5QZ3Cr0IcP3RnOyc9gsichbwF2C0MUafVBxABqbG84/L+rF0aymTPl6tlzO2cvXVNWw+/QK6FWxk0/OTyRh9ttWRlJvcKfRlQA8RSReRMGAcMMN1ARHpD7yGo8yLPR9TeduYk5K45+yefLqyiBfn6IMxWi1jWHPhFfRd9QNZf3icE2+91upE6gi0eJWLMaZORG4DZgF24A1jzFoReRTINsbMwHGIJQb40PnYqXxjzGgv5lZecPsZ3dm6p4LnZm8iLSGKMSfp3fNamzXX38GJsz9j3hW3MPKpX12hrPxci4UOYIyZCcw8ZNqDLh+f5eFcygIiwt8u6UthWSV//HAVXeIiGZQWb3Us5SObHv47J7z1EvNHjGH4f/6pzwQNQDpSVP1CeIid164eSFK7SCa+nc1WvZyxVSh64z26PzKJxcefTObM6YSE6IMqApEWuvqVdtFhvDFhEAb07oytQNmsOSTcNIG1XXuT+s0MYqIjrI6kjpIWumpSekI0k6/JpHBvJWNfy2L73kqrIykvKPo+m5CLL2J7bAfsX35J5y46YjiQaaGrZg1Oj+ft6wezq7yKS//1Azm79lsdSXnQ1tffI/bMkdQhVH4xkz79MqyOpI6RFro6rKEZ7Xn/pmHUNRguezWL5dvKrI6kPGDd5HdJ+d3VxFRX0Lahhj6i50qCgRa6alGfLrF8csvJtIsK5aqpi5m7YZfVkdQxmP/l93S582YEgwC2ujq9r3mQ0EJXbukaH8VHt5xMjw5tuPHt5Xy8XG+7G4hmvv45fceeT4gA4eF6X/Mgo4Wu3JYQE860iUMZmhHPvR/+xOQFm62OpNxkjOHTJ6Yw8pYrqI9pQ+jybGTePL2veZBxa2CRUo1iwkN4Y8Ig7vngJ/5v5gZ2H6hh0qje2PSp736rvsHw+R2PMfqVR9iZ1otOi+YQ0qWzY6YWeVDRQldHLDzEzj/H9SchOozJC/LYfaCapy7tR6hdf+HzN1U1dXwz/nYu+fhVtgw4hbR5M5HYWKtjKS/RQldHxWYTHh59PAkx4Tzz7SbKKmp4+aoBRIXpt5S/2FdRxZLzxzN6wafknnsx3WdMdxwvV0FLd6nUURMRbj+zB3+7pC/fbSrhqqlL9CEZfqK4uIw1w87h7AWfkjPh93T/+mMt81ZAC10dsysHp/DKVQNZu30fl+uoUssV5Bawc8gIhq5eRM4D/0ePN18GvdFWq6CFrjxi1AmddFSp1bKyKPn9XdgGDaJ3wUa2vfIGPR673+pUyoe00JXHuI4qvfw1HVXqU1lZNIw8nYR/vUCXvbsof+Ip0m+ZYHUq5WNa6MqjGkeVxkU6RpV+8dN2qyMFPdPQQNGkh5CaagTAZiexocrqWMoCWujK4xpHlfbuFMvt01Zy8zvLKd6nBeMNO3Lz+XHgSJIWfIsRG8ZuR8J15GdrpYWuvCIhJpyPbh7GpPN6M29jMWc9+x0fZBfoA6g9pKHBMPv5twnpfxJ9Vi9myZ0PwoIFiI78bNXEqv9gmZmZJjs725J1K9/KKznApI9Xs3RrKad2T+Bvl/Sla3yU1bECVl7hHtZNuJUL57xPYZd0QqZPo9PwIVbHUj4iIsuNMZlNzdM9dOV1GYkxTJ84lMcuOoGV+WWc89wC3vx+C/UNurd+JOrqG5j+1n+pHjiYC+e8T+7l15GUs0bLXP1MC135hM0mXDM0lW/uOY0hGfE88sU6Ln/1B3KL9fJGd+TN+IZl/U/j4t/9huTKUva+/zHdP3gLidLfdNT/aKErn0qKi+TNCYN47ooT2bK7gvNfWMRLc3OorW+wOppfqqqt57O/vkjqRecxbPUiwkwDbd59m7ixl1gdTfkhLXTlcyLCxf2T+fae0zjn+I48/c0mfvPPRawuLLc6ml9ZuXYbn4y6lgufuBubcfzAExFYs8biZMpfaaEryyTEhPPS+AFMvmYgpRU1XPTK9zz59QaqauutjmapisoaPrvzCZKG9Gf83PfYd+pIJCJCH0ahWqS3xlOWO+f4TgzJaM/fZq7n1e82M2vtTp68pC9DMlrRE+izsmD+fDbVhNAwZSoXFW2ioNeJVE75gvjhp/w8n5Ej9ZJE1Sy9bFH5le9zdzPpk1UUlFZycf8krh6awoCUdo5DDUHK/PAD5owzkGrHSM/S6Dj2PvEUGXfcqDfVUr9yuMsWdQ9d+ZVTuicw664RPD87h3cXb+PTlUV07xDDFZlduWRAEu1jwq2O6DHF+6pY+M4XDHjsj6Q5y7xBhDb33kn8nROtjqcCkO6hK79VUV3HV6t2MH1ZPivy9xJqF87u05GxmV0Z3iMRewA+9q5u0fds/egrftyxn7Qf5pJZuI6D4VFE1NUgGCQsTEd6qsM63B66FroKCDm79vP+sgI+WVlEaUUNXdpGcHlmVy7PTCa5nf9fi71tTwXLXn6HMY/cTkhDHQIcaJdA1R/+RMIdt8Dq1XqMXLlFC10Fjeq6emavK+b97AIW5pQAcGr3BMYNSuGsPh0ID7FbnPB/qmrryX7rE2TKZMK2FzGgaAN2HP/fjM2GPPoo/OUvFqdUgUaPoaugER5i54J+nbmgX2cKyw7yYXYhH2YXcOt7K4iPDuOS/klcMagrPTq2sSZgVhYl0z5m3c79hC9bwilbf0IAA1SfeTb27xdCba3j0MoZZ1iTUQUtt/bQRWQU8AJgB6YaY548ZH448DYwENgDXGGM2Xq499Q9dOUp9Q2GRbm7eX9ZPt+u20VtvaF3pzZ06xBD13ZRpMRH0TU+kq7tougSF0lYyDEMv3BePlgXF8++op1s6zuIsoIddPjqM2zFxfRevwwxBgGqwyIIq6ly3KPcbofHHnMcUtFDK+oYHNMhFxGxA5uAs4FCYBlwpTFmncsyvwf6GWNuFpFxwMXGmCsO975a6Mob9hyo5pMVRSzIKaGg9CCFZZXUudwEzCbQuW0kye0i6Rr/y7LvGh9FYkw4tox0GrZtw4hQHxtHTVQ0u1K6syu6HUPmf440NPy81y3OPzhf43xtbHbkxt/B229DTY1jQJCe7FQecKyFPgx42BhzrvP1/QDGmL+5LDPLuUyWiIQAO4FEc5g310JXvlDfYNi5r4qC0oPklx6ksPQgBWWVP78u3l/9i+UXvTKBpP27f/U+h15P01jmriWOzeb4Y8z/Chx0j1x51LEeQ08CClxeFwKH3q/z52WMMXUiUg60B37xP0NEJgITAVJSUtwKr9SxsNuEpLhIkuIiGdrEyNOq2noKyyopKHOUfZenHN+yLV0Q2VjgEhrqKPD6ekeJP/887NnzywLXIlc+4tOTosaYycBkcOyh+3LdSjUlItRO9w4xdO8Q45iQmgrbtjX/CSIQEgJ33w1xcf+7r4ruhSs/4E6hFwFdXV4nO6c1tUyh85BLWxwnR5UKLFu3Qlqao9RFoF07aNMG+veH88779d53Iy1y5QfcKfRlQA8RScdR3OOA8YcsMwO4DsgCLgPmHu74uVJ+betWqxModVRaLHTnMfHbgFk4Llt8wxizVkQeBbKNMTOA14F3RCQXKMVR+koppXzIrWPoxpiZwMxDpj3o8nEVcLlnoymllDoS+oALpZQKElroSikVJLTQlVIqSGihK6VUkLDs9rkiUgIcZgTHYSVwyChUP6G5jozmOnL+mk1zHZljyZVqjElsaoZlhX4sRCS7uXsZWElzHRnNdeT8NZvmOjLeyqWHXJRSKkhooSulVJAI1EKfbHWAZmiuI6O5jpy/ZtNcR8YruQLyGLpSSqlfC9Q9dKWUUofQQldKqSDht4UuIpeLyFoRaRCRZi/vEZFRIrJRRHJFZJLL9HQRWeKc/r6IhHkoV7yIfCsiOc6/2zWxzOki8qPLnyoRucg57y0R2eIy7yRf5XIuV++y7hku063cXieJSJbz671KRK5wmefR7dXc94vL/HDnvz/XuT3SXObd75y+UUTOPZYcR5HrHhFZ59w+c0Qk1WVek19TH+WaICIlLuv/ncu865xf9xwRuc7HuZ5zybRJRPa6zPPm9npDRIpFZE0z80VEXnTmXiUiA1zmHfv2Msb45R/gOKAXMB/IbGYZO7AZyADCgJ+APs55HwDjnB+/CtzioVx/ByY5P54EPNXC8vE4bikc5Xz9FnCZF7aXW7mAA81Mt2x7AT2BHs6PuwA7gDhPb6/Dfb+4LPN74FXnx+OA950f93EuHw6kO9/H7sNcp7t8D93SmOtwX1Mf5ZoAvNTE58YDec6/2zk/buerXIcsfzuO2357dXs533sEMABY08z884GvcTzFcCiwxJPby2/30I0x640xG1tYbDCQa4zJM8bUANOBMSIiwBnAR87l/g1c5KFoY5zv5+77XgZ8bYw56KH1N+dIc/3M6u1ljNlkjMlxfrwdKAaaHAl3jJr8fjlM3o+AM53bZwww3RhTbYzZAuQ6388nuYwx81y+hxbjeHKYt7mzvZpzLvCtMabUGFMGfAuMsijXlcA0D637sIwxC3DswDVnDPC2cVgMxIlIZzy0vfy20N3U1AOsk3A8oHqvMabukOme0NEYs8P58U6gYwvLj+PX30xPOH/dek5Ewn2cK0JEskVkceNhIPxoe4nIYBx7XZtdJntqezX3/dLkMs7t0fjAc3c+15u5XN2AYy+vUVNfU1/mutT59flIRBofV+kX28t5aCodmOsy2Vvbyx3NZffI9vLpQ6IPJSKzgU5NzPqLMeZzX+dpdLhcri+MMUZEmr3u0/mTty+Opz01uh9HsYXhuBb1PuBRH+ZKNcYUiUgGMFdEVuMoraPm4e31DnCdMabBOfmot1cwEpGrgUzgNJfJv/qaGmM2N/0OHvcFMM0YUy0iN+H47eYMH63bHeOAj4wx9S7TrNxeXmVpoRtjzjrGt2juAdZ7cPwqE+Lcy2rqwdZHlUtEdolIZ2PMDmcBFR/mrcYCnxpjal3eu3FvtVpE3gT+4Mtcxpgi5995IjIf6A98jMXbS0Riga9w/DBf7PLeR729mnAsDzx353O9mQsROQvHD8nTjDHVjdOb+Zp6oqBazGWMcX0Y/FQc50waP3fkIZ873wOZ3MrlYhxwq+sEL24vdzSX3SPbK9APufz8AGtxXJUxDphhHGcZ5uE4fg2OB1h7ao+/8YHY7rzvr47dOUut8bj1RUCTZ8O9kUtE2jUeshCRBOAUYJ3V28v5tfsUx7HFjw6Z58nt1eT3y2Hyuj7wfAYwThxXwaQDPYClx5DliHKJSH/gNWC0MabYZXqTX1Mf5urs8nI0sN758SzgHGe+dsA5/PI3Va/mcmbrjeMEY5bLNG9uL3fMAK51Xu0yFCh37rR4Znt562zvsf4BLsZxHKka2AXMck7vAsx0We58YBOOn7B/cZmegeM/XC7wIRDuoVztgTlADjAbiHdOzwSmuiyXhuOnru2Qz58LrMZRTP8BYnyVCzjZue6fnH/f4A/bC7gaqAV+dPlzkje2V1PfLzgO4Yx2fhzh/PfnOrdHhsvn/sX5eRuB8zz8/d5SrtnO/weN22dGS19TH+X6G7DWuf55QG+Xz73euR1zgd/6Mpfz9cPAk4d8nre31zQcV2nV4uivG4CbgZud8wV42Zl7NS5X8Hlie+nQf6WUChKBfshFKaWUkxa6UkoFCS10pZQKElroSikVJLTQlVIqSGihK6VUkNBCV0qpIPH/s/Ss5hTLXcMAAAAASUVORK5CYII=\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "execution_count": 10, + "id": "2f65ff78", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "from numpy import asarray\n", "from numpy import arange\n", @@ -2086,67 +2385,23 @@ }, { "cell_type": "markdown", - "id": "137dfd69", - "metadata": {}, + "id": "185959cb", + "metadata": { + "editable": true + }, "source": [ "## Same code but now with momentum gradient descent" ] }, { "cell_type": "code", - "execution_count": 2, - "id": "70ea5fd5", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - ">0 f([0.74724774]) = 0.55838\n", - ">1 f([0.54175461]) = 0.29350\n", - ">2 f([0.37175575]) = 0.13820\n", - ">3 f([0.24640494]) = 0.06072\n", - ">4 f([0.15951871]) = 0.02545\n", - ">5 f([0.1015491]) = 0.01031\n", - ">6 f([0.0638484]) = 0.00408\n", - ">7 f([0.03976851]) = 0.00158\n", - ">8 f([0.02459084]) = 0.00060\n", - ">9 f([0.01511937]) = 0.00023\n", - ">10 f([0.00925406]) = 0.00009\n", - ">11 f([0.00564365]) = 0.00003\n", - ">12 f([0.0034318]) = 0.00001\n", - ">13 f([0.00208188]) = 0.00000\n", - ">14 f([0.00126053]) = 0.00000\n", - ">15 f([0.00076202]) = 0.00000\n", - ">16 f([0.00046006]) = 0.00000\n", - ">17 f([0.00027746]) = 0.00000\n", - ">18 f([0.00016719]) = 0.00000\n", - ">19 f([0.00010067]) = 0.00000\n", - ">20 f([6.05804744e-05]) = 0.00000\n", - ">21 f([3.64373635e-05]) = 0.00000\n", - ">22 f([2.19069576e-05]) = 0.00000\n", - ">23 f([1.31664443e-05]) = 0.00000\n", - ">24 f([7.91100141e-06]) = 0.00000\n", - ">25 f([4.75216828e-06]) = 0.00000\n", - ">26 f([2.85408468e-06]) = 0.00000\n", - ">27 f([1.71384267e-06]) = 0.00000\n", - ">28 f([1.02900153e-06]) = 0.00000\n", - ">29 f([6.17748881e-07]) = 0.00000\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtxklEQVR4nO3dd3RUdf7/8ed7Jo0kBEihpRKq9BKaCqKggrhgoYPKVwTFsutaYXWVlbXvKrq6q4h9KdZdUXGRIgJKC106BEgCQgKEBALpn98fM/gbMSEBZnJnJu/HORwz996Z++ImvnK55XPFGINSSinfZ7M6gFJKKffQQldKKT+hha6UUn5CC10ppfyEFrpSSvmJAKtWHB0dbZKSkqxavVJK+aS1a9ceMcbElDfPskJPSkoiNTXVqtUrpZRPEpH9Fc3TQy5KKeUntNCVUspPaKErpZSf0EJXSik/oYWulFJ+otJCF5F3RCRLRH6qYL6IyKsisltENolIZ/fHVEopVZmq7KG/B/Q/x/wBQHPnnwnAvy4+llJKqfNVaaEbY5YCx86xyGDgA+OwEqgrIo3cFfBsGzOO89w329Fhf5VSvqaszPD011vZejDPI5/vjmPosUCGy+tM57TfEJEJIpIqIqnZ2dkXtLJNmcd54/s9bMrMvaD3K6WUVVakHeWtZXvZcdh7C73KjDHTjTEpxpiUmJhy71yt1OBOsdQKtDNrVbqb0ymllGfNWpVO3dBABrT1zEEMdxT6ASDe5XWcc5pHRIQEMrhjY+ZuPEju6WJPrUYppdwq60QB87ccYkjnOEIC7R5ZhzsKfS5wq/Nqlx5ArjHmZzd8boVGd0/kdHEp/13vsd8bSinlVp+kZlJSZhjZPcFj66jKZYuzgRVASxHJFJFxInKXiNzlXGQekAbsBt4C7vZYWqd2cXVoH1eHmav268lRpZTXKy0zzFqVzqVNo2gaE+6x9VQ62qIxZmQl8w1wj9sSVdGobglM+nwza/fnkJIUWd2rV0qpKlu6K5sDx08z+bpWHl2Pz94p+rsOjakdHMBMPTmqlPJyM1emEx0exDWtG3p0PT5b6GHBAdzYOZavN/9MTn6R1XGUUqpcB4+fZvH2wwxLiScowLOV67OFDjCqewJFJWV8ti7T6ihKKVWuj9ZkYICR3Tx3MvQMny70Vg0j6JJYj5mr0vXkqFLK65SUljFnTTq9m8cQHxnq8fX5dKEDjO6ewN4j+azYc9TqKEop9SuLtmdxOK+Q0R68VNGVzxf6de0aUTc0UE+OKqW8zsxV6TSMCOGqVvWrZX0+X+ghgXZu7hzH/C2HyD5RaHUcpZQCIP3oKZbtymZ413gC7NVTtT5f6OA4OVpSZvg4NaPyhZVSqhrMXpOOACO6xVe6rLv4RaE3jQmnZ3IUs1enU1amJ0eVUtYqKinjk9QM+l7SgEZ1alXbev2i0MGxl56Zc5qluy5sWF6llHKXb7ce4sjJIkZV08nQM/ym0K9t05CosCA9OaqUstzMlenE1atF7+YXNkz4hfKbQg8KsDGsazyLth3m59zTVsdRStVQu7NOsiLtKCO7JWC3SbWu228KHWBk1wQMMGe1nhxVSllj9up0AmzCsJTqOxl6hl8VekJUKL2ax/DRmgxKSsusjqOUqmEKikv5bF0m17ZpSEzt4Gpfv18VOjjuHD2UV8Di7VlWR1FK1TDzNv/M8VPF1XZn6Nn8rtD7tqpPg4hgPTmqlKp2M1elkxwdRs+mUZas3+8KPcBuY0TXBJbuyibj2Cmr4yilaojth/JYuz+HUd0TEKnek6Fn+F2hg+POLMFxckIpparDrFXpBAXYuLlznGUZ/LLQG9WpxVWtGvBxagZFJXpyVCnlWfmFJXy+7gAD2zWiXliQZTn8stABRvdI4MjJIr7desjqKEopP/flxoOcLCyx7GToGX5b6L2bxxBbtxaz9OSoUsrDZq1Op0WDcLok1rM0h98Wut0mjOqewI97jrIn+6TVcZRSfmpT5nE2ZeYyunuiZSdDz/DbQgcYmhJHgE2YrXvpSikPmbUqnVqBdm7sHGt1FP8u9Pq1Q7i2TUM+XZdJQXGp1XGUUn4mr6CYuRsPMqhDYyJCAq2O49+FDo5hdY+fKuabn362OopSys98sf4Ap4pKq32Y3Ir4faH3TI6iSXQYM1fqYRellPsYY5i5Kp22sRG0j6tjdRygBhS6zSaM6pZA6v4cdhw6YXUcpZSfWJeew/ZDJ7ziZOgZfl/oADd3iSPIbmPWqv1WR1FK+YmZq9IJDw5gUIfGVkf5RY0o9MiwIK5r15DP1x3gVFGJ1XGUUj7u+Kkivtr0Mzd0akxYcIDVcX5RIwodYHSPRE4UlvDlxoNWR1FK+bhP12ZSVFLGqG6JVkf5lRpT6CmJ9WjRIFyH1VVKXRRjDLNWp9M5oS6tG0dYHedXqlToItJfRHaIyG4RmVTO/AQR+U5E1ovIJhG5zv1RL46I4+TopsxcNmUetzqOUspHrdhzlLTsfEZ19669c6hCoYuIHXgdGAC0BkaKSOuzFnsc+NgY0wkYAfzT3UHd4aYucYQHBzBj2V6royilfNSM5XuJCgvi+vaNrI7yG1XZQ+8G7DbGpBljioA5wOCzljHAmX971AG88kB1REggI7vF8/Xmn8nM0YdfKKXOz67DJ1i8PYtbeyYREmi3Os5vVKXQY4EMl9eZzmmupgBjRCQTmAfcV94HicgEEUkVkdTs7OwLiHvx/u+yJgjw7g/7LFm/Usp3zVi2l+AAG7f09L7DLeC+k6IjgfeMMXHAdcCHIvKbzzbGTDfGpBhjUmJiYty06vPTuG4trm/fiDmr08k9XWxJBqWU78k6UcB/1h9gaEockRY+xOJcqlLoB4B4l9dxzmmuxgEfAxhjVgAhQLQ7AnrCHb2SyS8q1UfUKaWq7IMf91NcVsa4y5OtjlKhqhT6GqC5iDQRkSAcJz3nnrVMOtAXQEQuwVHo1hxTqYK2sXW4rFkU7/6wVx9Rp5Sq1KmiEj5cuZ9rWjegSXSY1XEqVGmhG2NKgHuB+cA2HFezbBGRp0RkkHOxB4HxIrIRmA2MNcYYT4V2h/G9kjmcV6g3GimlKvVJaia5p4uZ0Nt7984BqnTPqjFmHo6Tna7TnnD5eitwmXujedYVLWJo2aA2by1L46bOsV4zuI5SyruUlhlmLE+jc0JduiRGWh3nnGrMnaJnExHu6NWE7YdOsGzXEavjKKW81Pwth8g4dtrr986hBhc6wKCOjalfO5i3lqVZHUUp5YWMMby5NI3EqFCubt3Q6jiVqtGFHhxgZ+xlSSzbdYStB/OsjqOU8jKp+3PYmHGcOy5vgt3m/Ydla3ShA4zulkhokJ0Zy3UvXSn1a28tTaNeaCBDusRXvrAXqPGFXic0kOFd45m74SA/5562Oo5SykukZZ9kwbbD3NIjkVpB3nebf3lqfKED3H5ZE8qM4b0f91kdRSnlJd5evpdAu41beiZZHaXKtNCB+MhQrmvXiFkr0zlRoMMBKFXTHT1ZyKdrM7m5cywxtYOtjlNlWuhOE3onc6KwhI/WZFS+sFLKr324cj+FJd59m395tNCd2sfVpVuTSN79YR/FpTocgFI1VUFxKR+s2E/fVvVpVj/c6jjnRQvdxYReyRw4fpp5m3+2OopSyiKfrcvkWH4R433gRqKzaaG7uKpVfZJjwnhrWRpePhSNUsoDysoMM5btpX1cHbo38e7b/Mujhe7CZhPG90rmpwN5rEg7anUcpVQ1W7jtMHuP5DO+V7JPju+khX6WGzvFEh0exFtL9UYjpWqat5alEVu3FgPaev9t/uXRQj9LSKCdW3sm8d2ObHYePmF1HKVUNVmXnsOafTmMu7wJAXbfrEbfTO1hY3okEhJoY4YO2qVUjTFjWRoRIQEM6+obt/mXRwu9HJFhQQztEs9/1x8kK6/A6jhKKQ/bfzSf//10iNE9EgkPrtJjIrySFnoFxl3ehOKyMt5fsc/qKEopD3tn+V7sNmHspUlWR7koWugVSIoO49rWDfn3ynTyC0usjqOU8pCc/CI+Ts1kcMdYGkSEWB3nomihn8P43snkni7mk1QdDkApfzVz1X5OF5cyvpfv3Uh0Ni30c+iSWI8uifV4+4e9lOhwAEr5nYLiUt77cb/jGcMNa1sd56JpoVdifK9kMo6dZv6Ww1ZHUUq52RcbDnDkZKFPPC+0KrTQK3F16wYkRYUyfekeHQ5AKT9SVmZ4a9leWjeK4NKmUVbHcQst9ErYbcK4XslszMxlzb4cq+Mopdxkyc4sdmedZEJv37zNvzxa6FUwpHMc9UIDma7DASjlN6YvTaNRnRAGtm9kdRS30UKvglpBdm7pmcSi7YfZk33S6jhKqYu0OTOXlWnHuP2yJgT66G3+5fGfv4mH3dozkSC7jX9+t8fqKEqpi/T6d7upHRzA8G6+e5t/ebTQqyg6PJgxPRL5z/pM3UtXyof9dCCX/205xO2XNyEiJNDqOG6lhX4eJvZpSnCAnVcW7rI6ilLqAr20YCd1agUyrlcTq6O4nRb6eYgOD2bsZUl8uekgOw7p0LpK+Zp16Tks3p7FhN7Jfrd3Dlro521Cr2TCggJ4ecFOq6Mopc7TS9/uJCosyOcH4apIlQpdRPqLyA4R2S0ikypYZpiIbBWRLSIyy70xvUe9sCDGXd6E/205xE8Hcq2Oo5SqolVpR1m++wgT+zQlzIeHyD2XSgtdROzA68AAoDUwUkRan7VMc2AycJkxpg1wv/ujeo9xvZpQp1ag7qUr5SOMMfz9253Ur+24uMFfVWUPvRuw2xiTZowpAuYAg89aZjzwujEmB8AYk+XemN4lIiSQCb2TWbQ9i3XpeveoUt5u+e4jrN53jHuubEZIoN3qOB5TlUKPBVzHj810TnPVAmghIj+IyEoR6V/eB4nIBBFJFZHU7OzsC0vsJcZemkRkWJDupSvl5c7snTeuE8IIP7vu/GzuOikaADQH+gAjgbdEpO7ZCxljphtjUowxKTExMW5atTXCggOYeEVTlu06wqq0o1bHUUpVYPH2LDZkHOe+vs0JDvDfvXOoWqEfAFx/rcU5p7nKBOYaY4qNMXuBnTgK3q+N6ZFI/drB/H3BTh2JUSkvZIzhpQU7SYgMZUiXOKvjeFxVCn0N0FxEmohIEDACmHvWMv/FsXeOiETjOATj9yNZ1Qqyc8+VzVi99xg/7Na9dKW8zfwth9hyMI8/9G3uV2O2VKTSv6ExpgS4F5gPbAM+NsZsEZGnRGSQc7H5wFER2Qp8BzxsjKkRDTeiWzyN64Twt2936F66Ul6ktMyxd54cE8YNnc4+7eefqvQryxgzzxjTwhjT1BjztHPaE8aYuc6vjTHmAWNMa2NMO2PMHE+G9ibBAXbu69ucDRnH+W6HX1/co5RP+WrTQXYePskf+7XAbvOP8c4r4///BqkGQ7rEkRAZyt+/1WPpSnmDktIypi3cRauGtRnYzn/GO6+MFrobBNpt/L5vc7YczGP+lkNWx1Gqxls57V1umfMSU6LzsNWQvXPQQnebGzo2JjkmjJcW7KS0TPfSlbJK8bIf6PnwBP5v7Zd0nzAMVqywOlK10UJ3kwC7jfv7tWDn4ZN8temg1XGUqrH2v/gqdlOGAFJUBEuWWB2p2mihu9H17RrRskFtXlm4i5LSMqvjKFXjFOSfJnTZ9xjA2O0QFAR9+lgdq9poobuRzSb88eoWpB3J578bdC9dqeq2+ZGpND5+mH2PPIlMnQqLFkHPnlbHqjb+OYakha5t04C2sRG8smgngzs2rhE3MyjlDU7vy6D1jGmsbXcpnZ97EqTmnAw9Q9vGzUSEB69uScax03ySmml1HKVqjMzx9xJQWkLAq68iNbDMQQvdI/q0jKFTQl3+sXgXBcWlVsdRyu+dWrCY5gvnMm/ALXTo08XqOJbRQveAM3vpP+cWMGd1utVxlPJvxcWcvnMimRH1Sf77VKvTWEoL3UMuaxZF9yaRvL5kD6eLdC9dKU85Pe1Vovbu5L+3PUSHFo2tjmMpLXQPEREevKYl2ScK+XDlPqvjKOWfDh3CNmUKS5p04cpHx1udxnJa6B7UrUkkvZpH88b3aZwsLLE6jlJ+p/CBh5DCQhbf/RhtYutaHcdyWuge9sDVLTiWX8T7P+6zOopS/mX5coJnz+Stbjcy5pZ+VqfxClroHtYpoR59W9Xnze/3kHu62Oo4SvmHkhJKJt7NwYgY9k34Ay0a1LY6kVfQQq8Gf7y6BXkFJcxY5vcPcVKqevzrXwT8tJmn+97BxIHtrU7jNbTQq0Hb2Dpc374R05emkZlzyuo4Svm2w4cpffzPLE/qSO1Rw0mOCbc6kdfQQq8mk6+7BBF4+uttVkdRyqeZSZMw+fm8MPAeHurfyuo4XkULvZrE1q3FvVc245ufDrFsV7bVcZTyTStWIO+9x4yUwdw0si/R4cFWJ/IqWujV6I5eySRGhTJl7haKSnR4XaXOS2kpZffcS1ZENP+78Q7G9Ei0OpHX0UKvRiGBdp64vjV7svP1Mkalztf06djWr+OpPrczaWhXAnQk09/QLVLN+l7SgKta1Wfawp1k5RVYHUcp33DkCKV/eowVSR2wDR9Oj+QoqxN5JS10CzxxfWuKSw3PfbPd6ihK+YbJkzEn8nim/0T+NLC11Wm8lha6BZKiwxjfuwmfrz/Amn3HrI6jlHdbvRrz9tu803kQ1w3vR8M6IVYn8lpa6Ba558pmNKoTwpNfbKG0zFgdRynvVFpK2d33cLR2JHMHjeP2y5OsTuTVtNAtEhoUwGMDL2Hrz3nM0jHTlSrf229jW5vK1N5jeWhoV4ID7FYn8mpa6BYa2K4RPZOj+Nv8HRzLL7I6jlLe5ehRyiZNZk1CW04NGU6flvWtTuT1tNAtJCL8ZXAbThaW8Ldvd1gdRynv8vjjmNxc/nLtRJ74XRur0/gELXSLtWhQm9t6JjF7dTqbM3OtjqOUd1i7FvPmm7zfaSBXDe1HfGSo1Yl8gha6F7j/6uZEhQXxxNyfKNMTpKqmKyvD3H0POeH1mHP9OCZe0dTqRD5DC90LRIQE8mj/VqxPP87n6w9YHUcpa733HrJ6FX/tfRsPDOlKrSA9EVpVVSp0EekvIjtEZLeITDrHcjeLiBGRFPdFrBlu7hxHp4S6PPfNNvIK9EEYqobKyaHskUdZF9+a7BuGcW2bhlYn8imVFrqI2IHXgQFAa2CkiPzmVi0RqQ38AVjl7pA1gc0mPDWoLUfzi3hl4S6r4yhljT//GY4d44mrJ/LkoLaIiNWJfEpV9tC7AbuNMWnGmCJgDjC4nOWmAs8DOkDJBWoXV4cRXRN478d97Dx8wuo4SlWvDRsw//oXH3S6jktv7kuz+vrgivNVlUKPBTJcXmc6p/1CRDoD8caYr8/1QSIyQURSRSQ1O1vHBC/Pw9e2JDw4gCe/2IIxeoJU1RBlZZh77iE3NIIPBozjvquaWZ3IJ130SVERsQEvAQ9WtqwxZroxJsUYkxITE3Oxq/ZLkWFBPHRtS1akHWXe5kNWx1Gqevz738iPP/J0r9v4/c1dqR0SaHUin1SVQj8AxLu8jnNOO6M20BZYIiL7gB7AXD0xeuFGdUugdaMI/vr1Vk4VlVgdRynPys2l7OFH2BR3CfuvH8Lgjo2tTuSzqlLoa4DmItJERIKAEcDcMzONMbnGmGhjTJIxJglYCQwyxqR6JHENYLcJTw1uw8+5Bfzzuz1Wx1HKs558EsnO4vF+d/LkDe30ROhFqLTQjTElwL3AfGAb8LExZouIPCUigzwdsKZKSYrkxk6xTF+axr4j+VbHUcozNm/GvPYaszr2p8PgvrRpXMfqRD6tSsfQjTHzjDEtjDFNjTFPO6c9YYyZW86yfXTv3D0mD2hFoF146qutVkdRyv2MwdxzDydCwpl+ze08eE0LqxP5PL1T1IvVjwjh/n4tWLw9i0XbDlsdRyn3mjULWbaMZy6/hbtu6kbd0CCrE/k8LXQvd9ulSTSNCWPKl1vIL9QTpMpP5OVR9uBDbIltydYBQxmWEl/5e1SltNC9XFCAjWdvak9mzmn++rUeelH+wUyZAlmHebzfXTwzpAN2m54IdQctdB/QrUkkd/ZuyuzVGSzYqodelI/bsgXz6qvMaX8N/W67nraxeiLUXbTQfcQDV7egdaMIJn22iewThVbHUerCGEPBxLvJCwplwejfc5cOjetWWug+IijAxrQRHTlRWMKjn23SYQGUTyqb8xEhy5byypW38dS4Pnqoxc200H1Iiwa1mTygFYu3Z+mDpZXvOXmSU7+/n80NmtL6iQf1KUQeoIXuY27rmUSv5tH89attpGWftDqOUlV25NHHCT9ymHl3Ps6QbolWx/FLWug+xmYTXhzSgaAAG3/8eCPFpWVWR1KqUoWbt1D3jdf4onN/xj8ySm/v9xAtdB/UsE4Iz9zYjo0Zx3lt8W6r4yh1bsZw4Nbx5AeGEP2PvxEZpjcQeYoWuo8a2L4RN3WK5bXvdrMuPcfqOEpVaNtr75K8YQXLbv09l13axuo4fk2suloiJSXFpKbqkC8XI6+gmAHTlhFgF+b9vhdhwQFWR1LqV45n51DQvCUnwyKI3bWFWqHBVkfyeSKy1hhT7vDkuofuwyJCAnlpWAfSj53Su0iV1zHGsOqOB2mYm4289rqWeTXQQvdx3ZOj9C5S5ZXWPP8Gfb98n33detP0xmutjlMjaKH7Ab2LVHmb7G8W0uVP92A3ZSRuWg0rVlgdqUbQQvcDrneRTtK7SJXFysoMx+79I3ZjEECKi2HJEqtj1Qha6H6iRYPaTOrfikXbs5i9OsPqOKoG++GhqbRM+4kyux3sdggKgj59rI5VI2ih+5GxlyZxebNopn61Ve8iVZbYP+e/9Jz2FzZ1vBz57juYOhUWLYKePa2OViPoZYt+5lBuAddOW0pSdBif3tWTQLv+zlbVo3DTTxR378HBug2IXr+ayIZRVkfyS3rZYg3SsE4IT9/YVu8iVdXryBHyr72O0/ZAjsz6RMvcIlrofuj69o1/uYt0vd5FqjytsJDcAb8jLPsQ/3nydS69srPViWosLXQ/NWVwGxpGhPDHjzbos0iV5xhD0bjx1EldyQsjJ3HLH4ZbnahG00L3U2fuIt1/7BR//Xqb1XGUnzLPP0/QzA+Z1ms0Nzz7ILWC7FZHqtG00P1Y9+QoJvROZvbqdD5J1UsZlZt9/jkyeTJfXHIFAU8+Qbs4fTao1bTQ/dxD17Tk8mbR/Ok/m1mx56jVcZS/WLuW0tFjWNe4Jf974GkmXtnc6kQKLXS/F2i38froziRFhXHXv9eyR69PVxfrwAGKr/8dh4Nr8/LdL/D3W7vrs0G9hBZ6DVCnViDvjO1KgE24/b01HMsvsjqS8lX5+RQP/B1Fx47zyG1/5W/3XUNokA7b7C200GuI+MhQ3rothUO5BUz4IJWC4lKrIylfU1ZGyZhbsG3ayEM3TeKxScNpEBFidSrlQgu9BumcUI+XhnUkdX8Oj+ogXuo8lT32GAH//Q/PXDWOYU/exSWNIqyOpM6ihV7DDGzfiIevbckXGw7y8sJdVsdRvuL997E99xwzO/Yn8anJXNmqvtWJVDmqVOgi0l9EdojIbhGZVM78B0Rkq4hsEpFFIpLo/qjKXe7u05RhKXG8umgXn6/LtDqO8nbLllF6x3iWJ3Yg7YlnufXSJlYnUhWotNBFxA68DgwAWgMjRaT1WYutB1KMMe2BT4EX3B1UuY+I8Ncb2tEzOYpHP9vEqjS9nFFVYM8eigbdwP6I+nz8yN/50+AOVidS51CVPfRuwG5jTJoxpgiYAwx2XcAY850x5pTz5Uogzr0xlbsFBdh4Y0wXEiJDufPfa9l7JN/qSMrbHD9O4YCBnCos5tmJL/DsuCv08kQvV5VCjwVcbzPMdE6ryDjgm/JmiMgEEUkVkdTs7Oyqp1QeUSc0kHfHdsMmwv+9u5ocvZxRnVFSQuHNQ7Gl7WHy6ClMffgmwoL18kRv59aToiIyBkgBXixvvjFmujEmxRiTEhMT485VqwuUEBXKW7d24WBuAXd+uJbCEr2cUUHxfb8nePFCplx3H/f+ZRwN6+jlib6gKoV+AIh3eR3nnPYrItIPeAwYZIzRJxX7kC6Jkbw4pD2r9x1j0meb9XLGGq7sH/8g8I1/8Wb3m7jq+Udp01jHaPEVVfk31BqguYg0wVHkI4BRrguISCfgTaC/MSbL7SmVxw3uGMv+o6d4acFOkqLC+EM/HZujRvrf/+D++/m2eQ+Cnn+evpc0sDqROg+V7qEbY0qAe4H5wDbgY2PMFhF5SkQGORd7EQgHPhGRDSIy12OJlcfcd1Uzbuocy8sLd/LFht/8I0z5u59+ovjmoWyLTmL1U68wtndTqxOp81SlsxzGmHnAvLOmPeHydT8351IWEBGevakdmTmnefiTTTSuW4uuSZFWx1LVISuL0/0HkmcLYsYj03hxaAoiekWLr9E7RdWvBAfYeXNMF2Lr1WLCB6ns08sZ/V9BAaevH4QcPsTT459h6r0DCNCHi/sk/a6p36gXFsQ7Y7tiQEdn9HfGUDD2dmqtWcVfhjzCpCduJVwvT/RZWuiqXE2iw5h+SwqZx08z7M0VHDx+2upIygNyHptCyEezeaXPrYx68SEa161ldSR1EbTQVYW6NYnkg9u7cTi3gJv/9SO7Dp+wOpJyo/Q336fes0/xVfu+XPH+NH2EnB/QQlfn1CM5io/u7ElJmWHIGytYuz/H6kjKDTZ9voCYe+9kU2IbLvn6Izom1LM6knIDLXRVqdaNI/h84qXUCw1k9IyVLN5+2OpI6iKkvvIOySMGcapWGA0XzqNpXJTVkZSbaKGrKomPDOXTiZfSvH5txn+wls/W6rC7vmj5lGl0vn8cYcUFRBblUz9b7zfwJ1roqsqiw4OZPaEHPZIjefCTjUxfusfqSKqKjDF89edX6Tb1YQQcf0pKYMkSi5Mpd9JCV+clPDiAd8Z2ZWD7RjwzbzvPzNtGWZmO/eLNSotL+H7oBK7/6x842jgRQkLAboegIOjTx+p4yo30glN13oID7PxjRCeiw4KYvjSNIycLef7m9gTqzShepyD7KLuuHkyfjT+wsf8Q2v/nQ2T9eseeeZ8+0LOn1RGVG2mhqwtiswlTBrUhOjyYvy/YSU5+Ea+P7kxokP5IeYuT6zdyov/vaHXkID88OJXLXnwMRBwlrkXul3SXSl0wEeG+vs159qZ2fL8zm9EzVulDMrzE8dkfY+vZk8CTJ1j51sdc9rfHHWWu/JoWurpoI7sl8M/RXdhyMI+heleptcrKOP7wZOqOGs7uqHh2fbOEXrffZHUqVU200JVb9G/bUO8qtVpuLnn9B1L3b8/xRcerMUu+p2dvfahzTaKFrtzG9a7SoW/qXaXVascOTnVOIXTht7z0u/toO/8zOjRvaHUqVc200JVbnbmrtG4tx12lX248aHUkv2fmzqW4S1dOH8pm0sSXGfXhCzStX9vqWMoCWujK7c7cVdqqYQT3zV7PXR+uJSuvwOpY/qesjLzHnkAGD2Zb7Qb8+fF3+fOLE/WBzjWYFrryiOjwYD69qyeTBrTiux1Z9Hvpez5OzdAHULtJWW4e6VcOIOKZqXzRri8bZ87ltUdvoE5ooNXRlIW00JXHBNht3HVFU775Qy9aNYzgkU83ccvbq8k4dsrqaD4tfdUGDlzSkcbLFvLhiD/S+bsvuOWqS7DZ9LLEmk4LXXlcckw4cyb0YOoNbVmfnsM1Ly/l3R/2UqpDBpyXktIyvn7ubepecTnhx4+y/PWZjJn1d+KjwqyOpryEFrqqFjabcEuPRL594Aq6J0fyly+3MvSNH9mdpZc3VkXa3G/Z2LYHAybfwfEGsZSuXk2fiSP0Qc7qV7TQVbWKrVuLd8d25eXhHdh7JJ/rXlnOa4t3UVxaZnU0r1RQXMpXj75A4g396bJ9DdhsJLz3BtFtW1odTXkhLXRV7USEGzvFseCBK7imTQP+9u1OfveP5WzOzLU6mlfZvHw9S3sO4LoXHsXmPJlsE4GVKy1OpryVFrqyTHR4MK+N6sz0W7pwLL+IG/75A899s52C4lKro1nq1L4MVg8cRcsrunHFxu/Jvv5mRIe8VVWgha4sd02bhix44AqGdonjje/3MOCVZaxKO2p1rOqXk0P6Xfdja9Gczt98xKarb6R0x04afPkpLF4MU6fCokU6UqKqkFh1XXBKSopJTU21ZN3Ke/2w+wiTPt9ExrHT3NgpljE9EuicUM+vT/6ZkyfJeOoFol+fRuipEyzqdBX1X3qedn1SrI6mvJCIrDXGlPvDoYWuvM6pohKmLdzFzJX7yS8qpVn9cIanxHNT51iiwoOtjuc2WUfy2D71Jdq8+w+iThzj++bd2P/HPzHsjusJCbRbHU95KS105ZPyC0v4etPPzFmTzrr04wTahatbN2BYSjy9msdg98EbaUpKy1g/+yt49VXit62j4ckctjXrQPbkJ+g6ZjC1grTI1blpoSuft+vwCT5ak8Hn6w9wLL+IxnVCGJoSz9CUOOLqhVodr1L7j+bz7RfLiP7XPxic+g02DEaEo1OeJvrPk/ThE6rKtNCV3ygsKWXh1iw+Ss1g2a5sAC5vFs2Irgn0a12f4ADv2cMtKC5l2fxVHHnn37T/YT5tstIAMICA46qVqVNh8mQrYyofo4Wu/FJmzik+Sc3kk9QMDuYWEBkWxE2dYhneNZ7mDSwaPnbFCrL//RFp+7II27CWtgd3AnDokg6EjhlFRJuWMHIkFBU5LkHUq1bUebroQheR/sArgB2YYYx57qz5wcAHQBfgKDDcGLPvXJ+pha7cpbTMsHz3ET5ak86CrYcpLjW0alibpvXDia8XSkJkKPGRtYivF0rjurUICrj4q3WLl/3AifkL2d8mhQwJwbZmNbFLF9Bh7ZJfbgI61CiRgjG3kXDXWGzJTf7/m1esgCVLHNeTa5mr83RRhS4idmAncDWQCawBRhpjtrosczfQ3hhzl4iMAG40xgw/1+dqoStPOHqykM/XHWDprmwyjp0iM+c0JS6DgNkEGtWpRVy9WsRH/rrs4yNDiYmrj+3ECcqAsrAw8i69gpza9djY9Uryf84mbvUyivNPcdVPS7Ebx3AFZ45+F9kDCSwtRgBjtyN6OEV5wMUWek9gijHmWufryQDGmGddlpnvXGaFiAQAh4AYc44P10JX1aG0zHAor4CMY6dIP3aKzGOnyMg5/cvrrBOFvyy74aVh1Ckuf2jfs09ZnjkObkQouH4Qwc88jS33OFx9tR5OUR51rkIPqML7Y4EMl9eZQPeKljHGlIhILhAFHDkryARgAkBCQkKVwit1Mew2IbZuLWLr1qJHctRv5hcUl5KZc5qMnFNEPO8o8/KuNzE4xqDBGLDZEJsNjEGCgqg1+VFo28ax4KJFejhFWaYqhe42xpjpwHRw7KFX57qVKk9IoJ1m9cNpVj8cateGE+UP5yuBgY5LC0tLHXvf06bB0aO/Le6ePbXIlWWqUugHgHiX13HOaeUtk+k85FIHx8lRpXxHXh5ERPz/Ug8Ph379oGFDuPVWxzTd+1ZerCqFvgZoLiJNcBT3CGDUWcvMBW4DVgBDgMXnOn6ulNfKyzv3fC1y5cUqLXTnMfF7gfk4Llt8xxizRUSeAlKNMXOBt4EPRWQ3cAxH6SullKpGVTqGboyZB8w7a9oTLl8XAEPdG00ppdT50PHQlVLKT2ihK6WUn9BCV0opP6GFrpRSfsKy0RZFJBvYf4Fvj+asu1C9hOY6P5rr/HlrNs11fi4mV6IxJqa8GZYV+sUQkdSKxjKwkuY6P5rr/HlrNs11fjyVSw+5KKWUn9BCV0opP+GrhT7d6gAV0FznR3OdP2/NprnOj0dy+eQxdKWUUr/lq3voSimlzqKFrpRSfsJrC11EhorIFhEpE5EKL+8Rkf4iskNEdovIJJfpTURklXP6RyIS5KZckSKyQER2Of9br5xlrhSRDS5/CkTkBue890Rkr8u8jtWVy7lcqcu657pMt3J7dRSRFc7v9yYRGe4yz63bq6KfF5f5wc6//27n9khymTfZOX2HiFx7MTkuINcDIrLVuX0WiUiiy7xyv6fVlGusiGS7rP8Ol3m3Ob/vu0TktmrO9bJLpp0ictxlnie31zsikiUiP1UwX0TkVWfuTSLS2WXexW8vY4xX/gEuAVoCS4CUCpaxA3uAZCAI2Ai0ds77GBjh/PoNYKKbcr0ATHJ+PQl4vpLlI3EMKRzqfP0eMMQD26tKuYCTFUy3bHsBLYDmzq8bAz8Ddd29vc718+KyzN3AG86vRwAfOb9u7Vw+GGji/Bx7Nea60uVnaOKZXOf6nlZTrrHAa+W8NxJIc/63nvPretWV66zl78Mx7LdHt5fzs3sDnYGfKph/HfANjicd9gBWuXN7ee0eujFmmzFmRyWLdQN2G2PSjDFFwBxgsIgIcBXwqXO594Eb3BRtsPPzqvq5Q4BvjDHlP33Yfc431y+s3l7GmJ3GmF3Orw8CWUC5d8JdpHJ/Xs6R91Ogr3P7DAbmGGMKjTF7gd3Oz6uWXMaY71x+hlbieHKYp1Vle1XkWmCBMeaYMSYHWAD0tyjXSGC2m9Z9TsaYpTh24CoyGPjAOKwE6opII9y0vby20KuovAdYx+J4QPVxY0zJWdPdoYEx5mfn14eABpUsP4Lf/jA97fzn1ssiElzNuUJEJFVEVp45DIQXbS8R6YZjr2uPy2R3ba+Kfl7KXca5Pc488Lwq7/VkLlfjcOzlnVHe97Q6c93s/P58KiJnHlfpFdvLeWiqCbDYZbKntldVVJTdLdurWh8SfTYRWQg0LGfWY8aYL6o7zxnnyuX6whhjRKTC6z6dv3nb4Xja0xmTcRRbEI5rUR8FnqrGXInGmAMikgwsFpHNOErrgrl5e30I3GaMKXNOvuDt5Y9EZAyQAlzhMvk331NjzJ7yP8HtvgRmG2MKReROHP+6uaqa1l0VI4BPjTGlLtOs3F4eZWmhG2P6XeRHVPQA66M4/ikT4NzLKu/B1heUS0QOi0gjY8zPzgLKOsdHDQP+Y4wpdvnsM3urhSLyLvBQdeYyxhxw/jdNRJYAnYDPsHh7iUgE8DWOX+YrXT77grdXOS7mgedVea8ncyEi/XD8krzCGFN4ZnoF31N3FFSluYwxrg+Dn4HjnMmZ9/Y5671L3JCpSrlcjADucZ3gwe1VFRVld8v28vVDLr88wFocV2WMAOYax1mG73AcvwbHA6zdtcd/5oHYVfnc3xy7c5bamePWNwDlng33RC4RqXfmkIWIRAOXAVut3l7O791/cBxb/PSsee7cXuX+vJwjr+sDz+cCI8RxFUwToDmw+iKynFcuEekEvAkMMsZkuUwv93tajbkaubwcBGxzfj0fuMaZrx5wDb/+l6pHczmztcJxgnGFyzRPbq+qmAvc6rzapQeQ69xpcc/28tTZ3ov9A9yI4zhSIXAYmO+c3hiY57LcdcBOHL9hH3OZnozjf7jdwCdAsJtyRQGLgF3AQiDSOT0FmOGyXBKO37q2s96/GNiMo5j+DYRXVy7gUue6Nzr/O84bthcwBigGNrj86eiJ7VXezwuOQziDnF+HOP/+u53bI9nlvY8537cDGODmn/fKci10/n9wZvvMrex7Wk25ngW2ONf/HdDK5b23O7fjbuD/qjOX8/UU4Lmz3ufp7TUbx1VaxTj6axxwF3CXc74Arztzb8blCj53bC+99V8ppfyErx9yUUop5aSFrpRSfkILXSml/IQWulJK+QktdKWU8hNa6Eop5Se00JVSyk/8P2O4xqLPu9cnAAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "execution_count": 11, + "id": "84299e90", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "from numpy import asarray\n", "from numpy import arange\n", @@ -2215,8 +2470,10 @@ }, { "cell_type": "markdown", - "id": "3eac652d", - "metadata": {}, + "id": "cce1ff04", + "metadata": { + "editable": true + }, "source": [ "## Overview video on Stochastic Gradient Descent\n", "\n", @@ -2225,8 +2482,10 @@ }, { "cell_type": "markdown", - "id": "9ee2bcdd", - "metadata": {}, + "id": "ec171de6", + "metadata": { + "editable": true + }, "source": [ "## Batches and mini-batches\n", "\n", @@ -2244,8 +2503,10 @@ }, { "cell_type": "markdown", - "id": "a2a69883", - "metadata": {}, + "id": "61f771e2", + "metadata": { + "editable": true + }, "source": [ "## Stochastic Gradient Descent (SGD)\n", "\n", @@ -2274,8 +2535,10 @@ }, { "cell_type": "markdown", - "id": "3cb45498", - "metadata": {}, + "id": "167c0617", + "metadata": { + "editable": true + }, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2289,8 +2552,10 @@ }, { "cell_type": "markdown", - "id": "10c0c325", - "metadata": {}, + "id": "90e37c2b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2300,8 +2565,10 @@ }, { "cell_type": "markdown", - "id": "aa0d4c01", - "metadata": {}, + "id": "514288e1", + "metadata": { + "editable": true + }, "source": [ "## Computation of gradients\n", "\n", @@ -2311,8 +2578,10 @@ }, { "cell_type": "markdown", - "id": "9ada4734", - "metadata": {}, + "id": "bdb9af89", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2322,8 +2591,10 @@ }, { "cell_type": "markdown", - "id": "b187e9a4", - "metadata": {}, + "id": "3d7c4c2f", + "metadata": { + "editable": true + }, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2334,8 +2605,10 @@ }, { "cell_type": "markdown", - "id": "65cd1b71", - "metadata": {}, + "id": "8a5b43e2", + "metadata": { + "editable": true + }, "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", @@ -2354,8 +2627,10 @@ }, { "cell_type": "markdown", - "id": "37a27914", - "metadata": {}, + "id": "0d4cc828", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2367,8 +2642,10 @@ }, { "cell_type": "markdown", - "id": "06933ac2", - "metadata": {}, + "id": "00ecaf91", + "metadata": { + "editable": true + }, "source": [ "## The gradient step\n", "\n", @@ -2377,8 +2654,10 @@ }, { "cell_type": "markdown", - "id": "2c0cfb82", - "metadata": {}, + "id": "f0e3f62a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2388,8 +2667,10 @@ }, { "cell_type": "markdown", - "id": "f90b72f3", - "metadata": {}, + "id": "45411341", + "metadata": { + "editable": true + }, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2400,8 +2681,10 @@ }, { "cell_type": "markdown", - "id": "8c964be7", - "metadata": {}, + "id": "58c3927e", + "metadata": { + "editable": true + }, "source": [ "## Simple example code" ] @@ -2409,8 +2692,11 @@ { "cell_type": "code", "execution_count": 12, - "id": "4eac284e", - "metadata": {}, + "id": "eb9b91fb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2431,8 +2717,10 @@ }, { "cell_type": "markdown", - "id": "1074f7d4", - "metadata": {}, + "id": "9eba2696", + "metadata": { + "editable": true + }, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2445,8 +2733,10 @@ }, { "cell_type": "markdown", - "id": "982554ed", - "metadata": {}, + "id": "9b74dee2", + "metadata": { + "editable": true + }, "source": [ "## When do we stop?\n", "\n", @@ -2464,8 +2754,10 @@ }, { "cell_type": "markdown", - "id": "6e09fd45", - "metadata": {}, + "id": "1b0f3bea", + "metadata": { + "editable": true + }, "source": [ "## Slightly different approach\n", "\n", @@ -2482,8 +2774,10 @@ }, { "cell_type": "markdown", - "id": "28895eb5", - "metadata": {}, + "id": "d21909a4", + "metadata": { + "editable": true + }, "source": [ "## Time decay rate\n", "\n", @@ -2499,8 +2793,11 @@ { "cell_type": "code", "execution_count": 13, - "id": "07202d5c", - "metadata": {}, + "id": "c48e042b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import numpy as np \n", @@ -2531,8 +2828,10 @@ }, { "cell_type": "markdown", - "id": "cb6a7d32", - "metadata": {}, + "id": "c3e56e42", + "metadata": { + "editable": true + }, "source": [ "## Code with a Number of Minibatches which varies\n", "\n", @@ -2541,39 +2840,13 @@ }, { "cell_type": "code", - "execution_count": 3, - "id": "1ebde042", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Own inversion\n", - "[[4.]\n", - " [3.]]\n", - "Eigenvalues of Hessian Matrix:[0.26504701 4.42519896]\n", - "theta from own gd\n", - "[[4.]\n", - " [3.]]\n", - "theta from own sdg\n", - "[[3.97239309]\n", - " [3.02330208]]\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgbUlEQVR4nO3deZhddZ3n8fcnVSlClgaSCsiWCsgmrmgNIEIbDI6ILPY4zwiTsMRgNWRUVFokZtq1bR2XVrQFJ2okkBpUEB3FDQirbHaxyS4QUiFsKRK2kD317T/OueTk1q2qU6l776mqfF7Pk6fOPefcc773cKlv/XZFBGZmZv0ZVXQAZmY2PDhhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuThg2Ykn6oqRFRcdRTZKmSVpedBy2fXLCsLqStFTSWkmrJT0r6WJJ44uOy8z654RhRTghIsYDbwMOAeYWG872SVJj0THY8OKEYYWJiGeBP5EkDgAknS/pcUmvSHpQ0j9kjp0h6c+SviXpBUlPSHp/5vg+km5M33sN0Jy9n6QTJT0g6UVJN0h6Q+bYUkmfkfRXSa9K+omk3ST9Ib3etZJ2qfQ5StVEks6VtELSM5JmZY7fIOnM8s+ReR2S5kh6NL3XVyS9XtKtkl6W9AtJTWX3/Jyk59O4Z2T275A+n2WSnpP0Q0k7lsX5WUnPAj+V1CzpqvSZrJJ0syT/XrCK/MWwwkjaC3g/8Fhm9+PAUcBOwJeARZJ2zxw/DHiEJBl8A/iJJKXH/h9wZ3rsK8DpmXsdAFwGfBKYDPwe+G3ZL+IPAe8FDgBOAP4AfC49fxTwiT4+zuvSmPcEZgM/6C3B9OJ9wDuAw4HzgPnATGBv4E3AKWX3ak7vdTowX9KB6bGvp/G/DdgvPefzZe+dCLQAbcC5wPL0M+6Wfl7PF2QVOWFYEX4t6RXgSWAF8IXSgYi4PCKejojuiPg58ChwaOa9nRHxo4jYDCwEdgd2kzQF+C/AP0fE+oi4Cfht5n0fBn4XEddExEbgW8COwBGZc74fEc9FxFPAzcAdEXF3RKwDfkVSfdabjcCXI2JjRPweWA0c2Mf55b4RES9HxAPA/cDVEbEkIl4iSVzl9y59zhuB3wH/I02cbcCnImJVRLwC/CtwcuZ93cAX0veuTePeHWhJY785PMGc9cIJw4rwwYiYAEwDDiJTdSTpNEn3pFUkL5L8dZ2tWnq2tBERa9LN8cAewAsR8Wrm3M7M9h7Z1xHRTZKw9syc81xme22F1301zq+MiE2Z12v6Ob/cQO5d6XPuQVJKGAvcmXl+f0z3l3SlCbDkmyQlvKslLZF0/gBitu2ME4YVJv3r+GKSv/aR1AL8CPgYMCkidib5a1u9XCLrGWAXSeMy+6Zktp8mqYYhvZdIqnue2vZPkNurJL/IS143yOtV+pxPA8+TJJc3RsTO6b+d0g4GJVuVHiLilYg4NyL2BU4EPi1p+iDjsxHKCcOK9l3gvZLeCowj+YXWBZA2HL8pz0UiohPoAL4kqUnSkSTtECW/AD4gabqk0SR19+uBW6v1QfpwD/DfJI2VtB9JG8dglT7nUcDxwOVpqelHwHck7QogaU9J7+vtIpKOl7RfmkBfAjaTVFuZ9eCEYYWKiC7gEuDzEfEg8G3gNpIqmTcDtwzgcv+TpFF8FUm7yCWZ+zxC0oj8fZK/xE8g6d67oQofoz/fATaQfKaFQPsgr/cs8AJJqaIdOCsiHk6PfZakiul2SS8D19J3W8r+6TmrSZ77hRFx/SDjsxFKbt8yM7M8XMIwM7NcnDDMzCwXJwwzM8vFCcPMzHIZVpOPNTc3x9SpU4sOw8xsWLnzzjufj4jJ/Z/Zt2GVMKZOnUpHR0fRYZiZDSuSOvs/q3+ukjIzs1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCyXmicMSQskrZB0f4Vj50oKSc21jsPMzAanHiWMi4Fjy3dK2hv4r8CyOsRgZmaDVPOEERE3kayxXO47wHmA14g1MxsGCmnDkHQS8FRE3Jvj3DZJHZI6urq66hCdmZlVUveEIWks8Dng83nOj4j5EdEaEa2TJw96OnczM9tGRZQwXg/sA9wraSmwF3CXpNcVEIuZmeVU9wWUIuI+YNfS6zRptEbE8/WOxczM8qtHt9rLgNuAAyUtlzS71vc0M7Pqq3kJIyJO6ef41FrHYGZmg+eR3mZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrk4YZiZWS5OGGZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuNU8YkhZIWiHp/sy+b0p6WNJfJf1K0s61jsPMzAanHiWMi4Fjy/ZdA7wpIt4C/A2YW4c4zMxsEGqeMCLiJmBV2b6rI2JT+vJ2YK9ax2FmZoMzFNowPgL8obeDktokdUjq6OrqqmNYZmaWVWjCkDQP2AS093ZORMyPiNaIaJ08eXL9gjMzs600FnVjSWcAxwPTIyKKisPMzPIpJGFIOhY4D3h3RKwpIgYzMxuYenSrvQy4DThQ0nJJs4F/ByYA10i6R9IPax2HmZkNTs1LGBFxSoXdP6n1fc3MrLqGQi8pMzMbBpwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwzM8vFCcPMzHJxwjAzs1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwzM8vFCcPMzHKpecKQtEDSCkn3Z/ZNlHSNpEfTn7vUOg4zMxucepQwLgaOLdt3PrA4IvYHFqevzcxsW7S3w9SpMGpU8rO9vSa3qXnCiIibgFVlu08CFqbbC4EP1joOM7MRpb0dmptBgpkzobMTIqCzk+4zP8qfj/kCsw+4mX1GP1m1WxbVhrFbRDyTbj8L7NbbiZLaJHVI6ujq6qpPdGZmQ1l7O8yaBStXVjw8at1a9lq8kCsfezOHTH6qarctvNE7IgKIPo7Pj4jWiGidPHlyHSMzMytQqZpJgsbG5Gda3dR9/lzYuLHPt7ewjOfXTeDKpw+vWkhFJYznJO0OkP5cUVAcZmZDQ7Ydork5KUF0dibHNm9OfnZ2sm7mmbC8/2omtUyhoamhqiEWlTB+A5yebp8O/P+C4jAzK0ZvCSIiqWrqpQQxhnVEf7+6x46Fr3616iHXo1vtZcBtwIGSlkuaDXwdeK+kR4Fj0tdmZiNbtprp1FNzJYhKGuiG0aMrH5w0CebPhxkzqhNzRmPVr1gmIk7p5dD0Wt/bzKxQ7e0wbx4sWwYTJ8LLL29JDNFr023/WlqSEsQ552xp+J40CS64oCaJoqTmCcPMbLvR3r71L/GsXno0DVipumnGjJomh0oK7yVlZjYitLfDRz5SvcTQ1JSUGgAa0sbrlpaaVTfl4YRhZpZHP6Opu8+fCxs2bPv1SwlCShLDggXw/PNJ1dWmTcnPpUsLSxbgKikzs761t8NZZ8Hq1Vv2dXYSs2bx6HVPcvmyw1ncsRPXvrh84NeWkkRQapMoMBnk4RKGmRlsPdWGlGzPmZNUM2WTRUobN7LLgm/xv6+dxgvrd+TFpl37v0d5KeLSS4dEySEvJwwzs0rtDytXEhdd1Gc1UzMr6Xp4JXevOYiJC76dJITeZKuZuruHTZLIcsIws+1Ttk3i9NMrJgb1cwkBzQemDdMzZiQJodRQDcn2okXDqhTRFycMMxu5emuobm+HtrYtA+dKU28MVDY5QJIQSg3VEcn2ME8SWW70NrORqZQU1qxJXnd2Eh/9KPdeuYQpV/2AiRvWDO76jY3JQLntiEsYZjZ89dXVdd68LckipbVr2fnKn7DzhpzznY4eDWef3bMkMWkSXHzxiCo95OEShpkNP5VGVHd2Em1tLLnpKS7vPJTzOpdV/Iu4hWXEXntVnvF11KikQRq2nmrjwgtr8jGGG5cwzGzomzNny5oQDQ1w2mkVR1RrzRoa5l/I3D9N41l2r3gptUxh1Ne/lkyxkTV2LFxyyYhtf6gGJwwzG9rmzIGLLtrSMN3dvaUUUEELy3j2vi72WPSNykmhNEBu/vykq2tpTESBU24MF4rBzJhYZ62trdHR0VF0GGZWR9HQiLoH0IuppSXpwgpbzxY7ZcqwGE1dC5LujIjWwV7HJQwzq79sFVNjY/I69WLnS/x67h184q038sYxj8FAkkX5wkEzZiTJY5gOlBtq3OhtZvVVqmIq2byZuOgibvvl05yz+l+5a82BdHMYO7KGoyY9SKwfhei9Cuo1dVgPYnvnEoaZ1U6Fbq8xf36P0wQcuuIqxjRu5J/ffTM3fu9eXnipgT8938qos/+x8rXHjdvS/rBokRup68AlDDOrjvL2guOOg4ULtxo4t27mmexA5SqmBjZz80tv7Xmg1KV1/vyk4buhIRmQ566udedGbzMbuPJxEOPGJXMxZdalDirPxdTbfhoaknUfrOqq1ejtEoaZDUxpZtfsZH2vvtrjtN4m7ut1Qr+2tsFGZjVWaBuGpE9JekDS/ZIukzSmyHjMrEylqTfmzRvcynItLcl0G6VlRxsakteuYhryCksYkvYEPgG0RsSbgAbg5KLiMdvulSeHOXO2ntG1s5MNp86mu7Mz/zVVVp4odXu98MIty45u2uRkMUwU3UuqEdhRUiMwFni64HjMtg/l4yCOOaZHcoiLLuoxeV9TrCfy/toYOzZZ2tSjqUeMwtowIuIpSd8ClgFrgasj4ury8yS1AW0AU6ZMqW+QZiNFtgfT2LFbtzls3kwsXtyjbaG3toYGupOV5cqrpRoaYOedYdWq7XpU9UhWZJXULsBJwD7AHsA4STPLz4uI+RHRGhGtkydPrneYZsNXdo3qmTO3lB4G0EBdUWmp0fKV5RYuHNbLj1r/iqySOgZ4IiK6ImIjcCVwRIHxmA1v2TaI5uZk2dEKM7oOSG9tECN8ZTmrrMiEsQw4XNJYSQKmAw8VGI/Z8NPeDuPH9yxFrFy57cuOlrgNwsoU2YZxh6QrgLuATcDdQM85A8yssvb2pBQx2MRQMn06PPbYdj+zq/Wu0F5SEfGFiDgoIt4UEadGxPoi4zEbcrLtEFKynS5D2n3+3Ooki9I4iGuv9cyu1qd+E4akayRVmODFzGqqNKI62w6xciWbZp7Ol3b8Gixfvu3XHj06mbDP4yBsAPKUMD4LfFfSTyVVXvPQzAauvPQwfnzyOh04F+ecU3FEdSObaVv/fV5q2rX/e4wbt2V7VPq/e0sL/PSnLkHYgPWbMCLirog4GrgK+KOkL0jasfahmY1gc+YkjdTZ0sOrryav04FzffVw2p1n2WXBt7dMr1HJ2WfD6tVbejJt3pz8dHWTbaNcbRhpL6ZHgIuAjwOPSjq1loGZjQjlpYhSSSK7gFAv+hwbMWVK8kt/4cKtSxGQvF60yNVMVnX99pKSdAvJ4LoHgNuBM4CHgXMkHRURnmLSrKS9PemKunp17+dUGDg3IKNHb1mGdMYMlxasbvJ0q20DHoyeC2d8XJLHTZiVpt0YyKR8eZVGU5eqp7wMqRWo34QREQ/0cfgDVYzFbOgrX1Vuv/3guuuStoFqGzvWycGGlEEN3IuIJdUKxGzIa29PZnTNLDkanZ0Dm4epL01NMGGCJ++zIavo6c3Nhq7y9SHOOafHdN/bnCxGjUqql0pTbixaBOvXe/I+G9K8RKtZJZVKEwwiQWS5HcKGKScMszJP3vE0u5x1LuOrUZqQkl5T7uJqI4CrpGz70N4OY8ZsPR7imGMA6HroeS7/9G2cdfBNHND0BFMO34Oxq1ds232kZMBcabBcd7eThY0YLmHYyNfeDqee2qMnUyxezG06gndxK9DMBF7m3bs+zNnv7GTTLbvT9HyFFYMnTUoG3pV6SR13HPz+957h1bYLThg2MmTHQjQ0JNNgtLTAV79K9/lzGVWh26uAd3Ib/3LMDUz/cDOtMw+iccyh6fW+sXUbBribq2331HM83tDV2toaHR0dRYdhQ0F7e9JrqZ8V5dYxhibW9V332tv/A+VjLlx6sGFK0p0R0TrY67iEYcNDtgQxalTSNpDDGNaxmVFAvvO34mk3zLbiRm8bukrjIKSkDaI09UbOZFHSQHfPtalLpk8fXIxm2xEnDBsasmtTS0kp4rTTtiSJwVSdtrTApZfCDjtsvX/69GSVOTPLxQnDipEdRd3cnKwNkZ3FtdQldbDGjt3S9rBu3ZburhFOFmYD5IRh9VcaRd3Zmfzi7qfhesBK1U8tLTB/vtshzKqk0IQhaWdJV0h6WNJDkt5ZZDxWA+XzMZUar8tGUVfFpEnJnEzd3V5ZzqwGiu4ldQHwx4j475KagLEFx2PVVGE+po2nzaaxe/22T7MR0WOchZOCWX0UljAk7QT8PckKfkTEBqDnivc2LEV3sPFT59FUVpIY3b2eTTTQyOZ8Fxo3Lkk4HgdhVrgiq6T2AbqAn0q6W9KPJY0rP0lSm6QOSR1dXV31j9JyW/rn5SyYdTMzpt7CHqNX0Nj1TMXzGtmcNEZnNTUl/0pGjUrmZFq92tN9mw0RRSaMRuDtwEURcQjwKnB++UkRMT8iWiOidfLkyfWO0frw3P1d/OwTt/LRg27i9aM72eeovZh98VEsfnJ/jt77MV4dt2vlN5Yao1tatqwHsWBBsh5EqQfT5s2etM9siCmyDWM5sDwi7khfX0GFhGFDx0vLXuLG//swi3+3juse3oP71+8PTGYnXmLa6x7mnCOWMv20PTn4hNejUbtC+7crz8dUqlpyicFsWCksYUTEs5KelHRgRDwCTAceLCqe7VYf8yWtXbWWW378EIt/9TLX3ddMx6tvoJvD2JE1HDnxIWZMu4Hpp+zKIR8+gMYxh/W8dikheD4msxGh0MkHJb0N+DHQBCwBZkXEC72d78kHBymbHMaNS9oHymxuGsNv9juX7z39IW598WA2sAONbOTQCQ8x/W2rmP6hnTl81hvY4e92qHADMxuKRsTkgxFxDzDoD2E5lHdxrZAsABo2rOOQBy/lpR1n8PF33Mb0E8dx5JkHMWGPt9QxWDMbiooeh2H1MoDBci16krvWvAF4Q21jMrNhxVODjASVRlOnlv/HM1zS9meic1nuy2nKlOrHaGbDnksYw12F0dSbzpjNJec9zNe7PsKjG/cBdudo9mRvlvd/vVIvJjOzMi5hDHXt7clsrtlpv6XXShLdn53bo6qpcdN6jnl6IQfs0sW3T7yBe37+CHte8rWeg+XKjR/vyfrMrFcuYQxFfS0/WurV1tnJ2plnsgPrKl5iby3nqucOzew5EEZp615Sr766ZW6mtjYPlDOzPjlhDAXZ5UdLE+zlsGMfy49WbIfwYDkzGwRXSRUlu/zozJnbvLJcA909q5rcDmFmNeCEUU+9rVE9GJXmZXI7hJnVgKuk6qW8N1M1Rth7XiYzqyOXMKqtwpiIF554kTVz/qk6q8x5+VEzK4gTxrYo7+ra3JzsK1+rOu3J9LF9r2LMy89t+/1KS49GePlRMytMoZMPDtSQmHywvR1mzYKNG7fa3d0wmrXsyLjNL/d4y4s77MaYsaMY80LlBYUq8vKjZlYl1Zp80CWMSvqYaoN583okC4BRmzcytkKyANh5wwrGfP+bPXszlaqXGhqSny0tW0oSLkGY2RDjRu+S3sZCdHYSbW08ffcKfr3s7ZzduWzgWXbKFK8NYWbD3vZdJTWAAXNLaWEflvIke7EXT1U+adIkWLu25wpzbpw2swK5Smqwsg3U0G831xaW8fj1y9jz0v8Do0f3PKGpCS64wGMizGzEGvlVUuXzMk2aBBdcQPdn5zJqAN1c1TKFfadNAWaAqHjN1xKDE4SZjUAju0qqlx5NGxhNIxvzF69crWRmw5irpPqxcc1G1n3iMxV7NDWxkejvo3uAnJnZVoZXwli1qvKAOaB7Uzf3/uIR/u2kGzh+t78wadxamlY92+ulKk7al00Sl17q7q1mZhmFJwxJDZLulnRVvycvXbr1GhErV7L5tFlcMPGL7Na0ird9+EDO/c00Hn1hMjMOvoe1E3bt/VqVJu1zkjAz61XhbRiSPg20An8XEcf3dW6rFJVaMJazJ/P2/RnTp8PRs/dl78P2SA700oZBUxMsWOCkYGbbhWq1YRTaS0rSXsAHgK8Cn97W6+ypp1n4+JE9D5QSQl89mszMLJeiu9V+FzgPmNDbCZLagDaAd/R2TqXV5Uo89beZWVUU1oYh6XhgRUTc2dd5ETE/IlojovW1RumspiavLmdmVgdFNnq/CzhR0lLgZ8B7JC3q8x1TpyZVSiWTJrktwsysTgqrkoqIucBcAEnTgH+KiJl9vmniRFiypOaxmZlZT4V3qzUzs+Gh6EZvACLiBuCGgsMwM7M+uIRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrk4YZiZWS5OGGZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrk4YZiZWS5OGGZmlosThpmZ5VJYwpC0t6TrJT0o6QFJ5xQVi5mZ9a+xwHtvAs6NiLskTQDulHRNRDxYYExmZtaLwkoYEfFMRNyVbr8CPATsWVQ8ZmbWtyHRhiFpKnAIcEeFY22SOiR1dHV11T02MzNLFJ4wJI0Hfgl8MiJeLj8eEfMjojUiWidPnlz/AM3MDCg4YUgaTZIs2iPiyiJjMTOzvhXZS0rAT4CHIuLfiorDzMzyKbKE8S7gVOA9ku5J/x1XYDxmZtaHwrrVRsSfARV1fzMzG5jCG73NzGx4cMIwM7NcnDDMzCwXJwwzM8vFCcPMzHJxwjAzs1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwzM8vFCcPMzHJxwjAzs1ycMMzMLBcnDDMzy6XQhCHpWEmPSHpM0vlFxmJmZn0rLGFIagB+ALwfOBg4RdLBRcVjZmZ9K7KEcSjwWEQsiYgNwM+AkwqMx8zM+tBY4L33BJ7MvF4OHFZ+kqQ2oC19uV7S/XWIbbCageeLDiIHx1k9wyFGcJzVNlziPLAaFykyYeQSEfOB+QCSOiKiteCQ+uU4q2s4xDkcYgTHWW3DKc5qXKfIKqmngL0zr/dK95mZ2RBUZML4D2B/SftIagJOBn5TYDxmZtaHwqqkImKTpI8BfwIagAUR8UA/b5tf+8iqwnFW13CIczjECI6z2rarOBUR1biOmZmNcB7pbWZmuThhmJlZLkMmYfQ3TYikHST9PD1+h6SpmWNz0/2PSHpfgTF+WtKDkv4qabGklsyxzZLuSf/VtHE/R5xnSOrKxHNm5tjpkh5N/51ecJzfycT4N0kvZo7V5XlKWiBpRW/jf5T4XvoZ/irp7Zlj9XyW/cU5I43vPkm3Snpr5tjSdP891ep+OYg4p0l6KfPf9vOZY3WbSihHnJ/JxHh/+n2cmB6ry/OUtLek69PfOQ9IOqfCOdX9fkZE4f9IGr0fB/YFmoB7gYPLzpkD/DDdPhn4ebp9cHr+DsA+6XUaCorxaGBsun12Kcb09eoh9CzPAP69wnsnAkvSn7uk27sUFWfZ+R8n6RhR7+f598Dbgft7OX4c8AdAwOHAHfV+ljnjPKJ0f5LpeO7IHFsKNA+R5zkNuGqw35dax1l27gnAdfV+nsDuwNvT7QnA3yr8v17V7+dQKWHkmSbkJGBhun0FMF2S0v0/i4j1EfEE8Fh6vbrHGBHXR8Sa9OXtJGNL6m0wU668D7gmIlZFxAvANcCxQyTOU4DLahRLryLiJmBVH6ecBFwSiduBnSXtTn2fZb9xRsStaRxQ3Hczz/PsTV2nEhpgnEV9N5+JiLvS7VeAh0hm0Miq6vdzqCSMStOElH/w186JiE3AS8CknO+tV4xZs0kye8kYSR2Sbpf0wRrEV5I3zg+lRdQrJJUGUNbrWQ7oXmnV3j7AdZnd9Xqe/entc9TzWQ5U+XczgKsl3alkKp6ivVPSvZL+IOmN6b4h+TwljSX5RfvLzO66P08lVfSHAHeUHarq93PITw0yHEmaCbQC787sbomIpyTtC1wn6b6IeLyYCPktcFlErJf0jyQlt/cUFEseJwNXRMTmzL6h9DyHDUlHkySMIzO7j0yf5a7ANZIeTv/CLsJdJP9tV0s6Dvg1sH9BseRxAnBLRGRLI3V9npLGkySsT0bEy7W6DwydEkaeaUJeO0dSI7ATsDLne+sVI5KOAeYBJ0bE+tL+iHgq/bkEuIHkr4Fa6DfOiFiZie3HwDvyvreecWacTFmRv47Psz+9fY4hN/WNpLeQ/Pc+KSJWlvZnnuUK4FfUpko3l4h4OSJWp9u/B0ZLamYIPs9UX9/Nmj9PSaNJkkV7RFxZ4ZTqfj9r3TCTs/GmkaTRZR+2NGi9seyc/8XWjd6/SLffyNaN3kuoTaN3nhgPIWmY279s/y7ADul2M/AoNWqwyxnn7pntfwBujy0NYU+k8e6Sbk8sKs70vINIGhFVxPNM7zGV3htpP8DWjYp/qfezzBnnFJL2vSPK9o8DJmS2bwWOLTDO15X+W5P8ol2WPttc35d6xZke34mknWNcEc8zfS6XAN/t45yqfj9r9rC34cMfR9LK/zgwL933ZZK/1AHGAJenX/q/APtm3jsvfd8jwPsLjPFa4DngnvTfb9L9RwD3pV/y+4DZBT/LrwEPpPFcDxyUee9H0mf8GDCryDjT118Evl72vro9T5K/Hp8BNpLU884GzgLOSo+LZCGwx9NYWgt6lv3F+WPghcx3syPdv2/6HO9NvxPzCo7zY5nv5u1kElyl70tRcabnnEHS4Sb7vro9T5JqxQD+mvnvelwtv5+eGsTMzHIZKm0YZmY2xDlhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuThhmZpaLE4bZIKTrEbw33f4XSd8vOiazWvHkg2aD8wXgy+lEc4cAJxYcj1nNeKS32SBJuhEYD0yLZF0CsxHJVVJmgyDpzSQrn21wsrCRzgnDbBulK5e1k6xqtlpSzVbUMxsKnDDMtkG60tqVwLkR8RDwFZL2DLMRy20YZmaWi0sYZmaWixOGmZnl4oRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrn8JxI0W4aibDePAAAAAElFTkSuQmCC\n", - "text/plain": [ - "
" - ] - }, - "metadata": { - "needs_background": "light" - }, - "output_type": "display_data" - } - ], + "execution_count": 14, + "id": "27c60383", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -2583,7 +2856,7 @@ "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", - "y = 4+3*x#+np.random.randn(n,1)\n", + "y = 4+3*x+np.random.randn(n,1)\n", "\n", "X = np.c_[np.ones((n,1)), x]\n", "XT_X = X.T @ X\n", @@ -2645,8 +2918,10 @@ }, { "cell_type": "markdown", - "id": "99a3e2be", - "metadata": {}, + "id": "8eb3e862", + "metadata": { + "editable": true + }, "source": [ "## Replace or not\n", "\n", @@ -2658,8 +2933,10 @@ }, { "cell_type": "markdown", - "id": "12ad19f6", - "metadata": {}, + "id": "34c63f75", + "metadata": { + "editable": true + }, "source": [ "## Momentum based GD\n", "\n", @@ -2671,8 +2948,10 @@ }, { "cell_type": "markdown", - "id": "529fe4b5", - "metadata": {}, + "id": "b526eb97", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", @@ -2681,8 +2960,10 @@ }, { "cell_type": "markdown", - "id": "6f31d112", - "metadata": {}, + "id": "a65b9f7c", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2697,8 +2978,10 @@ }, { "cell_type": "markdown", - "id": "214661ae", - "metadata": {}, + "id": "501f2ee6", + "metadata": { + "editable": true + }, "source": [ "where we have introduced a momentum parameter $\\gamma$, with\n", "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", @@ -2714,8 +2997,10 @@ }, { "cell_type": "markdown", - "id": "157ac0bc", - "metadata": {}, + "id": "d7f9d1c1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", @@ -2724,16 +3009,20 @@ }, { "cell_type": "markdown", - "id": "0e4d8eb6", - "metadata": {}, + "id": "2f6d7af0", + "metadata": { + "editable": true + }, "source": [ "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." ] }, { "cell_type": "markdown", - "id": "5030a332", - "metadata": {}, + "id": "894ce2fd", + "metadata": { + "editable": true + }, "source": [ "## More on momentum based approaches\n", "\n", @@ -2746,8 +3035,10 @@ }, { "cell_type": "markdown", - "id": "1e3162a0", - "metadata": {}, + "id": "11ddd872", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", @@ -2756,16 +3047,20 @@ }, { "cell_type": "markdown", - "id": "39341fc6", - "metadata": {}, + "id": "c0cadfff", + "metadata": { + "editable": true + }, "source": [ "We can discretize this equation in the usual way to get" ] }, { "cell_type": "markdown", - "id": "4a925447", - "metadata": {}, + "id": "428438f5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", @@ -2774,16 +3069,20 @@ }, { "cell_type": "markdown", - "id": "5bc8857b", - "metadata": {}, + "id": "92e9357c", + "metadata": { + "editable": true + }, "source": [ "Rearranging this equation, we can rewrite this as" ] }, { "cell_type": "markdown", - "id": "3f3a3dca", - "metadata": {}, + "id": "5c0e73c6", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", @@ -2792,8 +3091,10 @@ }, { "cell_type": "markdown", - "id": "a1e7d5d2", - "metadata": {}, + "id": "0ea1cb78", + "metadata": { + "editable": true + }, "source": [ "## Momentum parameter\n", "\n", @@ -2806,8 +3107,10 @@ }, { "cell_type": "markdown", - "id": "258f174a", - "metadata": {}, + "id": "89f597f5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", @@ -2816,8 +3119,10 @@ }, { "cell_type": "markdown", - "id": "9405179c", - "metadata": {}, + "id": "954d762d", + "metadata": { + "editable": true + }, "source": [ "Thus, as the name suggests, the momentum parameter is proportional to\n", "the mass of the particle and effectively provides inertia.\n", @@ -2847,8 +3152,10 @@ }, { "cell_type": "markdown", - "id": "42647548", - "metadata": {}, + "id": "38641257", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", @@ -2857,8 +3164,10 @@ }, { "cell_type": "markdown", - "id": "fa8f3d45", - "metadata": {}, + "id": "22c5b321", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2873,16 +3182,20 @@ }, { "cell_type": "markdown", - "id": "757a5a92", - "metadata": {}, + "id": "c7309191", + "metadata": { + "editable": true + }, "source": [ "One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$." ] }, { "cell_type": "markdown", - "id": "ec07c646", - "metadata": {}, + "id": "56a16b68", + "metadata": { + "editable": true + }, "source": [ "## Second moment of the gradient\n", "\n", @@ -2910,8 +3223,10 @@ }, { "cell_type": "markdown", - "id": "c1d51956", - "metadata": {}, + "id": "b7c6d361", + "metadata": { + "editable": true + }, "source": [ "## RMS prop\n", "\n", @@ -2923,8 +3238,10 @@ }, { "cell_type": "markdown", - "id": "f8511d10", - "metadata": {}, + "id": "78d79a03", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2939,8 +3256,10 @@ }, { "cell_type": "markdown", - "id": "fd599d2a", - "metadata": {}, + "id": "ad384bea", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", @@ -2949,8 +3268,10 @@ }, { "cell_type": "markdown", - "id": "073455c5", - "metadata": {}, + "id": "a2b9882c", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", @@ -2959,8 +3280,10 @@ }, { "cell_type": "markdown", - "id": "0550cfc9", - "metadata": {}, + "id": "70d9b90b", + "metadata": { + "editable": true + }, "source": [ "where $\\beta$ controls the averaging time of the second moment and is\n", "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", @@ -2975,8 +3298,10 @@ }, { "cell_type": "markdown", - "id": "a3e891ab", - "metadata": {}, + "id": "6b4c7e03", + "metadata": { + "editable": true + }, "source": [ "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", "\n", @@ -2996,8 +3321,10 @@ }, { "cell_type": "markdown", - "id": "b56d6328", - "metadata": {}, + "id": "4fc5c09c", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -3012,8 +3339,10 @@ }, { "cell_type": "markdown", - "id": "a496b94c", - "metadata": {}, + "id": "6dac702f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", @@ -3022,8 +3351,10 @@ }, { "cell_type": "markdown", - "id": "0abe86a1", - "metadata": {}, + "id": "38fb872a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", @@ -3032,8 +3363,10 @@ }, { "cell_type": "markdown", - "id": "6d799dcf", - "metadata": {}, + "id": "48436706", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", @@ -3042,8 +3375,10 @@ }, { "cell_type": "markdown", - "id": "437c3cef", - "metadata": {}, + "id": "711b05c2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", @@ -3052,8 +3387,10 @@ }, { "cell_type": "markdown", - "id": "c3752fd1", - "metadata": {}, + "id": "7c5be7c4", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", @@ -3062,8 +3399,10 @@ }, { "cell_type": "markdown", - "id": "28e99f74", - "metadata": {}, + "id": "91fbb6a2", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -3077,8 +3416,10 @@ }, { "cell_type": "markdown", - "id": "c69e5b35", - "metadata": {}, + "id": "e25b8b4b", + "metadata": { + "editable": true + }, "source": [ "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", "second moment and are typically taken to be $0.9$ and $0.99$\n", @@ -3094,8 +3435,10 @@ }, { "cell_type": "markdown", - "id": "2207777f", - "metadata": {}, + "id": "bc510d5a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", @@ -3104,8 +3447,10 @@ }, { "cell_type": "markdown", - "id": "a9dd6e15", - "metadata": {}, + "id": "0957ade8", + "metadata": { + "editable": true + }, "source": [ "## Practical tips\n", "\n", @@ -3122,8 +3467,10 @@ }, { "cell_type": "markdown", - "id": "00b94d96", - "metadata": {}, + "id": "c0c4e70c", + "metadata": { + "editable": true + }, "source": [ "## Automatic differentiation\n", "\n", @@ -3158,8 +3505,10 @@ }, { "cell_type": "markdown", - "id": "40e322b8", - "metadata": {}, + "id": "2479e6ec", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(x) = \\sin\\left(2\\pi x + x^2\\right)\n", @@ -3168,16 +3517,20 @@ }, { "cell_type": "markdown", - "id": "6ef03583", - "metadata": {}, + "id": "25573059", + "metadata": { + "editable": true + }, "source": [ "which has the following derivative" ] }, { "cell_type": "markdown", - "id": "a90642b5", - "metadata": {}, + "id": "99867c27", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f'(x) = \\cos\\left(2\\pi x + x^2\\right)\\left(2\\pi + 2x\\right)\n", @@ -3186,8 +3539,10 @@ }, { "cell_type": "markdown", - "id": "f13e4196", - "metadata": {}, + "id": "ab095717", + "metadata": { + "editable": true + }, "source": [ "Using **autograd** we have" ] @@ -3195,8 +3550,11 @@ { "cell_type": "code", "execution_count": 15, - "id": "e91017bf", - "metadata": {}, + "id": "957de0bd", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3237,8 +3595,10 @@ }, { "cell_type": "markdown", - "id": "edb98a44", - "metadata": {}, + "id": "6b45687d", + "metadata": { + "editable": true + }, "source": [ "## Using autograd\n", "\n", @@ -3252,8 +3612,11 @@ { "cell_type": "code", "execution_count": 16, - "id": "8101c103", - "metadata": {}, + "id": "3250aae4", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3277,8 +3640,10 @@ }, { "cell_type": "markdown", - "id": "7925f3b4", - "metadata": {}, + "id": "4bb0ad31", + "metadata": { + "editable": true + }, "source": [ "## Autograd with more complicated functions\n", "\n", @@ -3290,8 +3655,11 @@ { "cell_type": "code", "execution_count": 17, - "id": "f2c69e0f", - "metadata": {}, + "id": "515ff31b", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3331,16 +3699,20 @@ }, { "cell_type": "markdown", - "id": "0fbbc5a9", - "metadata": {}, + "id": "774f5eda", + "metadata": { + "editable": true + }, "source": [ "Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable." ] }, { "cell_type": "markdown", - "id": "96d43a0a", - "metadata": {}, + "id": "68b8b8f7", + "metadata": { + "editable": true + }, "source": [ "## More complicated functions using the elements of their arguments directly" ] @@ -3348,8 +3720,11 @@ { "cell_type": "code", "execution_count": 18, - "id": "f53794e7", - "metadata": {}, + "id": "bcf539e8", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3373,8 +3748,10 @@ }, { "cell_type": "markdown", - "id": "cee7be4d", - "metadata": {}, + "id": "cb8ac207", + "metadata": { + "editable": true + }, "source": [ "Note that in this case, when sending an array as input argument, the\n", "output from Autograd is another array. This is the true gradient of\n", @@ -3386,8 +3763,10 @@ }, { "cell_type": "markdown", - "id": "df2680c1", - "metadata": {}, + "id": "c5f87a92", + "metadata": { + "editable": true + }, "source": [ "## Functions using mathematical functions from Numpy" ] @@ -3395,8 +3774,11 @@ { "cell_type": "code", "execution_count": 19, - "id": "e752fa81", - "metadata": {}, + "id": "7d46685e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3420,8 +3802,10 @@ }, { "cell_type": "markdown", - "id": "317db3e8", - "metadata": {}, + "id": "1f7f96c0", + "metadata": { + "editable": true + }, "source": [ "## More autograd" ] @@ -3429,8 +3813,11 @@ { "cell_type": "code", "execution_count": 20, - "id": "96609851", - "metadata": {}, + "id": "e49128c7", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3451,8 +3838,10 @@ }, { "cell_type": "markdown", - "id": "587bbcc3", - "metadata": {}, + "id": "03135da4", + "metadata": { + "editable": true + }, "source": [ "## And with loops" ] @@ -3460,8 +3849,11 @@ { "cell_type": "code", "execution_count": 21, - "id": "c62adaa8", - "metadata": {}, + "id": "968ba627", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3493,8 +3885,11 @@ { "cell_type": "code", "execution_count": 22, - "id": "75db0457", - "metadata": {}, + "id": "ebaba4fb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3510,8 +3905,10 @@ }, { "cell_type": "markdown", - "id": "c99c7e99", - "metadata": {}, + "id": "680f026e", + "metadata": { + "editable": true + }, "source": [ "## Using recursion" ] @@ -3519,8 +3916,11 @@ { "cell_type": "code", "execution_count": 23, - "id": "779140f3", - "metadata": {}, + "id": "092801c3", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3554,16 +3954,20 @@ }, { "cell_type": "markdown", - "id": "79051822", - "metadata": {}, + "id": "aaaf1ad3", + "metadata": { + "editable": true + }, "source": [ "Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input." ] }, { "cell_type": "markdown", - "id": "3644a47e", - "metadata": {}, + "id": "f5c6d17b", + "metadata": { + "editable": true + }, "source": [ "## Unsupported functions\n", "Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.\n", @@ -3574,8 +3978,11 @@ { "cell_type": "code", "execution_count": 24, - "id": "39e8bbff", - "metadata": {}, + "id": "1d32abfa", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3593,16 +4000,20 @@ }, { "cell_type": "markdown", - "id": "d42d292f", - "metadata": {}, + "id": "5f858e86", + "metadata": { + "editable": true + }, "source": [ "Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible." ] }, { "cell_type": "markdown", - "id": "65638c5a", - "metadata": {}, + "id": "1179e3dd", + "metadata": { + "editable": true + }, "source": [ "## The syntax a.dot(b) when finding the dot product" ] @@ -3610,8 +4021,11 @@ { "cell_type": "code", "execution_count": 25, - "id": "bec4592d", - "metadata": {}, + "id": "dba3ad31", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3629,8 +4043,10 @@ }, { "cell_type": "markdown", - "id": "0c4b4d89", - "metadata": {}, + "id": "2d71a680", + "metadata": { + "editable": true + }, "source": [ "Here we are told that the 'dot' function does not belong to Autograd's\n", "version of a Numpy array. To overcome this, an alternative syntax\n", @@ -3640,8 +4056,11 @@ { "cell_type": "code", "execution_count": 26, - "id": "520e4b18", - "metadata": {}, + "id": "6ac01a09", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3662,8 +4081,10 @@ }, { "cell_type": "markdown", - "id": "5d122ff1", - "metadata": {}, + "id": "5becf640", + "metadata": { + "editable": true + }, "source": [ "## Recommended to avoid\n", "The documentation recommends to avoid inplace operations such as" @@ -3672,8 +4093,11 @@ { "cell_type": "code", "execution_count": 27, - "id": "7f4d866e", - "metadata": {}, + "id": "6511ed46", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "a += b\n", @@ -3684,8 +4108,10 @@ }, { "cell_type": "markdown", - "id": "d66130d2", - "metadata": {}, + "id": "3f49669a", + "metadata": { + "editable": true + }, "source": [ "## Using Autograd with OLS\n", "\n", @@ -3697,8 +4123,11 @@ { "cell_type": "code", "execution_count": 28, - "id": "59144e98", - "metadata": {}, + "id": "7207a933", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients for OLS\n", @@ -3754,8 +4183,10 @@ }, { "cell_type": "markdown", - "id": "2b5ef0aa", - "metadata": {}, + "id": "eb71cd5b", + "metadata": { + "editable": true + }, "source": [ "## Including Stochastic Gradient Descent with Autograd\n", "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**." @@ -3764,8 +4195,11 @@ { "cell_type": "code", "execution_count": 29, - "id": "db6e0b30", - "metadata": {}, + "id": "f5fade01", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Using Autograd to calculate gradients using SGD\n", @@ -3845,8 +4279,10 @@ }, { "cell_type": "markdown", - "id": "e1b541fa", - "metadata": {}, + "id": "8a03d0c5", + "metadata": { + "editable": true + }, "source": [ "## And Logistic Regression" ] @@ -3854,8 +4290,11 @@ { "cell_type": "code", "execution_count": 30, - "id": "08d9299d", - "metadata": {}, + "id": "885dd6a7", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3895,8 +4334,10 @@ }, { "cell_type": "markdown", - "id": "4860575d", - "metadata": {}, + "id": "3168e44a", + "metadata": { + "editable": true + }, "source": [ "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n", "\n", @@ -3912,8 +4353,11 @@ { "cell_type": "code", "execution_count": 31, - "id": "fccac790", - "metadata": {}, + "id": "592c4321", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import jax.numpy as jnp\n", @@ -3928,25 +4372,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.14" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt index cf9948925..e626e3cfc 100644 --- a/doc/src/week39/week39.do.txt +++ b/doc/src/week39/week39.do.txt @@ -6,6 +6,7 @@ DATE: today ===== Plan for week 39 ===== * Thursday: Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent + * "Video of lecture":"https://youtu.be/rDBj50Lv3Go" * Friday: Stochastic Gradient descent with examples and automatic differentiation * Reading recommendations: