diff --git a/doc/HandWrittenNotes/2022/NotesSep292022pdf b/doc/HandWrittenNotes/2022/NotesSep292022pdf new file mode 100644 index 000000000..5255682ff Binary files /dev/null and b/doc/HandWrittenNotes/2022/NotesSep292022pdf differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 4e8920c3c..6cb26a370 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -2,10 +2,8 @@ "cells": [ { "cell_type": "markdown", - "id": "e6d28181", - "metadata": { - "editable": true - }, + "id": "4c73ece8", + "metadata": {}, "source": [ "\n", @@ -14,10 +12,8 @@ }, { "cell_type": "markdown", - "id": "fa5a3702", - "metadata": { - "editable": true - }, + "id": "22602ad0", + "metadata": {}, "source": [ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -29,10 +25,8 @@ }, { "cell_type": "markdown", - "id": "d109da99", - "metadata": { - "editable": true - }, + "id": "f2fcd724", + "metadata": {}, "source": [ "## Plan for week 39\n", "\n", @@ -53,10 +47,8 @@ }, { "cell_type": "markdown", - "id": "6962d370", - "metadata": { - "editable": true - }, + "id": "67125149", + "metadata": {}, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -74,10 +66,8 @@ }, { "cell_type": "markdown", - "id": "118ff3da", - "metadata": { - "editable": true - }, + "id": "0d07977c", + "metadata": {}, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -91,10 +81,8 @@ }, { "cell_type": "markdown", - "id": "290c6917", - "metadata": { - "editable": true - }, + "id": "973763d7", + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -106,20 +94,16 @@ }, { "cell_type": "markdown", - "id": "6e462b95", - "metadata": { - "editable": true - }, + "id": "d0a82293", + "metadata": {}, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] }, { "cell_type": "markdown", - "id": "ff55d62b", - "metadata": { - "editable": true - }, + "id": "de856843", + "metadata": {}, "source": [ "## The equations to solve\n", "\n", @@ -132,10 +116,8 @@ }, { "cell_type": "markdown", - "id": "47b40f46", - "metadata": { - "editable": true - }, + "id": "a60566a0", + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -144,10 +126,8 @@ }, { "cell_type": "markdown", - "id": "b42e0f80", - "metadata": { - "editable": true - }, + "id": "ef8826c0", + "metadata": {}, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -155,10 +135,8 @@ }, { "cell_type": "markdown", - "id": "27b035db", - "metadata": { - "editable": true - }, + "id": "566fdac8", + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -167,20 +145,16 @@ }, { "cell_type": "markdown", - "id": "6475213d", - "metadata": { - "editable": true - }, + "id": "69a2105f", + "metadata": {}, "source": [ "This defines what is called the Hessian matrix." ] }, { "cell_type": "markdown", - "id": "67c8df04", - "metadata": { - "editable": true - }, + "id": "0e766c2c", + "metadata": {}, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -191,10 +165,8 @@ }, { "cell_type": "markdown", - "id": "74a165b8", - "metadata": { - "editable": true - }, + "id": "900e0a5d", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -203,20 +175,16 @@ }, { "cell_type": "markdown", - "id": "ad20ab46", - "metadata": { - "editable": true - }, + "id": "fb485e9e", + "metadata": {}, "source": [ "or in matrix form as" ] }, { "cell_type": "markdown", - "id": "91787790", - "metadata": { - "editable": true - }, + "id": "c7110832", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -225,10 +193,8 @@ }, { "cell_type": "markdown", - "id": "939e7f11", - "metadata": { - "editable": true - }, + "id": "db0fd8b9", + "metadata": {}, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -237,10 +203,8 @@ }, { "cell_type": "markdown", - "id": "06bdab12", - "metadata": { - "editable": true - }, + "id": "73cc9db9", + "metadata": {}, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -257,10 +221,8 @@ }, { "cell_type": "markdown", - "id": "de5be449", - "metadata": { - "editable": true - }, + "id": "0483698c", + "metadata": {}, "source": [ "## The equations\n", "\n", @@ -273,10 +235,8 @@ }, { "cell_type": "markdown", - "id": "8cde618b", - "metadata": { - "editable": true - }, + "id": "0b41326e", + "metadata": {}, "source": [ "\n", "
\n", @@ -289,10 +249,8 @@ }, { "cell_type": "markdown", - "id": "2b2088d2", - "metadata": { - "editable": true - }, + "id": "bf22ca78", + "metadata": {}, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -300,10 +258,8 @@ }, { "cell_type": "markdown", - "id": "9eebba32", - "metadata": { - "editable": true - }, + "id": "e70a4b40", + "metadata": {}, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -312,20 +268,16 @@ }, { "cell_type": "markdown", - "id": "41f640fd", - "metadata": { - "editable": true - }, + "id": "e6ab42db", + "metadata": {}, "source": [ "yielding" ] }, { "cell_type": "markdown", - "id": "377a88a5", - "metadata": { - "editable": true - }, + "id": "60f49110", + "metadata": {}, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -334,20 +286,16 @@ }, { "cell_type": "markdown", - "id": "266e19d2", - "metadata": { - "editable": true - }, + "id": "4a5daac5", + "metadata": {}, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] }, { "cell_type": "markdown", - "id": "8f913de7", - "metadata": { - "editable": true - }, + "id": "dc406fad", + "metadata": {}, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -356,10 +304,8 @@ }, { "cell_type": "markdown", - "id": "b5688397", - "metadata": { - "editable": true - }, + "id": "4a01cef1", + "metadata": {}, "source": [ "## Simple geometric interpretation\n", "\n", @@ -378,10 +324,8 @@ }, { "cell_type": "markdown", - "id": "f419f089", - "metadata": { - "editable": true - }, + "id": "b9ab351f", + "metadata": {}, "source": [ "## Extending to more than one variable\n", "\n", @@ -391,10 +335,8 @@ }, { "cell_type": "markdown", - "id": "2094ec46", - "metadata": { - "editable": true - }, + "id": "c4a5c175", + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -404,20 +346,16 @@ }, { "cell_type": "markdown", - "id": "066054bd", - "metadata": { - "editable": true - }, + "id": "5a0ec73f", + "metadata": {}, "source": [ "which we Taylor expand to obtain" ] }, { "cell_type": "markdown", - "id": "17ab0c82", - "metadata": { - "editable": true - }, + "id": "43a0b666", + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -432,20 +370,16 @@ }, { "cell_type": "markdown", - "id": "1e5c616e", - "metadata": { - "editable": true - }, + "id": "9378b391", + "metadata": {}, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] }, { "cell_type": "markdown", - "id": "aca5997c", - "metadata": { - "editable": true - }, + "id": "742ab267", + "metadata": {}, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -457,20 +391,16 @@ }, { "cell_type": "markdown", - "id": "3c8988ea", - "metadata": { - "editable": true - }, + "id": "974e20a1", + "metadata": {}, "source": [ "we can rephrase Newton's method as" ] }, { "cell_type": "markdown", - "id": "0837e054", - "metadata": { - "editable": true - }, + "id": "65662f4e", + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -481,20 +411,16 @@ }, { "cell_type": "markdown", - "id": "7363768a", - "metadata": { - "editable": true - }, + "id": "9363cc25", + "metadata": {}, "source": [ "where we have defined" ] }, { "cell_type": "markdown", - "id": "ec159670", - "metadata": { - "editable": true - }, + "id": "6e0064bd", + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -505,10 +431,8 @@ }, { "cell_type": "markdown", - "id": "2ea2c173", - "metadata": { - "editable": true - }, + "id": "94fd2ee4", + "metadata": {}, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -520,10 +444,8 @@ }, { "cell_type": "markdown", - "id": "023920fd", - "metadata": { - "editable": true - }, + "id": "36b663f2", + "metadata": {}, "source": [ "## Steepest descent\n", "\n", @@ -537,10 +459,8 @@ }, { "cell_type": "markdown", - "id": "ca16c582", - "metadata": { - "editable": true - }, + "id": "9d0c9ad9", + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -549,10 +469,8 @@ }, { "cell_type": "markdown", - "id": "03d4d84c", - "metadata": { - "editable": true - }, + "id": "47f4fafb", + "metadata": {}, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -563,10 +481,8 @@ }, { "cell_type": "markdown", - "id": "dfaa60b4", - "metadata": { - "editable": true - }, + "id": "e380cfcc", + "metadata": {}, "source": [ "## More on Steepest descent\n", "\n", @@ -578,10 +494,8 @@ }, { "cell_type": "markdown", - "id": "363fdd3f", - "metadata": { - "editable": true - }, + "id": "db59d2b8", + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -590,10 +504,8 @@ }, { "cell_type": "markdown", - "id": "a0e35dff", - "metadata": { - "editable": true - }, + "id": "21c9aac9", + "metadata": {}, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -601,10 +513,8 @@ }, { "cell_type": "markdown", - "id": "5ffef34f", - "metadata": { - "editable": true - }, + "id": "327ae752", + "metadata": {}, "source": [ "## The ideal\n", "\n", @@ -629,10 +539,8 @@ }, { "cell_type": "markdown", - "id": "b84a98e4", - "metadata": { - "editable": true - }, + "id": "f1c424b9", + "metadata": {}, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -651,10 +559,8 @@ }, { "cell_type": "markdown", - "id": "dfa1b5d0", - "metadata": { - "editable": true - }, + "id": "c30dc8df", + "metadata": {}, "source": [ "## Convex functions\n", "\n", @@ -673,10 +579,8 @@ }, { "cell_type": "markdown", - "id": "c637d309", - "metadata": { - "editable": true - }, + "id": "6298f287", + "metadata": {}, "source": [ "## Convex function\n", "\n", @@ -685,10 +589,8 @@ }, { "cell_type": "markdown", - "id": "f96469d1", - "metadata": { - "editable": true - }, + "id": "ea9513ed", + "metadata": {}, "source": [ "## Conditions on convex functions\n", "\n", @@ -722,10 +624,8 @@ }, { "cell_type": "markdown", - "id": "b4dda17e", - "metadata": { - "editable": true - }, + "id": "2f218b22", + "metadata": {}, "source": [ "## More on convex functions\n", "\n", @@ -750,10 +650,8 @@ }, { "cell_type": "markdown", - "id": "5af5be77", - "metadata": { - "editable": true - }, + "id": "67110e2a", + "metadata": {}, "source": [ "## Some simple problems\n", "\n", @@ -780,10 +678,8 @@ }, { "cell_type": "markdown", - "id": "43becec9", - "metadata": { - "editable": true - }, + "id": "c7ec5012", + "metadata": {}, "source": [ "## Standard steepest descent\n", "\n", @@ -800,10 +696,8 @@ }, { "cell_type": "markdown", - "id": "9b9ba362", - "metadata": { - "editable": true - }, + "id": "54f15818", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -812,20 +706,16 @@ }, { "cell_type": "markdown", - "id": "3d03f376", - "metadata": { - "editable": true - }, + "id": "bb846136", + "metadata": {}, "source": [ "In the iterative process we end up with a problem like" ] }, { "cell_type": "markdown", - "id": "baff0f9d", - "metadata": { - "editable": true - }, + "id": "14a050e6", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -834,10 +724,8 @@ }, { "cell_type": "markdown", - "id": "c66fb64f", - "metadata": { - "editable": true - }, + "id": "b5be57be", + "metadata": {}, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", @@ -846,10 +734,8 @@ }, { "cell_type": "markdown", - "id": "21a18d29", - "metadata": { - "editable": true - }, + "id": "6a8ce76f", + "metadata": {}, "source": [ "## Gradient method\n", "\n", @@ -858,10 +744,8 @@ }, { "cell_type": "markdown", - "id": "2c321843", - "metadata": { - "editable": true - }, + "id": "594769d8", + "metadata": {}, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -870,10 +754,8 @@ }, { "cell_type": "markdown", - "id": "77463dbe", - "metadata": { - "editable": true - }, + "id": "05f1fe97", + "metadata": {}, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite." @@ -881,10 +763,8 @@ }, { "cell_type": "markdown", - "id": "e8e305e8", - "metadata": { - "editable": true - }, + "id": "6efaa986", + "metadata": {}, "source": [ "## Steepest descent method\n", "\n", @@ -894,10 +774,8 @@ }, { "cell_type": "markdown", - "id": "f2332a0e", - "metadata": { - "editable": true - }, + "id": "99e242c6", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -906,20 +784,16 @@ }, { "cell_type": "markdown", - "id": "534a1b51", - "metadata": { - "editable": true - }, + "id": "07d95ef1", + "metadata": {}, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "8bed151e", - "metadata": { - "editable": true - }, + "id": "b81f2f36", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -928,20 +802,16 @@ }, { "cell_type": "markdown", - "id": "678df355", - "metadata": { - "editable": true - }, + "id": "83c5e15f", + "metadata": {}, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "6f23d8bf", - "metadata": { - "editable": true - }, + "id": "0d5239fd", + "metadata": {}, "source": [ "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -949,10 +819,8 @@ }, { "cell_type": "markdown", - "id": "87918b7a", - "metadata": { - "editable": true - }, + "id": "95cf152f", + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -961,10 +829,8 @@ }, { "cell_type": "markdown", - "id": "ae32eb82", - "metadata": { - "editable": true - }, + "id": "18692235", + "metadata": {}, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -973,10 +839,8 @@ }, { "cell_type": "markdown", - "id": "8f157e23", - "metadata": { - "editable": true - }, + "id": "a5076cdd", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -985,10 +849,8 @@ }, { "cell_type": "markdown", - "id": "d83d8368", - "metadata": { - "editable": true - }, + "id": "daa366ec", + "metadata": {}, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." @@ -996,10 +858,8 @@ }, { "cell_type": "markdown", - "id": "a396e128", - "metadata": { - "editable": true - }, + "id": "75a681f3", + "metadata": {}, "source": [ "## Final expressions\n", "We can compute the residual iteratively as" @@ -1007,10 +867,8 @@ }, { "cell_type": "markdown", - "id": "865cdb3d", - "metadata": { - "editable": true - }, + "id": "98536b08", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1019,20 +877,16 @@ }, { "cell_type": "markdown", - "id": "33e21b01", - "metadata": { - "editable": true - }, + "id": "aff9b796", + "metadata": {}, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "eae3b871", - "metadata": { - "editable": true - }, + "id": "4c01226c", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -1041,20 +895,16 @@ }, { "cell_type": "markdown", - "id": "16dae522", - "metadata": { - "editable": true - }, + "id": "62d91969", + "metadata": {}, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "d3e3ca2a", - "metadata": { - "editable": true - }, + "id": "a5d9f79a", + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -1063,20 +913,16 @@ }, { "cell_type": "markdown", - "id": "d71254ca", - "metadata": { - "editable": true - }, + "id": "04fb0972", + "metadata": {}, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "4681b36b", - "metadata": { - "editable": true - }, + "id": "69f177ee", + "metadata": {}, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -1085,20 +931,16 @@ }, { "cell_type": "markdown", - "id": "8b55691e", - "metadata": { - "editable": true - }, + "id": "826c4c1a", + "metadata": {}, "source": [ "leading to the iterative scheme" ] }, { "cell_type": "markdown", - "id": "7e54bdbb", - "metadata": { - "editable": true - }, + "id": "5b65620e", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_{k},\n", @@ -1107,10 +949,8 @@ }, { "cell_type": "markdown", - "id": "cade0e16", - "metadata": { - "editable": true - }, + "id": "5ff7d0ef", + "metadata": {}, "source": [ "## Steepest descent example" ] @@ -1118,11 +958,8 @@ { "cell_type": "code", "execution_count": 1, - "id": "431eb09d", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "aeb6d5c9", + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -1151,10 +988,8 @@ }, { "cell_type": "markdown", - "id": "45abe8c8", - "metadata": { - "editable": true - }, + "id": "3ae5899d", + "metadata": {}, "source": [ "And then as countor plot" ] @@ -1162,11 +997,8 @@ { "cell_type": "code", "execution_count": 2, - "id": "918d5156", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "c6c3a008", + "metadata": {}, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1176,10 +1008,8 @@ }, { "cell_type": "markdown", - "id": "62ce3f74", - "metadata": { - "editable": true - }, + "id": "0c47c360", + "metadata": {}, "source": [ "Find guesses" ] @@ -1187,11 +1017,8 @@ { "cell_type": "code", "execution_count": 3, - "id": "0cf280cb", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "db86ff05", + "metadata": {}, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1200,10 +1027,8 @@ }, { "cell_type": "markdown", - "id": "f84d35a1", - "metadata": { - "editable": true - }, + "id": "9e873223", + "metadata": {}, "source": [ "Run it!" ] @@ -1211,11 +1036,8 @@ { "cell_type": "code", "execution_count": 4, - "id": "b823ed66", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "71c9d8c1", + "metadata": {}, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1229,10 +1051,8 @@ }, { "cell_type": "markdown", - "id": "f8c48a33", - "metadata": { - "editable": true - }, + "id": "8ffc7e06", + "metadata": {}, "source": [ "What happened?" ] @@ -1240,11 +1060,8 @@ { "cell_type": "code", "execution_count": 5, - "id": "9cb2845a", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "857c0d1d", + "metadata": {}, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1255,20 +1072,16 @@ }, { "cell_type": "markdown", - "id": "91589091", - "metadata": { - "editable": true - }, + "id": "78a9e67c", + "metadata": {}, "source": [ "Note that we did only one iteration here. We can easily add more using our previous guesses." ] }, { "cell_type": "markdown", - "id": "2be4a5ee", - "metadata": { - "editable": true - }, + "id": "a759a930", + "metadata": {}, "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", @@ -1279,10 +1092,8 @@ }, { "cell_type": "markdown", - "id": "0a4e1cdc", - "metadata": { - "editable": true - }, + "id": "f185c679", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1291,10 +1102,8 @@ }, { "cell_type": "markdown", - "id": "2b998c83", - "metadata": { - "editable": true - }, + "id": "bbef395d", + "metadata": {}, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1302,10 +1111,8 @@ }, { "cell_type": "markdown", - "id": "2dc34fcc", - "metadata": { - "editable": true - }, + "id": "1f1417d3", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1314,10 +1121,8 @@ }, { "cell_type": "markdown", - "id": "96c0a3f0", - "metadata": { - "editable": true - }, + "id": "8482b2fb", + "metadata": {}, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." @@ -1325,10 +1130,8 @@ }, { "cell_type": "markdown", - "id": "1e311e8a", - "metadata": { - "editable": true - }, + "id": "12aa0a01", + "metadata": {}, "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" @@ -1336,10 +1139,8 @@ }, { "cell_type": "markdown", - "id": "90c86903", - "metadata": { - "editable": true - }, + "id": "da7bd359", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1348,20 +1149,16 @@ }, { "cell_type": "markdown", - "id": "80db69c5", - "metadata": { - "editable": true - }, + "id": "85c05b4f", + "metadata": {}, "source": [ "which is zero unless $i=j$." ] }, { "cell_type": "markdown", - "id": "1811fa1a", - "metadata": { - "editable": true - }, + "id": "57b05592", + "metadata": {}, "source": [ "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", @@ -1370,10 +1167,8 @@ }, { "cell_type": "markdown", - "id": "9a9d7e34", - "metadata": { - "editable": true - }, + "id": "25edaca6", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1382,10 +1177,8 @@ }, { "cell_type": "markdown", - "id": "505ed405", - "metadata": { - "editable": true - }, + "id": "36dbf570", + "metadata": {}, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1394,10 +1187,8 @@ }, { "cell_type": "markdown", - "id": "43281aa2", - "metadata": { - "editable": true - }, + "id": "a8adfc31", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1406,10 +1197,8 @@ }, { "cell_type": "markdown", - "id": "df3930c9", - "metadata": { - "editable": true - }, + "id": "8210151d", + "metadata": {}, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1417,10 +1206,8 @@ }, { "cell_type": "markdown", - "id": "dae6d4e4", - "metadata": { - "editable": true - }, + "id": "06d06096", + "metadata": {}, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1429,20 +1216,16 @@ }, { "cell_type": "markdown", - "id": "3d11a09d", - "metadata": { - "editable": true - }, + "id": "d98b2f5d", + "metadata": {}, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] }, { "cell_type": "markdown", - "id": "4ec0b32d", - "metadata": { - "editable": true - }, + "id": "1b3d21d4", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1451,20 +1234,16 @@ }, { "cell_type": "markdown", - "id": "ed6c17a7", - "metadata": { - "editable": true - }, + "id": "0fa5c281", + "metadata": {}, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] }, { "cell_type": "markdown", - "id": "81f885b2", - "metadata": { - "editable": true - }, + "id": "36a79c2b", + "metadata": {}, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1473,10 +1252,8 @@ }, { "cell_type": "markdown", - "id": "d13b0cde", - "metadata": { - "editable": true - }, + "id": "7400bf9b", + "metadata": {}, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1493,10 +1270,8 @@ }, { "cell_type": "markdown", - "id": "44ddde9d", - "metadata": { - "editable": true - }, + "id": "05dcda06", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1505,20 +1280,16 @@ }, { "cell_type": "markdown", - "id": "fd476a0f", - "metadata": { - "editable": true - }, + "id": "f59f43f1", + "metadata": {}, "source": [ "or consider the system" ] }, { "cell_type": "markdown", - "id": "20588be3", - "metadata": { - "editable": true - }, + "id": "79375e76", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1527,20 +1298,16 @@ }, { "cell_type": "markdown", - "id": "3d610d43", - "metadata": { - "editable": true - }, + "id": "d8104364", + "metadata": {}, "source": [ "instead." ] }, { "cell_type": "markdown", - "id": "64198b50", - "metadata": { - "editable": true - }, + "id": "c014b000", + "metadata": {}, "source": [ "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1548,10 +1315,8 @@ }, { "cell_type": "markdown", - "id": "b4ce70c0", - "metadata": { - "editable": true - }, + "id": "ada74dc2", + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1560,10 +1325,8 @@ }, { "cell_type": "markdown", - "id": "7705277c", - "metadata": { - "editable": true - }, + "id": "183da2ae", + "metadata": {}, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1572,10 +1335,8 @@ }, { "cell_type": "markdown", - "id": "c17e7ac7", - "metadata": { - "editable": true - }, + "id": "edab709c", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1584,10 +1345,8 @@ }, { "cell_type": "markdown", - "id": "2469dda5", - "metadata": { - "editable": true - }, + "id": "ed04ea87", + "metadata": {}, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", @@ -1597,10 +1356,8 @@ }, { "cell_type": "markdown", - "id": "9a0ec4fa", - "metadata": { - "editable": true - }, + "id": "8ca6c92a", + "metadata": {}, "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" @@ -1608,10 +1365,8 @@ }, { "cell_type": "markdown", - "id": "a536e5fe", - "metadata": { - "editable": true - }, + "id": "e8186d4e", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1620,10 +1375,8 @@ }, { "cell_type": "markdown", - "id": "0bb1676f", - "metadata": { - "editable": true - }, + "id": "c0709a75", + "metadata": {}, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1636,10 +1389,8 @@ }, { "cell_type": "markdown", - "id": "aeaba424", - "metadata": { - "editable": true - }, + "id": "e1d261da", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1648,10 +1399,8 @@ }, { "cell_type": "markdown", - "id": "b23ea0a2", - "metadata": { - "editable": true - }, + "id": "dd0a7fcb", + "metadata": {}, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1659,10 +1408,8 @@ }, { "cell_type": "markdown", - "id": "eeccbcb7", - "metadata": { - "editable": true - }, + "id": "a9a717b8", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1671,20 +1418,16 @@ }, { "cell_type": "markdown", - "id": "1c8f9493", - "metadata": { - "editable": true - }, + "id": "b549f75f", + "metadata": {}, "source": [ "which equals" ] }, { "cell_type": "markdown", - "id": "4bbe9780", - "metadata": { - "editable": true - }, + "id": "c4aecd92", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1693,20 +1436,16 @@ }, { "cell_type": "markdown", - "id": "c7856672", - "metadata": { - "editable": true - }, + "id": "8df935ba", + "metadata": {}, "source": [ "or" ] }, { "cell_type": "markdown", - "id": "ff230aff", - "metadata": { - "editable": true - }, + "id": "3e641382", + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1715,20 +1454,16 @@ }, { "cell_type": "markdown", - "id": "e5ddd07d", - "metadata": { - "editable": true - }, + "id": "4478daea", + "metadata": {}, "source": [ "which gives" ] }, { "cell_type": "markdown", - "id": "a9cb9057", - "metadata": { - "editable": true - }, + "id": "a08581d0", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1737,10 +1472,8 @@ }, { "cell_type": "markdown", - "id": "3f8de819", - "metadata": { - "editable": true - }, + "id": "29a102a4", + "metadata": {}, "source": [ "## Revisiting our first homework\n", "\n", @@ -1761,11 +1494,8 @@ { "cell_type": "code", "execution_count": 6, - "id": "b6c82967", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "16f91d0d", + "metadata": {}, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1774,10 +1504,8 @@ }, { "cell_type": "markdown", - "id": "eb65e79c", - "metadata": { - "editable": true - }, + "id": "f165490e", + "metadata": {}, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1785,10 +1513,8 @@ }, { "cell_type": "markdown", - "id": "c9a1e5d8", - "metadata": { - "editable": true - }, + "id": "84498422", + "metadata": {}, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1797,20 +1523,16 @@ }, { "cell_type": "markdown", - "id": "ae4f1a9a", - "metadata": { - "editable": true - }, + "id": "f98e851e", + "metadata": {}, "source": [ "such that" ] }, { "cell_type": "markdown", - "id": "db33f468", - "metadata": { - "editable": true - }, + "id": "ab911929", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -1819,10 +1541,8 @@ }, { "cell_type": "markdown", - "id": "3562d9e9", - "metadata": { - "editable": true - }, + "id": "170e4189", + "metadata": {}, "source": [ "## Gradient descent example\n", "\n", @@ -1833,10 +1553,8 @@ }, { "cell_type": "markdown", - "id": "d07b80c6", - "metadata": { - "editable": true - }, + "id": "1a2651d2", + "metadata": {}, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -1849,20 +1567,16 @@ }, { "cell_type": "markdown", - "id": "3e9beaaa", - "metadata": { - "editable": true - }, + "id": "310ad769", + "metadata": {}, "source": [ "The cost/loss/risk function is given by (" ] }, { "cell_type": "markdown", - "id": "032534e5", - "metadata": { - "editable": true - }, + "id": "a5b0496c", + "metadata": {}, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -1871,20 +1585,16 @@ }, { "cell_type": "markdown", - "id": "c338adc5", - "metadata": { - "editable": true - }, + "id": "c0759118", + "metadata": {}, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] }, { "cell_type": "markdown", - "id": "042cb6a3", - "metadata": { - "editable": true - }, + "id": "c65d80ad", + "metadata": {}, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -1893,10 +1603,8 @@ }, { "cell_type": "markdown", - "id": "52bc4362", - "metadata": { - "editable": true - }, + "id": "30544a22", + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1907,20 +1615,16 @@ }, { "cell_type": "markdown", - "id": "dcd681f0", - "metadata": { - "editable": true - }, + "id": "f7e27f43", + "metadata": {}, "source": [ "where $X$ is the design matrix defined above." ] }, { "cell_type": "markdown", - "id": "650394f1", - "metadata": { - "editable": true - }, + "id": "0c7d8c1d", + "metadata": {}, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -1928,10 +1632,8 @@ }, { "cell_type": "markdown", - "id": "ef4eb0f1", - "metadata": { - "editable": true - }, + "id": "afa82cb3", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1943,20 +1645,16 @@ }, { "cell_type": "markdown", - "id": "61c1c429", - "metadata": { - "editable": true - }, + "id": "837c1e3f", + "metadata": {}, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] }, { "cell_type": "markdown", - "id": "45ec5fb5", - "metadata": { - "editable": true - }, + "id": "89fc16ec", + "metadata": {}, "source": [ "## Simple program\n", "\n", @@ -1965,10 +1663,8 @@ }, { "cell_type": "markdown", - "id": "070576d6", - "metadata": { - "editable": true - }, + "id": "983001de", + "metadata": {}, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -1977,10 +1673,8 @@ }, { "cell_type": "markdown", - "id": "f9832704", - "metadata": { - "editable": true - }, + "id": "915c0c75", + "metadata": {}, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -1992,10 +1686,8 @@ }, { "cell_type": "markdown", - "id": "7dfea6ff", - "metadata": { - "editable": true - }, + "id": "30d20294", + "metadata": {}, "source": [ "## Gradient Descent Example\n", "\n", @@ -2005,11 +1697,8 @@ { "cell_type": "code", "execution_count": 7, - "id": "bb304045", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "c6017dfa", + "metadata": {}, "outputs": [], "source": [ "\n", @@ -2062,10 +1751,8 @@ }, { "cell_type": "markdown", - "id": "e417c95f", - "metadata": { - "editable": true - }, + "id": "dcc6889f", + "metadata": {}, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -2073,11 +1760,8 @@ { "cell_type": "code", "execution_count": 8, - "id": "c01ffb63", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "8acb6aad", + "metadata": {}, "outputs": [], "source": [ "# Importing various packages\n", @@ -2100,10 +1784,8 @@ }, { "cell_type": "markdown", - "id": "99bf3bfa", - "metadata": { - "editable": true - }, + "id": "fcc08db2", + "metadata": {}, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -2112,10 +1794,8 @@ }, { "cell_type": "markdown", - "id": "190d5f89", - "metadata": { - "editable": true - }, + "id": "ad450622", + "metadata": {}, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -2124,20 +1804,16 @@ }, { "cell_type": "markdown", - "id": "e71de050", - "metadata": { - "editable": true - }, + "id": "d86008e8", + "metadata": {}, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] }, { "cell_type": "markdown", - "id": "a7bfa769", - "metadata": { - "editable": true - }, + "id": "8074070e", + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2148,20 +1824,16 @@ }, { "cell_type": "markdown", - "id": "99ec82bf", - "metadata": { - "editable": true - }, + "id": "05356160", + "metadata": {}, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] }, { "cell_type": "markdown", - "id": "0a94d0e3", - "metadata": { - "editable": true - }, + "id": "58c8ba7c", + "metadata": {}, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -2170,10 +1842,8 @@ }, { "cell_type": "markdown", - "id": "b62bd0f2", - "metadata": { - "editable": true - }, + "id": "abbd7c72", + "metadata": {}, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -2181,10 +1851,8 @@ }, { "cell_type": "markdown", - "id": "1597e821", - "metadata": { - "editable": true - }, + "id": "134a8134", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2196,10 +1864,8 @@ }, { "cell_type": "markdown", - "id": "493239f1", - "metadata": { - "editable": true - }, + "id": "b5b89343", + "metadata": {}, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -2210,10 +1876,8 @@ }, { "cell_type": "markdown", - "id": "e46daeca", - "metadata": { - "editable": true - }, + "id": "f79ceb7a", + "metadata": {}, "source": [ "## Program example for gradient descent with Ridge Regression" ] @@ -2221,11 +1885,8 @@ { "cell_type": "code", "execution_count": 9, - "id": "57b4ff9f", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "a9c7f630", + "metadata": {}, "outputs": [], "source": [ "from random import random, seed\n", @@ -2282,10 +1943,8 @@ }, { "cell_type": "markdown", - "id": "ddf6de67", - "metadata": { - "editable": true - }, + "id": "8c6aece2", + "metadata": {}, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -2304,10 +1963,8 @@ }, { "cell_type": "markdown", - "id": "b11c2361", - "metadata": { - "editable": true - }, + "id": "ea1e6f53", + "metadata": {}, "source": [ "## Improving gradient descent with momentum\n", "\n", @@ -2316,13 +1973,59 @@ }, { "cell_type": "code", - "execution_count": 10, - "id": "33b8ff87", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "execution_count": 1, + "id": "d1f044c9", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + ">0 f([0.74724774]) = 0.55838\n", + ">1 f([0.59779819]) = 0.35736\n", + ">2 f([0.47823856]) = 0.22871\n", + ">3 f([0.38259084]) = 0.14638\n", + ">4 f([0.30607268]) = 0.09368\n", + ">5 f([0.24485814]) = 0.05996\n", + ">6 f([0.19588651]) = 0.03837\n", + ">7 f([0.15670921]) = 0.02456\n", + ">8 f([0.12536737]) = 0.01572\n", + ">9 f([0.10029389]) = 0.01006\n", + ">10 f([0.08023512]) = 0.00644\n", + ">11 f([0.06418809]) = 0.00412\n", + ">12 f([0.05135047]) = 0.00264\n", + ">13 f([0.04108038]) = 0.00169\n", + ">14 f([0.0328643]) = 0.00108\n", + ">15 f([0.02629144]) = 0.00069\n", + ">16 f([0.02103315]) = 0.00044\n", + ">17 f([0.01682652]) = 0.00028\n", + ">18 f([0.01346122]) = 0.00018\n", + ">19 f([0.01076897]) = 0.00012\n", + ">20 f([0.00861518]) = 0.00007\n", + ">21 f([0.00689214]) = 0.00005\n", + ">22 f([0.00551372]) = 0.00003\n", + ">23 f([0.00441097]) = 0.00002\n", + ">24 f([0.00352878]) = 0.00001\n", + ">25 f([0.00282302]) = 0.00001\n", + ">26 f([0.00225842]) = 0.00001\n", + ">27 f([0.00180673]) = 0.00000\n", + ">28 f([0.00144539]) = 0.00000\n", + ">29 f([0.00115631]) = 0.00000\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtNUlEQVR4nO3dd3wUdf7H8ddnNz0hhJDQEtKooqBAaCqIHcuBFREbpyfq2fXuxDvP7u/0zn7qKWA5PQW7ouKhNAENJYDSISFACiWBhAAhPd/fH7vxVkzIArs7u5vP8/HgQXZmsvNmEt6ZzMx3RowxKKWUCnw2qwMopZTyDC10pZQKElroSikVJLTQlVIqSGihK6VUkAixasUJCQkmLS3NqtUrpVRAWr58+W5jTGJT8ywr9LS0NLKzs61avVJKBSQR2dbcPD3kopRSQUILXSmlgoQWulJKBQktdKWUChJa6EopFSRaLHQReUNEikVkTTPzRUReFJFcEVklIgM8H1MppVRL3NlDfwsYdZj55wE9nH8mAv869lhKKaWOVIuFboxZAJQeZpExwNvGYTEQJyKdPRXwUD8V7OXJrzegt/1VSgWahgbDE1+tY932fV55f08cQ08CClxeFzqn/YqITBSRbBHJLikpOaqVrSrcy6vfbWZVYflRfb5SSlklK28PUxZuYeMu/y10txljJhtjMo0xmYmJTY5cbdGY/klEhtp5b0m+h9MppZR3vbckn7ioUM47wTsHMTxR6EVAV5fXyc5pXhEbEcqYk7ow46ftlFfWems1SinlUcX7q5i1dieXDUgmItTulXV4otBnANc6r3YZCpQbY3Z44H2bddWQVCpr6/lspdd+biillEd9mF1IXYPhyiEpXluHO5ctTgOygF4iUigiN4jIzSJys3ORmUAekAtMAX7vtbROfZPb0i+5Le8u2aYnR5VSfq++wfDeknxO7taebokxXltPi3dbNMZc2cJ8A9zqsURuGj84hUmfrGb5tjIy0+J9vXqllHLbgpwSivZWcv/5vb26noAdKfqbE7vQJjyEd/XkqFLKz727OJ+EmDDO6dPJq+sJ2EKPDg/h4gFJfLV6B2UVNVbHUUqpJm3fW8ncDbsYm9mVsBDvVm7AFjrA+CEp1NQ18PGKQqujKKVUk95fVoABrhzsvZOhjQK60Ht3imVgajveXZKvJ0eVUn6nrr6B6cvyGdEjka7xUV5fX0AXOsBVQ1LYsruCrM17rI6ilFK/MGdDMbv2VXOVFy9VdBXwhX5+387ERYXqyVGllN95d0k+nWIjOKN3B5+sL+ALPSLUzqUDkpm1dicl+6utjqOUUgDk7znIwpwSrhjUlRC7b6o24AsdHCdH6xoMH2QXtLywUkr5wLRl+QgwbnDXFpf1lKAo9G6JMQzLaM+0pfk0NOjJUaWUtWrqGvgwu4Azj+tI57aRPltvUBQ6OPbSC8sqWZBzdLflVUopT/lm3U52H6hhvI9OhjYKmkI/9/hOtI8O05OjSinLvbs4n+R2kYzocXS3CT9aQVPoYSE2xg7qypz1u9hRXml1HKVUK5VbfICsvD1cOTgFu018uu6gKXSAKwelYIDpS/XkqFLKGtOW5hNiE8Zm+u5kaKOgKvSU9lEM75HI+8sKqKtvsDqOUqqVqaqt5+MVhZx7fCcS24T7fP1BVejgGDm6c18VczcUWx1FKdXKzFy9g70Ha302MvRQQVfoZ/buQMfYcD05qpTyuXeX5JOREM2wbu0tWX/QFXqI3ca4QSksyCmhoPSg1XGUUq3Ehp37WL6tjPFDUhDx7cnQRkFX6OAYmSU4Tk4opZQvvLckn7AQG5cOSLYsQ1AWeue2kZzRuyMfZBdQU6cnR5VS3lVRXccnK4q4oG9n2kWHWZYjKAsd4KqhKew+UMM363ZaHUUpFeS++Gk7B6rrLDsZ2ihoC31Ej0SS4iJ5T0+OKqW87L2l+fTsGMPA1HaW5gjaQrfbhPFDUvhh8x42lxywOo5SKkitKtzLqsJyrhqSatnJ0EZBW+gAl2cmE2ITpuleulLKS95bkk9kqJ2LByRZHSW4C71DmwjOPb4TH60opKq23uo4Sqkgs6+qlhk/bWf0iV2IjQi1Ok5wFzo4bqu792AtX6/ZYXUUpVSQ+XxlEQdr6n1+m9zmBH2hD8toT3pCNO8u1sMuSinPMcbw7pJ8TkiKpV9yW6vjAK2g0G02YfzgFLK3lbFx536r4yilgsSK/DI27NzvFydDGwV9oQNcOjCZMLuN95ZsszqKUipIvLskn5jwEEaf2MXqKD9rFYUeHx3G+X078cmKIg7W1FkdRykV4PYerOHLVTu4qH8XosNDrI7zs1ZR6ABXDU1lf3UdX/y03eooSqkA99HyQmrqGhg/ONXqKL/Qago9M7UdPTvG6G11lVLHxBjDe0vzGZASR58usVbH+QW3Cl1ERonIRhHJFZFJTcxPEZF5IrJSRFaJyPmej3psRBwnR1cVlrOqcK/VcZRSASpr8x7ySioYP8S/9s7BjUIXETvwMnAe0Ae4UkT6HLLYA8AHxpj+wDjgFU8H9YRLBiYTEx7C1IVbrI6ilApQUxdtoX10GBf262x1lF9xZw99MJBrjMkzxtQA04ExhyxjgMbfPdoCfnmgOjYilCsHd+Wr1TsoLNOHXyiljkzOrv3M3VDMtcPSiAi1Wx3nV9wp9CSgwOV1oXOaq4eBq0WkEJgJ3N7UG4nIRBHJFpHskpKSo4h77H57SjoCvPn9VkvWr5QKXFMXbiE8xMY1w/zvcAt47qTolcBbxphk4HzgHRH51XsbYyYbYzKNMZmJiYkeWvWR6RIXyYX9OjN9aT7llbWWZFBKBZ7i/VV8urKIyzOTibfwIRaH406hFwFdXV4nO6e5ugH4AMAYkwVEAAmeCOgNvxueQUVNvT6iTinltrd/2EZtQwM3nJphdZRmuVPoy4AeIpIuImE4TnrOOGSZfOBMABE5DkehW3NMxQ0nJLXllO7tefP7LfqIOqVUiw7W1PHO4m2c06cj6QnRVsdpVouFboypA24DZgHrcVzNslZEHhWR0c7F7gVuFJGfgGnABGOM8VZoT7hxeAa79lXrQCOlVIs+zC6kvLKWiSP8d+8cwK0xq8aYmThOdrpOe9Dl43XAKZ6N5l2n9UykV8c2TFmYxyUDkvzm5jpKKf9S32CYuiiPASlxDEyNtzrOYbWakaKHEhF+NzydDTv3szBnt9VxlFJ+atbanRSUVvr93jm04kIHGH1SFzq0CWfKwjyroyil/JAxhtcW5JHaPoqz+3SyOk6LWnWhh4fYmXBKGgtzdrNu+z6r4yil/Ez2tjJ+KtjL705Nx27z/8OyrbrQAa4anEpUmJ2pi3QvXSn1S1MW5NEuKpTLBnZteWE/0OoLvW1UKFcM6sqMH7ezo7zS6jhKKT+RV3KAb9fv4pqhqUSG+d8w/6a0+kIHuP6UdBqM4a0ftlodRSnlJ15ftIVQu41rhqVZHcVtWuhA1/gozu/bmfcW57O/Sm8HoFRrt+dANR8tL+TSAUkktgm3Oo7btNCdJo7IYH91He8vK2h5YaVUUHtn8Taq6/x7mH9TtNCd+iXHMTg9nje/30ptvd4OQKnWqqq2nreztnFm7w507xBjdZwjooXuYuLwDIr2VjJz9Q6royilLPLxikJKK2q4MQAGEh1KC93FGb07kJEYzZSFefj5rWiUUl7Q0GCYunAL/ZLbMiTdv4f5N0UL3YXNJtw4PIM1RfvIyttjdRyllI/NXr+LLbsruHF4RkDe30kL/RAX908iISaMKQt0oJFSrc2UhXkkxUVy3gn+P8y/KVroh4gItXPtsDTmbSxh0679VsdRSvnIivwylm0t44ZT0wmxB2Y1BmZqL7t6aCoRoTam6k27lGo1pi7MIzYihLGDAmOYf1O00JsQHx3G5QO78tnK7RTvq7I6jlLKy7btqeC/a3Zy1dBUYsLdekyEX9JCb8YNp6ZT29DAv7O2Wh1FKeVlbyzagt0mTDg5zeoox0QLvRlpCdGc26cT/1mcT0V1ndVxlFJeUlZRwwfZhYw5KYmOsRFWxzkmWuiHceOIDMora/kwW28HoFSwenfJNipr67lxeOANJDqUFvphDExtx8DUdrz+/Rbq9HYASgWdqtp63vphm+MZw53aWB3nmGmht+DG4RkUlFYya+0uq6MopTzs8x+L2H2gOiCeF+oOLfQWnN2nI2nto5i8YLPeDkCpINLQYJiycAt9Osdycrf2VsfxCC30Fthtwg3DM/ipsJxlW8usjqOU8pD5m4rJLT7AxBGBOcy/KVrobrhsQDLtokKZrLcDUCpoTF6QR+e2EVzQr7PVUTxGC90NkWF2rhmWxpwNu9hccsDqOEqpY7S6sJzFeaVcf0o6oQE6zL8pwfMv8bJrh6USZrfxyrzNVkdRSh2jl+fl0iY8hCsGB+4w/6ZoobspISacq4em8unKQt1LVyqArSkq579rd3L9qenERoRaHcejtNCPwC0juxEeYueF2TlWR1FKHaVnv91E28hQbhiebnUUj9NCPwIJMeFMOCWNL1ZtZ+NOvbWuUoFmRX4ZczcUM3FERtDtnYMW+hGbODyD6LAQnvt2k9VRlFJH6NlvNtE+Oizgb8LVHLcKXURGichGEckVkUnNLDNWRNaJyFoRec+zMf1Hu+gwbjg1nf+u3cmaonKr4yil3LQkbw+Lcndzy8huRAfwLXIPp8VCFxE78DJwHtAHuFJE+hyyTA/gfuAUY8zxwF2ej+o/bhieTtvIUN1LVypAGGN45ptNdGjjuLghWLmzhz4YyDXG5BljaoDpwJhDlrkReNkYUwZgjCn2bEz/EhsRysQRGczZUMyKfB09qpS/W5S7m6VbS7n19O5EhNqtjuM17hR6EuB6/9hC5zRXPYGeIvK9iCwWkVFNvZGITBSRbBHJLikpObrEfmLCyWnER4fpXrpSfq5x77xL2wjGBdl154fy1EnREKAHMBK4EpgiInGHLmSMmWyMyTTGZCYmJnpo1daIDg/hltO6sTBnN0vy9lgdRynVjLkbivmxYC+3n9mD8JDg3TsH9wq9CHD9sZbsnOaqEJhhjKk1xmwBNuEo+KB29dBUOrQJ55lvN+mdGJXyQ8YYnv12EynxUVw2MNnqOF7nTqEvA3qISLqIhAHjgBmHLPMZjr1zRCQBxyGYoL+TVWSYnVtP787SLaV8n6t76Ur5m1lrd7J2+z7uPLNHUN2zpTkt/guNMXXAbcAsYD3wgTFmrYg8KiKjnYvNAvaIyDpgHvBHY0yraLhxg7vSpW0ET3+zUffSlfIj9Q2OvfOMxGgu6n/oab/g5NaPLGPMTGNMT2NMN2PME85pDxpjZjg/NsaYe4wxfYwxfY0x070Z2p+Eh9i5/cwe/Fiwl3kbg/riHqUCypertrNp1wHuPqsndltw3O+8JcH/O4gPXDYwmZT4KJ75Ro+lK+UP6uobeH52Dr07teGCvsFzv/OWaKF7QKjdxh1n9mDt9n3MWrvT6jhKtXqLn3uTq6c/x8MJ+7C1kr1z0EL3mItO6kJGYjTPfruJ+gbdS1fKKrULv2fYnyZy/fIZDJk4FrKyrI7kM1roHhJit3HXWT3ZtOsAX67abnUcpVqtrc++gt00IIDU1MD8+VZH8hktdA+6sG9nenVswwuzc6irb7A6jlKtTlV1LSGLFmIAY7dDWBiMHGl1LJ/RQvcgm024++ye5O2u4LMfdS9dKV9b9tgLpO8uoOC2PyCPPQZz5sCwYVbH8hmx6qqMzMxMk52dbcm6vckYw29eWkR5ZS1z7x3ZKgYzKOUPKkvLOZCWwd72neieuwqxB+cwfxFZbozJbGqeto2HiQj3nt2LgtJKPswutDqOUq3G+rv+TOL+UmqeeTZoy7wlWuheMLJXIv1T4vjn3ByqauutjqNU0KvYtJnjp03hhyHncPwl51odxzJa6F7QuJe+o7yK6UvzrY6jVNDbftPtNGCj7QvPWB3FUlroXnJK9/YMSY/n5fmbqazRvXSlvOXAnPn0mP81315wDccPOcHqOJbSQvcSEeHec3pRsr+adxZvtTqOUsGpoYEDt9zG9jYJdH/mUavTWE4L3YsGp8czvEcCr36Xx4HqOqvjKBV0Kqa8Qaectfz3qjvp06313LOlOVroXnbP2T0prajh3z9stTqKUsFl/37M/fezoksvTn3oDqvT+AUtdC/rn9KOM3t34LXvNlNeWWt1HKWCRsWjjxNTtpv5t/yZnp1irY7jF7TQfeDus3uyr6qOqQuD/iFOSvnGli2Ev/A8nx5/OhfddInVafyGFroPnJDUlgv7dWbygjwKyw5aHUepgLf/rnupQVh/x/1kJMZYHcdvaKH7yP3nH4cIPPHVequjKBXQzHff0WbGp7x5ylgmjh9hdRy/ooXuI0lxkdx2ene+XrOThTklVsdRKjDV17PvltsoapNI7AOTSIgJtzqRX9FC96HfDc8gtX0UD89YS02d3l5XqSNV8/qbtF2/hncuupkrR/a2Oo7f0UL3oYhQOw9e2IfNJRV6GaNSR2r/fmon3c/yLr0Z+dAdhOidTH9Ft4iPnXlcR87o3YHnZ2+ieF+V1XGUChjlf32E6LLdzLv5zwztlmB1HL+khW6BBy/sQ2294cmvN1gdRanAsGULUS+/yOd9z+TqOy63Oo3f0kK3QFpCNDeOSOeTlUUs21pqdRyl/F7xzXdQKzbK//oIndpGWB3Hb2mhW+TW07vTuW0ED32+lvoGa54apVQgqJk7lw7ffMn008dzxcVDrY7j17TQLRIVFsJfLjiOdTv28Z7eM12pptXXU36T4zLF7n9/iPCQ1vkkIndpoVvogr6dGZbRnqdnbaS0osbqOEr5nfJXJpOYu56vrr6LESemWh3H72mhW0hEeGTM8RyoruPpbzZaHUcp/7JvH/LAAyxPPo7zHrvT6jQBQQvdYj07tuG6YWlMW5rP6sJyq+Mo5Te2//EBYveVsuG+R+naPtrqOAFBC90P3HV2D9pHh/HgjDU06AlSpajblEPC668yc8A5XDLxYqvjBAwtdD8QGxHKfaN6szJ/L5+sLLI6jlKWK7rxNmptdiL/8SSRYXoi1F1uFbqIjBKRjSKSKyKTDrPcpSJiRCTTcxFbh0sHJNM/JY4nv17Pvip9EIZqvcpnziJ1wTfMvOA6Rp5+ktVxAkqLhS4iduBl4DygD3CliPRpYrk2wJ3AEk+HbA1sNuHR0Sewp6KGF2bnWB1HKWvU11Px+zspbNuBAc8/hohYnSiguLOHPhjINcbkGWNqgOnAmCaWewx4CtAblBylvsltGTcohbd+2MqmXfutjqOUz+U//RJdtm1k2S330S010eo4AcedQk8CClxeFzqn/UxEBgBdjTFfHe6NRGSiiGSLSHZJid4TvCl/PLcXMeEhPPT5WozRE6Sq9Wgo20vs4w/zY+oJnPXgbVbHCUjHfFJURGzAs8C9LS1rjJlsjMk0xmQmJupP36bER4fxh3N7kZW3h5mrd1odRymf2XjHJGIPlFP2xN9pExlmdZyA5E6hFwFdXV4nO6c1agOcAMwXka3AUGCGnhg9euMHp9CncyyPf7WOgzV1VsdRyuv2r15P92mv892w8xg5fpTVcQKWO4W+DOghIukiEgaMA2Y0zjTGlBtjEowxacaYNGAxMNoYk+2VxK2A3SY8OuZ4dpRX8cq8zVbHUcrrCm+8jRpbCF1eflZPhB6DFgvdGFMH3AbMAtYDHxhj1orIoyIy2tsBW6vMtHgu7p/E5AV5bN1dYXUcpbxm24dfcNySufwwdiK9+veyOk5AE6tOvGVmZprsbN2JP5zifVWc/vR8hmS0540Jg6yOo5THmbo68tN6E1pxgKjNm4iLj7U6kt8TkeXGmCYPaetIUT/WITaCu87qydwNxcxZv8vqOEp5VlYWO86+kNSizeT96SEtcw/QQvdz152cRrfEaB7+Yi0V1XqCVAWJrCzMGWfQef4sGkQYNryf1YmCgha6nwsLsfG3S/pRWFbJ41+tszqOUh5h5s2DqioEx22k7QsXWB0pKGihB4DB6fHcNKIb05YW8O06PfSiAt/6/N0IYESQ8HAYOdLqSEFBCz1A3HN2T/p0jmXSx6so2V9tdRyljtqOH9fR9a3X2JrUDfPoozBnDgwbZnWsoKCFHiDCQmw8P+4k9lfXcd/Hq/S2ACog1dfUsu/SKwAI+/ILbA88oGXuQVroAaRnxzbcf15v5m4o1gdLq4C0cuK99Mpbw9q/PkWXk46zOk7Q0UIPMNcNS2N4jwQe/3I9eSUHrI6jlNu2fP4N/d9+mcWnnM+QP99qdZygpIUeYGw24R+XnUhYiI27P/iJ2voGqyMp1aKq3aVE/vZadsR1pOeH/9bh/V6ihR6AOrWN4P8u7stPBXt5aW6u1XGUalHOZdeSsLeE4n9NJb5zgtVxgpYWeoC6oF9nLumfxEvzclmRX2Z1HKWatfHpf9H3u6/47srfM+CK862OE9S00APYw2OOp1NsBHe//6OOIlV+ad/aDSQ/8AdWpffj5KnPWB0n6GmhB7DYiFCeHXsi+aUHdRSp8jumpoY9F42lHiFs2rtE6kMrvE4LPcANyWivo0iVX9p4259Iz11N1p+eoPeQE6yO0ypooQcBHUWq/E3JzNn0mPpP5g09j7MevsPqOK2GFnoQcB1FOklHkSqLNZSWwTVXUxTXkR7T38Ru00sUfUULPUj07NiGSaN6M2dDMdOWFlgdR7VWxpB3+bW0Kyth07Ovkpza0epErYoWehCZcHIap3ZP4LEv1+koUmWJohcn033ul3x56c2ced1vrI7T6mihBxGbTXj6ch1FqqxRvX4j7e67m+Vp/Rjx+tM6GtQCWuhBplPbCJ64+AQdRap8q7aW3RddTi02qt98i/jYSKsTtUpa6EHown5dfh5FulJHkSofKLjzPpI2rea/dzzCySP7Wx2n1dJCD1I6ilT5yoFv5pL06vN8PWgUox+/y+o4rZoWepBqHEW6rfQgj3+13uo4KkiZ0lJqxo8nP64zKe9MJTLMbnWkVk0LPYgNyWjPxBEZTFuaz4fZeimj8jBj2Hr5tbQp283Sx1/k+F5JVidq9bTQg9wfzunFqd0T+POnq8navMfqOCqIrHvyRdLnfsWMS2/m0psvsTqOQgs96IXabbx81QDS2kdz83+Ws1mvT1cekLNoBWkPTWJV9/6c9+9ndDSon9BCbwXaRobyxoRBhNiE699aRmlFjdWRVADb8/VsYi84FyM2Os/4gKjIcKsjKSct9Faia3wUU67LZGd5FRPfzqaqtt7qSCoAVc5fSNyFo+i4bzeRpo7EvSVWR1IutNBbkQEp7Xh27ElkbyvjPr2JlzpC9Q2G3Nv/hL3BsTNga6iH+fOtDaV+QQu9lbmgX2f+eG4vPv9xO8/NzrE6jgogM/7wFH3XLKbBZgO7HcLCYORIq2MpF24VuoiMEpGNIpIrIpOamH+PiKwTkVUiMkdEUj0fVXnK70d2Y2xmMi/OyeGTFYVWx1EBYPZz/+bCFx4gr98QbHPnwmOPwZw5MGyY1dGUi5CWFhARO/AycDZQCCwTkRnGGNdnnq0EMo0xB0XkFuDvwBXeCKyOnYjw+EV9KSit5L6PV5EUF8mQjPZWx1J+atnH3zLsvpvZkZRB6nezIK4tnHaa1bFUE9zZQx8M5Bpj8owxNcB0YIzrAsaYecaYg86Xi4Fkz8ZUnhYWYuPVqweSEh/FTf9ZzpbdFVZHUn4oN+tH0q8by4E2cSQsmIM9rq3VkdRhuFPoSYDrMMNC57Tm3AB83dQMEZkoItkikl1SomfHrdY2KpQ3JwzGJsJv31xKmV7OqFyUbM4n/DcXEtpQj23WLKLSulodSbXAoydFReRqIBP4R1PzjTGTjTGZxpjMxMRET65aHaWU9lFMuXYg28uruOmd5VTX6eWMCg6W7qXsjFEklO9mz/SPSczsZ3Uk5QZ3Cr0IcP3RnOyc9gsichbwF2C0MUafVBxABqbG84/L+rF0aymTPl6tlzO2cvXVNWw+/QK6FWxk0/OTyRh9ttWRlJvcKfRlQA8RSReRMGAcMMN1ARHpD7yGo8yLPR9TeduYk5K45+yefLqyiBfn6IMxWi1jWHPhFfRd9QNZf3icE2+91upE6gi0eJWLMaZORG4DZgF24A1jzFoReRTINsbMwHGIJQb40PnYqXxjzGgv5lZecPsZ3dm6p4LnZm8iLSGKMSfp3fNamzXX38GJsz9j3hW3MPKpX12hrPxci4UOYIyZCcw8ZNqDLh+f5eFcygIiwt8u6UthWSV//HAVXeIiGZQWb3Us5SObHv47J7z1EvNHjGH4f/6pzwQNQDpSVP1CeIid164eSFK7SCa+nc1WvZyxVSh64z26PzKJxcefTObM6YSE6IMqApEWuvqVdtFhvDFhEAb07oytQNmsOSTcNIG1XXuT+s0MYqIjrI6kjpIWumpSekI0k6/JpHBvJWNfy2L73kqrIykvKPo+m5CLL2J7bAfsX35J5y46YjiQaaGrZg1Oj+ft6wezq7yKS//1Azm79lsdSXnQ1tffI/bMkdQhVH4xkz79MqyOpI6RFro6rKEZ7Xn/pmHUNRguezWL5dvKrI6kPGDd5HdJ+d3VxFRX0Lahhj6i50qCgRa6alGfLrF8csvJtIsK5aqpi5m7YZfVkdQxmP/l93S582YEgwC2ujq9r3mQ0EJXbukaH8VHt5xMjw5tuPHt5Xy8XG+7G4hmvv45fceeT4gA4eF6X/Mgo4Wu3JYQE860iUMZmhHPvR/+xOQFm62OpNxkjOHTJ6Yw8pYrqI9pQ+jybGTePL2veZBxa2CRUo1iwkN4Y8Ig7vngJ/5v5gZ2H6hh0qje2PSp736rvsHw+R2PMfqVR9iZ1otOi+YQ0qWzY6YWeVDRQldHLDzEzj/H9SchOozJC/LYfaCapy7tR6hdf+HzN1U1dXwz/nYu+fhVtgw4hbR5M5HYWKtjKS/RQldHxWYTHh59PAkx4Tzz7SbKKmp4+aoBRIXpt5S/2FdRxZLzxzN6wafknnsx3WdMdxwvV0FLd6nUURMRbj+zB3+7pC/fbSrhqqlL9CEZfqK4uIw1w87h7AWfkjPh93T/+mMt81ZAC10dsysHp/DKVQNZu30fl+uoUssV5Bawc8gIhq5eRM4D/0ePN18GvdFWq6CFrjxi1AmddFSp1bKyKPn9XdgGDaJ3wUa2vfIGPR673+pUyoe00JXHuI4qvfw1HVXqU1lZNIw8nYR/vUCXvbsof+Ip0m+ZYHUq5WNa6MqjGkeVxkU6RpV+8dN2qyMFPdPQQNGkh5CaagTAZiexocrqWMoCWujK4xpHlfbuFMvt01Zy8zvLKd6nBeMNO3Lz+XHgSJIWfIsRG8ZuR8J15GdrpYWuvCIhJpyPbh7GpPN6M29jMWc9+x0fZBfoA6g9pKHBMPv5twnpfxJ9Vi9myZ0PwoIFiI78bNXEqv9gmZmZJjs725J1K9/KKznApI9Xs3RrKad2T+Bvl/Sla3yU1bECVl7hHtZNuJUL57xPYZd0QqZPo9PwIVbHUj4iIsuNMZlNzdM9dOV1GYkxTJ84lMcuOoGV+WWc89wC3vx+C/UNurd+JOrqG5j+1n+pHjiYC+e8T+7l15GUs0bLXP1MC135hM0mXDM0lW/uOY0hGfE88sU6Ln/1B3KL9fJGd+TN+IZl/U/j4t/9huTKUva+/zHdP3gLidLfdNT/aKErn0qKi+TNCYN47ooT2bK7gvNfWMRLc3OorW+wOppfqqqt57O/vkjqRecxbPUiwkwDbd59m7ixl1gdTfkhLXTlcyLCxf2T+fae0zjn+I48/c0mfvPPRawuLLc6ml9ZuXYbn4y6lgufuBubcfzAExFYs8biZMpfaaEryyTEhPPS+AFMvmYgpRU1XPTK9zz59QaqauutjmapisoaPrvzCZKG9Gf83PfYd+pIJCJCH0ahWqS3xlOWO+f4TgzJaM/fZq7n1e82M2vtTp68pC9DMlrRE+izsmD+fDbVhNAwZSoXFW2ioNeJVE75gvjhp/w8n5Ej9ZJE1Sy9bFH5le9zdzPpk1UUlFZycf8krh6awoCUdo5DDUHK/PAD5owzkGrHSM/S6Dj2PvEUGXfcqDfVUr9yuMsWdQ9d+ZVTuicw664RPD87h3cXb+PTlUV07xDDFZlduWRAEu1jwq2O6DHF+6pY+M4XDHjsj6Q5y7xBhDb33kn8nROtjqcCkO6hK79VUV3HV6t2MH1ZPivy9xJqF87u05GxmV0Z3iMRewA+9q5u0fds/egrftyxn7Qf5pJZuI6D4VFE1NUgGCQsTEd6qsM63B66FroKCDm79vP+sgI+WVlEaUUNXdpGcHlmVy7PTCa5nf9fi71tTwXLXn6HMY/cTkhDHQIcaJdA1R/+RMIdt8Dq1XqMXLlFC10Fjeq6emavK+b97AIW5pQAcGr3BMYNSuGsPh0ID7FbnPB/qmrryX7rE2TKZMK2FzGgaAN2HP/fjM2GPPoo/OUvFqdUgUaPoaugER5i54J+nbmgX2cKyw7yYXYhH2YXcOt7K4iPDuOS/klcMagrPTq2sSZgVhYl0z5m3c79hC9bwilbf0IAA1SfeTb27xdCba3j0MoZZ1iTUQUtt/bQRWQU8AJgB6YaY548ZH448DYwENgDXGGM2Xq499Q9dOUp9Q2GRbm7eX9ZPt+u20VtvaF3pzZ06xBD13ZRpMRH0TU+kq7tougSF0lYyDEMv3BePlgXF8++op1s6zuIsoIddPjqM2zFxfRevwwxBgGqwyIIq6ly3KPcbofHHnMcUtFDK+oYHNMhFxGxA5uAs4FCYBlwpTFmncsyvwf6GWNuFpFxwMXGmCsO975a6Mob9hyo5pMVRSzIKaGg9CCFZZXUudwEzCbQuW0kye0i6Rr/y7LvGh9FYkw4tox0GrZtw4hQHxtHTVQ0u1K6syu6HUPmf440NPy81y3OPzhf43xtbHbkxt/B229DTY1jQJCe7FQecKyFPgx42BhzrvP1/QDGmL+5LDPLuUyWiIQAO4FEc5g310JXvlDfYNi5r4qC0oPklx6ksPQgBWWVP78u3l/9i+UXvTKBpP27f/U+h15P01jmriWOzeb4Y8z/Chx0j1x51LEeQ08CClxeFwKH3q/z52WMMXUiUg60B37xP0NEJgITAVJSUtwKr9SxsNuEpLhIkuIiGdrEyNOq2noKyyopKHOUfZenHN+yLV0Q2VjgEhrqKPD6ekeJP/887NnzywLXIlc+4tOTosaYycBkcOyh+3LdSjUlItRO9w4xdO8Q45iQmgrbtjX/CSIQEgJ33w1xcf+7r4ruhSs/4E6hFwFdXV4nO6c1tUyh85BLWxwnR5UKLFu3Qlqao9RFoF07aNMG+veH88779d53Iy1y5QfcKfRlQA8RScdR3OOA8YcsMwO4DsgCLgPmHu74uVJ+betWqxModVRaLHTnMfHbgFk4Llt8wxizVkQeBbKNMTOA14F3RCQXKMVR+koppXzIrWPoxpiZwMxDpj3o8nEVcLlnoymllDoS+oALpZQKElroSikVJLTQlVIqSGihK6VUkLDs9rkiUgIcZgTHYSVwyChUP6G5jozmOnL+mk1zHZljyZVqjElsaoZlhX4sRCS7uXsZWElzHRnNdeT8NZvmOjLeyqWHXJRSKkhooSulVJAI1EKfbHWAZmiuI6O5jpy/ZtNcR8YruQLyGLpSSqlfC9Q9dKWUUofQQldKqSDht4UuIpeLyFoRaRCRZi/vEZFRIrJRRHJFZJLL9HQRWeKc/r6IhHkoV7yIfCsiOc6/2zWxzOki8qPLnyoRucg57y0R2eIy7yRf5XIuV++y7hku063cXieJSJbz671KRK5wmefR7dXc94vL/HDnvz/XuT3SXObd75y+UUTOPZYcR5HrHhFZ59w+c0Qk1WVek19TH+WaICIlLuv/ncu865xf9xwRuc7HuZ5zybRJRPa6zPPm9npDRIpFZE0z80VEXnTmXiUiA1zmHfv2Msb45R/gOKAXMB/IbGYZO7AZyADCgJ+APs55HwDjnB+/CtzioVx/ByY5P54EPNXC8vE4bikc5Xz9FnCZF7aXW7mAA81Mt2x7AT2BHs6PuwA7gDhPb6/Dfb+4LPN74FXnx+OA950f93EuHw6kO9/H7sNcp7t8D93SmOtwX1Mf5ZoAvNTE58YDec6/2zk/buerXIcsfzuO2357dXs533sEMABY08z884GvcTzFcCiwxJPby2/30I0x640xG1tYbDCQa4zJM8bUANOBMSIiwBnAR87l/g1c5KFoY5zv5+77XgZ8bYw56KH1N+dIc/3M6u1ljNlkjMlxfrwdKAaaHAl3jJr8fjlM3o+AM53bZwww3RhTbYzZAuQ6388nuYwx81y+hxbjeHKYt7mzvZpzLvCtMabUGFMGfAuMsijXlcA0D637sIwxC3DswDVnDPC2cVgMxIlIZzy0vfy20N3U1AOsk3A8oHqvMabukOme0NEYs8P58U6gYwvLj+PX30xPOH/dek5Ewn2cK0JEskVkceNhIPxoe4nIYBx7XZtdJntqezX3/dLkMs7t0fjAc3c+15u5XN2AYy+vUVNfU1/mutT59flIRBofV+kX28t5aCodmOsy2Vvbyx3NZffI9vLpQ6IPJSKzgU5NzPqLMeZzX+dpdLhcri+MMUZEmr3u0/mTty+Opz01uh9HsYXhuBb1PuBRH+ZKNcYUiUgGMFdEVuMoraPm4e31DnCdMabBOfmot1cwEpGrgUzgNJfJv/qaGmM2N/0OHvcFMM0YUy0iN+H47eYMH63bHeOAj4wx9S7TrNxeXmVpoRtjzjrGt2juAdZ7cPwqE+Lcy2rqwdZHlUtEdolIZ2PMDmcBFR/mrcYCnxpjal3eu3FvtVpE3gT+4Mtcxpgi5995IjIf6A98jMXbS0Riga9w/DBf7PLeR729mnAsDzx353O9mQsROQvHD8nTjDHVjdOb+Zp6oqBazGWMcX0Y/FQc50waP3fkIZ873wOZ3MrlYhxwq+sEL24vdzSX3SPbK9APufz8AGtxXJUxDphhHGcZ5uE4fg2OB1h7ao+/8YHY7rzvr47dOUut8bj1RUCTZ8O9kUtE2jUeshCRBOAUYJ3V28v5tfsUx7HFjw6Z58nt1eT3y2Hyuj7wfAYwThxXwaQDPYClx5DliHKJSH/gNWC0MabYZXqTX1Mf5urs8nI0sN758SzgHGe+dsA5/PI3Va/mcmbrjeMEY5bLNG9uL3fMAK51Xu0yFCh37rR4Znt562zvsf4BLsZxHKka2AXMck7vAsx0We58YBOOn7B/cZmegeM/XC7wIRDuoVztgTlADjAbiHdOzwSmuiyXhuOnru2Qz58LrMZRTP8BYnyVCzjZue6fnH/f4A/bC7gaqAV+dPlzkje2V1PfLzgO4Yx2fhzh/PfnOrdHhsvn/sX5eRuB8zz8/d5SrtnO/weN22dGS19TH+X6G7DWuf55QG+Xz73euR1zgd/6Mpfz9cPAk4d8nre31zQcV2nV4uivG4CbgZud8wV42Zl7NS5X8Hlie+nQf6WUChKBfshFKaWUkxa6UkoFCS10pZQKElroSikVJLTQlVIqSGihK6VUkNBCV0qpIPH/s/Ss5hTLXcMAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "from numpy import asarray\n", "from numpy import arange\n", @@ -2383,23 +2086,67 @@ }, { "cell_type": "markdown", - "id": "2b484919", - "metadata": { - "editable": true - }, + "id": "137dfd69", + "metadata": {}, "source": [ "## Same code but now with momentum gradient descent" ] }, { "cell_type": "code", - "execution_count": 11, - "id": "66b07a7b", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "execution_count": 2, + "id": "70ea5fd5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + ">0 f([0.74724774]) = 0.55838\n", + ">1 f([0.54175461]) = 0.29350\n", + ">2 f([0.37175575]) = 0.13820\n", + ">3 f([0.24640494]) = 0.06072\n", + ">4 f([0.15951871]) = 0.02545\n", + ">5 f([0.1015491]) = 0.01031\n", + ">6 f([0.0638484]) = 0.00408\n", + ">7 f([0.03976851]) = 0.00158\n", + ">8 f([0.02459084]) = 0.00060\n", + ">9 f([0.01511937]) = 0.00023\n", + ">10 f([0.00925406]) = 0.00009\n", + ">11 f([0.00564365]) = 0.00003\n", + ">12 f([0.0034318]) = 0.00001\n", + ">13 f([0.00208188]) = 0.00000\n", + ">14 f([0.00126053]) = 0.00000\n", + ">15 f([0.00076202]) = 0.00000\n", + ">16 f([0.00046006]) = 0.00000\n", + ">17 f([0.00027746]) = 0.00000\n", + ">18 f([0.00016719]) = 0.00000\n", + ">19 f([0.00010067]) = 0.00000\n", + ">20 f([6.05804744e-05]) = 0.00000\n", + ">21 f([3.64373635e-05]) = 0.00000\n", + ">22 f([2.19069576e-05]) = 0.00000\n", + ">23 f([1.31664443e-05]) = 0.00000\n", + ">24 f([7.91100141e-06]) = 0.00000\n", + ">25 f([4.75216828e-06]) = 0.00000\n", + ">26 f([2.85408468e-06]) = 0.00000\n", + ">27 f([1.71384267e-06]) = 0.00000\n", + ">28 f([1.02900153e-06]) = 0.00000\n", + ">29 f([6.17748881e-07]) = 0.00000\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtxklEQVR4nO3dd3RUdf7/8ed7Jo0kBEihpRKq9BKaCqKggrhgoYPKVwTFsutaYXWVlbXvKrq6q4h9KdZdUXGRIgJKC106BEgCQgKEBALpn98fM/gbMSEBZnJnJu/HORwz996Z++ImvnK55XPFGINSSinfZ7M6gFJKKffQQldKKT+hha6UUn5CC10ppfyEFrpSSvmJAKtWHB0dbZKSkqxavVJK+aS1a9ceMcbElDfPskJPSkoiNTXVqtUrpZRPEpH9Fc3TQy5KKeUntNCVUspPaKErpZSf0EJXSik/oYWulFJ+otJCF5F3RCRLRH6qYL6IyKsisltENolIZ/fHVEopVZmq7KG/B/Q/x/wBQHPnnwnAvy4+llJKqfNVaaEbY5YCx86xyGDgA+OwEqgrIo3cFfBsGzOO89w329Fhf5VSvqaszPD011vZejDPI5/vjmPosUCGy+tM57TfEJEJIpIqIqnZ2dkXtLJNmcd54/s9bMrMvaD3K6WUVVakHeWtZXvZcdh7C73KjDHTjTEpxpiUmJhy71yt1OBOsdQKtDNrVbqb0ymllGfNWpVO3dBABrT1zEEMdxT6ASDe5XWcc5pHRIQEMrhjY+ZuPEju6WJPrUYppdwq60QB87ccYkjnOEIC7R5ZhzsKfS5wq/Nqlx5ArjHmZzd8boVGd0/kdHEp/13vsd8bSinlVp+kZlJSZhjZPcFj66jKZYuzgRVASxHJFJFxInKXiNzlXGQekAbsBt4C7vZYWqd2cXVoH1eHmav268lRpZTXKy0zzFqVzqVNo2gaE+6x9VQ62qIxZmQl8w1wj9sSVdGobglM+nwza/fnkJIUWd2rV0qpKlu6K5sDx08z+bpWHl2Pz94p+rsOjakdHMBMPTmqlPJyM1emEx0exDWtG3p0PT5b6GHBAdzYOZavN/9MTn6R1XGUUqpcB4+fZvH2wwxLiScowLOV67OFDjCqewJFJWV8ti7T6ihKKVWuj9ZkYICR3Tx3MvQMny70Vg0j6JJYj5mr0vXkqFLK65SUljFnTTq9m8cQHxnq8fX5dKEDjO6ewN4j+azYc9TqKEop9SuLtmdxOK+Q0R68VNGVzxf6de0aUTc0UE+OKqW8zsxV6TSMCOGqVvWrZX0+X+ghgXZu7hzH/C2HyD5RaHUcpZQCIP3oKZbtymZ413gC7NVTtT5f6OA4OVpSZvg4NaPyhZVSqhrMXpOOACO6xVe6rLv4RaE3jQmnZ3IUs1enU1amJ0eVUtYqKinjk9QM+l7SgEZ1alXbev2i0MGxl56Zc5qluy5sWF6llHKXb7ce4sjJIkZV08nQM/ym0K9t05CosCA9OaqUstzMlenE1atF7+YXNkz4hfKbQg8KsDGsazyLth3m59zTVsdRStVQu7NOsiLtKCO7JWC3SbWu228KHWBk1wQMMGe1nhxVSllj9up0AmzCsJTqOxl6hl8VekJUKL2ax/DRmgxKSsusjqOUqmEKikv5bF0m17ZpSEzt4Gpfv18VOjjuHD2UV8Di7VlWR1FK1TDzNv/M8VPF1XZn6Nn8rtD7tqpPg4hgPTmqlKp2M1elkxwdRs+mUZas3+8KPcBuY0TXBJbuyibj2Cmr4yilaojth/JYuz+HUd0TEKnek6Fn+F2hg+POLMFxckIpparDrFXpBAXYuLlznGUZ/LLQG9WpxVWtGvBxagZFJXpyVCnlWfmFJXy+7gAD2zWiXliQZTn8stABRvdI4MjJIr7desjqKEopP/flxoOcLCyx7GToGX5b6L2bxxBbtxaz9OSoUsrDZq1Op0WDcLok1rM0h98Wut0mjOqewI97jrIn+6TVcZRSfmpT5nE2ZeYyunuiZSdDz/DbQgcYmhJHgE2YrXvpSikPmbUqnVqBdm7sHGt1FP8u9Pq1Q7i2TUM+XZdJQXGp1XGUUn4mr6CYuRsPMqhDYyJCAq2O49+FDo5hdY+fKuabn362OopSys98sf4Ap4pKq32Y3Ir4faH3TI6iSXQYM1fqYRellPsYY5i5Kp22sRG0j6tjdRygBhS6zSaM6pZA6v4cdhw6YXUcpZSfWJeew/ZDJ7ziZOgZfl/oADd3iSPIbmPWqv1WR1FK+YmZq9IJDw5gUIfGVkf5RY0o9MiwIK5r15DP1x3gVFGJ1XGUUj7u+Kkivtr0Mzd0akxYcIDVcX5RIwodYHSPRE4UlvDlxoNWR1FK+bhP12ZSVFLGqG6JVkf5lRpT6CmJ9WjRIFyH1VVKXRRjDLNWp9M5oS6tG0dYHedXqlToItJfRHaIyG4RmVTO/AQR+U5E1ovIJhG5zv1RL46I4+TopsxcNmUetzqOUspHrdhzlLTsfEZ19669c6hCoYuIHXgdGAC0BkaKSOuzFnsc+NgY0wkYAfzT3UHd4aYucYQHBzBj2V6royilfNSM5XuJCgvi+vaNrI7yG1XZQ+8G7DbGpBljioA5wOCzljHAmX971AG88kB1REggI7vF8/Xmn8nM0YdfKKXOz67DJ1i8PYtbeyYREmi3Os5vVKXQY4EMl9eZzmmupgBjRCQTmAfcV94HicgEEUkVkdTs7OwLiHvx/u+yJgjw7g/7LFm/Usp3zVi2l+AAG7f09L7DLeC+k6IjgfeMMXHAdcCHIvKbzzbGTDfGpBhjUmJiYty06vPTuG4trm/fiDmr08k9XWxJBqWU78k6UcB/1h9gaEockRY+xOJcqlLoB4B4l9dxzmmuxgEfAxhjVgAhQLQ7AnrCHb2SyS8q1UfUKaWq7IMf91NcVsa4y5OtjlKhqhT6GqC5iDQRkSAcJz3nnrVMOtAXQEQuwVHo1hxTqYK2sXW4rFkU7/6wVx9Rp5Sq1KmiEj5cuZ9rWjegSXSY1XEqVGmhG2NKgHuB+cA2HFezbBGRp0RkkHOxB4HxIrIRmA2MNcYYT4V2h/G9kjmcV6g3GimlKvVJaia5p4uZ0Nt7984BqnTPqjFmHo6Tna7TnnD5eitwmXujedYVLWJo2aA2by1L46bOsV4zuI5SyruUlhlmLE+jc0JduiRGWh3nnGrMnaJnExHu6NWE7YdOsGzXEavjKKW81Pwth8g4dtrr986hBhc6wKCOjalfO5i3lqVZHUUp5YWMMby5NI3EqFCubt3Q6jiVqtGFHhxgZ+xlSSzbdYStB/OsjqOU8jKp+3PYmHGcOy5vgt3m/Ydla3ShA4zulkhokJ0Zy3UvXSn1a28tTaNeaCBDusRXvrAXqPGFXic0kOFd45m74SA/5562Oo5SykukZZ9kwbbD3NIjkVpB3nebf3lqfKED3H5ZE8qM4b0f91kdRSnlJd5evpdAu41beiZZHaXKtNCB+MhQrmvXiFkr0zlRoMMBKFXTHT1ZyKdrM7m5cywxtYOtjlNlWuhOE3onc6KwhI/WZFS+sFLKr324cj+FJd59m395tNCd2sfVpVuTSN79YR/FpTocgFI1VUFxKR+s2E/fVvVpVj/c6jjnRQvdxYReyRw4fpp5m3+2OopSyiKfrcvkWH4R433gRqKzaaG7uKpVfZJjwnhrWRpePhSNUsoDysoMM5btpX1cHbo38e7b/Mujhe7CZhPG90rmpwN5rEg7anUcpVQ1W7jtMHuP5DO+V7JPju+khX6WGzvFEh0exFtL9UYjpWqat5alEVu3FgPaev9t/uXRQj9LSKCdW3sm8d2ObHYePmF1HKVUNVmXnsOafTmMu7wJAXbfrEbfTO1hY3okEhJoY4YO2qVUjTFjWRoRIQEM6+obt/mXRwu9HJFhQQztEs9/1x8kK6/A6jhKKQ/bfzSf//10iNE9EgkPrtJjIrySFnoFxl3ehOKyMt5fsc/qKEopD3tn+V7sNmHspUlWR7koWugVSIoO49rWDfn3ynTyC0usjqOU8pCc/CI+Ts1kcMdYGkSEWB3nomihn8P43snkni7mk1QdDkApfzVz1X5OF5cyvpfv3Uh0Ni30c+iSWI8uifV4+4e9lOhwAEr5nYLiUt77cb/jGcMNa1sd56JpoVdifK9kMo6dZv6Ww1ZHUUq52RcbDnDkZKFPPC+0KrTQK3F16wYkRYUyfekeHQ5AKT9SVmZ4a9leWjeK4NKmUVbHcQst9ErYbcK4XslszMxlzb4cq+Mopdxkyc4sdmedZEJv37zNvzxa6FUwpHMc9UIDma7DASjlN6YvTaNRnRAGtm9kdRS30UKvglpBdm7pmcSi7YfZk33S6jhKqYu0OTOXlWnHuP2yJgT66G3+5fGfv4mH3dozkSC7jX9+t8fqKEqpi/T6d7upHRzA8G6+e5t/ebTQqyg6PJgxPRL5z/pM3UtXyof9dCCX/205xO2XNyEiJNDqOG6lhX4eJvZpSnCAnVcW7rI6ilLqAr20YCd1agUyrlcTq6O4nRb6eYgOD2bsZUl8uekgOw7p0LpK+Zp16Tks3p7FhN7Jfrd3Dlro521Cr2TCggJ4ecFOq6Mopc7TS9/uJCosyOcH4apIlQpdRPqLyA4R2S0ikypYZpiIbBWRLSIyy70xvUe9sCDGXd6E/205xE8Hcq2Oo5SqolVpR1m++wgT+zQlzIeHyD2XSgtdROzA68AAoDUwUkRan7VMc2AycJkxpg1wv/ujeo9xvZpQp1ag7qUr5SOMMfz9253Ur+24uMFfVWUPvRuw2xiTZowpAuYAg89aZjzwujEmB8AYk+XemN4lIiSQCb2TWbQ9i3XpeveoUt5u+e4jrN53jHuubEZIoN3qOB5TlUKPBVzHj810TnPVAmghIj+IyEoR6V/eB4nIBBFJFZHU7OzsC0vsJcZemkRkWJDupSvl5c7snTeuE8IIP7vu/GzuOikaADQH+gAjgbdEpO7ZCxljphtjUowxKTExMW5atTXCggOYeEVTlu06wqq0o1bHUUpVYPH2LDZkHOe+vs0JDvDfvXOoWqEfAFx/rcU5p7nKBOYaY4qNMXuBnTgK3q+N6ZFI/drB/H3BTh2JUSkvZIzhpQU7SYgMZUiXOKvjeFxVCn0N0FxEmohIEDACmHvWMv/FsXeOiETjOATj9yNZ1Qqyc8+VzVi99xg/7Na9dKW8zfwth9hyMI8/9G3uV2O2VKTSv6ExpgS4F5gPbAM+NsZsEZGnRGSQc7H5wFER2Qp8BzxsjKkRDTeiWzyN64Twt2936F66Ul6ktMyxd54cE8YNnc4+7eefqvQryxgzzxjTwhjT1BjztHPaE8aYuc6vjTHmAWNMa2NMO2PMHE+G9ibBAXbu69ucDRnH+W6HX1/co5RP+WrTQXYePskf+7XAbvOP8c4r4///BqkGQ7rEkRAZyt+/1WPpSnmDktIypi3cRauGtRnYzn/GO6+MFrobBNpt/L5vc7YczGP+lkNWx1Gqxls57V1umfMSU6LzsNWQvXPQQnebGzo2JjkmjJcW7KS0TPfSlbJK8bIf6PnwBP5v7Zd0nzAMVqywOlK10UJ3kwC7jfv7tWDn4ZN8temg1XGUqrH2v/gqdlOGAFJUBEuWWB2p2mihu9H17RrRskFtXlm4i5LSMqvjKFXjFOSfJnTZ9xjA2O0QFAR9+lgdq9poobuRzSb88eoWpB3J578bdC9dqeq2+ZGpND5+mH2PPIlMnQqLFkHPnlbHqjb+OYakha5t04C2sRG8smgngzs2rhE3MyjlDU7vy6D1jGmsbXcpnZ97EqTmnAw9Q9vGzUSEB69uScax03ySmml1HKVqjMzx9xJQWkLAq68iNbDMQQvdI/q0jKFTQl3+sXgXBcWlVsdRyu+dWrCY5gvnMm/ALXTo08XqOJbRQveAM3vpP+cWMGd1utVxlPJvxcWcvnMimRH1Sf77VKvTWEoL3UMuaxZF9yaRvL5kD6eLdC9dKU85Pe1Vovbu5L+3PUSHFo2tjmMpLXQPEREevKYl2ScK+XDlPqvjKOWfDh3CNmUKS5p04cpHx1udxnJa6B7UrUkkvZpH88b3aZwsLLE6jlJ+p/CBh5DCQhbf/RhtYutaHcdyWuge9sDVLTiWX8T7P+6zOopS/mX5coJnz+Stbjcy5pZ+VqfxClroHtYpoR59W9Xnze/3kHu62Oo4SvmHkhJKJt7NwYgY9k34Ay0a1LY6kVfQQq8Gf7y6BXkFJcxY5vcPcVKqevzrXwT8tJmn+97BxIHtrU7jNbTQq0Hb2Dpc374R05emkZlzyuo4Svm2w4cpffzPLE/qSO1Rw0mOCbc6kdfQQq8mk6+7BBF4+uttVkdRyqeZSZMw+fm8MPAeHurfyuo4XkULvZrE1q3FvVc245ufDrFsV7bVcZTyTStWIO+9x4yUwdw0si/R4cFWJ/IqWujV6I5eySRGhTJl7haKSnR4XaXOS2kpZffcS1ZENP+78Q7G9Ei0OpHX0UKvRiGBdp64vjV7svP1Mkalztf06djWr+OpPrczaWhXAnQk09/QLVLN+l7SgKta1Wfawp1k5RVYHUcp33DkCKV/eowVSR2wDR9Oj+QoqxN5JS10CzxxfWuKSw3PfbPd6ihK+YbJkzEn8nim/0T+NLC11Wm8lha6BZKiwxjfuwmfrz/Amn3HrI6jlHdbvRrz9tu803kQ1w3vR8M6IVYn8lpa6Ba558pmNKoTwpNfbKG0zFgdRynvVFpK2d33cLR2JHMHjeP2y5OsTuTVtNAtEhoUwGMDL2Hrz3nM0jHTlSrf229jW5vK1N5jeWhoV4ID7FYn8mpa6BYa2K4RPZOj+Nv8HRzLL7I6jlLe5ehRyiZNZk1CW04NGU6flvWtTuT1tNAtJCL8ZXAbThaW8Ldvd1gdRynv8vjjmNxc/nLtRJ74XRur0/gELXSLtWhQm9t6JjF7dTqbM3OtjqOUd1i7FvPmm7zfaSBXDe1HfGSo1Yl8gha6F7j/6uZEhQXxxNyfKNMTpKqmKyvD3H0POeH1mHP9OCZe0dTqRD5DC90LRIQE8mj/VqxPP87n6w9YHUcpa733HrJ6FX/tfRsPDOlKrSA9EVpVVSp0EekvIjtEZLeITDrHcjeLiBGRFPdFrBlu7hxHp4S6PPfNNvIK9EEYqobKyaHskUdZF9+a7BuGcW2bhlYn8imVFrqI2IHXgQFAa2CkiPzmVi0RqQ38AVjl7pA1gc0mPDWoLUfzi3hl4S6r4yhljT//GY4d44mrJ/LkoLaIiNWJfEpV9tC7AbuNMWnGmCJgDjC4nOWmAs8DOkDJBWoXV4cRXRN478d97Dx8wuo4SlWvDRsw//oXH3S6jktv7kuz+vrgivNVlUKPBTJcXmc6p/1CRDoD8caYr8/1QSIyQURSRSQ1O1vHBC/Pw9e2JDw4gCe/2IIxeoJU1RBlZZh77iE3NIIPBozjvquaWZ3IJ130SVERsQEvAQ9WtqwxZroxJsUYkxITE3Oxq/ZLkWFBPHRtS1akHWXe5kNWx1Gqevz738iPP/J0r9v4/c1dqR0SaHUin1SVQj8AxLu8jnNOO6M20BZYIiL7gB7AXD0xeuFGdUugdaMI/vr1Vk4VlVgdRynPys2l7OFH2BR3CfuvH8Lgjo2tTuSzqlLoa4DmItJERIKAEcDcMzONMbnGmGhjTJIxJglYCQwyxqR6JHENYLcJTw1uw8+5Bfzzuz1Wx1HKs558EsnO4vF+d/LkDe30ROhFqLTQjTElwL3AfGAb8LExZouIPCUigzwdsKZKSYrkxk6xTF+axr4j+VbHUcozNm/GvPYaszr2p8PgvrRpXMfqRD6tSsfQjTHzjDEtjDFNjTFPO6c9YYyZW86yfXTv3D0mD2hFoF146qutVkdRyv2MwdxzDydCwpl+ze08eE0LqxP5PL1T1IvVjwjh/n4tWLw9i0XbDlsdRyn3mjULWbaMZy6/hbtu6kbd0CCrE/k8LXQvd9ulSTSNCWPKl1vIL9QTpMpP5OVR9uBDbIltydYBQxmWEl/5e1SltNC9XFCAjWdvak9mzmn++rUeelH+wUyZAlmHebzfXTwzpAN2m54IdQctdB/QrUkkd/ZuyuzVGSzYqodelI/bsgXz6qvMaX8N/W67nraxeiLUXbTQfcQDV7egdaMIJn22iewThVbHUerCGEPBxLvJCwplwejfc5cOjetWWug+IijAxrQRHTlRWMKjn23SYQGUTyqb8xEhy5byypW38dS4Pnqoxc200H1Iiwa1mTygFYu3Z+mDpZXvOXmSU7+/n80NmtL6iQf1KUQeoIXuY27rmUSv5tH89attpGWftDqOUlV25NHHCT9ymHl3Ps6QbolWx/FLWug+xmYTXhzSgaAAG3/8eCPFpWVWR1KqUoWbt1D3jdf4onN/xj8ySm/v9xAtdB/UsE4Iz9zYjo0Zx3lt8W6r4yh1bsZw4Nbx5AeGEP2PvxEZpjcQeYoWuo8a2L4RN3WK5bXvdrMuPcfqOEpVaNtr75K8YQXLbv09l13axuo4fk2suloiJSXFpKbqkC8XI6+gmAHTlhFgF+b9vhdhwQFWR1LqV45n51DQvCUnwyKI3bWFWqHBVkfyeSKy1hhT7vDkuofuwyJCAnlpWAfSj53Su0iV1zHGsOqOB2mYm4289rqWeTXQQvdx3ZOj9C5S5ZXWPP8Gfb98n33detP0xmutjlMjaKH7Ab2LVHmb7G8W0uVP92A3ZSRuWg0rVlgdqUbQQvcDrneRTtK7SJXFysoMx+79I3ZjEECKi2HJEqtj1Qha6H6iRYPaTOrfikXbs5i9OsPqOKoG++GhqbRM+4kyux3sdggKgj59rI5VI2ih+5GxlyZxebNopn61Ve8iVZbYP+e/9Jz2FzZ1vBz57juYOhUWLYKePa2OViPoZYt+5lBuAddOW0pSdBif3tWTQLv+zlbVo3DTTxR378HBug2IXr+ayIZRVkfyS3rZYg3SsE4IT9/YVu8iVdXryBHyr72O0/ZAjsz6RMvcIlrofuj69o1/uYt0vd5FqjytsJDcAb8jLPsQ/3nydS69srPViWosLXQ/NWVwGxpGhPDHjzbos0iV5xhD0bjx1EldyQsjJ3HLH4ZbnahG00L3U2fuIt1/7BR//Xqb1XGUnzLPP0/QzA+Z1ms0Nzz7ILWC7FZHqtG00P1Y9+QoJvROZvbqdD5J1UsZlZt9/jkyeTJfXHIFAU8+Qbs4fTao1bTQ/dxD17Tk8mbR/Ok/m1mx56jVcZS/WLuW0tFjWNe4Jf974GkmXtnc6kQKLXS/F2i38froziRFhXHXv9eyR69PVxfrwAGKr/8dh4Nr8/LdL/D3W7vrs0G9hBZ6DVCnViDvjO1KgE24/b01HMsvsjqS8lX5+RQP/B1Fx47zyG1/5W/3XUNokA7b7C200GuI+MhQ3rothUO5BUz4IJWC4lKrIylfU1ZGyZhbsG3ayEM3TeKxScNpEBFidSrlQgu9BumcUI+XhnUkdX8Oj+ogXuo8lT32GAH//Q/PXDWOYU/exSWNIqyOpM6ihV7DDGzfiIevbckXGw7y8sJdVsdRvuL997E99xwzO/Yn8anJXNmqvtWJVDmqVOgi0l9EdojIbhGZVM78B0Rkq4hsEpFFIpLo/qjKXe7u05RhKXG8umgXn6/LtDqO8nbLllF6x3iWJ3Yg7YlnufXSJlYnUhWotNBFxA68DgwAWgMjRaT1WYutB1KMMe2BT4EX3B1UuY+I8Ncb2tEzOYpHP9vEqjS9nFFVYM8eigbdwP6I+nz8yN/50+AOVidS51CVPfRuwG5jTJoxpgiYAwx2XcAY850x5pTz5Uogzr0xlbsFBdh4Y0wXEiJDufPfa9l7JN/qSMrbHD9O4YCBnCos5tmJL/DsuCv08kQvV5VCjwVcbzPMdE6ryDjgm/JmiMgEEUkVkdTs7Oyqp1QeUSc0kHfHdsMmwv+9u5ocvZxRnVFSQuHNQ7Gl7WHy6ClMffgmwoL18kRv59aToiIyBkgBXixvvjFmujEmxRiTEhMT485VqwuUEBXKW7d24WBuAXd+uJbCEr2cUUHxfb8nePFCplx3H/f+ZRwN6+jlib6gKoV+AIh3eR3nnPYrItIPeAwYZIzRJxX7kC6Jkbw4pD2r9x1j0meb9XLGGq7sH/8g8I1/8Wb3m7jq+Udp01jHaPEVVfk31BqguYg0wVHkI4BRrguISCfgTaC/MSbL7SmVxw3uGMv+o6d4acFOkqLC+EM/HZujRvrf/+D++/m2eQ+Cnn+evpc0sDqROg+V7qEbY0qAe4H5wDbgY2PMFhF5SkQGORd7EQgHPhGRDSIy12OJlcfcd1Uzbuocy8sLd/LFht/8I0z5u59+ovjmoWyLTmL1U68wtndTqxOp81SlsxzGmHnAvLOmPeHydT8351IWEBGevakdmTmnefiTTTSuW4uuSZFWx1LVISuL0/0HkmcLYsYj03hxaAoiekWLr9E7RdWvBAfYeXNMF2Lr1WLCB6ns08sZ/V9BAaevH4QcPsTT459h6r0DCNCHi/sk/a6p36gXFsQ7Y7tiQEdn9HfGUDD2dmqtWcVfhjzCpCduJVwvT/RZWuiqXE2iw5h+SwqZx08z7M0VHDx+2upIygNyHptCyEezeaXPrYx68SEa161ldSR1EbTQVYW6NYnkg9u7cTi3gJv/9SO7Dp+wOpJyo/Q336fes0/xVfu+XPH+NH2EnB/QQlfn1CM5io/u7ElJmWHIGytYuz/H6kjKDTZ9voCYe+9kU2IbLvn6Izom1LM6knIDLXRVqdaNI/h84qXUCw1k9IyVLN5+2OpI6iKkvvIOySMGcapWGA0XzqNpXJTVkZSbaKGrKomPDOXTiZfSvH5txn+wls/W6rC7vmj5lGl0vn8cYcUFRBblUz9b7zfwJ1roqsqiw4OZPaEHPZIjefCTjUxfusfqSKqKjDF89edX6Tb1YQQcf0pKYMkSi5Mpd9JCV+clPDiAd8Z2ZWD7RjwzbzvPzNtGWZmO/eLNSotL+H7oBK7/6x842jgRQkLAboegIOjTx+p4yo30glN13oID7PxjRCeiw4KYvjSNIycLef7m9gTqzShepyD7KLuuHkyfjT+wsf8Q2v/nQ2T9eseeeZ8+0LOn1RGVG2mhqwtiswlTBrUhOjyYvy/YSU5+Ea+P7kxokP5IeYuT6zdyov/vaHXkID88OJXLXnwMRBwlrkXul3SXSl0wEeG+vs159qZ2fL8zm9EzVulDMrzE8dkfY+vZk8CTJ1j51sdc9rfHHWWu/JoWurpoI7sl8M/RXdhyMI+heleptcrKOP7wZOqOGs7uqHh2fbOEXrffZHUqVU200JVb9G/bUO8qtVpuLnn9B1L3b8/xRcerMUu+p2dvfahzTaKFrtzG9a7SoW/qXaXVascOTnVOIXTht7z0u/toO/8zOjRvaHUqVc200JVbnbmrtG4tx12lX248aHUkv2fmzqW4S1dOH8pm0sSXGfXhCzStX9vqWMoCWujK7c7cVdqqYQT3zV7PXR+uJSuvwOpY/qesjLzHnkAGD2Zb7Qb8+fF3+fOLE/WBzjWYFrryiOjwYD69qyeTBrTiux1Z9Hvpez5OzdAHULtJWW4e6VcOIOKZqXzRri8bZ87ltUdvoE5ooNXRlIW00JXHBNht3HVFU775Qy9aNYzgkU83ccvbq8k4dsrqaD4tfdUGDlzSkcbLFvLhiD/S+bsvuOWqS7DZ9LLEmk4LXXlcckw4cyb0YOoNbVmfnsM1Ly/l3R/2UqpDBpyXktIyvn7ubepecTnhx4+y/PWZjJn1d+KjwqyOpryEFrqqFjabcEuPRL594Aq6J0fyly+3MvSNH9mdpZc3VkXa3G/Z2LYHAybfwfEGsZSuXk2fiSP0Qc7qV7TQVbWKrVuLd8d25eXhHdh7JJ/rXlnOa4t3UVxaZnU0r1RQXMpXj75A4g396bJ9DdhsJLz3BtFtW1odTXkhLXRV7USEGzvFseCBK7imTQP+9u1OfveP5WzOzLU6mlfZvHw9S3sO4LoXHsXmPJlsE4GVKy1OpryVFrqyTHR4MK+N6sz0W7pwLL+IG/75A899s52C4lKro1nq1L4MVg8cRcsrunHFxu/Jvv5mRIe8VVWgha4sd02bhix44AqGdonjje/3MOCVZaxKO2p1rOqXk0P6Xfdja9Gczt98xKarb6R0x04afPkpLF4MU6fCokU6UqKqkFh1XXBKSopJTU21ZN3Ke/2w+wiTPt9ExrHT3NgpljE9EuicUM+vT/6ZkyfJeOoFol+fRuipEyzqdBX1X3qedn1SrI6mvJCIrDXGlPvDoYWuvM6pohKmLdzFzJX7yS8qpVn9cIanxHNT51iiwoOtjuc2WUfy2D71Jdq8+w+iThzj++bd2P/HPzHsjusJCbRbHU95KS105ZPyC0v4etPPzFmTzrr04wTahatbN2BYSjy9msdg98EbaUpKy1g/+yt49VXit62j4ckctjXrQPbkJ+g6ZjC1grTI1blpoSuft+vwCT5ak8Hn6w9wLL+IxnVCGJoSz9CUOOLqhVodr1L7j+bz7RfLiP7XPxic+g02DEaEo1OeJvrPk/ThE6rKtNCV3ygsKWXh1iw+Ss1g2a5sAC5vFs2Irgn0a12f4ADv2cMtKC5l2fxVHHnn37T/YT5tstIAMICA46qVqVNh8mQrYyofo4Wu/FJmzik+Sc3kk9QMDuYWEBkWxE2dYhneNZ7mDSwaPnbFCrL//RFp+7II27CWtgd3AnDokg6EjhlFRJuWMHIkFBU5LkHUq1bUebroQheR/sArgB2YYYx57qz5wcAHQBfgKDDcGLPvXJ+pha7cpbTMsHz3ET5ak86CrYcpLjW0alibpvXDia8XSkJkKPGRtYivF0rjurUICrj4q3WLl/3AifkL2d8mhQwJwbZmNbFLF9Bh7ZJfbgI61CiRgjG3kXDXWGzJTf7/m1esgCVLHNeTa5mr83RRhS4idmAncDWQCawBRhpjtrosczfQ3hhzl4iMAG40xgw/1+dqoStPOHqykM/XHWDprmwyjp0iM+c0JS6DgNkEGtWpRVy9WsRH/rrs4yNDiYmrj+3ECcqAsrAw8i69gpza9djY9Uryf84mbvUyivNPcdVPS7Ebx3AFZ45+F9kDCSwtRgBjtyN6OEV5wMUWek9gijHmWufryQDGmGddlpnvXGaFiAQAh4AYc44P10JX1aG0zHAor4CMY6dIP3aKzGOnyMg5/cvrrBOFvyy74aVh1Ckuf2jfs09ZnjkObkQouH4Qwc88jS33OFx9tR5OUR51rkIPqML7Y4EMl9eZQPeKljHGlIhILhAFHDkryARgAkBCQkKVwit1Mew2IbZuLWLr1qJHctRv5hcUl5KZc5qMnFNEPO8o8/KuNzE4xqDBGLDZEJsNjEGCgqg1+VFo28ax4KJFejhFWaYqhe42xpjpwHRw7KFX57qVKk9IoJ1m9cNpVj8cateGE+UP5yuBgY5LC0tLHXvf06bB0aO/Le6ePbXIlWWqUugHgHiX13HOaeUtk+k85FIHx8lRpXxHXh5ERPz/Ug8Ph379oGFDuPVWxzTd+1ZerCqFvgZoLiJNcBT3CGDUWcvMBW4DVgBDgMXnOn6ulNfKyzv3fC1y5cUqLXTnMfF7gfk4Llt8xxizRUSeAlKNMXOBt4EPRWQ3cAxH6SullKpGVTqGboyZB8w7a9oTLl8XAEPdG00ppdT50PHQlVLKT2ihK6WUn9BCV0opP6GFrpRSfsKy0RZFJBvYf4Fvj+asu1C9hOY6P5rr/HlrNs11fi4mV6IxJqa8GZYV+sUQkdSKxjKwkuY6P5rr/HlrNs11fjyVSw+5KKWUn9BCV0opP+GrhT7d6gAV0FznR3OdP2/NprnOj0dy+eQxdKWUUr/lq3voSimlzqKFrpRSfsJrC11EhorIFhEpE5EKL+8Rkf4iskNEdovIJJfpTURklXP6RyIS5KZckSKyQER2Of9br5xlrhSRDS5/CkTkBue890Rkr8u8jtWVy7lcqcu657pMt3J7dRSRFc7v9yYRGe4yz63bq6KfF5f5wc6//27n9khymTfZOX2HiFx7MTkuINcDIrLVuX0WiUiiy7xyv6fVlGusiGS7rP8Ol3m3Ob/vu0TktmrO9bJLpp0ictxlnie31zsikiUiP1UwX0TkVWfuTSLS2WXexW8vY4xX/gEuAVoCS4CUCpaxA3uAZCAI2Ai0ds77GBjh/PoNYKKbcr0ATHJ+PQl4vpLlI3EMKRzqfP0eMMQD26tKuYCTFUy3bHsBLYDmzq8bAz8Ddd29vc718+KyzN3AG86vRwAfOb9u7Vw+GGji/Bx7Nea60uVnaOKZXOf6nlZTrrHAa+W8NxJIc/63nvPretWV66zl78Mx7LdHt5fzs3sDnYGfKph/HfANjicd9gBWuXN7ee0eujFmmzFmRyWLdQN2G2PSjDFFwBxgsIgIcBXwqXO594Eb3BRtsPPzqvq5Q4BvjDHlP33Yfc431y+s3l7GmJ3GmF3Orw8CWUC5d8JdpHJ/Xs6R91Ogr3P7DAbmGGMKjTF7gd3Oz6uWXMaY71x+hlbieHKYp1Vle1XkWmCBMeaYMSYHWAD0tyjXSGC2m9Z9TsaYpTh24CoyGPjAOKwE6opII9y0vby20KuovAdYx+J4QPVxY0zJWdPdoYEx5mfn14eABpUsP4Lf/jA97fzn1ssiElzNuUJEJFVEVp45DIQXbS8R6YZjr2uPy2R3ba+Kfl7KXca5Pc488Lwq7/VkLlfjcOzlnVHe97Q6c93s/P58KiJnHlfpFdvLeWiqCbDYZbKntldVVJTdLdurWh8SfTYRWQg0LGfWY8aYL6o7zxnnyuX6whhjRKTC6z6dv3nb4Xja0xmTcRRbEI5rUR8FnqrGXInGmAMikgwsFpHNOErrgrl5e30I3GaMKXNOvuDt5Y9EZAyQAlzhMvk331NjzJ7yP8HtvgRmG2MKReROHP+6uaqa1l0VI4BPjTGlLtOs3F4eZWmhG2P6XeRHVPQA66M4/ikT4NzLKu/B1heUS0QOi0gjY8zPzgLKOsdHDQP+Y4wpdvnsM3urhSLyLvBQdeYyxhxw/jdNRJYAnYDPsHh7iUgE8DWOX+YrXT77grdXOS7mgedVea8ncyEi/XD8krzCGFN4ZnoF31N3FFSluYwxrg+Dn4HjnMmZ9/Y5671L3JCpSrlcjADucZ3gwe1VFRVld8v28vVDLr88wFocV2WMAOYax1mG73AcvwbHA6zdtcd/5oHYVfnc3xy7c5bamePWNwDlng33RC4RqXfmkIWIRAOXAVut3l7O791/cBxb/PSsee7cXuX+vJwjr+sDz+cCI8RxFUwToDmw+iKynFcuEekEvAkMMsZkuUwv93tajbkaubwcBGxzfj0fuMaZrx5wDb/+l6pHczmztcJxgnGFyzRPbq+qmAvc6rzapQeQ69xpcc/28tTZ3ov9A9yI4zhSIXAYmO+c3hiY57LcdcBOHL9hH3OZnozjf7jdwCdAsJtyRQGLgF3AQiDSOT0FmOGyXBKO37q2s96/GNiMo5j+DYRXVy7gUue6Nzr/O84bthcwBigGNrj86eiJ7VXezwuOQziDnF+HOP/+u53bI9nlvY8537cDGODmn/fKci10/n9wZvvMrex7Wk25ngW2ONf/HdDK5b23O7fjbuD/qjOX8/UU4Lmz3ufp7TUbx1VaxTj6axxwF3CXc74Arztzb8blCj53bC+99V8ppfyErx9yUUop5aSFrpRSfkILXSml/IQWulJK+QktdKWU8hNa6Eop5Se00JVSyk/8P2O4xqLPu9cnAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "from numpy import asarray\n", "from numpy import arange\n", @@ -2468,10 +2215,8 @@ }, { "cell_type": "markdown", - "id": "e876cfa2", - "metadata": { - "editable": true - }, + "id": "3eac652d", + "metadata": {}, "source": [ "## Overview video on Stochastic Gradient Descent\n", "\n", @@ -2480,10 +2225,8 @@ }, { "cell_type": "markdown", - "id": "6249ba96", - "metadata": { - "editable": true - }, + "id": "9ee2bcdd", + "metadata": {}, "source": [ "## Batches and mini-batches\n", "\n", @@ -2501,10 +2244,8 @@ }, { "cell_type": "markdown", - "id": "da24d433", - "metadata": { - "editable": true - }, + "id": "a2a69883", + "metadata": {}, "source": [ "## Stochastic Gradient Descent (SGD)\n", "\n", @@ -2533,10 +2274,8 @@ }, { "cell_type": "markdown", - "id": "4bdc03a8", - "metadata": { - "editable": true - }, + "id": "3cb45498", + "metadata": {}, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2550,10 +2289,8 @@ }, { "cell_type": "markdown", - "id": "663d5c76", - "metadata": { - "editable": true - }, + "id": "10c0c325", + "metadata": {}, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2563,10 +2300,8 @@ }, { "cell_type": "markdown", - "id": "d708404c", - "metadata": { - "editable": true - }, + "id": "aa0d4c01", + "metadata": {}, "source": [ "## Computation of gradients\n", "\n", @@ -2576,10 +2311,8 @@ }, { "cell_type": "markdown", - "id": "7c887c77", - "metadata": { - "editable": true - }, + "id": "9ada4734", + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2589,10 +2322,8 @@ }, { "cell_type": "markdown", - "id": "78f7e779", - "metadata": { - "editable": true - }, + "id": "b187e9a4", + "metadata": {}, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2603,10 +2334,8 @@ }, { "cell_type": "markdown", - "id": "07c20bb7", - "metadata": { - "editable": true - }, + "id": "65cd1b71", + "metadata": {}, "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", @@ -2625,10 +2354,8 @@ }, { "cell_type": "markdown", - "id": "29268af5", - "metadata": { - "editable": true - }, + "id": "37a27914", + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2640,10 +2367,8 @@ }, { "cell_type": "markdown", - "id": "6555a699", - "metadata": { - "editable": true - }, + "id": "06933ac2", + "metadata": {}, "source": [ "## The gradient step\n", "\n", @@ -2652,10 +2377,8 @@ }, { "cell_type": "markdown", - "id": "386851b0", - "metadata": { - "editable": true - }, + "id": "2c0cfb82", + "metadata": {}, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2665,10 +2388,8 @@ }, { "cell_type": "markdown", - "id": "7ceea54c", - "metadata": { - "editable": true - }, + "id": "f90b72f3", + "metadata": {}, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2679,10 +2400,8 @@ }, { "cell_type": "markdown", - "id": "b21be97d", - "metadata": { - "editable": true - }, + "id": "8c964be7", + "metadata": {}, "source": [ "## Simple example code" ] @@ -2690,11 +2409,8 @@ { "cell_type": "code", "execution_count": 12, - "id": "dbcc29fa", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "4eac284e", + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -2715,10 +2431,8 @@ }, { "cell_type": "markdown", - "id": "8fd3d410", - "metadata": { - "editable": true - }, + "id": "1074f7d4", + "metadata": {}, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2731,10 +2445,8 @@ }, { "cell_type": "markdown", - "id": "a173ce09", - "metadata": { - "editable": true - }, + "id": "982554ed", + "metadata": {}, "source": [ "## When do we stop?\n", "\n", @@ -2752,10 +2464,8 @@ }, { "cell_type": "markdown", - "id": "32a527a7", - "metadata": { - "editable": true - }, + "id": "6e09fd45", + "metadata": {}, "source": [ "## Slightly different approach\n", "\n", @@ -2772,10 +2482,8 @@ }, { "cell_type": "markdown", - "id": "fc9eb6b3", - "metadata": { - "editable": true - }, + "id": "28895eb5", + "metadata": {}, "source": [ "## Time decay rate\n", "\n", @@ -2791,11 +2499,8 @@ { "cell_type": "code", "execution_count": 13, - "id": "2fc5afba", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "07202d5c", + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -2826,10 +2531,8 @@ }, { "cell_type": "markdown", - "id": "6d58db52", - "metadata": { - "editable": true - }, + "id": "cb6a7d32", + "metadata": {}, "source": [ "## Code with a Number of Minibatches which varies\n", "\n", @@ -2838,13 +2541,39 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "464646e8", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "execution_count": 3, + "id": "1ebde042", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.]\n", + " [3.]]\n", + "Eigenvalues of Hessian Matrix:[0.26504701 4.42519896]\n", + "theta from own gd\n", + "[[4.]\n", + " [3.]]\n", + "theta from own sdg\n", + "[[3.97239309]\n", + " [3.02330208]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAgbUlEQVR4nO3deZhddZ3n8fcnVSlClgaSCsiWCsgmrmgNIEIbDI6ILPY4zwiTsMRgNWRUVFokZtq1bR2XVrQFJ2okkBpUEB3FDQirbHaxyS4QUiFsKRK2kD317T/OueTk1q2qU6l776mqfF7Pk6fOPefcc773cKlv/XZFBGZmZv0ZVXQAZmY2PDhhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuThg2Ykn6oqRFRcdRTZKmSVpedBy2fXLCsLqStFTSWkmrJT0r6WJJ44uOy8z654RhRTghIsYDbwMOAeYWG872SVJj0THY8OKEYYWJiGeBP5EkDgAknS/pcUmvSHpQ0j9kjp0h6c+SviXpBUlPSHp/5vg+km5M33sN0Jy9n6QTJT0g6UVJN0h6Q+bYUkmfkfRXSa9K+omk3ST9Ib3etZJ2qfQ5StVEks6VtELSM5JmZY7fIOnM8s+ReR2S5kh6NL3XVyS9XtKtkl6W9AtJTWX3/Jyk59O4Z2T275A+n2WSnpP0Q0k7lsX5WUnPAj+V1CzpqvSZrJJ0syT/XrCK/MWwwkjaC3g/8Fhm9+PAUcBOwJeARZJ2zxw/DHiEJBl8A/iJJKXH/h9wZ3rsK8DpmXsdAFwGfBKYDPwe+G3ZL+IPAe8FDgBOAP4AfC49fxTwiT4+zuvSmPcEZgM/6C3B9OJ9wDuAw4HzgPnATGBv4E3AKWX3ak7vdTowX9KB6bGvp/G/DdgvPefzZe+dCLQAbcC5wPL0M+6Wfl7PF2QVOWFYEX4t6RXgSWAF8IXSgYi4PCKejojuiPg58ChwaOa9nRHxo4jYDCwEdgd2kzQF+C/AP0fE+oi4Cfht5n0fBn4XEddExEbgW8COwBGZc74fEc9FxFPAzcAdEXF3RKwDfkVSfdabjcCXI2JjRPweWA0c2Mf55b4RES9HxAPA/cDVEbEkIl4iSVzl9y59zhuB3wH/I02cbcCnImJVRLwC/CtwcuZ93cAX0veuTePeHWhJY785PMGc9cIJw4rwwYiYAEwDDiJTdSTpNEn3pFUkL5L8dZ2tWnq2tBERa9LN8cAewAsR8Wrm3M7M9h7Z1xHRTZKw9syc81xme22F1301zq+MiE2Z12v6Ob/cQO5d6XPuQVJKGAvcmXl+f0z3l3SlCbDkmyQlvKslLZF0/gBitu2ME4YVJv3r+GKSv/aR1AL8CPgYMCkidib5a1u9XCLrGWAXSeMy+6Zktp8mqYYhvZdIqnue2vZPkNurJL/IS143yOtV+pxPA8+TJJc3RsTO6b+d0g4GJVuVHiLilYg4NyL2BU4EPi1p+iDjsxHKCcOK9l3gvZLeCowj+YXWBZA2HL8pz0UiohPoAL4kqUnSkSTtECW/AD4gabqk0SR19+uBW6v1QfpwD/DfJI2VtB9JG8dglT7nUcDxwOVpqelHwHck7QogaU9J7+vtIpKOl7RfmkBfAjaTVFuZ9eCEYYWKiC7gEuDzEfEg8G3gNpIqmTcDtwzgcv+TpFF8FUm7yCWZ+zxC0oj8fZK/xE8g6d67oQofoz/fATaQfKaFQPsgr/cs8AJJqaIdOCsiHk6PfZakiul2SS8D19J3W8r+6TmrSZ77hRFx/SDjsxFKbt8yM7M8XMIwM7NcnDDMzCwXJwwzM8vFCcPMzHIZVpOPNTc3x9SpU4sOw8xsWLnzzjufj4jJ/Z/Zt2GVMKZOnUpHR0fRYZiZDSuSOvs/q3+ukjIzs1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCyXmicMSQskrZB0f4Vj50oKSc21jsPMzAanHiWMi4Fjy3dK2hv4r8CyOsRgZmaDVPOEERE3kayxXO47wHmA14g1MxsGCmnDkHQS8FRE3Jvj3DZJHZI6urq66hCdmZlVUveEIWks8Dng83nOj4j5EdEaEa2TJw96OnczM9tGRZQwXg/sA9wraSmwF3CXpNcVEIuZmeVU9wWUIuI+YNfS6zRptEbE8/WOxczM8qtHt9rLgNuAAyUtlzS71vc0M7Pqq3kJIyJO6ef41FrHYGZmg+eR3mZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrk4YZiZWS5OGGZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuNU8YkhZIWiHp/sy+b0p6WNJfJf1K0s61jsPMzAanHiWMi4Fjy/ZdA7wpIt4C/A2YW4c4zMxsEGqeMCLiJmBV2b6rI2JT+vJ2YK9ax2FmZoMzFNowPgL8obeDktokdUjq6OrqqmNYZmaWVWjCkDQP2AS093ZORMyPiNaIaJ08eXL9gjMzs600FnVjSWcAxwPTIyKKisPMzPIpJGFIOhY4D3h3RKwpIgYzMxuYenSrvQy4DThQ0nJJs4F/ByYA10i6R9IPax2HmZkNTs1LGBFxSoXdP6n1fc3MrLqGQi8pMzMbBpwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwzM8vFCcPMzHJxwjAzs1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwzM8vFCcPMzHKpecKQtEDSCkn3Z/ZNlHSNpEfTn7vUOg4zMxucepQwLgaOLdt3PrA4IvYHFqevzcxsW7S3w9SpMGpU8rO9vSa3qXnCiIibgFVlu08CFqbbC4EP1joOM7MRpb0dmptBgpkzobMTIqCzk+4zP8qfj/kCsw+4mX1GP1m1WxbVhrFbRDyTbj8L7NbbiZLaJHVI6ujq6qpPdGZmQ1l7O8yaBStXVjw8at1a9lq8kCsfezOHTH6qarctvNE7IgKIPo7Pj4jWiGidPHlyHSMzMytQqZpJgsbG5Gda3dR9/lzYuLHPt7ewjOfXTeDKpw+vWkhFJYznJO0OkP5cUVAcZmZDQ7Ydork5KUF0dibHNm9OfnZ2sm7mmbC8/2omtUyhoamhqiEWlTB+A5yebp8O/P+C4jAzK0ZvCSIiqWrqpQQxhnVEf7+6x46Fr3616iHXo1vtZcBtwIGSlkuaDXwdeK+kR4Fj0tdmZiNbtprp1FNzJYhKGuiG0aMrH5w0CebPhxkzqhNzRmPVr1gmIk7p5dD0Wt/bzKxQ7e0wbx4sWwYTJ8LLL29JDNFr023/WlqSEsQ552xp+J40CS64oCaJoqTmCcPMbLvR3r71L/GsXno0DVipumnGjJomh0oK7yVlZjYitLfDRz5SvcTQ1JSUGgAa0sbrlpaaVTfl4YRhZpZHP6Opu8+fCxs2bPv1SwlCShLDggXw/PNJ1dWmTcnPpUsLSxbgKikzs761t8NZZ8Hq1Vv2dXYSs2bx6HVPcvmyw1ncsRPXvrh84NeWkkRQapMoMBnk4RKGmRlsPdWGlGzPmZNUM2WTRUobN7LLgm/xv6+dxgvrd+TFpl37v0d5KeLSS4dEySEvJwwzs0rtDytXEhdd1Gc1UzMr6Xp4JXevOYiJC76dJITeZKuZuruHTZLIcsIws+1Ttk3i9NMrJgb1cwkBzQemDdMzZiQJodRQDcn2okXDqhTRFycMMxu5emuobm+HtrYtA+dKU28MVDY5QJIQSg3VEcn2ME8SWW70NrORqZQU1qxJXnd2Eh/9KPdeuYQpV/2AiRvWDO76jY3JQLntiEsYZjZ89dXVdd68LckipbVr2fnKn7DzhpzznY4eDWef3bMkMWkSXHzxiCo95OEShpkNP5VGVHd2Em1tLLnpKS7vPJTzOpdV/Iu4hWXEXntVnvF11KikQRq2nmrjwgtr8jGGG5cwzGzomzNny5oQDQ1w2mkVR1RrzRoa5l/I3D9N41l2r3gptUxh1Ne/lkyxkTV2LFxyyYhtf6gGJwwzG9rmzIGLLtrSMN3dvaUUUEELy3j2vi72WPSNykmhNEBu/vykq2tpTESBU24MF4rBzJhYZ62trdHR0VF0GGZWR9HQiLoH0IuppSXpwgpbzxY7ZcqwGE1dC5LujIjWwV7HJQwzq79sFVNjY/I69WLnS/x67h184q038sYxj8FAkkX5wkEzZiTJY5gOlBtq3OhtZvVVqmIq2byZuOgibvvl05yz+l+5a82BdHMYO7KGoyY9SKwfhei9Cuo1dVgPYnvnEoaZ1U6Fbq8xf36P0wQcuuIqxjRu5J/ffTM3fu9eXnipgT8938qos/+x8rXHjdvS/rBokRup68AlDDOrjvL2guOOg4ULtxo4t27mmexA5SqmBjZz80tv7Xmg1KV1/vyk4buhIRmQ566udedGbzMbuPJxEOPGJXMxZdalDirPxdTbfhoaknUfrOqq1ejtEoaZDUxpZtfsZH2vvtrjtN4m7ut1Qr+2tsFGZjVWaBuGpE9JekDS/ZIukzSmyHjMrEylqTfmzRvcynItLcl0G6VlRxsakteuYhryCksYkvYEPgG0RsSbgAbg5KLiMdvulSeHOXO2ntG1s5MNp86mu7Mz/zVVVp4odXu98MIty45u2uRkMUwU3UuqEdhRUiMwFni64HjMtg/l4yCOOaZHcoiLLuoxeV9TrCfy/toYOzZZ2tSjqUeMwtowIuIpSd8ClgFrgasj4ury8yS1AW0AU6ZMqW+QZiNFtgfT2LFbtzls3kwsXtyjbaG3toYGupOV5cqrpRoaYOedYdWq7XpU9UhWZJXULsBJwD7AHsA4STPLz4uI+RHRGhGtkydPrneYZsNXdo3qmTO3lB4G0EBdUWmp0fKV5RYuHNbLj1r/iqySOgZ4IiK6ImIjcCVwRIHxmA1v2TaI5uZk2dEKM7oOSG9tECN8ZTmrrMiEsQw4XNJYSQKmAw8VGI/Z8NPeDuPH9yxFrFy57cuOlrgNwsoU2YZxh6QrgLuATcDdQM85A8yssvb2pBQx2MRQMn06PPbYdj+zq/Wu0F5SEfGFiDgoIt4UEadGxPoi4zEbcrLtEFKynS5D2n3+3Ooki9I4iGuv9cyu1qd+E4akayRVmODFzGqqNKI62w6xciWbZp7Ol3b8Gixfvu3XHj06mbDP4yBsAPKUMD4LfFfSTyVVXvPQzAauvPQwfnzyOh04F+ecU3FEdSObaVv/fV5q2rX/e4wbt2V7VPq/e0sL/PSnLkHYgPWbMCLirog4GrgK+KOkL0jasfahmY1gc+YkjdTZ0sOrryav04FzffVw2p1n2WXBt7dMr1HJ2WfD6tVbejJt3pz8dHWTbaNcbRhpL6ZHgIuAjwOPSjq1loGZjQjlpYhSSSK7gFAv+hwbMWVK8kt/4cKtSxGQvF60yNVMVnX99pKSdAvJ4LoHgNuBM4CHgXMkHRURnmLSrKS9PemKunp17+dUGDg3IKNHb1mGdMYMlxasbvJ0q20DHoyeC2d8XJLHTZiVpt0YyKR8eZVGU5eqp7wMqRWo34QREQ/0cfgDVYzFbOgrX1Vuv/3guuuStoFqGzvWycGGlEEN3IuIJdUKxGzIa29PZnTNLDkanZ0Dm4epL01NMGGCJ++zIavo6c3Nhq7y9SHOOafHdN/bnCxGjUqql0pTbixaBOvXe/I+G9K8RKtZJZVKEwwiQWS5HcKGKScMszJP3vE0u5x1LuOrUZqQkl5T7uJqI4CrpGz70N4OY8ZsPR7imGMA6HroeS7/9G2cdfBNHND0BFMO34Oxq1ds232kZMBcabBcd7eThY0YLmHYyNfeDqee2qMnUyxezG06gndxK9DMBF7m3bs+zNnv7GTTLbvT9HyFFYMnTUoG3pV6SR13HPz+957h1bYLThg2MmTHQjQ0JNNgtLTAV79K9/lzGVWh26uAd3Ib/3LMDUz/cDOtMw+iccyh6fW+sXUbBribq2331HM83tDV2toaHR0dRYdhQ0F7e9JrqZ8V5dYxhibW9V332tv/A+VjLlx6sGFK0p0R0TrY67iEYcNDtgQxalTSNpDDGNaxmVFAvvO34mk3zLbiRm8bukrjIKSkDaI09UbOZFHSQHfPtalLpk8fXIxm2xEnDBsasmtTS0kp4rTTtiSJwVSdtrTApZfCDjtsvX/69GSVOTPLxQnDipEdRd3cnKwNkZ3FtdQldbDGjt3S9rBu3ZburhFOFmYD5IRh9VcaRd3Zmfzi7qfhesBK1U8tLTB/vtshzKqk0IQhaWdJV0h6WNJDkt5ZZDxWA+XzMZUar8tGUVfFpEnJnEzd3V5ZzqwGiu4ldQHwx4j475KagLEFx2PVVGE+po2nzaaxe/22T7MR0WOchZOCWX0UljAk7QT8PckKfkTEBqDnivc2LEV3sPFT59FUVpIY3b2eTTTQyOZ8Fxo3Lkk4HgdhVrgiq6T2AbqAn0q6W9KPJY0rP0lSm6QOSR1dXV31j9JyW/rn5SyYdTMzpt7CHqNX0Nj1TMXzGtmcNEZnNTUl/0pGjUrmZFq92tN9mw0RRSaMRuDtwEURcQjwKnB++UkRMT8iWiOidfLkyfWO0frw3P1d/OwTt/LRg27i9aM72eeovZh98VEsfnJ/jt77MV4dt2vlN5Yao1tatqwHsWBBsh5EqQfT5s2etM9siCmyDWM5sDwi7khfX0GFhGFDx0vLXuLG//swi3+3juse3oP71+8PTGYnXmLa6x7mnCOWMv20PTn4hNejUbtC+7crz8dUqlpyicFsWCksYUTEs5KelHRgRDwCTAceLCqe7VYf8yWtXbWWW378EIt/9TLX3ddMx6tvoJvD2JE1HDnxIWZMu4Hpp+zKIR8+gMYxh/W8dikheD4msxGh0MkHJb0N+DHQBCwBZkXEC72d78kHBymbHMaNS9oHymxuGsNv9juX7z39IW598WA2sAONbOTQCQ8x/W2rmP6hnTl81hvY4e92qHADMxuKRsTkgxFxDzDoD2E5lHdxrZAsABo2rOOQBy/lpR1n8PF33Mb0E8dx5JkHMWGPt9QxWDMbiooeh2H1MoDBci16krvWvAF4Q21jMrNhxVODjASVRlOnlv/HM1zS9meic1nuy2nKlOrHaGbDnksYw12F0dSbzpjNJec9zNe7PsKjG/cBdudo9mRvlvd/vVIvJjOzMi5hDHXt7clsrtlpv6XXShLdn53bo6qpcdN6jnl6IQfs0sW3T7yBe37+CHte8rWeg+XKjR/vyfrMrFcuYQxFfS0/WurV1tnJ2plnsgPrKl5iby3nqucOzew5EEZp615Sr766ZW6mtjYPlDOzPjlhDAXZ5UdLE+zlsGMfy49WbIfwYDkzGwRXSRUlu/zozJnbvLJcA909q5rcDmFmNeCEUU+9rVE9GJXmZXI7hJnVgKuk6qW8N1M1Rth7XiYzqyOXMKqtwpiIF554kTVz/qk6q8x5+VEzK4gTxrYo7+ra3JzsK1+rOu3J9LF9r2LMy89t+/1KS49GePlRMytMoZMPDtSQmHywvR1mzYKNG7fa3d0wmrXsyLjNL/d4y4s77MaYsaMY80LlBYUq8vKjZlYl1Zp80CWMSvqYaoN583okC4BRmzcytkKyANh5wwrGfP+bPXszlaqXGhqSny0tW0oSLkGY2RDjRu+S3sZCdHYSbW08ffcKfr3s7ZzduWzgWXbKFK8NYWbD3vZdJTWAAXNLaWEflvIke7EXT1U+adIkWLu25wpzbpw2swK5Smqwsg3U0G831xaW8fj1y9jz0v8Do0f3PKGpCS64wGMizGzEGvlVUuXzMk2aBBdcQPdn5zJqAN1c1TKFfadNAWaAqHjN1xKDE4SZjUAju0qqlx5NGxhNIxvzF69crWRmw5irpPqxcc1G1n3iMxV7NDWxkejvo3uAnJnZVoZXwli1qvKAOaB7Uzf3/uIR/u2kGzh+t78wadxamlY92+ulKk7al00Sl17q7q1mZhmFJwxJDZLulnRVvycvXbr1GhErV7L5tFlcMPGL7Na0ird9+EDO/c00Hn1hMjMOvoe1E3bt/VqVJu1zkjAz61XhbRiSPg20An8XEcf3dW6rFJVaMJazJ/P2/RnTp8PRs/dl78P2SA700oZBUxMsWOCkYGbbhWq1YRTaS0rSXsAHgK8Cn97W6+ypp1n4+JE9D5QSQl89mszMLJeiu9V+FzgPmNDbCZLagDaAd/R2TqXV5Uo89beZWVUU1oYh6XhgRUTc2dd5ETE/IlojovW1RumspiavLmdmVgdFNnq/CzhR0lLgZ8B7JC3q8x1TpyZVSiWTJrktwsysTgqrkoqIucBcAEnTgH+KiJl9vmniRFiypOaxmZlZT4V3qzUzs+Gh6EZvACLiBuCGgsMwM7M+uIRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrk4YZiZWS5OGGZmlosThpmZ5eKEYWZmuThhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrk4YZiZWS5OGGZmlosThpmZ5VJYwpC0t6TrJT0o6QFJ5xQVi5mZ9a+xwHtvAs6NiLskTQDulHRNRDxYYExmZtaLwkoYEfFMRNyVbr8CPATsWVQ8ZmbWtyHRhiFpKnAIcEeFY22SOiR1dHV11T02MzNLFJ4wJI0Hfgl8MiJeLj8eEfMjojUiWidPnlz/AM3MDCg4YUgaTZIs2iPiyiJjMTOzvhXZS0rAT4CHIuLfiorDzMzyKbKE8S7gVOA9ku5J/x1XYDxmZtaHwrrVRsSfARV1fzMzG5jCG73NzGx4cMIwM7NcnDDMzCwXJwwzM8vFCcPMzHJxwjAzs1ycMMzMLBcnDDMzy8UJw8zMcnHCMDOzXJwwzMwsFycMMzPLxQnDzMxyccIwM7NcnDDMzCwXJwwzM8vFCcPMzHJxwjAzs1ycMMzMLBcnDDMzy6XQhCHpWEmPSHpM0vlFxmJmZn0rLGFIagB+ALwfOBg4RdLBRcVjZmZ9K7KEcSjwWEQsiYgNwM+AkwqMx8zM+tBY4L33BJ7MvF4OHFZ+kqQ2oC19uV7S/XWIbbCageeLDiIHx1k9wyFGcJzVNlziPLAaFykyYeQSEfOB+QCSOiKiteCQ+uU4q2s4xDkcYgTHWW3DKc5qXKfIKqmngL0zr/dK95mZ2RBUZML4D2B/SftIagJOBn5TYDxmZtaHwqqkImKTpI8BfwIagAUR8UA/b5tf+8iqwnFW13CIczjECI6z2rarOBUR1biOmZmNcB7pbWZmuThhmJlZLkMmYfQ3TYikHST9PD1+h6SpmWNz0/2PSHpfgTF+WtKDkv4qabGklsyxzZLuSf/VtHE/R5xnSOrKxHNm5tjpkh5N/51ecJzfycT4N0kvZo7V5XlKWiBpRW/jf5T4XvoZ/irp7Zlj9XyW/cU5I43vPkm3Snpr5tjSdP891ep+OYg4p0l6KfPf9vOZY3WbSihHnJ/JxHh/+n2cmB6ry/OUtLek69PfOQ9IOqfCOdX9fkZE4f9IGr0fB/YFmoB7gYPLzpkD/DDdPhn4ebp9cHr+DsA+6XUaCorxaGBsun12Kcb09eoh9CzPAP69wnsnAkvSn7uk27sUFWfZ+R8n6RhR7+f598Dbgft7OX4c8AdAwOHAHfV+ljnjPKJ0f5LpeO7IHFsKNA+R5zkNuGqw35dax1l27gnAdfV+nsDuwNvT7QnA3yr8v17V7+dQKWHkmSbkJGBhun0FMF2S0v0/i4j1EfEE8Fh6vbrHGBHXR8Sa9OXtJGNL6m0wU668D7gmIlZFxAvANcCxQyTOU4DLahRLryLiJmBVH6ecBFwSiduBnSXtTn2fZb9xRsStaRxQ3Hczz/PsTV2nEhpgnEV9N5+JiLvS7VeAh0hm0Miq6vdzqCSMStOElH/w186JiE3AS8CknO+tV4xZs0kye8kYSR2Sbpf0wRrEV5I3zg+lRdQrJJUGUNbrWQ7oXmnV3j7AdZnd9Xqe/entc9TzWQ5U+XczgKsl3alkKp6ivVPSvZL+IOmN6b4h+TwljSX5RfvLzO66P08lVfSHAHeUHarq93PITw0yHEmaCbQC787sbomIpyTtC1wn6b6IeLyYCPktcFlErJf0jyQlt/cUFEseJwNXRMTmzL6h9DyHDUlHkySMIzO7j0yf5a7ANZIeTv/CLsJdJP9tV0s6Dvg1sH9BseRxAnBLRGRLI3V9npLGkySsT0bEy7W6DwydEkaeaUJeO0dSI7ATsDLne+sVI5KOAeYBJ0bE+tL+iHgq/bkEuIHkr4Fa6DfOiFiZie3HwDvyvreecWacTFmRv47Psz+9fY4hN/WNpLeQ/Pc+KSJWlvZnnuUK4FfUpko3l4h4OSJWp9u/B0ZLamYIPs9UX9/Nmj9PSaNJkkV7RFxZ4ZTqfj9r3TCTs/GmkaTRZR+2NGi9seyc/8XWjd6/SLffyNaN3kuoTaN3nhgPIWmY279s/y7ADul2M/AoNWqwyxnn7pntfwBujy0NYU+k8e6Sbk8sKs70vINIGhFVxPNM7zGV3htpP8DWjYp/qfezzBnnFJL2vSPK9o8DJmS2bwWOLTDO15X+W5P8ol2WPttc35d6xZke34mknWNcEc8zfS6XAN/t45yqfj9r9rC34cMfR9LK/zgwL933ZZK/1AHGAJenX/q/APtm3jsvfd8jwPsLjPFa4DngnvTfb9L9RwD3pV/y+4DZBT/LrwEPpPFcDxyUee9H0mf8GDCryDjT118Evl72vro9T5K/Hp8BNpLU884GzgLOSo+LZCGwx9NYWgt6lv3F+WPghcx3syPdv2/6HO9NvxPzCo7zY5nv5u1kElyl70tRcabnnEHS4Sb7vro9T5JqxQD+mvnvelwtv5+eGsTMzHIZKm0YZmY2xDlhmJlZLk4YZmaWixOGmZnl4oRhZma5OGGYmVkuThhmZpaLE4bZIKTrEbw33f4XSd8vOiazWvHkg2aD8wXgy+lEc4cAJxYcj1nNeKS32SBJuhEYD0yLZF0CsxHJVVJmgyDpzSQrn21wsrCRzgnDbBulK5e1k6xqtlpSzVbUMxsKnDDMtkG60tqVwLkR8RDwFZL2DLMRy20YZmaWi0sYZmaWixOGmZnl4oRhZma5OGGYmVkuThhmZpaLE4aZmeXihGFmZrn8JxI0W4aibDePAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -2854,7 +2583,7 @@ "\n", "n = 100\n", "x = 2*np.random.rand(n,1)\n", - "y = 4+3*x+np.random.randn(n,1)\n", + "y = 4+3*x#+np.random.randn(n,1)\n", "\n", "X = np.c_[np.ones((n,1)), x]\n", "XT_X = X.T @ X\n", @@ -2916,10 +2645,8 @@ }, { "cell_type": "markdown", - "id": "709dd39d", - "metadata": { - "editable": true - }, + "id": "99a3e2be", + "metadata": {}, "source": [ "## Replace or not\n", "\n", @@ -2931,10 +2658,8 @@ }, { "cell_type": "markdown", - "id": "41a56c5f", - "metadata": { - "editable": true - }, + "id": "12ad19f6", + "metadata": {}, "source": [ "## Momentum based GD\n", "\n", @@ -2946,10 +2671,8 @@ }, { "cell_type": "markdown", - "id": "84a39002", - "metadata": { - "editable": true - }, + "id": "529fe4b5", + "metadata": {}, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", @@ -2958,10 +2681,8 @@ }, { "cell_type": "markdown", - "id": "d8367a57", - "metadata": { - "editable": true - }, + "id": "6f31d112", + "metadata": {}, "source": [ "\n", "
\n", @@ -2976,10 +2697,8 @@ }, { "cell_type": "markdown", - "id": "4310b258", - "metadata": { - "editable": true - }, + "id": "214661ae", + "metadata": {}, "source": [ "where we have introduced a momentum parameter $\\gamma$, with\n", "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", @@ -2995,10 +2714,8 @@ }, { "cell_type": "markdown", - "id": "71b3eed4", - "metadata": { - "editable": true - }, + "id": "157ac0bc", + "metadata": {}, "source": [ "$$\n", "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", @@ -3007,20 +2724,16 @@ }, { "cell_type": "markdown", - "id": "2b06dee9", - "metadata": { - "editable": true - }, + "id": "0e4d8eb6", + "metadata": {}, "source": [ "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." ] }, { "cell_type": "markdown", - "id": "e9e8f2ef", - "metadata": { - "editable": true - }, + "id": "5030a332", + "metadata": {}, "source": [ "## More on momentum based approaches\n", "\n", @@ -3033,10 +2746,8 @@ }, { "cell_type": "markdown", - "id": "e52cd8aa", - "metadata": { - "editable": true - }, + "id": "1e3162a0", + "metadata": {}, "source": [ "$$\n", "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", @@ -3045,20 +2756,16 @@ }, { "cell_type": "markdown", - "id": "94b41623", - "metadata": { - "editable": true - }, + "id": "39341fc6", + "metadata": {}, "source": [ "We can discretize this equation in the usual way to get" ] }, { "cell_type": "markdown", - "id": "6bb769bb", - "metadata": { - "editable": true - }, + "id": "4a925447", + "metadata": {}, "source": [ "$$\n", "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", @@ -3067,20 +2774,16 @@ }, { "cell_type": "markdown", - "id": "f8d21536", - "metadata": { - "editable": true - }, + "id": "5bc8857b", + "metadata": {}, "source": [ "Rearranging this equation, we can rewrite this as" ] }, { "cell_type": "markdown", - "id": "fed1c380", - "metadata": { - "editable": true - }, + "id": "3f3a3dca", + "metadata": {}, "source": [ "$$\n", "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", @@ -3089,10 +2792,8 @@ }, { "cell_type": "markdown", - "id": "764ef6b3", - "metadata": { - "editable": true - }, + "id": "a1e7d5d2", + "metadata": {}, "source": [ "## Momentum parameter\n", "\n", @@ -3105,10 +2806,8 @@ }, { "cell_type": "markdown", - "id": "aae9659a", - "metadata": { - "editable": true - }, + "id": "258f174a", + "metadata": {}, "source": [ "$$\n", "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", @@ -3117,10 +2816,8 @@ }, { "cell_type": "markdown", - "id": "0efeaa52", - "metadata": { - "editable": true - }, + "id": "9405179c", + "metadata": {}, "source": [ "Thus, as the name suggests, the momentum parameter is proportional to\n", "the mass of the particle and effectively provides inertia.\n", @@ -3150,10 +2847,8 @@ }, { "cell_type": "markdown", - "id": "3ef43651", - "metadata": { - "editable": true - }, + "id": "42647548", + "metadata": {}, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", @@ -3162,10 +2857,8 @@ }, { "cell_type": "markdown", - "id": "137df2fd", - "metadata": { - "editable": true - }, + "id": "fa8f3d45", + "metadata": {}, "source": [ "\n", "
\n", @@ -3180,20 +2873,16 @@ }, { "cell_type": "markdown", - "id": "eede99a9", - "metadata": { - "editable": true - }, + "id": "757a5a92", + "metadata": {}, "source": [ "One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$." ] }, { "cell_type": "markdown", - "id": "fd619ef7", - "metadata": { - "editable": true - }, + "id": "ec07c646", + "metadata": {}, "source": [ "## Second moment of the gradient\n", "\n", @@ -3221,10 +2910,8 @@ }, { "cell_type": "markdown", - "id": "38bcf670", - "metadata": { - "editable": true - }, + "id": "c1d51956", + "metadata": {}, "source": [ "## RMS prop\n", "\n", @@ -3236,10 +2923,8 @@ }, { "cell_type": "markdown", - "id": "f5d6f8a4", - "metadata": { - "editable": true - }, + "id": "f8511d10", + "metadata": {}, "source": [ "\n", "
\n", @@ -3254,10 +2939,8 @@ }, { "cell_type": "markdown", - "id": "66d5f5ae", - "metadata": { - "editable": true - }, + "id": "fd599d2a", + "metadata": {}, "source": [ "$$\n", "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", @@ -3266,10 +2949,8 @@ }, { "cell_type": "markdown", - "id": "c6d4d5c4", - "metadata": { - "editable": true - }, + "id": "073455c5", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", @@ -3278,10 +2959,8 @@ }, { "cell_type": "markdown", - "id": "6f6273ca", - "metadata": { - "editable": true - }, + "id": "0550cfc9", + "metadata": {}, "source": [ "where $\\beta$ controls the averaging time of the second moment and is\n", "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", @@ -3296,10 +2975,8 @@ }, { "cell_type": "markdown", - "id": "806d7b87", - "metadata": { - "editable": true - }, + "id": "a3e891ab", + "metadata": {}, "source": [ "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", "\n", @@ -3319,10 +2996,8 @@ }, { "cell_type": "markdown", - "id": "2f72bdeb", - "metadata": { - "editable": true - }, + "id": "b56d6328", + "metadata": {}, "source": [ "\n", "
\n", @@ -3337,10 +3012,8 @@ }, { "cell_type": "markdown", - "id": "58e6a172", - "metadata": { - "editable": true - }, + "id": "a496b94c", + "metadata": {}, "source": [ "$$\n", "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", @@ -3349,10 +3022,8 @@ }, { "cell_type": "markdown", - "id": "62f683ee", - "metadata": { - "editable": true - }, + "id": "0abe86a1", + "metadata": {}, "source": [ "$$\n", "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", @@ -3361,10 +3032,8 @@ }, { "cell_type": "markdown", - "id": "d39f0e06", - "metadata": { - "editable": true - }, + "id": "6d799dcf", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", @@ -3373,10 +3042,8 @@ }, { "cell_type": "markdown", - "id": "33a61b53", - "metadata": { - "editable": true - }, + "id": "437c3cef", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", @@ -3385,10 +3052,8 @@ }, { "cell_type": "markdown", - "id": "cff3a5f1", - "metadata": { - "editable": true - }, + "id": "c3752fd1", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", @@ -3397,10 +3062,8 @@ }, { "cell_type": "markdown", - "id": "63b3db72", - "metadata": { - "editable": true - }, + "id": "28e99f74", + "metadata": {}, "source": [ "\n", "
\n", @@ -3414,10 +3077,8 @@ }, { "cell_type": "markdown", - "id": "30e92488", - "metadata": { - "editable": true - }, + "id": "c69e5b35", + "metadata": {}, "source": [ "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", "second moment and are typically taken to be $0.9$ and $0.99$\n", @@ -3433,10 +3094,8 @@ }, { "cell_type": "markdown", - "id": "86e56bc8", - "metadata": { - "editable": true - }, + "id": "2207777f", + "metadata": {}, "source": [ "$$\n", "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", @@ -3445,10 +3104,8 @@ }, { "cell_type": "markdown", - "id": "560fd0ca", - "metadata": { - "editable": true - }, + "id": "a9dd6e15", + "metadata": {}, "source": [ "## Practical tips\n", "\n", @@ -3465,10 +3122,8 @@ }, { "cell_type": "markdown", - "id": "c1fdbbf2", - "metadata": { - "editable": true - }, + "id": "00b94d96", + "metadata": {}, "source": [ "## Automatic differentiation\n", "\n", @@ -3503,10 +3158,8 @@ }, { "cell_type": "markdown", - "id": "4c993089", - "metadata": { - "editable": true - }, + "id": "40e322b8", + "metadata": {}, "source": [ "$$\n", "f(x) = \\sin\\left(2\\pi x + x^2\\right)\n", @@ -3515,20 +3168,16 @@ }, { "cell_type": "markdown", - "id": "2116a487", - "metadata": { - "editable": true - }, + "id": "6ef03583", + "metadata": {}, "source": [ "which has the following derivative" ] }, { "cell_type": "markdown", - "id": "49f8e249", - "metadata": { - "editable": true - }, + "id": "a90642b5", + "metadata": {}, "source": [ "$$\n", "f'(x) = \\cos\\left(2\\pi x + x^2\\right)\\left(2\\pi + 2x\\right)\n", @@ -3537,10 +3186,8 @@ }, { "cell_type": "markdown", - "id": "adc93d64", - "metadata": { - "editable": true - }, + "id": "f13e4196", + "metadata": {}, "source": [ "Using **autograd** we have" ] @@ -3548,11 +3195,8 @@ { "cell_type": "code", "execution_count": 15, - "id": "4c28a649", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "e91017bf", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3593,10 +3237,8 @@ }, { "cell_type": "markdown", - "id": "7db36c68", - "metadata": { - "editable": true - }, + "id": "edb98a44", + "metadata": {}, "source": [ "## Using autograd\n", "\n", @@ -3610,11 +3252,8 @@ { "cell_type": "code", "execution_count": 16, - "id": "2817c98b", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "8101c103", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3638,10 +3277,8 @@ }, { "cell_type": "markdown", - "id": "19372f3b", - "metadata": { - "editable": true - }, + "id": "7925f3b4", + "metadata": {}, "source": [ "## Autograd with more complicated functions\n", "\n", @@ -3653,11 +3290,8 @@ { "cell_type": "code", "execution_count": 17, - "id": "1acc2c25", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "f2c69e0f", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3697,20 +3331,16 @@ }, { "cell_type": "markdown", - "id": "03a7e411", - "metadata": { - "editable": true - }, + "id": "0fbbc5a9", + "metadata": {}, "source": [ "Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable." ] }, { "cell_type": "markdown", - "id": "f3456894", - "metadata": { - "editable": true - }, + "id": "96d43a0a", + "metadata": {}, "source": [ "## More complicated functions using the elements of their arguments directly" ] @@ -3718,11 +3348,8 @@ { "cell_type": "code", "execution_count": 18, - "id": "695b0d60", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "f53794e7", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3746,10 +3373,8 @@ }, { "cell_type": "markdown", - "id": "7de20a87", - "metadata": { - "editable": true - }, + "id": "cee7be4d", + "metadata": {}, "source": [ "Note that in this case, when sending an array as input argument, the\n", "output from Autograd is another array. This is the true gradient of\n", @@ -3761,10 +3386,8 @@ }, { "cell_type": "markdown", - "id": "a101eb60", - "metadata": { - "editable": true - }, + "id": "df2680c1", + "metadata": {}, "source": [ "## Functions using mathematical functions from Numpy" ] @@ -3772,11 +3395,8 @@ { "cell_type": "code", "execution_count": 19, - "id": "041ae7db", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "e752fa81", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3800,10 +3420,8 @@ }, { "cell_type": "markdown", - "id": "0e2f0cc2", - "metadata": { - "editable": true - }, + "id": "317db3e8", + "metadata": {}, "source": [ "## More autograd" ] @@ -3811,11 +3429,8 @@ { "cell_type": "code", "execution_count": 20, - "id": "751f2479", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "96609851", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3836,10 +3451,8 @@ }, { "cell_type": "markdown", - "id": "d0d4e63d", - "metadata": { - "editable": true - }, + "id": "587bbcc3", + "metadata": {}, "source": [ "## And with loops" ] @@ -3847,11 +3460,8 @@ { "cell_type": "code", "execution_count": 21, - "id": "749f217e", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "c62adaa8", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3883,11 +3493,8 @@ { "cell_type": "code", "execution_count": 22, - "id": "a3f94af6", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "75db0457", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3903,10 +3510,8 @@ }, { "cell_type": "markdown", - "id": "14980c30", - "metadata": { - "editable": true - }, + "id": "c99c7e99", + "metadata": {}, "source": [ "## Using recursion" ] @@ -3914,11 +3519,8 @@ { "cell_type": "code", "execution_count": 23, - "id": "ba20ab50", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "779140f3", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3952,20 +3554,16 @@ }, { "cell_type": "markdown", - "id": "6ceaf1b5", - "metadata": { - "editable": true - }, + "id": "79051822", + "metadata": {}, "source": [ "Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input." ] }, { "cell_type": "markdown", - "id": "0c3672e8", - "metadata": { - "editable": true - }, + "id": "3644a47e", + "metadata": {}, "source": [ "## Unsupported functions\n", "Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.\n", @@ -3976,11 +3574,8 @@ { "cell_type": "code", "execution_count": 24, - "id": "ac76278c", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "39e8bbff", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3998,20 +3593,16 @@ }, { "cell_type": "markdown", - "id": "793c54aa", - "metadata": { - "editable": true - }, + "id": "d42d292f", + "metadata": {}, "source": [ "Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible." ] }, { "cell_type": "markdown", - "id": "fcfc92f5", - "metadata": { - "editable": true - }, + "id": "65638c5a", + "metadata": {}, "source": [ "## The syntax a.dot(b) when finding the dot product" ] @@ -4019,11 +3610,8 @@ { "cell_type": "code", "execution_count": 25, - "id": "fda8b086", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "bec4592d", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4041,10 +3629,8 @@ }, { "cell_type": "markdown", - "id": "07c039bd", - "metadata": { - "editable": true - }, + "id": "0c4b4d89", + "metadata": {}, "source": [ "Here we are told that the 'dot' function does not belong to Autograd's\n", "version of a Numpy array. To overcome this, an alternative syntax\n", @@ -4054,11 +3640,8 @@ { "cell_type": "code", "execution_count": 26, - "id": "8045a474", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "520e4b18", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4079,10 +3662,8 @@ }, { "cell_type": "markdown", - "id": "3d68f43a", - "metadata": { - "editable": true - }, + "id": "5d122ff1", + "metadata": {}, "source": [ "## Recommended to avoid\n", "The documentation recommends to avoid inplace operations such as" @@ -4091,11 +3672,8 @@ { "cell_type": "code", "execution_count": 27, - "id": "384186c3", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "7f4d866e", + "metadata": {}, "outputs": [], "source": [ "a += b\n", @@ -4106,10 +3684,8 @@ }, { "cell_type": "markdown", - "id": "e278c944", - "metadata": { - "editable": true - }, + "id": "d66130d2", + "metadata": {}, "source": [ "## Using Autograd with OLS\n", "\n", @@ -4121,11 +3697,8 @@ { "cell_type": "code", "execution_count": 28, - "id": "037a0df8", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "59144e98", + "metadata": {}, "outputs": [], "source": [ "# Using Autograd to calculate gradients for OLS\n", @@ -4181,10 +3754,8 @@ }, { "cell_type": "markdown", - "id": "7f217252", - "metadata": { - "editable": true - }, + "id": "2b5ef0aa", + "metadata": {}, "source": [ "## Including Stochastic Gradient Descent with Autograd\n", "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**." @@ -4193,11 +3764,8 @@ { "cell_type": "code", "execution_count": 29, - "id": "4f9696c6", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "db6e0b30", + "metadata": {}, "outputs": [], "source": [ "# Using Autograd to calculate gradients using SGD\n", @@ -4277,10 +3845,8 @@ }, { "cell_type": "markdown", - "id": "31965b63", - "metadata": { - "editable": true - }, + "id": "e1b541fa", + "metadata": {}, "source": [ "## And Logistic Regression" ] @@ -4288,11 +3854,8 @@ { "cell_type": "code", "execution_count": 30, - "id": "9d920e06", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "08d9299d", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4332,10 +3895,8 @@ }, { "cell_type": "markdown", - "id": "e698f20a", - "metadata": { - "editable": true - }, + "id": "4860575d", + "metadata": {}, "source": [ "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n", "\n", @@ -4351,11 +3912,8 @@ { "cell_type": "code", "execution_count": 31, - "id": "1390dd29", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "fccac790", + "metadata": {}, "outputs": [], "source": [ "import jax.numpy as jnp\n", @@ -4370,7 +3928,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.14" + } + }, "nbformat": 4, "nbformat_minor": 5 }