diff --git a/doc/HandWrittenNotes/2022/NotesSep302022.pdf b/doc/HandWrittenNotes/2022/NotesSep302022.pdf new file mode 100644 index 000000000..70cc8d7ec Binary files /dev/null and b/doc/HandWrittenNotes/2022/NotesSep302022.pdf differ diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index a4cfff437..d9ad75146 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "9b7db121", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "f1671d88", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -30,9 +26,7 @@ { "cell_type": "markdown", "id": "d8fe7be9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plan for week 39\n", "\n", @@ -56,9 +50,7 @@ { "cell_type": "markdown", "id": "bb4365c9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -77,9 +69,7 @@ { "cell_type": "markdown", "id": "426ca5ed", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -94,9 +84,7 @@ { "cell_type": "markdown", "id": "b42f8fa9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -109,9 +97,7 @@ { "cell_type": "markdown", "id": "2c9b3ece", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] @@ -119,9 +105,7 @@ { "cell_type": "markdown", "id": "308a0e16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations to solve\n", "\n", @@ -135,9 +119,7 @@ { "cell_type": "markdown", "id": "c31bc92a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -147,9 +129,7 @@ { "cell_type": "markdown", "id": "a627316d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -158,9 +138,7 @@ { "cell_type": "markdown", "id": "89c7fb27", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -170,9 +148,7 @@ { "cell_type": "markdown", "id": "88cafcf5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This defines what is called the Hessian matrix." ] @@ -180,9 +156,7 @@ { "cell_type": "markdown", "id": "b6fd7c9c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -194,9 +168,7 @@ { "cell_type": "markdown", "id": "2669fb99", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -206,9 +178,7 @@ { "cell_type": "markdown", "id": "70518a94", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or in matrix form as" ] @@ -216,9 +186,7 @@ { "cell_type": "markdown", "id": "62d2845c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -228,9 +196,7 @@ { "cell_type": "markdown", "id": "5c8b6f2f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -240,9 +206,7 @@ { "cell_type": "markdown", "id": "95636683", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -260,9 +224,7 @@ { "cell_type": "markdown", "id": "a0f23f0e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations\n", "\n", @@ -276,9 +238,7 @@ { "cell_type": "markdown", "id": "b5808734", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -292,9 +252,7 @@ { "cell_type": "markdown", "id": "20900d60", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -303,9 +261,7 @@ { "cell_type": "markdown", "id": "3c036156", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -315,9 +271,7 @@ { "cell_type": "markdown", "id": "bffe992a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "yielding" ] @@ -325,9 +279,7 @@ { "cell_type": "markdown", "id": "a9299a85", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -337,9 +289,7 @@ { "cell_type": "markdown", "id": "ff34f290", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] @@ -347,9 +297,7 @@ { "cell_type": "markdown", "id": "84912a6e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -359,9 +307,7 @@ { "cell_type": "markdown", "id": "0881b8b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple geometric interpretation\n", "\n", @@ -381,9 +327,7 @@ { "cell_type": "markdown", "id": "c79f5156", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Extending to more than one variable\n", "\n", @@ -394,9 +338,7 @@ { "cell_type": "markdown", "id": "fdb0735d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -407,9 +349,7 @@ { "cell_type": "markdown", "id": "ae2b1267", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we Taylor expand to obtain" ] @@ -417,9 +357,7 @@ { "cell_type": "markdown", "id": "9842f857", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -435,9 +373,7 @@ { "cell_type": "markdown", "id": "efb235b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] @@ -445,9 +381,7 @@ { "cell_type": "markdown", "id": "71cf2200", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -460,9 +394,7 @@ { "cell_type": "markdown", "id": "dfbdf302", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we can rephrase Newton's method as" ] @@ -470,9 +402,7 @@ { "cell_type": "markdown", "id": "bbfb119f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -484,9 +414,7 @@ { "cell_type": "markdown", "id": "7e9002d9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined" ] @@ -494,9 +422,7 @@ { "cell_type": "markdown", "id": "5dd52f69", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -508,9 +434,7 @@ { "cell_type": "markdown", "id": "df62542f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -523,9 +447,7 @@ { "cell_type": "markdown", "id": "dca132d7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent\n", "\n", @@ -540,9 +462,7 @@ { "cell_type": "markdown", "id": "77fe3e13", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -552,9 +472,7 @@ { "cell_type": "markdown", "id": "a4e6dc6b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -566,9 +484,7 @@ { "cell_type": "markdown", "id": "f5fe2fa6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on Steepest descent\n", "\n", @@ -581,9 +497,7 @@ { "cell_type": "markdown", "id": "951106f4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -593,9 +507,7 @@ { "cell_type": "markdown", "id": "8cc0b2d3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -604,9 +516,7 @@ { "cell_type": "markdown", "id": "8681dec5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The ideal\n", "\n", @@ -632,9 +542,7 @@ { "cell_type": "markdown", "id": "e05315b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -654,9 +562,7 @@ { "cell_type": "markdown", "id": "40b35380", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex functions\n", "\n", @@ -676,9 +582,7 @@ { "cell_type": "markdown", "id": "67b75c7f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex function\n", "\n", @@ -688,9 +592,7 @@ { "cell_type": "markdown", "id": "6b773fb4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conditions on convex functions\n", "\n", @@ -725,9 +627,7 @@ { "cell_type": "markdown", "id": "486bb678", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on convex functions\n", "\n", @@ -753,9 +653,7 @@ { "cell_type": "markdown", "id": "2b7a2bd1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Some simple problems\n", "\n", @@ -783,9 +681,7 @@ { "cell_type": "markdown", "id": "bc478b05", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Standard steepest descent\n", "\n", @@ -803,9 +699,7 @@ { "cell_type": "markdown", "id": "7b297f5c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -815,9 +709,7 @@ { "cell_type": "markdown", "id": "f6d83262", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the iterative process we end up with a problem like" ] @@ -825,9 +717,7 @@ { "cell_type": "markdown", "id": "8848ed37", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -837,9 +727,7 @@ { "cell_type": "markdown", "id": "c1616654", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", @@ -849,9 +737,7 @@ { "cell_type": "markdown", "id": "a6e6a0f0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient method\n", "\n", @@ -861,9 +747,7 @@ { "cell_type": "markdown", "id": "9cd06502", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -873,9 +757,7 @@ { "cell_type": "markdown", "id": "90ae67c8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite." @@ -884,9 +766,7 @@ { "cell_type": "markdown", "id": "750282d6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent method\n", "\n", @@ -897,9 +777,7 @@ { "cell_type": "markdown", "id": "b3f33101", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -909,9 +787,7 @@ { "cell_type": "markdown", "id": "6f8d78d8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or consider the system" ] @@ -919,9 +795,7 @@ { "cell_type": "markdown", "id": "368212e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -931,9 +805,7 @@ { "cell_type": "markdown", "id": "5aa9e02d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "instead." ] @@ -941,9 +813,7 @@ { "cell_type": "markdown", "id": "e79412dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -952,9 +822,7 @@ { "cell_type": "markdown", "id": "208ec5a4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -964,9 +832,7 @@ { "cell_type": "markdown", "id": "edd640fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -976,9 +842,7 @@ { "cell_type": "markdown", "id": "45ad3912", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -988,9 +852,7 @@ { "cell_type": "markdown", "id": "859cc5e1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." @@ -999,9 +861,7 @@ { "cell_type": "markdown", "id": "dcea544e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expressions\n", "We can compute the residual iteratively as" @@ -1010,9 +870,7 @@ { "cell_type": "markdown", "id": "c0633405", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1022,9 +880,7 @@ { "cell_type": "markdown", "id": "a439d140", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which equals" ] @@ -1032,9 +888,7 @@ { "cell_type": "markdown", "id": "66790308", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -1044,9 +898,7 @@ { "cell_type": "markdown", "id": "8217e1bf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or" ] @@ -1054,9 +906,7 @@ { "cell_type": "markdown", "id": "429cfb67", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -1066,9 +916,7 @@ { "cell_type": "markdown", "id": "a8f75971", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives" ] @@ -1076,9 +924,7 @@ { "cell_type": "markdown", "id": "d16e21f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -1088,9 +934,7 @@ { "cell_type": "markdown", "id": "684202b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "leading to the iterative scheme" ] @@ -1098,9 +942,7 @@ { "cell_type": "markdown", "id": "9f7ed189", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_{k},\n", @@ -1110,9 +952,7 @@ { "cell_type": "markdown", "id": "a4e5089e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent example" ] @@ -1121,10 +961,7 @@ "cell_type": "code", "execution_count": 1, "id": "ccc7aacb", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -1154,9 +991,7 @@ { "cell_type": "markdown", "id": "205c783b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "And then as countor plot" ] @@ -1165,10 +1000,7 @@ "cell_type": "code", "execution_count": 2, "id": "9f418efd", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1179,9 +1011,7 @@ { "cell_type": "markdown", "id": "041a6dd4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Find guesses" ] @@ -1190,10 +1020,7 @@ "cell_type": "code", "execution_count": 3, "id": "2c92d8f7", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1203,9 +1030,7 @@ { "cell_type": "markdown", "id": "681d14f6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Run it!" ] @@ -1214,10 +1039,7 @@ "cell_type": "code", "execution_count": 4, "id": "7be438b5", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1232,9 +1054,7 @@ { "cell_type": "markdown", "id": "8efb0322", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "What happened?" ] @@ -1243,10 +1063,7 @@ "cell_type": "code", "execution_count": 5, "id": "73a6827e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1258,9 +1075,7 @@ { "cell_type": "markdown", "id": "03e91eb1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that we did only one iteration here. We can easily add more using our previous guesses." ] @@ -1268,9 +1083,7 @@ { "cell_type": "markdown", "id": "ac4293db", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", @@ -1282,9 +1095,7 @@ { "cell_type": "markdown", "id": "371a0891", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1294,9 +1105,7 @@ { "cell_type": "markdown", "id": "778382e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1305,9 +1114,7 @@ { "cell_type": "markdown", "id": "efec93a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1317,9 +1124,7 @@ { "cell_type": "markdown", "id": "f0d3d6fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." @@ -1328,9 +1133,7 @@ { "cell_type": "markdown", "id": "782a3efa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" @@ -1339,9 +1142,7 @@ { "cell_type": "markdown", "id": "473ef2fe", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1351,9 +1152,7 @@ { "cell_type": "markdown", "id": "fd2f6a1a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is zero unless $i=j$." ] @@ -1361,9 +1160,7 @@ { "cell_type": "markdown", "id": "5b573e30", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", @@ -1373,9 +1170,7 @@ { "cell_type": "markdown", "id": "4dbf34b5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1385,9 +1180,7 @@ { "cell_type": "markdown", "id": "ae9cb823", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1397,9 +1190,7 @@ { "cell_type": "markdown", "id": "e02a8a1b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1409,9 +1200,7 @@ { "cell_type": "markdown", "id": "1b72466f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1420,9 +1209,7 @@ { "cell_type": "markdown", "id": "623a1d89", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1432,9 +1219,7 @@ { "cell_type": "markdown", "id": "6a7682f7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] @@ -1442,9 +1227,7 @@ { "cell_type": "markdown", "id": "270d0947", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1454,9 +1237,7 @@ { "cell_type": "markdown", "id": "1296bb8d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] @@ -1464,9 +1245,7 @@ { "cell_type": "markdown", "id": "4f2a2f4c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1476,9 +1255,7 @@ { "cell_type": "markdown", "id": "a1cdaf0c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1496,9 +1273,7 @@ { "cell_type": "markdown", "id": "94f0eae7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1508,9 +1283,7 @@ { "cell_type": "markdown", "id": "16dcb85d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or consider the system" ] @@ -1518,9 +1291,7 @@ { "cell_type": "markdown", "id": "8a505735", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1530,9 +1301,7 @@ { "cell_type": "markdown", "id": "274bbb55", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "instead." ] @@ -1540,9 +1309,7 @@ { "cell_type": "markdown", "id": "f23740c2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1551,9 +1318,7 @@ { "cell_type": "markdown", "id": "a0d3cb10", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1563,9 +1328,7 @@ { "cell_type": "markdown", "id": "2097ee27", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1575,9 +1338,7 @@ { "cell_type": "markdown", "id": "690435dc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1587,9 +1348,7 @@ { "cell_type": "markdown", "id": "fa980b59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", @@ -1600,9 +1359,7 @@ { "cell_type": "markdown", "id": "5ba92010", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" @@ -1611,9 +1368,7 @@ { "cell_type": "markdown", "id": "9dd09dc4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1623,9 +1378,7 @@ { "cell_type": "markdown", "id": "973d17fb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1639,9 +1392,7 @@ { "cell_type": "markdown", "id": "7440e5bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1651,9 +1402,7 @@ { "cell_type": "markdown", "id": "37f47c63", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1662,9 +1411,7 @@ { "cell_type": "markdown", "id": "1b6292a9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1674,9 +1421,7 @@ { "cell_type": "markdown", "id": "ab9a1d7d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which equals" ] @@ -1684,9 +1429,7 @@ { "cell_type": "markdown", "id": "5536e380", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1696,9 +1439,7 @@ { "cell_type": "markdown", "id": "758f15c4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or" ] @@ -1706,9 +1447,7 @@ { "cell_type": "markdown", "id": "8b262faf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1718,9 +1457,7 @@ { "cell_type": "markdown", "id": "82ff3f5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives" ] @@ -1728,9 +1465,7 @@ { "cell_type": "markdown", "id": "98208abd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1740,9 +1475,7 @@ { "cell_type": "markdown", "id": "ea471c60", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our first homework\n", "\n", @@ -1764,10 +1497,7 @@ "cell_type": "code", "execution_count": 6, "id": "71d40f6d", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1777,9 +1507,7 @@ { "cell_type": "markdown", "id": "204e4160", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1788,9 +1516,7 @@ { "cell_type": "markdown", "id": "11437293", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1800,9 +1526,7 @@ { "cell_type": "markdown", "id": "0a4bdc7e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "such that" ] @@ -1810,9 +1534,7 @@ { "cell_type": "markdown", "id": "dd893e5d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -1822,9 +1544,7 @@ { "cell_type": "markdown", "id": "497eece1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent example\n", "\n", @@ -1836,9 +1556,7 @@ { "cell_type": "markdown", "id": "6a8c35f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -1852,9 +1570,7 @@ { "cell_type": "markdown", "id": "3891b919", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The cost/loss/risk function is given by (" ] @@ -1862,9 +1578,7 @@ { "cell_type": "markdown", "id": "b9bb6c9b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -1874,9 +1588,7 @@ { "cell_type": "markdown", "id": "956f953c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] @@ -1884,9 +1596,7 @@ { "cell_type": "markdown", "id": "01f67ff6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -1896,9 +1606,7 @@ { "cell_type": "markdown", "id": "ca425e41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1910,9 +1618,7 @@ { "cell_type": "markdown", "id": "92d0d224", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $X$ is the design matrix defined above." ] @@ -1920,9 +1626,7 @@ { "cell_type": "markdown", "id": "3b47851c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -1931,9 +1635,7 @@ { "cell_type": "markdown", "id": "18712689", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1946,9 +1648,7 @@ { "cell_type": "markdown", "id": "8544cef4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] @@ -1956,9 +1656,7 @@ { "cell_type": "markdown", "id": "23e1a45b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple program\n", "\n", @@ -1968,9 +1666,7 @@ { "cell_type": "markdown", "id": "c2d44a27", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -1980,9 +1676,7 @@ { "cell_type": "markdown", "id": "485749f6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -1995,9 +1689,7 @@ { "cell_type": "markdown", "id": "7855a6d2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient Descent Example\n", "\n", @@ -2008,10 +1700,7 @@ "cell_type": "code", "execution_count": 7, "id": "ead28149", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -2065,9 +1754,7 @@ { "cell_type": "markdown", "id": "3ab38d8a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -2076,10 +1763,7 @@ "cell_type": "code", "execution_count": 8, "id": "1f02088e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Importing various packages\n", @@ -2103,9 +1787,7 @@ { "cell_type": "markdown", "id": "90cd160c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -2115,9 +1797,7 @@ { "cell_type": "markdown", "id": "f2426889", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -2127,9 +1807,7 @@ { "cell_type": "markdown", "id": "70204e4a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] @@ -2137,9 +1815,7 @@ { "cell_type": "markdown", "id": "439376b7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2151,9 +1827,7 @@ { "cell_type": "markdown", "id": "da18f4fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] @@ -2161,9 +1835,7 @@ { "cell_type": "markdown", "id": "100886dd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -2173,9 +1845,7 @@ { "cell_type": "markdown", "id": "77261942", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -2184,9 +1854,7 @@ { "cell_type": "markdown", "id": "4c4bb3a0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2199,9 +1867,7 @@ { "cell_type": "markdown", "id": "20eb20b6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -2213,9 +1879,7 @@ { "cell_type": "markdown", "id": "1b720b14", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Program example for gradient descent with Ridge Regression" ] @@ -2224,10 +1888,7 @@ "cell_type": "code", "execution_count": 9, "id": "9db2edcb", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from random import random, seed\n", @@ -2285,9 +1946,7 @@ { "cell_type": "markdown", "id": "831a479d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -2307,9 +1966,7 @@ { "cell_type": "markdown", "id": "6511842d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Improving gradient descent with momentum\n", "\n", @@ -2320,10 +1977,7 @@ "cell_type": "code", "execution_count": 10, "id": "6e519237", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from numpy import asarray\n", @@ -2386,9 +2040,7 @@ { "cell_type": "markdown", "id": "26883d81", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Same code but now with momentum gradient descent" ] @@ -2397,10 +2049,7 @@ "cell_type": "code", "execution_count": 11, "id": "ecdc701c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "from numpy import asarray\n", @@ -2471,9 +2120,7 @@ { "cell_type": "markdown", "id": "704b1d12", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Overview video on Stochastic Gradient Descent\n", "\n", @@ -2483,9 +2130,7 @@ { "cell_type": "markdown", "id": "f487bc53", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Batches and mini-batches\n", "\n", @@ -2504,9 +2149,7 @@ { "cell_type": "markdown", "id": "564faf53", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Stochastic Gradient Descent (SGD)\n", "\n", @@ -2536,9 +2179,7 @@ { "cell_type": "markdown", "id": "118060f3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2553,9 +2194,7 @@ { "cell_type": "markdown", "id": "abfe7400", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2566,9 +2205,7 @@ { "cell_type": "markdown", "id": "6612d50a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Computation of gradients\n", "\n", @@ -2579,9 +2216,7 @@ { "cell_type": "markdown", "id": "528c070a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2592,9 +2227,7 @@ { "cell_type": "markdown", "id": "07043458", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2606,9 +2239,7 @@ { "cell_type": "markdown", "id": "8ecc203a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", @@ -2628,9 +2259,7 @@ { "cell_type": "markdown", "id": "d588c576", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2643,9 +2272,7 @@ { "cell_type": "markdown", "id": "af8db60c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The gradient step\n", "\n", @@ -2655,9 +2282,7 @@ { "cell_type": "markdown", "id": "4fa5774d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2668,9 +2293,7 @@ { "cell_type": "markdown", "id": "443a1c0b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2682,9 +2305,7 @@ { "cell_type": "markdown", "id": "cfb4a964", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example code" ] @@ -2693,10 +2314,7 @@ "cell_type": "code", "execution_count": 12, "id": "a6e4c197", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -2718,9 +2336,7 @@ { "cell_type": "markdown", "id": "f72af4fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2734,9 +2350,7 @@ { "cell_type": "markdown", "id": "f3f93ce6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## When do we stop?\n", "\n", @@ -2755,9 +2369,7 @@ { "cell_type": "markdown", "id": "18b40e9b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Slightly different approach\n", "\n", @@ -2775,9 +2387,7 @@ { "cell_type": "markdown", "id": "ce2fa974", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Time decay rate\n", "\n", @@ -2794,10 +2404,7 @@ "cell_type": "code", "execution_count": 13, "id": "ba26d9e7", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -2829,9 +2436,7 @@ { "cell_type": "markdown", "id": "42f625b8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code with a Number of Minibatches which varies\n", "\n", @@ -2840,13 +2445,39 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 7, "id": "8f62bd25", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[3.76140842]\n", + " [3.2351429 ]]\n", + "Eigenvalues of Hessian Matrix:[0.26907999 4.46569763]\n", + "theta from own gd\n", + "[[3.76140842]\n", + " [3.2351429 ]]\n", + "theta from own sdg\n", + "[[3.74868945]\n", + " [3.25626547]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAApo0lEQVR4nO3deZwcdZ3/8ddnJnfCEXIfzIT7FDnyQ0Q5FJFBkEN0BSLLChjBxVXWAyQKKsZdjxVW5TAcImRkvQAvCIdOQiAEGO5AOALMJCEnhDA5J5mZz++P6gk9nT6qu6u7umfez8ejHzPdXcenamq+n299v9+qMndHREQkl5q4AxARkeqghCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhSK9lZt81s5lxxxElMzvWzJbGHYf0TUoYUlZm1mJmm8xsvZmtMLNbzWxY3HGJSG5KGBKHT7r7MOBg4BDgW/GG0zeZWb+4Y5DqooQhsXH3FcB9BIkDADO7zMxeM7N1ZvaimZ2e9N2/mdnDZvZTM3vHzN4wsxOTvt/NzOYk5n0AGJm8PjM7xcxeMLO1ZjbbzPZL+q7FzL5hZs+Z2QYzu9nMxpjZvYnlPWhmw9NtR3czkZl9zcxWmdlyM/t80vezzeyC1O1Ieu9m9iUzezWxrqvMbA8ze9TM2szs92Y2IGWdl5vZW4m4pyR9PjCxfxab2Uozu8HMBqfEeamZrQB+bWYjzexviX2yxszmmpnKBUlLB4bExswmAicCi5I+fg04CtgJ+B4w08zGJX3/AeBlgmTwY+BmM7PEd78Fnkx8dxVwbtK69gbuAL4KjALuAf6aUhCfARwP7A18ErgXuDyxvBrgP7JszthEzBOA84FrMyWYDBqAw4AjgG8CM4ApwK7AgcBZKesamVjXucAMM9sn8d2PEvEfDOyZmOaKlHl3AeqBqcDXgKUE+2RMYnt1vyBJSwlD4nC3ma0DlgCrgCu7v3D3P7j7MnfvcvffAa8ChyfN2+ruN7p7J/AbYBwwxszqgP8HfMfd2939IeCvSfN9Fvi7uz/g7luBnwKDgSOTpvmFu6909zeBucBj7v60u7cDdxE0n2WyFfi+u29193uA9cA+WaZP9SN3b3P3F4AFwP3u/rq7v0uQuFLX3b2dc4C/A/+SSJxfAC5x9zXuvg74IXBm0nxdwJWJeTcl4h4H1Cdin+u6wZxkoIQhcTjN3XcAjgX2JanpyMz+1cyeSTSRrCWoXSc3La3o/sXdNyZ+HQaMB95x9w1J07Ym/T4++b27dxEkrAlJ06xM+n1TmvfZOuffdveOpPcbc0yfKp91p9vO8QRnCUOAJ5P236zE591Wu/vmpPc/ITjDu9/MXjezy/KIWfoYJQyJTaJ2fCtBbR8zqwduBC4GRrj7zgS1bcuwiGTLgeFmNjTps7qk35cRNMOQWJcRNPe8WfgWhLaBoCDvNrbI5aXbzmXAWwTJ5QB33znx2ikxwKBbj7MHd1/n7l9z990JmuH+08yOKzI+6aWUMCRu1wDHm9nBwFCCAm01QKLj+MAwC3H3VqAZ+J6ZDTCzDxMUgN1+D5xkZseZWX+Ctvt2YF5E25HNM8CnzGyIme1J0MdRrO7tPAo4GfhD4qzpRuBqMxsNYGYTzOyETAsxs5PNbM9EAm0DOhMvke0oYUis3H01cBtBm/yLwP8AjxI0ybwPeCSPxZ1N0Cm+hqBf5Lak9bwMfA74BUFN/JMEw3u3RLAZuVwNbCHYpt8AjUUubwXwDsFZRSNwobu/lPjuUoImpvlm1gY8SPa+lL0S06wn2O/XufvsIuOTXsrUvyUiImHoDENEREJRwhARkVCUMEREJBQlDBERCaWqbj42cuRInzRpUtxhiIhUlSeffPItdx+Ve8rsqiphTJo0iebm5rjDEBGpKmbWmnuq3NQkJSIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISCglTxhmdouZrTKzBWm++7qZuZmNLHUcIiJSnHKcYdwKNKR+aGa7AscDi8sQg4iIFKnkCcPdHyJ4xnKqq4FvAnpGrIhIFYilD8PMTgHedPdnQ0w71cyazax59erVZYhORETSKXvCMLMhwDTgijDTu/sMd5/s7pNHjSr6du4iIlKgOM4w9gB2A541sxZgIvCUmY2NIRYREQmp7A9QcvfngdHd7xNJY7K7v1XuWEREJLxyDKu9A3gU2MfMlprZ+aVep4iIRK/kZxjuflaO7yeVOgYRESmervQWEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQil5wjCzW8xslZktSPrsJ2b2kpk9Z2Z3mdnOpY5DRESKU44zjFuBhpTPHgAOdPeDgFeAb5UhDhERKULJE4a7PwSsSfnsfnfvSLydD0wsdRwiIlKcSujDOA+4N9OXZjbVzJrNrHn16tVlDEtERJLFmjDMbBrQATRmmsbdZ7j7ZHefPGrUqPIFJyIiPfSLa8Vmdi5wMnCcu3tccYiISDixJAwzawAuBY5x941xxCAiIvkpx7DaO4BHgX3MbKmZnQ/8EtgBeMDMnjGzG0odh4iIFKfkZxjuflaaj28u9XpFRCRalTBKSkREqoAShohItWlshEmToKYm+NmYcaBppGIbJSUiIgVobISpU2FjYrxQa2vwHmDKlJKuWmcYIiLVZNq095JFt40bg89LTAlDRKRcomhKWrw4v88jpIQhIlIO3U1Jra3g/l5TUr5Jo64uv88jpIQhIlIOUTUlTZ8OQ4b0/GzIkODzElPCEBEph0xNRq2t+S1nyhSYMQPq68Es+DljRsk7vEEJQ0SkPLI1GY0cmV+/xpQp0NICXV3BzzIkC1DCEBEpj+nTgzOCdN5+u7h+jTJRwhARKYcpU4KkkEuZhsgWQglDRKRc6uvDTVeGIbKFUMIQkepWyttkRL3sdCOc0oliiGxS7AfB+4pfoG4NIiLVrJS3ySjFsrvnmzYtOIvYZRdYtw62bHlvmiiGyKbE3h8GFLfAgFXTw+4mT57szc3NcYchIpVi0qT0w1Lr64PRQ5W67GSNje8lkLq6IFkUm+xSYp8MNLtn6HEPTwlDRKpXTU36jmSzYMhppS671FJijyphqA9DRKpXKW+TEeMtOIpWohiVMESkepXyNhlRL7ucz7AI27meJyUMEalepbxNRpTLzvfGg8Uml0TsvmsdjrEVtuSeKTf1YYiIlFo+Heipo7MgOFsImayWPLaM+657jVn/6MeDb+7Pu+wE2JPuPrmYTYAyJAwzuwU4GVjl7gcmPtsF+B0wCWgB/sXd38m1LCUMEalK+XSg5zk6a/Pazcy94QVm/XEd9y2YwAvtewEwoWY5DXsu4oST+vEvVx9ZNQnjaGA9cFtSwvgxsMbd/9vMLgOGu/uluZalhCEiVSmfJJApuSSm98WL6Rg1jn/UfZ6ft57K7NUHsIkhDKCdo3dZQMOR6zjhvAkccOqeWE0wMMosmjOMkl+45+4PmdmklI9PBY5N/P4bYDaQM2GIiFSlT3wCrr8+/eep6urSJhcHrLUVA/qvWsZRq67m/pqR7HnQRho+NYRjLtqfoaMPizz0ZHFd6T3G3ZcDuPtyMxudaUIzmwpMBairhuFsIiKp7rkn/OfTp+NTp2JJfRhdGDX0POsYykZ+tus18GxLdHHmUPGjpNx9hrtPdvfJo0aNijscEZH8hXgO91svv80dX57HuVfU8+uNn6GDWhzooBYjQxNVmW9SGFfCWGlm4wASP1fFFIdI9SnneH6JRobWkc3Dx3LF0bP5wLAFjN53OGf/8kgGvv4iU+z/6EcnBsHPTM/RKHOrS1wJ4y/AuYnfzwX+HFMcItUl3/H8UhnSXEi3kcGct+YnTJ97FA0df2PNwPF0UcOvar/EQG/vOb/79g9fKtNzvJOVPGGY2R3Ao8A+ZrbUzM4H/hs43sxeBY5PvBfpXUpxJjBtWs/x+VDRD9yJVDH7M6azsva2dh788VN8/eoJXNr+fVqopwtjKROYOebrnH7J7rz7s5v5Xu1V7Ny+EsOxzs70C3OP5TneKTF41bwOO+wwF6kKM2e6DxniHvybB68hQ4LPi2HWc5ndL7No4q5UxezPUv0t0ujq7PJX7n/Df37GbD9p9GM+hPUO7gPY7McNf9J/clKTP3/nK97V2fXeTPX16f+mqa/6+uzbWF8fHAf19dttG9DsEZTBsSeBfF5KGFI1MhUC2f7p41xupStmu0u8z9rebPM/Xz7fLzpgju/er2Xb4vfs/4Zf/L7Z/rcrH/f1K9dnXkCmSkDYBBciISphiFSyUp0JlLG2XFGK2Z8R/y26bp/p7SPHexfmy22cf45bHdyHss4/OWa+X/vZ2b7oHy3hF5gpodXWZjxjCDV/UkJUwhCpZKWs1eZofuiVYj7DeOuVt/2OLz/i14/+jm9gcI/ltNcM9AVn/8A3v7u5sG0rthIQIiEqYYhUsr56JlAqZe7D2Lppq8/71XN+xdFNfvjQ593odHBvZWJpKgLFVAJ0hqGEIb1AXzwTKKVi9meIeZc+scxv/reH/DMT5/lwW+PgXkOHHzHsOf/usU0+/6bnvasSBx2UsQ9DtzcXkeiV4jnVEWtva+fhX73ArN+3cd+C8Ty/eW8AxtWsoGGPV2k4qZaPXbwfu+wx/L2ZyvWc73zl2N9R3XxQCUNEolXk8xxKadE/Wpl1QwuzHhpM06oD2MhQ+rOFo4YvoOGDbTScN54DT99r211et5Nu2wYMgB12gDVrKjY5KmGISGWKsxaeUtPe/M0reGD5gcy6exOzXprE6x31AOzRr5WG/VpoOG0wx35pf4aNHVbYOnbZBdraYOvW976vkOSYTAlDRCpTPg8LilJj43Z3ed3AEL7ADP7MaXx09As0HLOJE744iT2Pq49mnZXaRJWiap6HISJ9TIbnOZTqRnlrXnuHB36+kOOu+xojO3reNmUoG7l552/w69ZPM3DHw6NdcWNj+u2Est9FtlyUMEQkWtOnp+/DiOhGeZ1bOnnitoXMmvkWs5pH8sSG/ejiSDoz3PR68LsrYMeBkax7m+6+jEx66bN7lDBEJFrdbfcRjpJa9tQK7vvlq8x6sJYHlu7HO34gRheHD32R7xwzl4bPjcSu2jV9zb4UhXe6m0B2i+EusuWihCEi0Q+DnTKlqPnb29p55MYXmfW7d5n1fPeQ17GMq1nBqXssoOETtXzs4n0ZsdeB7800+IclPbPpIVuTU4V1eEcqios5yvXShXsiJRDHVelpLqRb9I8W/+VnZvvJox/zoaxzcO9Pu39k56f8Ryc2+bN/eLnnXV5DLrckquwmkOhKb5ES6WtXaJe78EuToDYw2M9ipoP77v1a/EsHzva/fPsxX7d8XWliKFaV3folqoShYbUiySr4orOSyfT4z4iHwXqXs+CuV6n73NHstHnldt+3DRnDqr88Ft2Q11KrgqvZu0U1rDauR7SKVKa+9kS7xsbMCSNbZ3HIJ9itee0dfn/JPM7bey4T+6/goE/vzQ6b049m2nHTqupJFhAkh5aWIKm2tFRssoiSOr1FkmXqzOyl4+qZNi3zRXaZOotTz8K6nysOdH7mTJpv7x7yOoLH1+/PZ3mD/+YybuJNNg4djdfuBG1rt19uLx2K2pvoDEMkWaZCq9oKs7DPsM6UCN0z15gznIWtPvdrjB70LkdccCDfm3007sbv9rmS2wdcQB1LqcEZtmEltZs3Qv/+PefvxUNRexMlDJFk06cHhVeyaivMus8AWluDgr/7DCBd0siUCOszNw15hiQzonMVn9z9Re748jxWv7KW+esP5NObZ1K7ZXPPCbdsgR13DNZhFvzszX1EvUisnd5mdglwAeDA88Dn3X1zpunV6S1lUUWdmWmFub9R9zZmurXFccfBokXb9sHKsy/hT68fzKzZg/jlys9Qx5LtZvG6Oix1eXHdV0p6qPqbD5rZBOBhYH9332RmvwfucfdbM82jhCESQq5COt1IsBRuhiUto/smfvP7HcW3x9zIv678Kf06kup2mUaSVcnN+Xq73jJKqh8w2Mz6AUOAZTHHI3EL2/Zebesqp1z9MNlua5FgKQlnKBv5zehLea19V85behX9br0pXJNSb2jik/dEcTFHoS/gK8B6YDXQmGGaqUAz0FxXVxfFNSxSqcp5MVSVXXi1TZiLCi+6KP2FeBddFHyf6TGjuV6FPoa0mi6ErKZY80C1X+kNDAf+CYwC+gN3A5/LNo+u9O7lynnFcZXd2sHdwye5DNvWVVfnj92ywN8ZOCZ3Yqi2fROFaq1EhBBVwoizSepjwBvuvtrdtwJ3AkfGGI/ELdMQz9bW6JuNqvF6i0wXFZ57bs/9kmEbfPESPnDeAfx7+0/ZxKCeX3ZfvFdfDxde2DebkfraRZsFiDNhLAaOMLMhZmbAccDCGOORuGW71sFzDA+Nal2VfL1FpmTW2bltv2xZv4XNO49NO9mafqP57cXz+N+XTmTwzJQ+iNtvD/ZxSwtcd13QJ1EJw17L2c9UjZWIcoviNKXQF/A94CVgAXA7MDDb9GqS6uXSNQmUqmmkGpsfMjWjJV4ra8b6MNr8LGb6enpuWxe4jxhR2duXqtx/o2pspgyJau/DKOSlhNEHJHc6Rt35mm1d1dDBmSOhdmJ+4f5z/O5vzfcNP78xSBCp01V6UkxWAXfRrar9lUXZEgbwIPD+KFZW7EsJo8rlW0D34hpfPro6u3zxV3/qaweP8c7us4V0ndq1tT33abXvv0yVhqgqDOlUWyUipHImjEMJRjP9GhgXxUoLfSlhVLFCam+9uMaXy5rX3/E//Oc8P3/vh/zf+WX6JqZ0r+T9E0eBG6VqT3gVpOxNUsAZwLPAlcDgKFae70sJo4oV+s/fS2t8qTraO/yxWxb49z/a5Efu8KzX0OHgvhNrfWVtjmGwmfZpuiap7r6MatCHKwxRiyphhBollRjF9DJwPfBl4FUzOyeKTncpg0q4ornQESi9+JkDK55bxW1TH+bsSY8wZtBaPnDeAVz5z6PZ2lXL5R+ey8PXPcdbm4Yyuiv98yMyymdUTyUcG5lMmVI5o7UkkCujENzvaRnwAHAVcDKwJ/ALYEYUWSvsS2cYBaiUWpqaF7x9XbvPvuZpv+yIJj948MJtu2C0rfJzdp/rjV962Fe9uHr7GTPtu9ra7Ps0V5NUFMdGHzkDrHaUsQ/jQBI3KUzz3cIoggj7UsIoQKUU1OVOXFEVZEUu5/U5i/36s+b4qWPn+zDaHNz7scWP2elp/+HHm/yp3y70zq2duWNIt+8uuij7Ps31ty/22KiUyojkVLaEkXVm2D2KIMK+lDAKUEkdn+WqjaYryMzeu5dSMcvJUSBuWL3B7/ne4/4f75/te/d//b0yuHaJX7j/HL/rsvn+7pJ3C9umdPsu2z7NNAy3+3qMYo+NSqmMSE4VkTDK/VLCKEC+/9S9oYkh0zab5bc9IfZdV2eXL7j7Vf+fU5r8+F2afSCbHNwHsdFPHPW4X3P6bH/pnte8q7Mr4o30cH+rmTMzX4+RqVM8bIFfSZURyUoJQ8LJp5Zc7U0M3QVoukKskNpvhgKxy8z/+PV5fsE+c3xi7Zvbvtp/4Kt+yaFNft8Pm33j2xtLtZWBfP5WmfbJiBHF/b0r5QyjN1RySkwJQ8IL+w9VKQVAIcLeViSf2m+G/dFCnYP7jqz1MybM8xnnzPHWeUtLtmn5xJb2b5XtTKCYwrYSKhiVEEMVUMKQ6FVzE0OuM4s8k9+K51f5Qx/9jm+2gT3m38gg/91e03zutc/6lg1bwsVWihpwPn+rfJJLvrHGXbuv5kpOGSlhSPSq+Z8vzEOBstQ8t2zY4rOvedq/9cEmP2Twi9tm+SLX++p+Y7wL845xE/MvEEtVA843CYSJoRpr69VcySkjJQyJXjUWGN2yXauQofb7xtwlfsPZc/y0cY/6DrwbTM5WP3qnp3368U3+5MwXcw95LTSuYpNwvn+rMGcC1VhhqMaYY6CEIbkV0lwQVxNDruGhYUYD5ShAu4e8fuXg2b7PgNe2TVZXu8Sn7jvH7/zmo762dW2021XKGnDUf6tqrK1XcyWnjJQwJLtq+kfKFmu+o7ySCtCu22f6C39+1X92apN/fMQTPYa8Nox83K8+bbYv/HuJhrx2q+QacGrCKXaYbVzi7kepAkoYkl0lF1SpssWa53asbV3rf/rGo/6Ffef4rrVLt02+34BFfsmhTT7rB0+UfshrsnIl7kI6q1Pj6t/ffcCA6qhkSF6UMCS7ampeyBZrts7s+nrvMvPNI8b5nQdM8w/v+IzXstUhGPL6qfGP+q+mzPGWh5dEF2slNvMVkpSyXZuh2nqvo4Qh2fXyM4zU50GsZ4hf3v9HPu1DTf7QL54JP+S1W0T9JLEo5G9dTRUKKZoShmRXqYVbOjn6MLoG9/yukwyFXaG147D7Kq4knCuZFVL4V1OFQoqmhCG5lbMzsNh1pcy/6oqf+w1nz/HTxz3qn+dGf4N678R8uY3L/LS51FfYBBm28IzrkaG5klkhhX81VSikaL0iYQA7A38EXgIWAh/MNr0SRoWKoPDZ+PZGv/eqJ/yrh8z2fQcs2raY7iGvf/pG0pDXsFd1h60xh00EcdTKw6yz0P2v0UV9Rm9JGL8BLkj8PgDYOdv0ShgVqoCCtKuzy1/86yK/+rTZfsKIJ3wQGx3cB7LJTxjxhP/s1CZ/8a+L0g95DXvfqORCP1vhGDb+OGrlYZOZCn/JouoTBrAj8EamhzOleylhVKiQhVr3kNep+87xutol2ybbd8Ai/+ohs/3eq57wDas3hFvnzJnhEkZ34ZmtoC/iWo+SF8zqa5AI9IaEcTDwOHAr8DRwEzA0zXRTgWagua6uLuLdKJHINJKprs6bb3/Rpx/f5Eft+IxP4bZtfREra8b6g0d+u7ghr7maproL/bDNOpVYQ1dfg0SgNySMyUAH8IHE+/8Frso2j84wKlSaQm2zDfQvcv22jy7v/2Nvr+l559eiC750hWn32U5yoV/tQ0grNZlJ1YgqYdQQn6XAUnd/LPH+j8ChMcYjBdi6cStz3zmQ30+4hOU2ji6MFur5D7+G9ZPex21ffJgVz69m+vhrGdDV3nPmjRth2rTCVz5lCsyYAfX1YBb8vP32IB20tATfA9TVpZ8/0+eVZsqUYHu6unpul0iZ9Ytrxe6+wsyWmNk+7v4ycBzwYlzxSA6NjUHhvngxHaPH89Cen+faN07iwWX70cb7qeUAPrjjZ2g4/B1OOGc015+9LzX9kuojixenX25ra7DsQgvBKVNyzzt9OkydGiSobkOGBJ+LSHhRnKYU+iLox2gGngPuBoZnm15NUhEooHmj/Ve3eEf/QT2ac9YzxC+2X/gX9p3jf/z6PH+nZW32ddXWpm8WKlebvJp1pA8joiYpC5ZVHSZPnuzNzc1xh1G9GhvT17RnzOhRS/cu5+VZbzDrxiXc9/AQbnjrDOpZst3ivK4Oa20Nv65s6uuD5hYRiZyZPenuk4tdTpx9GOXV2AiTJkFNTfCzsTHuiMpv2rTtC/BEP0Lb0jbuunQ+F+7/ELsNeJP9TtqdS+4+hjfaRlDH0rSLsyXbJ5Gs68omU5OViFSM2Powyiq1ttvaGryHvtWBmKFQ7mpdzIhdB9PBEQxjHR8b9wKXHfs6J1y4G7sdvTtMqgv2Wapsncb5JoBq6YAW6cP6xhlGlpp11SniTKlz7IS0n6+wsXzjg48w+5pneHvdQO5adgQX/vZodjt612CC6dODpqtkuTqNMyWAESPyX5aIVIYoOkLK9crZ6Z2pY7Pax+F3y/Mirq2btvrca5/1b3+4yScPecHP5nZfT8qdXwcNLs0zHXI9RU8d0CJlQ7VfuFfIK2vCyFZA9ZbbK4TYjtZ5S33GOXP8jAnzfCfWOrjX0OEf2uFZv+q4Jl/xkc96V/eIpdpa94suKl28pUwMSjoioSlhpMpWmEZxe4VKKKAynCl1YX7JoU2+/8BXt308sfZNv2CflCGvveU2E71lO0TKRAkjVa5mp2IK/EopoDIkxTeo9wFs9uN3afb/OaXJF9z9avq7vOZ7plUJSTKd3nLGKFImShipSlmIxFxAvbvkXb/rsvl+8/hv+wYG94ih3Qb6U5/6vq9fuT73gvLpy6mUJJlOufqkKjVhiuRJCSNVKQu4Uj2TIMP0nVs7/anfLvQffrzJj9npae/HFgf3YbT5T3f8rrcNGeNdhRRi+SS+Sq7FlyO2Sk6YInlSwkinVDXCUjz1LM30W/sN9OtGfcfH1Kzc9vHBgxf6ZUc0edPVT3v7uvbitiOfGCt5ZFk5CvNKTpgieVLCKJeZM91HjNi+4Cjyucpdu9alnX4xE/2s+of9N1+Y68ufXVma7QmTVCu9wCx1c5GedCe9iBJGOWR6FOiIEdsXDCEKmMXz3/Qb//UhP2PCPO8kw4inSqjBu6tJppTP0hYpMyWMcoigzX/jzmP9Pw9r8gMGvrLt4wk1y3x1vzHhl92t3LXZvlx7DpMMKv0sTCRBCaMc8hxV1JVSwKxnsJ/FTB/AZv/YLs3+05Ob/Pk7XwmGvEbQ56HabEiFJr5c81VyP49IEiWMcghRg2x7s83v/tZ8v+iAOf4V+99tz6xeyni/dddp/vfvPp55yGs+BZlqs4UpZaLV30SqhBJGWBFfsNc1ZIi/cfFP/L9OaPJjd35q25DXoazzU8bO9+vOnOOvNbXmH2cuqs0WppSFus76pEooYbjnTgbp/qG7C96wyWPmTO8YP9G7MF/db4xfxLXbFvX+QS/5pR+IaMhrLqrNFqbUibYv9/NI1VDCKKZTMkdtcOumrf7IDc/5d45q8sOHPu9Gp4P7Lva2n1n3iN96wVxf9vSKcH+pqKg2WxglWhEljFAFQabaZZrplzy+zG869yH/9IR5vrO94xDc5fWDw57z732kyeff9Lx3tHfk+3eKtgaq2mz+lGhFIksY1ftM75qa4N8/lRl0dQW/T5qU/klxSRzjoIEvs6B9LwDG1yynYY9FNJzcj+P+fV922WN44QGHfIa2lFhjY/CwrMWLgwc7TZ+u/S99SlTP9I49YZhZLdAMvOnuJ2ebtkfCyJQM6uuhpSX4PV2BnaKVOs4ffhcNR7bRcP4EDjh1T6zGCtqW7YSJUUSkxKJKGJXwiNavAAvznivEY0PXfeQUnvzYN3m7/xi6gC56JoKO/oMYc92VPLjmUL7+t2M58PS9oksWkPm51vk+7zofRTzCVUQkm1gThplNBE4Cbgo9U3eBeM45MHhw8IxoM6ivx381g2f7T+ZHJ87mI8OfZsSEgUz+y5XUb13EaWPm8+ARl7N19IRt0/f79U0Muui8Um1e5udaZ/q8WN1nVK2tQXNda2vwXklDRCIQa5OUmf0R+C9gB+Dr6ZqkzGwqMBVg7xEjDnt506YeTUw+aDCPHv1NZrz2Me57Yy9WdI0B4KBBL9Nw0HIaPrsTR16wPwN3HFiOTeqp3H0YagITkTSqvg/DzE4GPuHuXzKzY8mQMJJNHjjQm7ds2e7zFuo51J7m4xMX0vDxLj7+pT0Zf+jYksSdt3J2uIYZCCAifU5vSBj/BZwDdACDgB2BO939c5nmmWzmzWk+d4yu9q3UDqgtTbDVQmcYIpJG1Xd6u/u33H2iu08CzgT+mS1ZAGyhf9rPrb5OyQJCDQQQESlUJYySCm3AbhNVIGYzZUrQP1Jfv61jX9d8iEhUYr8OIx+TJ0/25ksu0UVYIiJ5iKpJql8UwZTVlClKECIiMaiqJikREYmPEoaIiISihCEiIqEoYYiISChKGMmivnGfbgQoIr1I9Y2SKpXU+z5137gPChuVFfXyRERiVn3XYTSnuzlIBKK+rYZu0yEiFaLqbw1ScaJ+dkUcz8IQESkhJYxuUT+7otzPwhARKTEljG5R37hPNwIUkV5GCaNb1Dfu040ARaSX6Xud3uV8oJGISAXouzcfLIaGuoqIFKxvNUlNm9bz+doQvJ82LZ54RESqSN9KGBrqKiJSsL6VMDTUVUSkYH0rYWioq4hIwfpWwtBQVxGRgvWtUVKgR7yKiBQotjMMM9vVzJrMbKGZvWBmX4krFhERyS3OM4wO4Gvu/pSZ7QA8aWYPuPuLMcYkIiIZxHaG4e7L3f2pxO/rgIXAhLjiERGR7Cqi09vMJgGHAI+l+W6qmTWbWfPq1avLHpuIiARiTxhmNgz4E/BVd29L/d7dZ7j7ZHefPGrUqPIHKCIiQMwJw8z6EySLRne/M85YREQkuzhHSRlwM7DQ3X8WVxwiIhJOnGcYHwLOAT5qZs8kXp+IMR4REckitmG17v4wYHGtX0RE8hN7p7eIiFQHJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUGJNGGbWYGYvm9kiM7sszlhERCS72BKGmdUC1wInAvsDZ5nZ/nHFIyIi2cV5hnE4sMjdX3f3LcD/AafGGI+IiGTRL8Z1TwCWJL1fCnwgdSIzmwpMTbxtN7MFZYitWCOBt+IOIgTFGZ1qiBEUZ9SqJc59olhInAnD0nzm233gPgOYAWBmze4+udSBFUtxRqsa4qyGGEFxRq2a4oxiOXE2SS0Fdk16PxFYFlMsIiKSQ5wJ4wlgLzPbzcwGAGcCf4kxHhERySK2Jil37zCzi4H7gFrgFnd/IcdsM0ofWSQUZ7SqIc5qiBEUZ9T6VJzmvl23gYiIyHZ0pbeIiISihCEiIqFURMLIdYsQC/w88f1zZnZo2HnLHOeURHzPmdk8M3t/0nctZva8mT0T1RC3IuI81szeTcTyjJldEXbeMsf5jaQYF5hZp5ntkviuLPvTzG4xs1WZrv+poGMzV5yVcmzmirNSjs1ccVbCsbmrmTWZ2UIze8HMvpJmmmiPT3eP9UXQ4f0asDswAHgW2D9lmk8A9xJcu3EE8FjYecsc55HA8MTvJ3bHmXjfAoyskP15LPC3QuYtZ5wp038S+GcM+/No4FBgQYbvYz82Q8YZ+7EZMs7Yj80wcVbIsTkOODTx+w7AK6UuOyvhDCPMLUJOBW7zwHxgZzMbF3LessXp7vPc/Z3E2/kE15aUWzH7pKL2Z4qzgDtKFEtG7v4QsCbLJJVwbOaMs0KOzTD7M5OK2p8p4jo2l7v7U4nf1wELCe6gkSzS47MSEka6W4SkbnSmacLMG5V813U+QWbv5sD9ZvakBbc7KZWwcX7QzJ41s3vN7IA8541C6HWZ2RCgAfhT0sfl2p+5VMKxma+4js2w4j42Q6uUY9PMJgGHAI+lfBXp8RnnrUG6hblFSKZpQt1eJCKh12VmHyH4p/xw0scfcvdlZjYaeMDMXkrUYuKI8ymg3t3Xm9kngLuBvULOG5V81vVJ4BF3T67xlWt/5lIJx2ZoMR+bYVTCsZmP2I9NMxtGkLC+6u5tqV+nmaXg47MSzjDC3CIk0zTlvL1IqHWZ2UHATcCp7v529+fuvizxcxVwF8EpYSxxunubu69P/H4P0N/MRoaZt5xxJjmTlFP+Mu7PXCrh2AylAo7NnCrk2MxHrMemmfUnSBaN7n5nmkmiPT5L3TETouOmH/A6sBvvdb4ckDLNSfTsuHk87LxljrMOWAQcmfL5UGCHpN/nAQ0xxjmW9y7aPBxYnNi3FbU/E9PtRNCWPDSO/ZlYxyQyd9LGfmyGjDP2YzNknLEfm2HirIRjM7FfbgOuyTJNpMdn7E1SnuEWIWZ2YeL7G4B7CHr7FwEbgc9nmzfGOK8ARgDXmRlAhwd3shwD3JX4rB/wW3efFWOcnwYuMrMOYBNwpgdHUaXtT4DTgfvdfUPS7GXbn2Z2B8HInZFmthS4EuifFGPsx2bIOGM/NkPGGfuxGTJOiPnYBD4EnAM8b2bPJD67nKByUJLjU7cGERGRUCqhD0NERKqAEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiBQh8TyC4xO//8DMfh53TCKlEvuV3iJV7krg+4kbzR0CnBJzPCIloyu9RYpkZnOAYcCxHjyXQKRXUpOUSBHM7H0ETz5rV7KQ3k4JQ6RAiSeXNRI8qWyDmZ0Qc0giJaWEIVKAxJPW7gS+5u4LgauA78YalEiJqQ9DRERC0RmGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISCj/H/sk4sq+lc4mAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -2919,9 +2550,7 @@ { "cell_type": "markdown", "id": "c7ef262f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Replace or not\n", "\n", @@ -2934,9 +2563,7 @@ { "cell_type": "markdown", "id": "4d00735a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Momentum based GD\n", "\n", @@ -2949,9 +2576,7 @@ { "cell_type": "markdown", "id": "fa136972", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", @@ -2961,9 +2586,7 @@ { "cell_type": "markdown", "id": "67d45e2a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -2979,9 +2602,7 @@ { "cell_type": "markdown", "id": "e3c9063b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have introduced a momentum parameter $\\gamma$, with\n", "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", @@ -2998,9 +2619,7 @@ { "cell_type": "markdown", "id": "fea528b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", @@ -3010,9 +2629,7 @@ { "cell_type": "markdown", "id": "ec1cbddf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." ] @@ -3020,9 +2637,7 @@ { "cell_type": "markdown", "id": "77134c29", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on momentum based approaches\n", "\n", @@ -3036,9 +2651,7 @@ { "cell_type": "markdown", "id": "becf5080", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", @@ -3048,9 +2661,7 @@ { "cell_type": "markdown", "id": "73157461", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can discretize this equation in the usual way to get" ] @@ -3058,9 +2669,7 @@ { "cell_type": "markdown", "id": "5d7111b3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", @@ -3070,9 +2679,7 @@ { "cell_type": "markdown", "id": "8962fb88", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Rearranging this equation, we can rewrite this as" ] @@ -3080,9 +2687,7 @@ { "cell_type": "markdown", "id": "c37e8563", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", @@ -3092,9 +2697,7 @@ { "cell_type": "markdown", "id": "9d59ea2e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Momentum parameter\n", "\n", @@ -3108,9 +2711,7 @@ { "cell_type": "markdown", "id": "0d015b64", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", @@ -3120,9 +2721,7 @@ { "cell_type": "markdown", "id": "eef52144", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Thus, as the name suggests, the momentum parameter is proportional to\n", "the mass of the particle and effectively provides inertia.\n", @@ -3153,9 +2752,7 @@ { "cell_type": "markdown", "id": "b973698a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", @@ -3165,9 +2762,7 @@ { "cell_type": "markdown", "id": "698130a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3183,9 +2778,7 @@ { "cell_type": "markdown", "id": "19b4312f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$." ] @@ -3193,9 +2786,7 @@ { "cell_type": "markdown", "id": "f2b1ce56", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Second moment of the gradient\n", "\n", @@ -3224,9 +2815,7 @@ { "cell_type": "markdown", "id": "01000999", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## RMS prop\n", "\n", @@ -3239,9 +2828,7 @@ { "cell_type": "markdown", "id": "f2f20692", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3257,9 +2844,7 @@ { "cell_type": "markdown", "id": "4e2b4bd1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", @@ -3269,9 +2854,7 @@ { "cell_type": "markdown", "id": "83c5eab2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", @@ -3281,9 +2864,7 @@ { "cell_type": "markdown", "id": "40ef84a6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\beta$ controls the averaging time of the second moment and is\n", "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", @@ -3299,9 +2880,7 @@ { "cell_type": "markdown", "id": "d8dcef44", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n", "\n", @@ -3328,9 +2907,7 @@ { "cell_type": "markdown", "id": "073a3253", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3346,9 +2923,7 @@ { "cell_type": "markdown", "id": "10c9590c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", @@ -3358,9 +2933,7 @@ { "cell_type": "markdown", "id": "c90b6aaf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", @@ -3370,9 +2943,7 @@ { "cell_type": "markdown", "id": "b5c003bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", @@ -3382,9 +2953,7 @@ { "cell_type": "markdown", "id": "9527a589", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", @@ -3394,9 +2963,7 @@ { "cell_type": "markdown", "id": "b7d1a178", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", @@ -3406,9 +2973,7 @@ { "cell_type": "markdown", "id": "0423c3eb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3423,9 +2988,7 @@ { "cell_type": "markdown", "id": "4db72a8e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", "second moment and are typically taken to be $0.9$ and $0.99$\n", @@ -3442,9 +3005,7 @@ { "cell_type": "markdown", "id": "197b823d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", @@ -3454,9 +3015,7 @@ { "cell_type": "markdown", "id": "3c07201a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Algorithms and codes for Adagrad, RMSprop and Adam\n", "\n", @@ -3468,9 +3027,7 @@ { "cell_type": "markdown", "id": "a676af2a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Practical tips\n", "\n", @@ -3488,9 +3045,7 @@ { "cell_type": "markdown", "id": "5d437b79", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Automatic differentiation\n", "\n", @@ -3526,9 +3081,7 @@ { "cell_type": "markdown", "id": "7997afc7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(x) = \\sin\\left(2\\pi x + x^2\\right)\n", @@ -3538,9 +3091,7 @@ { "cell_type": "markdown", "id": "79f772bc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which has the following derivative" ] @@ -3548,9 +3099,7 @@ { "cell_type": "markdown", "id": "068fa7f3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f'(x) = \\cos\\left(2\\pi x + x^2\\right)\\left(2\\pi + 2x\\right)\n", @@ -3560,22 +3109,37 @@ { "cell_type": "markdown", "id": "c77feb11", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using **autograd** we have" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 19, "id": "2134397b", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdwAAAEWCAYAAADM0CYnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABDhklEQVR4nO3dd3hUZdrH8e+dRggJIZDQO9IhBAgCFrBgwQIqFhQFK7ouln2ti2vbteuuXVGxN1QQBVFRVLDREgih9xZCCYFAKCEked4/zgHHmE4yz5T7c125MjPnzJnfeebMc582c8QYg1JKKaVqVojtAEoppVQw0IKrlFJKeYEWXKWUUsoLtOAqpZRSXqAFVymllPICLbhKKaWUF1gpuCJysoisrKFpjxWR8TUx7UAnIu+IyCNlDH9ERHaKyDZv5gpEInKKiGTYzmGTiBgROe4Yp7FURE4pY/hMEbn+GKb/kIh8UNXne8uxtKWIfCMio47x9a8WkV8rMf4GERlUxddqKSL7RCS0Ks8vZ9ojROS76p7uEeUWXLdhDopIrojkiMjvInKTiFS5WBtjfjHGdKzq8z2y/aXTMsY8Zoyp8gfMn1VHB1bGtFsAdwBdjDGNa+I1Kpjjanc+L63k88pcmVD+yRjT1RgzE469OAbDSlBJbWSMGWyMeddWpsoyxmwyxkQbYwqPZToi0trtS8I8pv2hMebMY09ZsooWzfONMTFAK+AJ4B7gzaq8oOfMKb/SCsg2xuwoaaAX39dRwC73v0/QZVrbQHmH3y9nxpgy/4ANwKBijx0PFAHd3Pu1gGeATcB2YBxQ2x12CpCBU6S3Ae8fecwdfi8wsdj0nwdecG9fAywHcoF1wI3u43WAg26Ofe5fU+Ah4AN3nG+BMcWmvQi4yL3dCfgepwNfCVxaRjvUB94GMoHdwBcew24A1rjTmQI09RhmgJuB1e48/AdoB8wG9gKfAhHF2mossNNt+xEe05oJXO9x/2rgV/f2z+5r7Xfb4jL38fOANCAH+B1I9Hh+T2CBm+sTYALwSAnzPqhYW78DtHZf7zr3ff8ZZwXuX8BGYAfwHhDrTuPI+NcAm902vAnoA6S7+V4qZ1ls5WYYBhQAjUpqi2JtfxwwGjgM5Lv5p7rDO7ttmgMsBYZ4PLcBMNV9j+YDj3hO35323933db3HcrvZfU4qcLLH+LXddtsNLAPuwv0MlDKvXflj2dwOjPX4rD2HsxxmurdrFVt+7nbbfytwAXAOsMqd1liP13gImOi+97nustDDY/i9wFp32DLgwmLt/RvwrDvdRyijH3Cfc5ebKRO49sj7U8K8nwos9rg/A5jncf9X4ALP/gk4231/D7vv8SKPz8x/3Ky5wHdAfAmvWVZ/8inOspzrLifJHs9rCkwCsoD1wK1lvKfnAgvd5WMz8JDHsNZue4xy228ncF+xPnc2zrK6FXgJt98otqz3cds+zGPYMJw+oKw28uxXbuCPPncZ0KuCy8OvZcz7VTj9QjZwHx51BaffODLtbLe96xdrF89+5shjYcBwIKXYa/0DmFKBNt/kTufI+92fP/ep44Bnik37S+D/KvveH31+uSOUUHA9wv7Nvf0cTqGpD8TgdFSPe3QCBcCTOB/I2vy54LYCDgB13fuh7gLVz6PB2gECDHTH7eUx7YxiuR7ij4I7EvjNY1gXnAW2Fs4HbDNOAQgDeuEs5F1LaYdpOB1THBAODHQfP819Xi93ui8CPxf7IEwB6uJ0ooeAH4C2QCzOgjuqWFv9z53WQJwC2rGUD8bRhcPzQ+dxvxdOx9vXbddR7vtZC4jA+QD8w52fi3E+iH8puCW1NX8s9O+5bVkbpxNd485bNPA58H6x8ccBkcCZQB7wBdAQaOZmHVjGsng/bscLLMZd8Ev7wHu2B06xe8RjWLibdazbFqfhdCRH2nqC+xeFs9xsLqGtv8dZ5o+sXF6JU6jDcHa/bwMi3WFPAL+447cAllBKwcX5DG11pxHp3u/rDvs3MMdtswSclaj/FFt+HnDn7waczuAjdxpd3TZv6/FZOey+9+HAnTgdR7g7/BKcTiUEuAxnWWzi0d4FwC3u/Nam7H7gbJxC0A1nefmI0gtuJE7xi3envQ2nSMe4r3MQaFC8f8Ljs+8xrZk4HXkH97kzgScqsox7TDMPZ6UlFHgcmONRKFLd9o7AWe7XAWeVMf3u7vMS3fY4suLQ2m2PN9ycPXD6is7u8N5AP7c9WuMUxNtLWdaXAYM9hk0G7iinja73eM+34BRuwSnirSq4PJRYcHE+P/uAATh9z/9wlp0j79vtOMt0c3f4a8DHZfQzRx4Lw/l85gLtPV5vPjC8Em3uuXJydD7cvJsBce/H4Sx7R9qgwu/90emXNbD4Al3s8Tk4ayriNnw7j2H9+WOt/xSctarIMjrvX4GR7u0zgLVl5PkCuK2cD8iRghvjZjuywDwKvOXevgz4pdhzXwMeLOE1m+Cs+caVMOxN4CmP+9E4nVhrjw/CiR7DU4F7PO7/F3iuWIdZx2P4p8D9xT8YJS3k/LXgvorbGXs8thKnkA/A6cTEY9jvVL7gtvV47AfgZo/7Hd22CPMYv5nH8GzcLXH3/iQ8OpESMqw+Mhz4J+4aemkfeMouuCfjdOQhHo997C4/oW7ujh7DStrCPa2cz85u3C1GnA/j2R7DRlN6wb0cWFjKsLXAOR73zwI2eLxHB4FQj+Xf4BZrj+XvSIfzEG7xcO+H4BT6k0t57TRgqEd7b/IYVl4/8BYehQ6nAJZYcN3hvwAX4RSZ73A+B2fjbP2me4y3gfIL7r887t8MfFuRZdxjmjM87ncBDrq3+3q2gcdy+XZZy4XHuM8Bzxb7PDX3GD4Pt3CU8NzbgcmlLOv3AB+6t+vjbKQ08ZifsgrudNz+tQL5iy8PpRXcB4AJHvfr4NSEI+/bcuB0j+FN+Gu/4dnPHHkszL3/AfCAe7s9TgGOqkSbl1ZwBWfDcoB7/wbgx2N574/lLOVmOLuSEnDWMlLdk6pycHblJniMm2WMyStjWh/hdDIAV7j3ARCRwSIyR0R2udM+B2fNt1zGmFycLdPh7kPDgQ/d262Avkcyu9MeAZR0QlALYJcxZncJw5ribCkeec19OIWkmcc42z1uHyzhfrTH/d3GmP0e9ze6r1EVrYA7is1jC3d6TYEtxl1SPF6rsjZ73P5TW7i3w4BGHo9Vpi2OEpETgTY4W53gLCPdRSSpCpmPZN1sjCkqlrcZzrIbxp/nzfN2iY+JyB0islxE9rhtHcsfy2rTYuOX1dYtcAprabmLt7Hn8pFt/jiZ5KD7v6w2PprJbYuMI9MTkZEikuax7HTjz589z/kprx+ozPwDzMIpgAPc2zNxVhQHuvcrw/Os+gOUsoxV4vmR7rHEVkDTYp+vsfx5eT9KRPqKyE8ikiUie3AOqRTvy0rMKiIdROQrEdkmInuBx0p47hEfAOeLSDRwKc6GxdYKzmupy14FlofS/Om9d/u3bI/hrYDJHtNdDhTy53Ys6fN3RPH68YUx5oCbuSJtXiK3b5xQbNqe9aPC7/0RVSq4ItIHp2P6FWd36kGcXbH13L9YY4znQm1Kmo6Hz4BTRKQ5cCFuwRWRWjhbPc/gHK+rB3yNs+ZRkemCs9VyuYj0x9kd8ZP7+GZglkfmesY58+1vJUxjM1BfROqVMCwTp/FxM9fB2a24pQLZShLnTuOIlu5rgLMFEeUxrLyzhTcDjxabxyhjzMc4WzLNREQ8xm9Zhbye78Gf2sKdXgF/7vCrahTO+57mfi1prvv4SPf/n9pGRIq3TfFlJRNoUexs+5Y471uWm7u5x7AWJWQ6Ok0RORlny+JSnD0h9YA9/LGsbi02jbLaejPOYZSSlNTGmaWMWxFHM7lt0RzIFJFWOLs3x+Dsvq2Hsxvcc3nxbNPy+oHKzD/8teDOovyCW5H+oCyVff5mnC14z89XjDHmnFLG/whnl3sLY0wszuEVKWXc4l4FVuDsOq2L07mX+FxjzBac470X4hw7fd9zcAXm6S/LXgWXh9L86b0XkSicPtLzNQcXa8dIdz4qkvs7IN5d+b4cjw02ym7zitaPi93574tTj45krsx7D1Sy4IpIXRE5D6fqf2CMWeyuFb8BPCsiDd3xmonIWRWdrjEmC2cN9m13Jpa7gyJw9ulnAQUiMhjn2N8R24EGIhJbxuS/xumg/g184rFF8xXQQUSuEpFw96+PiHQuId9W4BvgFRGJc8cd4A7+CLhGRJLcFYTHgLnGmA0Vnf8SPCwiEW4nfh7OCgk4u3AuEpEo9+s/1xV73nacYwlHvAHc5K7liYjUEZFzRSQG5wNZANwqImEichHOiRnH4mPgHyLSxl27fgynzQuOZaIiEolTyEYDSR5/twAj3K2NRUBX932IxNl15ql428zFKdJ3u+/nKcD5OLu+CnGOPz/ktnUn/ijspYnBac8sIExEHsA5bn/Ep8A/3eWnuZu9NF8BjUXkdhGpJSIxItLXHfYx8C8RSRCReJzddcfyPdHeInKR24a34xw3nIOz28+484OIXIOzRVOiCvQDnwJXi0gXt8N9sJxcv+Mckjge57j9Uty9UjgnzpRkO9Baqv6VxYr0J57mAXtF5B4RqS0ioSLSzd0gKUkMzp6yPBE5HmeLqaJicE782ecujyVtGHh6D+fkue44x3CPKK+NxgN3ikhvt884zi02lVoeipkInCciJ4lIBE5f7Pn644BH3dfBXbaHVnDauP3LROBpnF3o33sMLqvNs3AOFXr2C8WnvdAdbzww3RiT4w6q7HsPVLzgThWRXJyqfh/OQe9rPIbfg3MCyhxxdnfMwPmwVMZHOGcbHl07cXcJ34rzYd2N01hTPIavwOmA1rmb9X/Z9WqMOYTTeZY07TNxdjNn4uzKOXJiV0muwjmusALn5J7b3en8gHMyzyScNbl2/LELuyq24cxrJs7ui5vc+QTnjNB8nA/Nu/yxe+OIh4B33ba41BiTgnPc4SV3mmtwjlFgjMnHOUZ2tTvsMpx2OhZv4axN/4xz8k0eZReWiroAZ+vpPWPMtiN/OMfPQ3GOja7C+SDPwDnWW/xL+G8CXdy2+cKd/yHAYJyts1dwziM40tZjcHYJHzmz/mOcYlSa6TgrZatwdpfm8efdYA+7j6/HWSN/v/gEjnCXzTNwVgC2ufNzqjv4ESAF58zuxThnFh/L94u/xHnvd+Ms4xcZYw4bY5bhnF8wG2d5645zpm9ZSu0HjDHf4Bw/+9Ed58eyJuTudlwALHXfK9wsG00pX03jjxXTbBFZUE7Wkl6z3P6k2PiFOO9REs77uhOnYy6tYN8M/NvtSx/A6dcq6k6c/i8XZ8Xmk3LGn4y7q7bYIaoy28gY8xnOuS4fua/1Bc4Zw1VZHo5McynOGf0f4fSRu3EOXRzxPE6//p3bNnNwVqwq40j9+KzYCn6pbe7udn4U+M19v/uVMu2P+Wv9qOx7D/xx9pXyAe5W1gfGmObljKq8TESeBBobY0bZzlJdROQhnBNtrrSdRVU/EVmL8zXKGbazKIf+lrJSJRCRTiKS6O5WOx5n9/3k8p6nlC8QkWE4u4DL3JOgvMu/f7VDqZoTg7MrqSnOIYT/4ux+VcqnichMnK8vXVXsLHxlme5SVkoppbxAdykrpZRSXqC7lMsQHx9vWrdubTuGUkr5ldTU1J3GmITyxwwuWnDL0Lp1a1JSUmzHUEopvyIiVfnVuoCnu5SVUkopL9CCq5RSSnmBFlyllFLKC7TgKqWUUl6gBVcppZTyAi24SimllBcEVcEVkX+IyFIRWSIiH7uXcVNKKaVqXNB8D1dEmuFc6q+LMeagiHyKcxm9d6wGU0oFnYLD+WxcsYBd6xZQmLsDk38AqRVNWL0mJLTrTfPjEgkNC5ruOWgE2zsaBtQWkcNAFM41Z5VSqsaZoiIWz/qc/IUf02nPb7STg7QracT5kE0saxqcQtyJ19Kh1yleTqpqStAUXGPMFhF5BtiEczHz74wx3xUfT0RGA6MBWrZs6d2QSqmAY4qKWPDtu9RPeZbEoo3kEM2y+qcR0nYgCR36EteoBVF1Yti/by/ZW9ayc/U8QtfOoPvOb4ma8iVLv+1O6JkP0yn5dNuzoo5R0FwtSETigEnAZUAO8Bkw0RjzQWnPSU5ONvrTjkqpqtq0Ko29k26j26E0NoS0JKvH3+hx9rVE1Cr/9JF9e3ezZOoLtFv9NgnsZl69c+g46kVi4+K9kPzYiEiqMSbZdg5fE0wnTQ0C1htjsowxh4HPgRMsZ1JKBSBTVMS8Sc/S8MNBtDy0mrmdx9Ji7EL6DL25QsUWILpuHP1GPEjUHWnMbnIlPXdP58Dz/Vgxf0YNp1c1JZgK7iagn4hEiYgApwPLLWdSSgWYvIP7SXl+OMcvfojVkd3Iv2kOfS+7p8onQdWJqUf/G19m3fkTMSK0++pS5n/xcjWnVt4QNAXXGDMXmAgsABbjzPvrVkMppQLKnl1ZrHv2LPrsmc7sljfS5a4ZxDeunnNBOiafRp1bZ7Mysjt90sYy+627MUVF1TJt5R1Bcwy3KvQYrlKqonZu28y+18+haeEW0o9/kuRzb6iR18k/lMeiV0bRZ8+3zG46kn7XP4+E+Na2kx7DLZlvvUtKKeWHdmdtJff1c2lYuJ1Vg96usWILEFErkuTbPmZug6H0z3yPOe/cW2OvpaqXFlyllDoGe3bvJHvcuTQtzGTdoDfodvLQGn9NCQmhz81vM7/eYPpveo25nzxR46+pjp0WXKWUqqL8Q3lkvHohLQs2sOKUV71SbI8ICQ2l15gPWBh1AsnLniD9p4lee21VNVpwlVKqCkxREWmvXkPX/HTSez9Gj1Mv8XqG0LAwOvztYzaEtaHNzDFsXJ7q9Qyq4rTgKqVUFcz96GGOz/maOc2vI3nITdZy1ImpR/Q1EzkkteCzUezPzbGWRZVNC65SSlXSkl+ncPzq51kQPZDjr3nadhwaNW/HtkEv0qIwg2Xjb9CvC/koLbhKKVUJO7dtpvGMW9gc2oyON75HSGio7UgAdDtpCHNb3UCfPd8x/8uXbMdRJdCCq5RSFVRUWMjWt0cSbfZTNOwt6sTUsx3pT44f+ThLI7rTJe0xtm1abTuOKkYLrlJKVdC89++n+6EFpHcfS5uufW3H+YvQsDDqDX+DEIrI+lB3LfsaLbhKKVUBa9N/p/f6caTGnEqfi263HadUzdp2ZnHXO+l+aCHzJv3PdhzlQQuuUkqVI/9QHnx5M3skhnajxvncTykW12fYHSyplUS3JU+zPWOt7TjK5dtLjVJK+YDUD/5Fu8L1bD7hUerFN7Ydp1whoaHEXTaOUArJmPB/tuMolxZcpZQqw9r030ne9BYpdQfR88wrbcepsGZtO7Ow1bX03jeTxT9/aTuOQguuUkqVqrCggMIpt7FHYmg/6hXbcSqt5+UPkiGNqffTvRzKO2A7TtDTgquUUqVImfwcHQpWsaHXP4lt0Mh2nEqLrF2HXQMepYXJZOGnj9uOE/TCbAcIRHM/+y/Rq78gvCiP8KJDFEg4h0Nrcyg8lkPRLZD6bajXrg+tu/WjVmSU7bhKqRLs2rGFTkv/x9KIRHqfN9p2nCpLPPVi0ua9QZd148nZ+Xe/OAYdqLTg1oTCfEJMIXlhddkfUosQc5iwwoPUP7iRRvvmErn9MCyH/KlhLK3VmdxWZ9Ci/8U0a9vVdnKllGvNR3fS0+QRfeFzPn9WcnnihjxGnQmnM//T++l38xu24wQtMcbYzuA1IlIPGA90AwxwrTFmdmnjJycnm5SUlGrNYIqK2L5lHVuX/cah9XNptOM32hRtAGBZeDf2d72C7meOIjIqulpfVylVcSvmfU+nry9mdpMr6X/jy7bjVIt5L1xJUvbXZI36pcZX7kUk1RiTXKMv4oeCreC+C/xijBkvIhFAlDEmp7Txa6LgliRz/Qo2/vIhzddPpIXJZCf1WHPcNXQfervP/XScUoGuqLCQNY/3I7Ygm+g7FgTMZ3Bn5kaiXuvDiph+9LpzSo2+lhbckvn3fpJKEJG6wADgTQBjTH5ZxdabmrbpRP+R/6H5/UtZMuh9ttVqTb81z5L/3+7M/eQJCg7n246oVNBY8M2bdChYxaaedwZMsQWIb9qKRS1H0mvfLFYtmGU7TlAKmi1cEUkCXgeWAT2AVOA2Y8z+YuONBkYDtGzZsvfGjRu9nNSxYv4MCmb8h26H0lgf0poDgx6n6wnnWMmiVLDIO7if3U/2YH9oXdqOne8zVwKqLrl7dlH4bHc21u5Cj3u+r7HX0S3ckgXNFi7OCWK9gFeNMT2B/cC9xUcyxrxujEk2xiQnJCR4O+NRnfoMous9P7Gw/wtEFu2n63eXM/elaziwb4+1TEoFurTPnqAJWeSd+lDAFVuAmNj6LG8zih4H57Ey5UfbcYJOMBXcDCDDGDPXvT8RpwD7LAkJoedZo4i7O405jS6nT9Zkdv23Lyvmz7AdTamAsztrK13XvsGi2n3pdtIQ23FqTPcL72I3MRya8ZjtKEEnaAquMWYbsFlEOroPnY6ze9nnRUZF0+9v41h+1keEUEi7ry5lzseP6qW3lKpGqz79F1HkUW9IYP9ARHTdOFa0vYbEvPm68u5lQVNwXbcAH4pIOpAE+NUqXtcTziH6tjksqdOXfiufIvXZi3UXs1LVIHPDSnrumExKg/Np1bm37Tg1LvHCO9hNXfJ/8Ksu0O8FVcE1xqS5x2cTjTEXGGN2285UWXXrNaDHHV8xp/Xf6bX3RzKfPZWd2zbZjqWUX8v44iEMIbQZ9m/bUbyiTkw9VrS9msS8VFan/WI7TtAIqoIbKEJCQ+l39WMsHjiOpgUZHB53GhtXLLAdSym/tHnNYnrt/paFjS6kYbM2tuN4Tdcht5NrarN3xjO2owQNLbh+rMdpw9lywUTCOUzchPP0eIxSVbB9ykMcJozjLrrfdhSvqluvAUuaDiMpdxZb1i21HScoaMH1c+17DiD/6u/YK3Vp8dUIls+dbjuSUn5j4/JUeu35gUVNLiG+cUvbcbzuuPPvopBQMqY9ZTtKUNCCGwCatu5IxPXfkB3agFZfX8XS36bZjqSUX8ie9jAHqUXHYcG1dXtEQtPWpNU/ix47p5G9PcN2nICnBTdANGzWhqjR37IjtCFtv7tai65S5Vi/dC699s0ivfnlxCU0sR3HmsaD7yaCAlZN0WO5NU0LbgCJb9ySujdNZ3toI1p9d52efahUGXZ9+wT7TSRdLvqn7ShWteyQxKLoE+m65VP9mmEN04IbYOo3bEbUdVPJlWjiv7iCTavSbEdSyudsXrOYpL0/kd70YmIbNLIdx7rIAbdSl/0s/kavlVuTtOAGoIbN2lBw5WSKECI+Gsb2jLW2IynlU7ZOe5zDhNF+6F9+Tj0odepzBmtC29Fo+Tv6C3Y1SAtugGpxXHdyLppAtNnPvrcvZn9uju1ISvmEbZvXkLTrWxYlnE984xa24/gECQlhV7draF20maW/TbUdJ2BpwQ1g7RJPYN0pL9G6YD2rXr2cwoIC25GUsm7D1CcRoOX5unXrKfHsa9lFXQ7PHmc7SsDSghvgEk+9mJTO99DzwO/MH3+r7ThKWbVrxxZ6bP+CtLgzadKqY/lPCCKRteuwsvnF9Ng/my3rltuOE5C04AaBvsP/ydz4YfTb9iHzJj1nO45S1qz88ilqcZhG5+jWbUnaDb6VQkLYPP1521ECkhbcINH7xnGkRyaTlP4fVi2YZTuOUl63NyebbhmfkBYzgJYdkmzH8UkNm7Uhve5Aumz/Us/7qAFacINEWHgErW74iGyJo+6Ua9mdtdV2JKW8atnU54mRg9Q9427bUXxanQF/py4HWDrjfdtRAo4W3CAS26AR+y54mzizh4zxV+hJVCpoHM4/RJu1H7A0ogfH9TjJdhyf1rH3aSw7awLJQ262HSXgaMENMu2TTmZR4n10P7SA+e/omr4KDou+fZtGZFPQT4tIeSQkhC79BxMSGmo7SsAJuoIrIqEislBEvrKdxZY+F97GvHrn0C/jTRb/PNl2HKVqlCkqot6i19kY0pzuAy+xHUcFsaAruMBtQFCf8y4hIXS/4XU2hrSg8Y//YNeOLbYjKVVjls6exnGFa9ne5TrdalNWBVXBFZHmwLnAeNtZbKtdJ4aCC8cTa3LZ9M51+nNuKmAV/voiu6hL4rk32o6iglxQFVzgOeBuQKsL0K57PxZ0/AdJB2Yz77OnbcdRqtptXLGAHgfnsrLlcCJr17EdRwW5oCm4InIesMMYk1rOeKNFJEVEUrKysryUzp6+w8eyKLIPPZY9zfpl823HUapabf/uf+SZcDqed7vtKEoFT8EFTgSGiMgGYAJwmoh8UHwkY8zrxphkY0xyQkKCtzN6nYSE0Ozqt9kvUZiJ15N/KM92JKWqRfb2DHpkf8uiBoOp37CZ7ThKBU/BNcb80xjT3BjTGhgO/GiMudJyLJ8Q37gFm098grZFG0j94D7bcZSqFqumvUAtOUzjs+6wHUUpIIgKripb0hlXkFL3DJI3vc2aRb/ZjqPUMck/lEf7TZ+QHtmHVh2TbMdRCgjSgmuMmWmMOc92Dl/TftTL7JEYQr68WXctK7+WPuN94snBHD/adhSljgrKgqtKFtugERknObuWF7w/1nYcpaosOu0tMqQJ3QcOsx1FqaO04Ko/SRp0OfNjzyR589usWfSr7ThKVdqaRb/S6fAyMtqP0B+6UD5FC676iw6jXmG3xCJTxlBwON92HKUqZfdPL3HA1KLLOfq7ycq3aMFVfxFbP4GM/g/TrnA9KZ88ZjuOUhW2a8cWEnfPYHH8YOrWa2A7jlJ/ogVXlSjpjKtYGHUCiatfIXPDSttxlKqQld+87HwVaNCttqMo9RdacFWJJCSEJsNfACDrkzH6W8vK5xUczqfN+k9YUiuJVp17246j1F9owVWlatyyPekdxtDj4DwWTn/XdhylyrT4x49pzE4O977BdhSlSqQFV5Up+dJ7WRPajhZzH2ZvTrbtOEqVKiJ1PJnSkMTThtuOolSJtOCqMoWFR2DOf576JoflH9xpO45SJVq/dC5d89PZ1PZyQsPCbMdRqkRacFW52iedTErCRSRnTWbt4jm24yj1Fzt+eIk8E06nwfpVIOW7tOCqCul0xZPslRgOTblDT6BSPmXP7p10z55Oev0zqRff2HYcpUqlBVdVSGz9BFZ3+wddDi8hddobtuModdSK6W8QJYeIG/g321GUKpMWXFVhvS+4ldWhx9Ey9Qn25+bYjqMUpqiIRqs+YlVYB9onnWw7jlJl0oKrKiw0LIzCwU/RkF2kf3S/7ThKsXL+DFoXbSKn8wjbUZQqlxZcVSmdkk9nfuzZ9M78kM2rF9mOo4Lcvt9eJ9fUpttZ19iOolS5tOCqSmtz+dPkE8GuSXoClbInZ+c2uu+ZybKEc4iKjrUdR6lyacFVlRbfuCVLOvyNHnnzSZ/5qe04KkitmP4ateQwDU/Tk6WUfwiagisiLUTkJxFZLiJLReQ225n8Wa+L72GzNKXer4/oJfyU15miIpqtmcDy8C606dLHdhylKiRoCi5QANxhjOkM9AP+LiJdLGfyWxG1ItnZ/z5aFW0mdfJztuOoILP0969oYTLZ1+0q21GUqrCgKbjGmK3GmAXu7VxgOdDMbir/ljToCpZGdKfDshf1d5aVVx2a8yY5RNP9jJG2oyhVYUFTcD2JSGugJzDXchS/JiEhRJzzOHHsZeknD9mOo4LEzm2bScz9hRWNzicyKtp2HKUqLOgKrohEA5OA240xe0sYPlpEUkQkJSsry/sB/Uz7pJOZH3sWvTM/0gvVK69YPf1VwqWQpqffZDuKUpUSVAVXRMJxiu2HxpjPSxrHGPO6MSbZGJOckJDg3YB+quUlj1NICFsn3Ws7igpwRYWFtNrwGUsjetCyQ5LtOEpVStAUXBER4E1guTHmf7bzBJJGzduR1uJKeuf+yIqUH2zHUQFsyc+TaWp2kJc0ynYUpSotaAoucCJwFXCaiKS5f+fYDhUoul/6ADupB9/epz+GoWpM4fw3ySaW7qfrTzkq/xM0BdcY86sxRowxicaYJPfva9u5AkV03TjWdbuNTgXLSfv+fdtxVADanrGW7vvnsKrpUCJqRdqOo1SlBU3BVTWv19AxbAxpToO5T+qPYahqt+67cYRgaDlILzKv/JMWXFVtwsIjyO57Ly2LtrBgysu246gAUnA4n3abJrKkdm+ate1sO45SVaIFV1WrnmeMYGVYJ1ovfoG8A/tsx1EBYsnMiTRkF4W99KpAyn9pwVXVSkJCKDjtQRqyi7SJT9qOowKEpL7FDurT/dRLbUdRqsq04Kpq1/WEc1gU2Ycu695kzy798RB1bDI3rKT7wRTWtriIsPAI23GUqjItuKpGRJ/7CNHmAMs++7ftKMrPbfz+FQzQ5ky9DJ/yb1pwVY1o170fC2JPp2fmx+zYst52HOWn8g/l0X7LFyyu04/GLY6zHUepY6IFV9WYphc+SghFbJh0v+0oyk8t/uFj4slBkq+1HUWpY6YFV9WYpm06saDhRfTKnsamVWm24yg/VGvRO2wlgW4DLrIdRaljpgVX1aj2Fz/EISLY+aVu5arK2bx6Ed0OpbGh9cWEhoXZjqPUMdOCq2pUg0bNSW95Fb32/8zqtF9sx1F+ZMuMVzlsQmmvJ0upAKEFV9W4bhePJYdoDkzXM5ZVxeQd3E/H7V+xOOZE4pu2sh1HqWqhBVfVuJjY+qxoey09Ds5jxbzvbcdRfmDJ9+8TRy7hx19vO4pS1UYLrvKKxIvuZCf1KJzxH9tRlB+os/g9MqQJXU88z3YUpaqNTxZcERkjInG2c6jqExUdy5qOo+mav4glv06xHUf5sA3LU+h8eCkZbS8lJDTUdhylqo1PFlygMTBfRD4VkbNFRGwHUscu6YLb2U4DwmY+phepV6Xa/uOr5JswOp6tJ0upwOKTBdcY8y+gPfAmcDWwWkQeE5F2VoOpYxJZuw4bu42hU8Fy0md+ajuO8kEH9+fSOetr0mNPIS6hie04SlUrnyy4AMYYA2xz/wqAOGCiiDxV1Wm6W8srRWSNiNxbTVFVJfQc8ne2SCPq/PYURYWFtuMoH7P4u7epywGiTrjBdhSlqp1PFlwRuVVEUoGngN+A7saYvwG9gWFVnGYo8DIwGOgCXC4iXaopsqqg8IhaZCbdznGFa0n7/n3bcZSPqbf0fTaEtKDz8WfajqJUtfPJggvEAxcZY84yxnxmjDkMYIwpAqp62uLxwBpjzDpjTD4wARhaPXFVZfQ6dzQbQ1rQYN4zFBYU2I6jfMSaRb/SoWAV29pfjoT4atekVNX55FJtjHnAGLOxlGHLqzjZZsBmj/sZ7mN/IiKjRSRFRFKysvRarjUhNCyM7D530KpoMwu/fsN2HOUjsme9xkETQeezRtuOolSN8MmCW0NKOtPZ/OUBY143xiQbY5ITEhK8ECs4JZ05krWhbWm88DkO5x+yHUdZlrtnF92zp7M4bhCx9fVzpwJTMBXcDKCFx/3mQKalLEEvJDSU3BPuobnZxsIpL9uOoyxbNv1NouQQ9QbcaDuKUjUmmArufKC9iLQRkQhgOKC/wGBRj1MvZWVYJ1oteYm8g/ttx1GWmKIi4ld+yNrQtrRPGmA7jlI1JmgKrjGmABgDTAeWA58aY5baTRXcJCSEwwPH0ohs0r54znYcZcnKBT/RrnA9OzuN0JOlVEALqqXbGPO1MaaDMaadMeZR23kUdD3xfJZGJHLcytc5uD/XdhxlQe4vr7PfRNL1rOtsR1GqRgVVwVW+R0JCCDn9X8STw6LPn7YdR3nZnl1ZdM/5gSXxZxNdV38+XQU2LbjKus59zyI9MpmOa99i397dtuMoL1r+7WtEymHiB+rJUirwacFVPiHyzAeII5fFk56wHUV5iSkqosmaj1kZ1pF2iSfYjqNUjdOCq3xCh14DWRh1Al03vseeXfqDI8Fg2ZxvaVWUwZ6uV9mOopRXaMFVPiP2nIeoywGWff6Y7SjKCw7OHs9eouh+5tW2oyjlFVpwlc9o260vqdGnkLj5I3bt2GI7jqpB2dszSNw7k2UJ51K7ToztOEp5hRZc5VPiz3uQSA6x6nP91lYgWz39NSKkkCan6UXmVfDQgqt8SqtOvVhQ70yStn7KzswSr1+h/FxhQQEt1n/CsvButOrc23YcpbxGC67yOU2HPkgYhayd/G/bUVQNWPLzRJqZ7RxMusZ2FKW8Sguu8jnN2nZlQYNz6bnjC7ZtWm07jqpu88aTRRzdB11pO4lSXqUFV/mklhc8CMCmLx62nERVp4w1S+iRN581LS8holak7ThKeZUWXOWTGrdsz8KEofTM/pot6/QaE4Ei4/uXOGxCaX/2GNtRlPI6LbjKZ7W76EEKCSHzS93KDQQH9+fSZfsU0usOIL5pK9txlPI6LbjKZ8U3bUVak0vplfMdG1cssB1HHaPF346nLvupfeJNtqMoZYUWXOXTOlx0H3nUYudXupXrz0xREQ2Wvcu6kNZ0Pv5M23GUskILrvJp9Rs2I7355fTeN5N1S+bajqOqaMX872lXuJ6sziP1IvMqaOmSr3xel2H3sZco9nz9kO0oqooO/DrO+d3kwdfbjqKUNUFRcEXkaRFZISLpIjJZROrZzqQqLrZ+AktbjaTngd9ZtWCW7TiqknZmbiRx7yyWNTyfqOhY23GUsiYoCi7wPdDNGJMIrAL+aTmPqqTuw+5lNzHkfae/PuVvVn/7MuFSSLMzb7EdRSmrgqLgGmO+M8YUuHfnAM1t5lGVF103jpXtriUxL4Xlc6fbjqMq6HD+IY7b9CnpkX1ocVx323GUsiooCm4x1wLflDZQREaLSIqIpGRl6YXQfUmPi+5iJ/Uo+uE/mKIi23FUBaR//z4J7Kaojx67VSpgCq6IzBCRJSX8DfUY5z6gAPiwtOkYY143xiQbY5ITEhK8EV1VUO06MazpOJqu+YtZ+ttU23FUBUQvfJ0MaULiKZfYjqKUdQFTcI0xg4wx3Ur4+xJAREYB5wEjjDHGblpVVUkX3M52GhA263HdyvVxK+bPoGPBSrZ0HEVIaKjtOEpZFzAFtywicjZwDzDEGHPAdh5VdZG167Ch69/pVLCc9JkTbcdRZdg/60X2EkW3c/Ui80pBkBRc4CUgBvheRNJEZJztQKrqeg0dwxZpRJ3fntCtXB+1bdNqeuT+zLLGF1Inpp7tOEr5hKAouMaY44wxLYwxSe6f/pirHwuPqEVmj1s5rnAtad+/bzuOKsGGr58FoPU5/7CcRCnfERQFVwWenueOZlNIM+LmPkNhQUH5T1Besz83hy7bJrMoZgCNW7a3HUcpn6EFV/mlsPAIsnr/H62LNrHw27dsx1Eelkx7lbocoM4pt9qOopRP0YKr/FbPs69hfUhrGqU+S8HhfNtxFFBUWEizle+yMqwjnZJPtx1HKZ+iBVf5rZDQUHL63UULk8mCKa/YjqOA9B8/obnZyr6eo21HUcrnaMFVfi1p0BWsDOtI68XPc3B/ru04QS98/qtsI54eZ460HUUpn6MFV/k1CQmh8PSHacgu0iY+YTtOUFu1YBZd89PZcNyVhIVH2I6jlM/Rgqv8Xpf+g0mr3Y+u698iZ+c223GC1r4fnmEvUXQ9/zbbUZTySVpwVUCoN+RR6piDrPjsQdtRgtLm1YtI2vcLS5teQkxsfdtxlPJJWnBVQGjdOZnUuMH02jaRzA0rbccJOlu/forDhNF+yF22oyjls7TgqoDR6uJHKELI/Hys7ShBZWfmRpJ2fUta/LnEN25hO45SPksLrgoYjZq3Y2HTy0neO4M1i36zHSdorJ76FKEU0vzcu21HUcqnacFVAaXLpQ+SQzQHvrnfdpSgsDcnm26Zk0iLGUiztl1tx1HKp2nBVQElNi6eFe1Hk5iXyuKfv7QdJ+AtnfIcMXKQ2DP02K1S5dGCqwJOz2F3sZUEas96mKLCQttxAtbB/bm0X/cei2v15LgeJ9mOo5TP04KrAk6tyCi29LqD4wrXkjpVL31cUxZ9+Tzx5BB6yj22oyjlF7TgqoDU69zRrArrQKu0Z9ifm2M7TsDJO7ifdqvGszSiO136D7YdRym/oAVXBaSQ0FCKznqchuwi/ZOHbccJOIumvEgCuzEDdetWqYoKqoIrIneKiBGReNtZVM3r1GcQKTGn03Pz+2zdqD+GUV0O5R2g9fLXWR7ela79z7UdRym/ETQFV0RaAGcAm2xnUd7T/NKnMAiZE++1HSVgpE15mUZkU3DyXUhI0HQhSh2zYPq0PAvcDRjbQZT3NG5xHGktR9I790dWzP3Odhy/l38oj1bLXmNFWGe6nTTUdhyl/EpQFFwRGQJsMcYsqsC4o0UkRURSsrKyvJBO1bQelz3ADuoT+v1Y/ZrQMVo45WUak0X+iXfq1q1SlRQwnxgRmSEiS0r4GwrcBzxQkekYY143xiQbY5ITEhJqNrTyiqjoWDb1upv2BatJnfqq7Th+K+/APtosfYmVYZ3oPvAi23GU8jsBU3CNMYOMMd2K/wHrgDbAIhHZADQHFohIY5t5lXf1Onc0K8M60SbtafbmZNuO45fSPn+Ghuyi4NQHdOtWqSoI+E+NMWaxMaahMaa1MaY1kAH0MsbolcqDSEhoKCHnPUOc2cPyD/VH9isrd88uOq4ZT3pkb7qeqGcmK1UVAV9wlTqifdLJpCRcSPKOSXo1oUpaMvFR4sil9tn6nWalqiroCq67pbvTdg5lR6cRT5MjdSmc+g89gaqCdu3YQuKmD1gQPYD2SSfbjqOU3wq6gquCW2xcPOt63kvHgpWkTH7edhy/sGriw0RyiAbn/8d2FKX8mhZcFXSSz7+JZRHd6bDkv+zO2mo7jk/LWLOEXtsnklr/HFp1TLIdRym/pgVXBR0JCaH2Bc9Sxxxk9Ud32I7j07I+v5sCwmh7yeO2oyjl97TgqqDUpksfUpsM5/jd01j62zTbcXzSkt+m0vPAb6S3uZ74pq1sx1HK72nBVUGrx1VPskUaETvj/zi4P9d2HJ9SWFBA7R8fYBsJJF061nYcpQKCFlwVtGrXiWH3oP/S3Gxj0ft6mTlPC6a8TLvCdWT0uYfIqGjbcZQKCFpwVVDrduL5zK0/hD5bP2LVglm24/iEfXt30yb9f6wI60zvwdfZjqNUwNCCq4Je55HPkS1xhH91C/mH8mzHsW7JB/dQ3+wh5Jwn9CcclapGYbYD+JvDhw+TkZFBXp52zFURGRlJ8+bNCQ8Ptx3lqLr1GrDu5MdJ+uVGZn9wH/2v+6/tSNasWfQbfbZ/yvz4ofTtdYrtOEoFFC24lZSRkUFMTAytW7dGRGzH8SvGGLKzs8nIyKBNmza24/xJ0unDmZ8+iT6b3mJlyrl0TD7NdiSvKywooGjq7eRIXTqNeMZ2HKUCju4vqqS8vDwaNGigxbYKRIQGDRr47N6BjteMY6c0oM60m9mfm2M7jtelfP4sHQpWsb7XWGLr66UplapuWnCrQItt1fly29Wt14DsM1+gadE2lrx9i+04XrVz2yY6L/sfS2ol0fu80bbjKBWQtOAq5aHrCecwt+kI+u6aQtqMj23H8QpTVMTm926kljlM3WHP64lSStUQ/WQFgS+++IJly5Z5/XUfeughnnnG/44F9hr1NGtD29Dy17vJytxgO06NS/3qdXoe+J2F7cfQskOS7ThKBSwtuEGgOgtuQUFBtUzHl9WKjCLskreINIfIensEBYfzbUeqMTszN9J+wb9ZEd6FPsP/ZTuOUgFNz1I+Bg9PXcqyzL3VOs0uTevy4Pldyx3vggsuYPPmzeTl5XHbbbcxevRooqOj2bdvHwATJ07kq6++YvTo0UyZMoVZs2bxyCOPMGnSJHJzc7nppps4cOAA7dq146233iIuLo758+dz3XXXUadOHU466SS++eYblixZwjvvvMO0adPIy8tj//79TJkyhaFDh7J7924OHz7MI488wtChQwF49NFHee+992jRogUJCQn07t27WtvHW1p16kVK8n9ITr2b2W/9g/43vmw7UrUzRUVkfHAjnUw+dS4dR2iYdgdK1STdwvVTb731FqmpqaSkpPDCCy+QnZ1d4ngnnHACQ4YM4emnnyYtLY127doxcuRInnzySdLT0+nevTsPP/wwANdccw3jxo1j9uzZhIaG/mk6s2fP5t133+XHH38kMjKSyZMns2DBAn766SfuuOMOjDGkpqYyYcIEFi5cyOeff878+fNrvB1qUvL5NzK3wQX03/oBad9/ZDtOtZv32VMkHZhNWsfbadG+h+04SgW8oFmlFZFbgDFAATDNGHP3sU6zIluiNeWFF15g8uTJAGzevJnVq1dX6Hl79uwhJyeHgQMHAjBq1CguueQScnJyyM3N5YQTTgDgiiuu4Kuvvjr6vDPOOIP69esDzvdpx44dy88//0xISAhbtmxh+/bt/PLLL1x44YVERUUBMGTIkGqbX1t6XP8Ka55ZTNvf7mRLu540a9vZdqRqsTb9d3oue5pFUX3pO1wvTqCUNwTFFq6InAoMBRKNMV0B/zuTx8PMmTOZMWMGs2fPZtGiRfTs2ZO8vLw/feWmst91NcaUObxOnTpHb3/44YdkZWWRmppKWloajRo1Ovp6vvy1n6qIrF2HqCs/wgCHP7iEvTkl70nwJ/tzcwj/4npypC4trnlbz0pWykuC5ZP2N+AJY8whAGPMDst5jsmePXuIi4sjKiqKFStWMGfOHAAaNWrE8uXLKSoqOrr1CxATE0NurnP5udjYWOLi4vjll18AeP/99xk4cCBxcXHExMQcndaECRPKfP2GDRsSHh7OTz/9xMaNGwEYMGAAkydP5uDBg+Tm5jJ16tQamX9va9qmE5vPeI1mhZlsGHeZX59EZYqKWP7G9TQvzGTHGS9Qv2Ez25GUChrBUnA7ACeLyFwRmSUifUobUURGi0iKiKRkZWV5MWLFnX322RQUFJCYmMj9999Pv379AHjiiSc477zzOO2002jSpMnR8YcPH87TTz9Nz549Wbt2Le+++y533XUXiYmJpKWl8cADDwDw5ptvMnr0aPr3748xhtjY2BJff8SIEaSkpJCcnMyHH35Ip06dAOjVqxeXXXYZSUlJDBs2jJNPPrmGW8J7up14Pgu7/4vEvPmkvPF323GqbO5H/yZ57/fMbX0j3U4833YcpYKKlLcr0V+IyAygcQmD7gMeBX4EbgP6AJ8AbU05M5+cnGxSUlL+9Njy5cvp3DkwjuMVt2/fPqKjnWufPvHEE2zdupXnn3++2l/Hn9twziuj6bfjE+Z0vJt+l99nO06lpM+cRNefrmNR9Mkk/d8XhBQ7MU6p6iIiqcaYZNs5fE3AnDRljBlU2jAR+RvwuVtg54lIERAP+OYmrCXTpk3j8ccfp6CggFatWvHOO+/YjuRz+ox+hYX/y6DfyqdImVKf5CF/sx2pQjavXkTrmbewMbQVHW96X4utUhYETMEtxxfAacBMEekARAA7rSbyQZdddhmXXXaZ7Rg+LTQsjM63fMbSZweTlDqWtDpxJJ0+3HasMmVlbiDsw2EUEErkyE+pE1PPdiSlglKwHMN9C2grIkuACcCo8nYnK1WayNp1aDXmS9aHt6PTz2NInznJdqRS7dmVxb7xQ6hrcsm+4COatu5oO5JSQSsoCq4xJt8Yc6Uxppsxppcx5kfbmZR/i64bR/yNU9kS1oJOP41m4Xcf2I70F7l7drH11SE0K9zC+kGv0z4pcE5iU8ofBUXBVaomxCU0If7v37EhvC3dfruV1GnjbUc6as+uLLa+eBbt8ley9IT/0e3kobYjKRX0tOAqdQxi6yfQ5JbprI7oTO/5dzDnvfsxRUVWM2VvzyDr5bNofXgdS096iZ5njbKaRynl0IIbxN555x3GjBlT7jiZmZlH719//fVWLvXny2Ji69P2H9NJjTmVfuteYP4LI8g/VLlf+qou65fN59C4U2lesInlp4wj6YwrrORQSv2VFlxVpuIFd/z48XTp0sViIt8UGRVNz9snMbv5tRyf8zUbnj6ZLeuWezXDoh8nkPDJ+YSbw2waOokep17i1ddXSpUtWL4WVDO+uRe2La7eaTbuDoOfKHe00i7Pd9ttt/HVV19Ru3ZtvvzySxo1asTUqVN55JFHyM/Pp0GDBnz44Yc0atTo6LRyc3NJTExk1apVhIeHs3fvXhITE3n66adJSUlhxIgR1K5dm9mzZzN48GCeeeYZkpOT+fbbbxk7diyFhYXEx8fzww8/VG9b+JmQ0FD6X/8sC6b34rjZ9yDvnkpK74fofd7oGv294kN5B1j41u302/EJa8LaEXPNZ3Ro3q7GXk8pVTW6heunSro83/79++nXrx+LFi1iwIABvPHGGwCcdNJJzJkzh4ULFzJ8+HCeeuqpP00rJiaGU045hWnTpgHO7ygPGzaMSy655OjPN6alpVG7du2jz8nKyuKGG25g0qRJLFq0iM8++8x7M+/jep11Ffuv+YktEW1IXnAPi58axJZ1S2vktZbPnU7mU/3ot+MT5sYPo/mdv9BIi61SPkm3cI9FBbZEa0pJl+eLiIjgvPPOA6B37958//33AGRkZHDZZZexdetW8vPzadOmzV+md/311/PUU09xwQUX8Pbbbx8t1qWZM2cOAwYMODqtI5fuU44mrTrS8J5fmDvxabouf57wdwcyp/Ew2l90Pw0aNT/m6W9Zt5zMLx+kz57pbCOetJNfo6+P/wCHUsFOt3D9UGmX5wsPDz96ebzQ0FAKCgoAuOWWWxgzZgyLFy/mtddeK/HSfSeeeCIbNmxg1qxZFBYW0q1btzIzGGMC7lJ81S00LIy+w//JgdGzSa93On22fULtV3ox98WRrE3/vdLTM0VFrJj3PfOfvZRG755AYs6PzG5yJTF3pPr8r10ppXQL1y+Vdnm+ssZv1sy5DNu7775b6ngjR47k8ssv5/777z/6mOel/Tz179+fv//976xfv542bdqwa9cu3cotRcNmbWj4j0/YtCqN7V8/QY+dXxP5+Zds/KI5WxsOJLL9AJp07EPDpm3+dKy3qLCQHZnr2bYqhbxVP9Fsx890MpkcMLVIaXQJ7S4YS/+mre3NmFKqUrTg+qGzzz6bcePGkZiYSMeOHY9enq80Dz30EJdccgnNmjWjX79+rF+/vsTxRowYwb/+9S8uv/zyo49dffXV3HTTTUdPmjoiISGB119/nYsuuoiioiIaNmx4dBe2KlnLDkm07DCBPbuyWPTdeKLWT6fX1glEbPsQfoFDJpw9EkO+1CLCHCLG7KOx5NMYZ9iqyG5s63gTnQeNpF/dONuzo5SqpIC5PF9NCLbL802cOJEvv/yS999/v0ZfJ5DbsLL25+awedk89qxPxeRsJiRvNyGFhygKrUVRrVgk/jiim3WhXc+BRNauYzuuUhWil+crmW7hKsA5zvvNN9/w9ddf244SVOrE1KNT3zOh75m2oyilapgWXAXAiy++aDuCUkoFND1LuQp0N3zVadsppYKVFtxKioyMJDs7WwtHFRhjyM7OJjIy0nYUpZTyOt2lXEnNmzcnIyODrKws21H8UmRkJM2bH/sPPyillL/RgltJ4eHhJf5Sk1JKKVUW3aWslFJKeYEWXKWUUsoLtOAqpZRSXqC/NFUGEckCNlbx6fHAzmqM4w90noODznNwOJZ5bmWMSajOMIFAC24NEZGUYPtpM53n4KDzHByCcZ5rmu5SVkoppbxAC65SSinlBVpwa87rtgNYoPMcHHSeg0MwznON0mO4SimllBfoFq5SSinlBVpwlVJKKS/QgnuMRORsEVkpImtE5N4ShouIvOAOTxeRXjZyVqcKzPMId17TReR3EelhI2d1Km+ePcbrIyKFInKxN/NVt4rMr4icIiJpIrJURGZ5O2N1q8ByHSsiU0VkkTvP19jIWZ1E5C0R2SEiS0oZHnD9l1XGGP2r4h8QCqwF2gIRwCKgS7FxzgG+AQToB8y1ndsL83wCEOfeHhwM8+wx3o/A18DFtnPX8HtcD1gGtHTvN7Sd2wvzPBZ40r2dAOwCImxnP8b5HgD0ApaUMjyg+i/bf7qFe2yOB9YYY9YZY/KBCcDQYuMMBd4zjjlAPRFp4u2g1ajceTbG/G6M2e3enQP4+/X4KvI+A9wCTAJ2eDNcDajI/F4BfG6M2QRgjAmGeTZAjIgIEI1TcAu8G7N6GWN+xpmP0gRa/2WVFtxj0wzY7HE/w32ssuP4k8rOz3U4a8j+rNx5FpFmwIXAOC/mqikVeY87AHEiMlNEUkVkpNfS1YyKzPNLQGcgE1gM3GaMKfJOPGsCrf+ySq+He2ykhMeKf8+qIuP4kwrPj4icilNwT6rRRDWvIvP8HHCPMabQ2QDyaxWZ3zCgN3A6UBuYLSJzjDGrajpcDanIPJ8FpAGnAe2A70XkF2PM3hrOZlOg9V9WacE9NhlAC4/7zXHWfis7jj+p0PyISCIwHhhsjMn2UraaUpF5TgYmuMU2HjhHRAqMMV94JWH1quhyvdMYsx/YLyI/Az0Afy24FZnna4AnjHNwc42IrAc6AfO8E9GKQOu/rNJdysdmPtBeRNqISAQwHJhSbJwpwEj3bL9+wB5jzFZvB61G5c6ziLQEPgeu8uMtHk/lzrMxpo0xprUxpjUwEbjZT4stVGy5/hI4WUTCRCQK6Ass93LO6lSRed6Es0WPiDQCOgLrvJrS+wKt/7JKt3CPgTGmQETGANNxznJ8yxizVERucoePwzlj9RxgDXAAZy3Zb1Vwnh8AGgCvuFt8BcaPrzpSwXkOGBWZX2PMchH5FkgHioDxxpgSv1riDyr4Hv8HeEdEFuPsar3HGOPXl+wTkY+BU4B4EckAHgTCITD7L9v0px2VUkopL9BdykoppZQXaMFVSimlvEALrlJKKeUFWnCVUkopL9CCq5RSSnmBFlyllFLKC7TgKqWUUl6gBVcpP+JebzddRCJFpI57XdZutnMppcqnP3yhlJ8RkUeASJyLBmQYYx63HEkpVQFacJXyM+5v/c4H8oATjDGFliMppSpAdykr5X/q41wAPQZnS1cp5Qd0C1cpPyMiU4AJQBugiTFmjOVISqkK0KsFKeVHRGQkztWXPhKRUOB3ETnNGPOj7WxKqbLpFq5SSinlBXoMVymllPICLbhKKaWUF2jBVUoppbxAC65SSinlBVpwlVJKKS/QgquUUkp5gRZcpZRSygv+H5eG7m/qSqURAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The max absolute difference is: 1.77636e-15\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "\n", @@ -3616,9 +3180,7 @@ { "cell_type": "markdown", "id": "e271876e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using autograd\n", "\n", @@ -3631,13 +3193,19 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 20, "id": "a9bc2740", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The gradient of f1 evaluated at a = 1 using autograd is: 3\n", + "The gradient of f1 evaluated at a = 1 by finding the analytic expression is: 3\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -3661,9 +3229,7 @@ { "cell_type": "markdown", "id": "8b3edc43", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Autograd with more complicated functions\n", "\n", @@ -3674,13 +3240,24 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 21, "id": "d205e7eb", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Evaluating at x1 = 1, x2 = 3\n", + "------------------------------\n", + "The derivative of f2 w.r.t x1: 12\n", + "The analytical derivative of f2 w.r.t x1: 12\n", + "\n", + "The derivative of f2 w.r.t x2: -4\n", + "The analytical derivative of f2 w.r.t x2: -4\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -3720,9 +3297,7 @@ { "cell_type": "markdown", "id": "a2f9d1cc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable." ] @@ -3730,9 +3305,7 @@ { "cell_type": "markdown", "id": "77a185bb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More complicated functions using the elements of their arguments directly" ] @@ -3741,10 +3314,7 @@ "cell_type": "code", "execution_count": 18, "id": "1c4a4d77", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3769,9 +3339,7 @@ { "cell_type": "markdown", "id": "739eae4d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that in this case, when sending an array as input argument, the\n", "output from Autograd is another array. This is the true gradient of\n", @@ -3784,9 +3352,7 @@ { "cell_type": "markdown", "id": "6f4276ea", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Functions using mathematical functions from Numpy" ] @@ -3795,10 +3361,7 @@ "cell_type": "code", "execution_count": 19, "id": "aa1c1e62", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3823,9 +3386,7 @@ { "cell_type": "markdown", "id": "661c9456", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More autograd" ] @@ -3834,10 +3395,7 @@ "cell_type": "code", "execution_count": 20, "id": "f4538a03", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3859,9 +3417,7 @@ { "cell_type": "markdown", "id": "3a2398ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And with loops" ] @@ -3870,10 +3426,7 @@ "cell_type": "code", "execution_count": 21, "id": "2c98ad0b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3906,10 +3459,7 @@ "cell_type": "code", "execution_count": 22, "id": "02c3d631", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3926,9 +3476,7 @@ { "cell_type": "markdown", "id": "71bb1f87", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using recursion" ] @@ -3937,10 +3485,7 @@ "cell_type": "code", "execution_count": 23, "id": "4aaeea32", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3975,9 +3520,7 @@ { "cell_type": "markdown", "id": "ae549235", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input." ] @@ -3985,9 +3528,7 @@ { "cell_type": "markdown", "id": "4a9b3bf5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Unsupported functions\n", "Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.\n", @@ -3999,10 +3540,7 @@ "cell_type": "code", "execution_count": 24, "id": "36fb2a7e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4021,9 +3559,7 @@ { "cell_type": "markdown", "id": "de241d13", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible." ] @@ -4031,9 +3567,7 @@ { "cell_type": "markdown", "id": "396b2523", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The syntax a.dot(b) when finding the dot product" ] @@ -4042,10 +3576,7 @@ "cell_type": "code", "execution_count": 25, "id": "6de53da2", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4064,9 +3595,7 @@ { "cell_type": "markdown", "id": "6f4d7c50", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we are told that the 'dot' function does not belong to Autograd's\n", "version of a Numpy array. To overcome this, an alternative syntax\n", @@ -4077,10 +3606,7 @@ "cell_type": "code", "execution_count": 26, "id": "bccf2c33", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4102,9 +3628,7 @@ { "cell_type": "markdown", "id": "0c75b991", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Recommended to avoid\n", "The documentation recommends to avoid inplace operations such as" @@ -4114,10 +3638,7 @@ "cell_type": "code", "execution_count": 27, "id": "33d78552", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "a += b\n", @@ -4129,9 +3650,7 @@ { "cell_type": "markdown", "id": "ace05b46", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using Autograd with OLS\n", "\n", @@ -4142,13 +3661,36 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 22, "id": "96581910", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[3.90171693]\n", + " [3.15636147]]\n", + "Eigenvalues of Hessian Matrix:[0.37651284 4.18879738]\n", + "theta from own gd\n", + "[[3.90171693]\n", + " [3.15636147]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAplklEQVR4nO3deZgcZbn38e89k0ySSSDLJCQhMBMCCAQQhAgooEEPERDBXTEiKpqXeDhuKKJRURE8R1EQ3E5AZJmRc9QDuIEO6IQEQoAJAkkISyAL2ScJ2deZvt8/uhM6nV6qe6q7qie/z3XVlenqWp6qrtRdz1LPY+6OiIhIITVRJ0BERKqDAoaIiASigCEiIoEoYIiISCAKGCIiEogChoiIBKKAIT2WmX3HzJqjTkeYzGy8mS2NOh2yf1LAkIoys0Vmts3MNpvZSjO73cwGRJ0uESlMAUOi8B53HwCcCLwJ+Hq0ydk/mVmvqNMg1UUBQyLj7iuBv5MMHACY2VVm9rKZbTKz58zsfWnffdLMHjGz683sNTNbaGbnpn1/mJk9nFr3QWBo+v7M7AIzm2dm681smpkdk/bdIjP7qpk9a2ZbzOzXZjbczB5Ibe8hMxuc7Th2FxOZ2RVmttrMVpjZp9K+n2Zmn8k8jrTPbmafM7OXUvu6xswON7PHzGyjmf3OzOoy9vkNM1uTSvfEtPl9UudniZmtMrNfmVm/jHR+zcxWAr8xs6Fm9pfUOVlnZjPMTPcFyUoXhkTGzA4BzgUWpM1+GTgTGAh8F2g2s5Fp358KvEAyGPwQ+LWZWeq73wKzU99dA1yStq83AHcDXwSGAfcDf864EX8AOBt4A/Ae4AHgG6nt1QCfz3M4I1JpHgVcCvw8V4DJ4RzgZOA04EpgKjAROBQ4DrgoY19DU/u6BJhqZkelvvuvVPpPBI5ILfPtjHWHAE3AJOAKYCnJczI8dbzqL0iyUsCQKNxnZpuAV4HVwNW7v3D337v7cndPuPv/Ai8Bp6Stu9jdb3H3LuAOYCQw3MwagTcD33L3He4+Hfhz2nofAf7q7g+6+y7geqAf8Na0ZW5291XuvgyYATzu7v9y9x3AvSSLz3LZBXzP3Xe5+/3AZuCoPMtn+i933+ju84C5QKu7v+LuG0gGrsx97z7Oh4G/Ah9OBc7PAl9y93Xuvgm4Dvho2noJ4OrUuttS6R4JNKXSPsPVwZzkoIAhUXivux8AjAeOJq3oyMw+YWZPp4pI1pN8uk4vWlq5+w9335r6cwBwMPCau29JW3Zx2t8Hp3929wTJgDUqbZlVaX9vy/I5X+X8WnfvTPu8tcDymYrZd7bjPJhkLqEemJ12/v6Wmr9bh7tvT/v8I5I5vFYze8XMrioizbKfUcCQyKSejm8n+bSPmTUBtwCXAw3uPojk07bl2ES6FcBgM+ufNq8x7e/lJIthSO3LSBb3LCv9CALbQvJGvtuIbm4v23EuB9aQDC7Huvug1DQw1cBgt71yD+6+yd2vcPcxJIvhvmxm7+xm+qSHUsCQqN0InG1mJwL9Sd7QOgBSFcfHBdmIuy8G2oHvmlmdmZ1B8ga42++Ad5vZO82sN8my+x3AzJCOI5+ngfebWb2ZHUGyjqO7dh/nmcD5wO9TuaZbgBvM7CAAMxtlZu/KtREzO9/MjkgF0I1AV2oS2YcChkTK3TuAO0mWyT8H/Bh4jGSRzPHAo0Vs7mMkK8XXkawXuTNtPy8AHwduJvkk/h6SzXt3hnAYhdwA7CR5THcALd3c3krgNZK5ihbgMnd/PvXd10gWMc0ys43AQ+SvSzkytcxmkuf9F+4+rZvpkx7KVL8lIiJBKIchIiKBKGCIiEggChgiIhKIAoaIiARSVZ2PDR061EePHh11MkREqsrs2bPXuPuwwkvmV1UBY/To0bS3t0edDBGRqmJmiwsvVZiKpEREJBAFDBERCUQBQ0REAlHAEBGRQBQwREQkEAUMEREJRAFDREQCUcAQEZFAFDBERCQQBQwREQmk7AHDzG4zs9VmNjfLd18xMzezoeVOh4iIdE8lchi3A+dkzjSzQ4GzgSUVSIOIiHRT2QOGu08nOcZyphuAKwGNESsiUgUiqcMwswuAZe7+TIBlJ5lZu5m1d3R0VCB1IiKSTcUDhpnVA1OAbwdZ3t2nuvs4dx83bFi3u3MXEZESRZHDOBw4DHjGzBYBhwBPmdmICNIiIiIBVXwAJXefAxy0+3MqaIxz9zWVTouIiARXiWa1dwOPAUeZ2VIzu7Tc+xQRkfCVPYfh7hcV+H50udMgIiLdpze9RUQkEAUMEREJRAFDREQCUcAQEZFAFDBERCQQBQwREQlEAUNERAJRwBARkUAUMEREJBAFDBERCUQBQ0REAlHAEBGRQBQwREQkEAUMEREJRAFDREQCUcAQEZFAFDBERCQQBQwREQlEAUNERAJRwBARkUDKHjDM7DYzW21mc9Pm/cjMnjezZ83sXjMbVO50iIhI91Qih3E7cE7GvAeB49z9jcCLwNcrkA4REemGsgcMd58OrMuY1+runamPs4BDyp0OERHpnjjUYXwaeCDXl2Y2yczazay9o6OjgskSEZF0kQYMM5sCdAItuZZx96nuPs7dxw0bNqxyiRMRkb30imrHZnYJcD7wTnf3qNIhIiLBRJLDMLNzgK8BF7j71ijSICISSy0tMHo01NQk/23JWQBTcWXPYZjZ3cB4YKiZLQWuJtkqqg/woJkBzHL3y8qdFhGRWGtpgUmTYGvqOXrx4uRngIkTo0tXilVTadC4ceO8vb096mSIiJTH6NHJIJGpqQkWLSp5s2Y2293HlbyBlDi0khIREYAlS4qbX2EKGCIicdHYWNz8ClPAEBGJi2uvhfr6vefV1yfnx4AChohIXEycCFOnJusszJL/Tp0aiwpvUMAQkWoR4+amoZo4MVnBnUgk/41JsIAIX9wTEQks5s1N9xfKYYhI/E2Z8nqw2G3r1uR8qRgFDBGJv5g3N91fKGCISPzFvLlpLFSgjkcBQ0TiL+bNTSO3u45n8WJwf72OJ+SgoYAhIvEX8+amkatQHY/6khIRqXY1NcmcRSYzSCTUl5SIiKQMGZJ9fsh1PAoYIiLVrKUFNm3ad35NTeh1PAoYIiLVbMoU2Llz3/mJROi7UsAQEalm+d5FCbnSWwFDRKSa5aunCPnFRgUMEZFqk/6S3ubNuZcLudJbnQ+KiFSTzI4Y165NBo7MOovUi407Nu4IbdfKYYiIVJNsL+klEtDQAE1NuBk7hx3MA2O/xLu/9AaGDOwMbddlDxhmdpuZrTazuWnzhpjZg2b2UurfweVOh4hIj5CjXsLXruPSurtorFlKn45lnNf+fRasH8qnjg/vZedK5DBuB87JmHcV8A93PxL4R+qziEh8RT2A0+795+idYzGN3LPgeE4bsZipF09n4YylvLDzMH727NtDS0LZA4a7TwfWZcy+ELgj9fcdwHvLnQ4RkZKVs3O/AIHIm1tIXPrZ5H6z2FnTh87PTGbN9gP4/dK38Nk738boMw7pftoyRFWHMdzdVwCk/j0o14JmNsnM2s2svaOjo2IJFJEyivppvVjl6twvTyBa9/Jr/P7Lj/HZo6ez7OKvUbNj2z6rO0BTE3V3/pojbvkatXW13UtPARXpfNDMRgN/cffjUp/Xu/ugtO9fc/eC9RjqfFCkB8hs5QPJFj1x7n22QOd+JRs9OmuuYYWNZJQvxalhIBtYx2BqKH3/1d754CozGwmQ+nd1ROkQkUqrxuFWyzCAkyccz1GBPdxXcvX46cz87zms2dafmqZ4DCAVVcD4E3BJ6u9LgD9GlA4RqbRqHG41pAGcXlu4nj9c8RiTjpnOmD5LWePZe5mtaWrk6rbxvGXS8fTq2ys2A0iV/cU9M7sbGA8MNbOlwNXAfwK/M7NLgSXAh8qdDhGJicbG7JW3cR5udXdR2ZQpycDW2Ji8WRcoQtu1dReP3z6f1v9ZR+tTQ3lyyzEkeAsHsoGrB97I4I3r2aekqa5u30BQ4v7DpgGURKSyqrEOIyBPOC+3LaH1lkW0PtyXf648hk0cSA1dnDrgOSacvJazPzKEUy45ht5jj8weOBsaYM2aUNNV7XUYIhIX3W2xVOz61Tjcap5jXL94A/dcOYvLxk7n8D6vcuS/NfHv//t2nllzMB875mn+76uzWPPKJmZuOp7vTBvP6ZPfSO/63rmL4NZlvoUQI+5eNdPJJ5/sIhKi5mb3+nr3ZBug5FRfn5zf3Oze1ORulvy3ubm49XuKLMfYWdfX/3D0FH/LgGe9hk4H9wPY4BeOmOU//8g0f+mhRZ7oSuTfblPT3udt99TUFPohAO0ewj048iBQzKSAIRKyXDethoZggaCCN73I5DjGRTT6qf3n+LfObPMZP3/Gd27ZWdx2uxtsgwT0FAUMEek+s+w3/FxTQ0Ow9c2iOZ5SZdx8t/z0Fr/3qlk++diHvYvsx5gI4xiLuOnvs14RwUYBQ6SnKPWmEYZcOYR8U3r6ekIOo7nZExk3383U+0U0+wA2+qqaEfE7xiLPe1gBQ5XeIlEqZx9FQeRq39/QkHud9BfsYvJ+QCkWTn+VqR+fzupLrsAyXiTsz1ZuG/xV1m7qw0F3Xh/dMeaqbI/qXZYwok6lJuUwpMeJwxN6thxOc3PuHEZmUUxYOaQy57Q2vLrB//iNWf7vx0/zI3u/sudwchU57XWcUeQC8xU7RZTDiDwIFDMpYEiPE+c6gIaGygWzMrS26tzR6bNunePfe0ebn3Hg0/5x7vCFNHkX5qtqRvgD46b4/L++7InGxuiDdjb5goLqMBQwZD8UZQ6j0FNzJZvMhnQeFj3yqk+9+GH/4KiZPtjWJWMvXf6N3v/lO2r6ZD+WYo+zUrmNfPVIRaZDAUOkJ4jqPYag+y3XzTFzu0GLvzJsXLbR//TNx/3y46f5G9KKmUbVLPdPHznd7/6PR331cx2FA1LQ46zk71Vbmz3NtbVFbyqsgKGuQUSi1tJS+T6CcnSrTVMTLFpU3n1n6xrELHk7LJCerp1dPHX3C7Q2r6b1yUHM3HAsnfSmni2MH/YcE87YyoRPH8LR543Bauz17YTVPXklz5tZ7u+KvG+H1TWIAobI/qhc4zsEkeummxk0Uv1LvXrEWbT+8mVa/1nLQ0uPZl2qh9eT+s1nwgmrmPChgbz1M2Ppc2Cf4vdZ7I2+kuctxOCkvqREpHRlGN8hsFxNP92hqQk3Y+vAEdzZ8CWO+fRpNJ52MJ+540weWXYYFxw+j99ePpNVczuYvfUYfvDYeM768pvyBwvI3vzXDM477/XPQfrEquR5i2OT5TDKtSo1qQ5DJCSllsWHUaeRoz5hfd/hPn7QU96bHQ7u/dji5wx9wn9yYZvPve+lwn0zFTJ58r6t0oqt+K50nVNIdUio0ltEuqXYm1FYN8vmZu/q22+v7ex+s/rEfvP9ylPa/KEfzvZtr20r9ciyy1fxXUwrrSjfzC+RAoZITxbHm1I3mr5uXrXZ7//uE/7FN03zsX1e8oto3vNOREev4T7jnd/2lXNWv75COY4/3zsvcX4fJgRhBYyyj7gnIkXKbEW0u7sQiHbMiCK6o0h0Jnj6f1+g9a5VtD4+kEfXj2Unb6Yv23hbwzxOPnsUmz7Zir3vSIbWGGekr1yu4y800l+1jQIYAbWSEombKJu85lMgXcvaV/Dgr16m9SHjwSVHscaHAvDGvi8w4bgVTPjAAZwxaSz9hvTr1n5Klm+kP+ixowBCeK2kIi9mKmZSkZTsF+JaPJKlDmNX775+V9MUP7bPi3tmD69Z5RePmeF3XfaIr3hmVfH7Kefx5yvqimMxoHso6UJ1GCI9VBw6JMyia1eXL7z8R76+73DvwnwRjX4Rzd6HbX5d/2t8fb/hnjBL9s3UnZttTI8/EiE1NAgrYET6HoaZfcnM5pnZXDO728z6RpkekViIUfv7FU+v4s5Jj/Dxwx5lZN1aDvvZVxi0fSUn9n2em0++g6s+tJCtg0fx9S3fYuC2VZg7tmRJ97poj9HxR27KlL2LySD5Ob2L+UoKI+qUMgGjgIVAv9Tn3wGfzLeOchiy34ioeGTr2q3+9+va/YqT2/z4vi/seag9yFb7xNGP+B2fneHLZq94PY2ZT79h5QjiWjxUaSEVz1HtfUmZ2ShgFnACsBG4D7jJ3VtzraNKb4mtKPqDCoEnnDn3vETr7ctpfewApq87lh30pY4dnDl4HhNO28iES0byxg8cSU2vjAKJXJXTu1Wim5GeLqQGAD2i0hv4ArAZ6ABaciwzCWgH2hsbG4uKqiIVEVWPsyVa8cwqv+uyR/ziMTN8RM3KPUk+ts+L/qWT2vyBa570LR1bCm+o0HjgpeQwlLPYW8zqMKIMFoOBfwLDgN4kcxgfz7eOiqQklmJeSbt17VZv/UG7f2Vcm5/Q9/k9yRtqHX5R0yP+m0tn+NInlxe/4Xzdkpf4Bng1Bd6KUSspB/gQ8Ou0z58AfpFvHQUMieUTaMyawSa6Ev7sH17w689v8wkNT3pftjq492aHnzXoKf/Bu9p8dvNz3rWrq7QdpA8Rmu3YGxpC7WNqz+9c7O8ex2slIj0hYJwKzAPqAQPuAP4j3zoKGPu5uD6BxiCHsXLOam+e/Ih/YswMH1mzYk8SxvZ5yb9w4jT/63ee8M2rNnd/R9l+g91Bo7s35XxFXMX+7nG9ViJS9QEjeQx8F3gemAvcBfTJt7wCxn4uBjfmrCK4OW17bZs/9MPZfuUpbX5iv/mvP9zbGv9o46N+26em+6tPLH89fWE9aZfzN8i17Vwjz+XbZ1yvlYj0iIBR7KSAsZ+LWdHPXooZ4rOEm3eiK+Fz73vJf3Jhm58z9Anvx5Y9xUzjBz3l101o8/a7shQzhR3Myv0Wdra05sp15NtnnK+VCFQsYAAPASeEsbPuTgoY+7lqf2os8ua9+rkO/+3lj/onj5juB9cs37PK0XUL/PMnTPO/XP2Eb1qxKf8+wz5nxW6vlC7UM5cv5Riq/VoJWSUDxkmp1ky/AUaGsdNSJwWM/Vxcy6WD3hQL3MS2b9ju//zxU37VaW1+Ur/n9nw9xNb6hw991G+9ZLovnrm0uLSV8qRdqL+loL9BiONnqA6jeypeJAV8AHgGuJrU29mVnhQwYiDqlidh77+5OdmqZ08lQJEtfArdmNLTm6NoJYH5ecOe8Ho2O7j3Yqe/beC//Pv/1uZP3D7PO3d0ln58peQICt1oQwqQRVErqW6paMBItWI6DrgMWAMsBS4OIwHFTAoYEetpT23Nze51dfve0Hr3Dn5MhZqC5iuDT00LafI39H7FLz9+mv/pm4/7xmUbwz3GbL/Z5MnZb6Zh3uRVjxAblSySegRYDjwIXAOcDxwB3AxMDSMRQScFjIj1tHLhfC+eBT2mfDfFfNtPTTtr+/jqb91UxoP0fZ+0J0/OHfjDvMn3tOulilUyYBxHaqClLN/NDyMRQafYB4wwKvjirKc9MeZr9x/0mHLcFDf0S3YBnr0ICk9E+ZuHNbZ1IT0tR1rFYtGsFhgTRiKCTrEOGMX+56jG/0w97YkxjBxGc7Mn+vbba93N1PtFNPtSRsXzfOUL/GFfl9X2UNRDxSJgVHqKdcAo9mZajTffagxy+ZRYh7Fj0w5/+Kanfcrpbf7m/nP9Y9zlC2nyLsxX1Q73B9/yTX+5bXF8z1eha083+R5HASNuii2uqdbinZ52MwnQSirRlfAX/vaK3/zBaf6e4bN8ABsd3GvZ5acf8Ix/96w2f+yWOb5r267s24/b+YprIJOyCStgRDYeRiliPR5Gsf3Wl2ugewnFawvX84+fzaf1/l20vjSGxV2HAHB4r8VMOGoxE97Th7MmH83AxoERp7REVTp+h5SmR4yHUewU6xxGnOow4vhUG1epc5Uw822DR/rvjprip/af4zV0OrgfyHp/38jH/JcXPewL/rEo6tSKlAQVScVQHFpJqbghu4xznbir2Vd8/UbfWdtnr3O1mXr/dp8f+NVvb/NHf/XsvsVMCsZShRQwJLtqrUwv5004SxDdQj9fTUNx50rBWKpUWAGjpkCJlVSbJUuKmx+1lhaYNClZn+Oe/HfSpOT8UrY1ejTU1MDo0XTdfhcz/3sO6y+9ArZu3WvRerYxlLXZt5PrXE2Zss922Lo1OT9uMs5FSedTJIMqvXuaaqtMDyu9uwNP2g19K/34DLfQzMXUUMR1nmvfNTXJoJbJDBKJ4Nsvtyzngvp6mDpVFdv7qbAqvZXD6GmuvTZ5c0hXX5+cH0fdzBFtWLKBe782i7WfzJ6LuOWAK/CDR2VfuaGhuHPV2Fjc/KhUU05IqooCRk8zcWLySbKpKfnk29QU7yfLIm/Cnds7eWzqHL571jROP/BZGpr68/4fnsbgztVZl++/eTW1P/zP7IHhpz8Nfq5aWmDz5n3nxzEYV1uxpFSPMCpCKjWp0jsEcWvlE6Ai+ZWHl/ivPvawv//gx3wg65PvN9Llp/Sf4988o82n3/y0Jw5tzF+B3Z3jztXrbLFdoVdKNTZ8kLJCraT2I0FudkGXiWMrn4y0b7npFr/v67P8c8dN8yN6L9yT1ENrl/pnjnrYf/elmb7mxbX7bqNcx1ZtN+C4/s4SGQWMOAvzKT7ogDZBbhCVuPGVcOydOzp91q1z/HvvaPMzDnzaa9nl4N6fTX7+QY/7TR+Y5s/f/7In7iqw7XLlnqqxG5e45SQlUj0iYACDgD8AzwPzgbfkW74qAkbYT3dBbvJBA0G5b3xFHPuiR171qRc/7B8cNdMH2Wt7ipnG1c/zb7y1zafd+C/fsWlHSdsOXbXlMEQyhBUwIm1Wa2Z3ADPc/VYzqwPq3X19ruWrollt2M1agzTlDNrcs9xNbvNsf9PMOUz75Xxa/7iN1ucbeXHXYQAcUrucCWNeZsK5tbzzc0cx9KiGordd9ubCaqYqVa7q+5ICDgQWkmNwpmxTVeQwwn6KD/J02xDwjeVyP6XnOPYuzHuxM7k7Nvt5w57wG983zZ/78wJPdCW6te2KFQupiEeqGD3gTe8xQAfwGzP7l5ndamb9Mxcys0lm1m5m7R0dHZVPZbHCbqtf6L2KlhbYtGnf9Wpqks1A09/0zdXkFkJ5K7jzoIOzzl9pI/jKaY/yzx//i3UbevHX1W/mC/e8nWPOPxyrsWAbj/odiIkTkzmZRCL5r3IWsj8KI+qUMgHjgE7g1NTnnwLX5FunKnIY5XiKz/d0G2Dc6Lxp6EZ6N63Y5H/+1uP+H2+c5kfVvewX0eyb2Xtbib791KmiSMSo9kpvYASwKO3zmcBf861TFQHDvbLFF/nGpQ5SSVtEhW7Xri5/8o55fu3Zbf72gf/y3uxwcO/HFj932BN+w3un+atf/rEnGhvLc+wqFhIpSdUHjOQxMAM4KvX3d4Af5Vu+agJGLuW44QXNYWQr729uLrjsklnL/NefnO4fOfRRb7A1e75+U7/n/Guntvk/rn/Kt722rfvHEZSChkjRekrAOBFoB54F7gMG51u+qgNGKUUqpb6MlyvXkZ5ryPX2cmpa23u4H1O3YM+skTUr/JLDZ3jL5x7xVXNXh3xyAlKxlEhJekTAKHaq6oBRbFv+Ym6OmYFl8uTC6+bJmWym3j/Bbf6uhif9xxe0+Zx7Xgzemqmc9D6ESEnCChjq3rxSiu0au7vvHWSO2XzeeXD//bBkCZ3DD6Z25TKytU9yYO5Hvs+Rv7qCvoP6Ft5PJVVL9+IiMRPWexgKGJVSbAAI8ea441e/ofbzn6PXru175iWw7GNExHXcDKi+sT5EYkLjYVSbYsepGDIk+/xC7x20tOBNTXhNDRv6jeC6/teycvJ39goWADU4bhl5jDh21Z2u2sb6EOlpwijXqtRU1XUY7sFb+DQ3u/fuvW9ZfV1dznWWzV7hD4//lm+3Pnuts5W+nijU1LaaWhyplZRI0VAdRg+Wq+iloQHWrAFg27ptTP/lPFrv3Uzr3FHM3XEkCxnNaLKsV1sLXV37zldRjsh+IawiqV5hJEZClmNkNF+3juvfPY3WWQcwY92x7GAcfdjOmUPmcfE7ptH0tyVkHbo6W7BQUY6IFEkBI44aG7PmMBZ7I1feP57j+rzEv588iwnvG8CZ/28s9UNPTi4wOvt6+2hoSA5Pqv6QRKQIqvSOkW3rtvHgf87mtzUfZxt7N2ndYX1YMv4TLH1yBXO2H8mP28fzrinjqB+aVgmcrVI4mwEDuh8sWlqK77CwlHVEJDaUw4iQJ5y5975E6+3LaX1sANPXHst2TqaO41hY34/LEzdz4I7VcOih9LnuOt5W6Ca/+/vd71/kqp/KUeQVWOb4EIsXJz+npyGMdUQkVlTpXWGr5nbw0C9epPXvTuuiI1mZGA7A2D4LmDB2GRPeW8/bLhtL/4P26em9eOV6b6GU7eodCpHIqNK7Smxfv51Hb3mO1j9spHXOCJ7edjQwjAZby9mHvsCEd77I2ZOP4JA3HwEcEe7Or702+0hx3a3szpVDyZdzKWUdEYmV/bsOowxl6p5w5v1xATe8dxrnDnuSIYMT/NuVJ3HDE29lUJ9tXDdhGu13zWf1zsHcvfitfOq2MznkzSO7vd+scg2Y1N0ioFIGM8r13ZAhqtcQqRZhvMxRqSnUF/dC7Pl09XMd/tvLH/VPHjHdD65ZvmdzR9ct8M+fMM3/cvUTvmnFpvDSHrVSe97NXKeubt8XFNX7rEjoUG+1AZQyUl2Ank+3b9ju/7j+Kb/qtDY/qd9ze1YdYmv9w4c+6rdeMt0Xz1xaXFqrTSlvXGeuE3QschHplrACRs+t9M5slQPJ8vvdRTJFdO7nCWfZV2/gwF/+kAHbVrOUQ7iKH/B7PszpA+cx4ZT1nD3xIE666Chq62q7cYT7GfU+K1IR6q22kEKtcvJ9f+21dF15FTXLl7G210H8pfMcPsTv6c/rwaezd192/eRn9Lv80lIORUAtp0QqRL3VFpKvVU5LC6xdu89XDry4roFtH/8MtcuXYjhDO1dxCXfsFSwAeu3aTr/rrylDwvcj6n1WpKr03ICRr1XOpEmwefM+XxkwZtMz9GP7PvOzUpPQ7ilXKy4RKYueGzCyPL06kFi7bu96jQy1ZOmoL5dCY1NIYRMnJoufEonkvwoWIrHVcwPGxIl03vwLdtYP3NOBq0H2UebS5MxNZFLRiYjsZyIPGGZWa2b/MrO/FFx4zpy8L3h5wnnx7wv52Yce5oIRjzP40vezfOug4EGgcGKT/6roRET2Q5G3kjKzLwPjgAPd/fx8y44z8z1tpMzAna6DD6H9hEv59cJ30PrSGBZ3HQLA4b0WM+Goxfx83nisQK6iKGrBIyJVpke0kjKzQ4B3A7cWvXIq0NUuX8pxD/yIrudf5KSDlvLLi6az4B+LWbCriV/MfRvWlKOeobb29YrWYqiiW0T2U1F3PngjcCVwQK4FzGwSMAng5BzL9Gcrtx56DbYkS5v+XB3wpRcp5XofIBtVdIvIfiqyHIaZnQ+sdvfZ+ZZz96nuPq5QdspeXZK947ogTTezvQ/QuzfU1e09TxXdIrIfi7JI6nTgAjNbBPwP8A4za+7WFidNyh00rr02mTtYsiQ5wFD6ctmCym9+A7fdpncEwqQR90SqWuSV3gBmNh74SlGV3rlkq5Qu1K+UlJ9+A5HI9Ki+pEINGNk6rlOfRdHTbyASmR4VMIIaZ+btqea0OWW7AalX1OjpNxCJTI9oVlu0urr8wSJXpXQpI8RJuPQbiFS96goYO3fm/i5fpbR6RY2efgORqhf1exjFqavLHjQKlYPvDiJTpiRbSTU2Jm9UqmytHP0GIlWvuuowxozx9lWr1NJGRKQI+2cdxpAhGj9BRCQi1VUkBcngoAAhIlJx1ZXDEBGRyChgiIhIIAoYIiISiAKGiIgE0jMDhnpFFREJXfW1kioks1fUxYuTn0Gtq0REuqFn5DDScxSXXLL3i32Q/DxlSiRJExHpKao/h5GZo+jqyr6cxuIWEemW6s9hTJmyb44imzj3iqo6FxGpAtWfwwiSc+jdO769oqrORUSqRPXnMILkHMzKn45SZcshqc5FRGKo+gNGtnEWMu3cGd8bcK4ckupcRCRmqj9gTJy4dw+2ucT1BqyR6ESkSlR/wIBk0Fi0KDk2dFNT9mXiegPWSHQiUiUiCxhmdqiZtZnZfDObZ2ZfCGXD1XYDzswhaYwPEYmpKFtJdQJXuPtTZnYAMNvMHnT357q11WocClRjfIhIFYgsYLj7CmBF6u9NZjYfGAV0L2CAbsAiImUQizoMMxsNvAl4PMt3k8ys3czaOzo6Kp42ERFJijxgmNkA4P+AL7r7xszv3X2qu49z93HDhg2rfAJFRASIOGCYWW+SwaLF3e+JMi0iIpJflK2kDPg1MN/dfxJVOkREJJgocxinAxcD7zCzp1PTeRGmR0RE8oiyldQjQIw7eRIRkXSRV3qLiEh1UMAQEZFAFDBERCQQBQwREQlEAUNERAJRwBARkUAUMEREJBAFDBERCUQBQ0REAlHAEBGRQBQwREQkEAUMEREJRAFDREQCUcAQEZFAFDBERCQQBQwREQlEAUNERAJRwBARkUAUMEREJBAFDBERCSTSgGFm55jZC2a2wMyuijItIiKSX2QBw8xqgZ8D5wJjgYvMbGxU6RERkfyizGGcAixw91fcfSfwP8CFEaZHRETy6BXhvkcBr6Z9XgqcmrmQmU0CJqU+7jCzuRVIW3cNBdZEnYgAlM7wVEMaQekMW7Wk86gwNhJlwLAs83yfGe5TgakAZtbu7uPKnbDuUjrDVQ3prIY0gtIZtmpKZxjbibJIailwaNrnQ4DlEaVFREQKiDJgPAkcaWaHmVkd8FHgTxGmR0RE8oisSMrdO83scuDvQC1wm7vPK7Da1PKnLBRKZ7iqIZ3VkEZQOsO2X6XT3PepNhAREdmH3vQWEZFAFDBERCSQWASMQl2EWNJNqe+fNbOTgq5b4XROTKXvWTObaWYnpH23yMzmmNnTYTVx60Y6x5vZhlRanjazbwddt8Lp/GpaGueaWZeZDUl9V5HzaWa3mdnqXO//xOjaLJTOuFybhdIZl2uzUDrjcG0eamZtZjbfzOaZ2ReyLBPu9enukU4kK7xfBsYAdcAzwNiMZc4DHiD57sZpwONB161wOt8KDE79fe7udKY+LwKGxuR8jgf+Usq6lUxnxvLvAf4Zwfl8G3ASMDfH95FfmwHTGfm1GTCdkV+bQdIZk2tzJHBS6u8DgBfLfe+MQw4jSBchFwJ3etIsYJCZjQy4bsXS6e4z3f211MdZJN8tqbTunJNYnc8MFwF3lyktObn7dGBdnkXicG0WTGdMrs0g5zOXWJ3PDFFdmyvc/anU35uA+SR70EgX6vUZh4CRrYuQzIPOtUyQdcNS7L4uJRnZd3Og1cxmW7K7k3IJms63mNkzZvaAmR1b5LphCLwvM6sHzgH+L212pc5nIXG4NosV1bUZVNTXZmBxuTbNbDTwJuDxjK9CvT6j7BpktyBdhORaJlD3IiEJvC8zO4vkf8oz0maf7u7Lzewg4EEzez71FBNFOp8Cmtx9s5mdB9wHHBlw3bAUs6/3AI+6e/oTX6XOZyFxuDYDi/jaDCIO12YxIr82zWwAyYD1RXffmPl1llVKvj7jkMMI0kVIrmUq2b1IoH2Z2RuBW4EL3X3t7vnuvjz172rgXpJZwkjS6e4b3X1z6u/7gd5mNjTIupVMZ5qPkpHlr+D5LCQO12YgMbg2C4rJtVmMSK9NM+tNMli0uPs9WRYJ9/osd8VMgIqbXsArwGG8XvlybMYy72bvipsngq5b4XQ2AguAt2bM7w8ckPb3TOCcCNM5gtdf2jwFWJI6t7E6n6nlBpIsS+4fxflM7WM0uStpI782A6Yz8mszYDojvzaDpDMO12bqvNwJ3JhnmVCvz8iLpDxHFyFmdlnq+18B95Os7V8AbAU+lW/dCNP5baAB+IWZAXR6sifL4cC9qXm9gN+6+98iTOcHgclm1glsAz7qyasobucT4H1Aq7tvSVu9YufTzO4m2XJnqJktBa4GeqelMfJrM2A6I782A6Yz8mszYDoh4msTOB24GJhjZk+n5n2D5MNBWa5PdQ0iIiKBxKEOQ0REqoAChoiIBKKAISIigShgiIhIIAoYIiISiAKGiIgEooAhIiKBKGCIdENqPIKzU39/38xuijpNIuUS+ZveIlXuauB7qY7m3gRcEHF6RMpGb3qLdJOZPQwMAMZ7clwCkR5JRVIi3WBmx5Mc+WyHgoX0dAoYIiVKjVzWQnKksi1m9q6IkyRSVgoYIiVIjbR2D3CFu88HrgG+E2miRMpMdRgiIhKIchgiIhKIAoaIiASigCEiIoEoYIiISCAKGCIiEogChoiIBKKAISIigfx/Hm7fJBVxtDcAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Using Autograd to calculate gradients for OLS\n", "from random import random, seed\n", @@ -4204,22 +3746,94 @@ { "cell_type": "markdown", "id": "c8917005", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Same code but now with momentum gradient descent" ] }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 11, "id": "8cdf3bb8", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.]\n", + " [3.]]\n", + "Eigenvalues of Hessian Matrix:[0.33073635 4.59244252]\n", + "0 [-12.86901412] [-14.65511131]\n", + "1 [-0.62619801] [0.50248093]\n", + "2 [-0.58110078] [0.4662935]\n", + "3 [-0.53925134] [0.43271219]\n", + "4 [-0.50041579] [0.40154933]\n", + "5 [-0.46437708] [0.37263074]\n", + "6 [-0.4309338] [0.3457948]\n", + "7 [-0.39989901] [0.32089151]\n", + "8 [-0.37109927] [0.2977817]\n", + "9 [-0.34437362] [0.2763362]\n", + "10 [-0.31957268] [0.25643515]\n", + "11 [-0.29655785] [0.23796732]\n", + "12 [-0.27520049] [0.22082951]\n", + "13 [-0.25538123] [0.20492591]\n", + "14 [-0.23698931] [0.19016765]\n", + "15 [-0.21992192] [0.17647225]\n", + "16 [-0.20408369] [0.16376316]\n", + "17 [-0.18938609] [0.15196934]\n", + "18 [-0.17574697] [0.14102488]\n", + "19 [-0.16309011] [0.13086862]\n", + "20 [-0.15134476] [0.12144378]\n", + "21 [-0.14044529] [0.1126977]\n", + "22 [-0.13033076] [0.10458149]\n", + "23 [-0.12094466] [0.09704979]\n", + "24 [-0.11223453] [0.09006051]\n", + "25 [-0.10415168] [0.08357457]\n", + "26 [-0.09665093] [0.07755574]\n", + "27 [-0.08969037] [0.07197037]\n", + "28 [-0.08323109] [0.06678724]\n", + "29 [-0.07723699] [0.06197739]\n", + "theta from own gd\n", + "[[3.78328788]\n", + " [3.17389661]]\n", + "0 [-0.07167458] [0.05751393]\n", + "1 [-0.06651275] [0.05337192]\n", + "2 [-0.06017412] [0.0482856]\n", + "3 [-0.05393894] [0.0432823]\n", + "4 [-0.04818384] [0.03866422]\n", + "5 [-0.04298722] [0.0344943]\n", + "6 [-0.03833241] [0.03075913]\n", + "7 [-0.03417536] [0.02742338]\n", + "8 [-0.03046702] [0.02444769]\n", + "9 [-0.02716036] [0.02179432]\n", + "10 [-0.02421234] [0.01942874]\n", + "11 [-0.02158422] [0.01731985]\n", + "12 [-0.01924134] [0.01543985]\n", + "13 [-0.01715276] [0.01376392]\n", + "14 [-0.01529089] [0.01226989]\n", + "15 [-0.01363112] [0.01093804]\n", + "16 [-0.01215151] [0.00975075]\n", + "17 [-0.0108325] [0.00869234]\n", + "18 [-0.00965667] [0.00774881]\n", + "19 [-0.00860847] [0.00690771]\n", + "20 [-0.00767405] [0.0061579]\n", + "21 [-0.00684106] [0.00548948]\n", + "22 [-0.00609848] [0.00489361]\n", + "23 [-0.00543651] [0.00436243]\n", + "24 [-0.0048464] [0.0038889]\n", + "25 [-0.00432034] [0.00346678]\n", + "26 [-0.00385138] [0.00309047]\n", + "27 [-0.00343333] [0.00275501]\n", + "28 [-0.00306065] [0.00245596]\n", + "29 [-0.00272843] [0.00218938]\n", + "theta from own gd wth momentum\n", + "[[3.99264591]\n", + " [3.00590115]]\n" + ] + } + ], "source": [ "# Using Autograd to calculate gradients for OLS\n", "from random import random, seed\n", @@ -4279,22 +3893,36 @@ { "cell_type": "markdown", "id": "29cd040f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ - "## But noen of these can compete with Newton's method" + "## But none of these can compete with Newton's method" ] }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 13, "id": "e24e4dff", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.52521419]\n", + " [2.64532661]]\n", + "Eigenvalues of Hessian Matrix:[0.27967231 4.7957411 ]\n", + "0 [-14.06500537] [-16.29352315]\n", + "1 [1.09912079e-14] [1.10675358e-14]\n", + "2 [-8.50014503e-17] [-2.94902991e-17]\n", + "3 [-8.50014503e-17] [-2.94902991e-17]\n", + "4 [-8.50014503e-17] [-2.94902991e-17]\n", + "beta from own Newton code\n", + "[[4.52521419]\n", + " [2.64532661]]\n" + ] + } + ], "source": [ "# Using Newton's method\n", "from random import random, seed\n", @@ -4339,9 +3967,7 @@ { "cell_type": "markdown", "id": "7c4cabd0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Including Stochastic Gradient Descent with Autograd\n", "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**." @@ -4349,13 +3975,45 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 14, "id": "ac8d1b74", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.0041616 ]\n", + " [2.90398269]]\n", + "Eigenvalues of Hessian Matrix:[0.27700763 4.28666339]\n", + "theta from own gd\n", + "[[4.0041616 ]\n", + " [2.90398269]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqzklEQVR4nO3deXhU1f0G8PebDQibQBAQSAIVcV+juBva2qKidrMVo9XalkoXa2trVfpzqdLW2n0vLhVNamtbtXXfOhN2ISj7JkvYl7DIEjAhme/vjzuJwzDLmZm7zcz7eZ55SGbucu7l5rz3nHPvHVFVEBERJVPgdQGIiCg7MDCIiMgIA4OIiIwwMIiIyAgDg4iIjDAwiIjICAODcpaI3CcitV6Xw04iUi0iG70uB+UnBga5SkQaReSgiOwXka0i8oSI9PC6XESUHAODvHClqvYAcDqAMwDc5W1x8pOIFHldBsouDAzyjKpuBfAarOAAAIjInSKyWkT2ichSEfl0xGc3ich0Efm5iOwWkbUiclnE58NEpD487xsAyiLXJyJXicgSEXlfRIIickLEZ40i8n0RWSgizSLymIgMEJFXwst7U0T6xNqOjm4iEbldRLaLyBYR+VLE50ER+Ur0dkT8riLydRF5L7yuB0TkIyIyS0T2isgzIlIStc67RWRHuNw1Ee93Ce+f9SKyTUT+LCLdosr5AxHZCuCvIlImIi+G98kuEZkmIqwXKCYeGOQZERkC4DIAqyLeXg3gIgC9AdwPoFZEBkV8PgrAClhh8DMAj4mIhD/7G4B54c8eAHBjxLqOA/A0gNsA9AfwMoAXoirizwK4FMBxAK4E8AqAu8PLKwBwa4LNGRgu82AAXwbwh3gBE8cYAGcBOBfAHQAmA6gBMBTAyQDGRa2rLLyuGwFMFpGR4c8eCpf/dADHhqe5J2revgAqAIwHcDuAjbD2yYDw9vJ5QRQTA4O88LyI7AOwAcB2APd2fKCq/1TVzaoaUtV/AHgPwDkR865T1UdUtR3AFACDAAwQkXIAZwP4P1VtUdWpAF6ImO8LAF5S1TdU9RCAnwPoBuD8iGl+p6rbVHUTgGkA3lbVd1W1BcBzsLrP4jkE4EeqekhVXwawH8DIBNNHe0hV96rqEgCLAbyuqmtUdQ+s4Iped8d21gN4CcDnw8H5VQDfUdVdqroPwI8BXBsxXwjAveF5D4bLPQhARbjs05QPmKM4GBjkhU+pak8A1QCOR0TXkYh8UUTmh7tI3od1dh3ZtbS14wdVPRD+sQeAYwDsVtXmiGnXRfx8TOTvqhqCFViDI6bZFvHzwRi/Jxqc36mqbRG/H0gyfbRU1h1rO4+B1UooBTAvYv+9Gn6/Q5OqfhDx+8OwWnivi8gaEbkzhTJTnmFgkGfCZ8dPwDrbh4hUAHgEwDcB9FPVo2CdbUucRUTaAqCPiHSPeK884ufNsLphEF6XwOru2ZT+FhhrhlWRdxiY4fJibedmADtghctJqnpU+NU7fIFBh8NaD6q6T1VvV9XhsLrhvisiH8uwfJSjGBjktV8DuFRETgfQHVaF1gQA4YHjk00WoqrrADQAuF9ESkTkQlgVYIdnAFwhIh8TkWJYffctAGbatB2JzAfwGREpFZFjYY1xZKpjOy8CMBbAP8OtpkcA/EpEjgYAERksIp+MtxARGSsix4YDdC+A9vCL6AgMDPKUqjYBeBJWn/xSAL8AMAtWl8wpAGaksLjrYA2K74I1LvJkxHpWALgewO9gnYlfCevy3lYbNiOZXwFohbVNUwDUZbi8rQB2w2pV1AG4RVWXhz/7AawuptkishfAm0g8ljIiPM1+WPv9j6oazLB8lKOE41tERGSCLQwiIjLCwCAiIiMMDCIiMsLAICIiI1n18LGysjKtrKz0uhhERFll3rx5O1S1f/IpE8uqwKisrERDQ4PXxSAiyioisi75VMmxS4qIiIwwMIiIyAgDg4iIjDAwiIjICAODiIiMMDCIiMgIA4OIiIwwMIiIyAgDg4iIjDAwiIjIiOOBISKPi8h2EVkc47PviYiKSJnT5SAiosy40cJ4AsCY6DdFZCiASwGsd6EMRESUIccDQ1WnwvqO5Wi/AnAHAH5HLBFRFvBkDENErgKwSVUXGEw7XkQaRKShqanJhdIREVEsrgeGiJQCmAjgHpPpVXWyqlapalX//hk/zp2IiNLkRQvjIwCGAVggIo0AhgB4R0QGelAWIiIy5PoXKKnqIgBHd/weDo0qVd3hdlmIiMicG5fVPg1gFoCRIrJRRL7s9DqJiMh+jrcwVHVcks8rnS4DERFljnd6ExGREQYGEREZYWAQEZERBgYRERlhYBARkREGBhERGWFgEBGREQYGEREZYWAQEZERBgYRERlhYBARkREGBhERGWFgEBGREQYGEREZYWAQEZERBgYRERlhYBARkREGBhERGWFgEBGREQYGEREZcTwwRORxEdkuIosj3ntYRJaLyEIReU5EjnK6HERElBk3WhhPABgT9d4bAE5W1VMBrARwlwvlICKiDDgeGKo6FcCuqPdeV9W28K+zAQxxuhxERJQZP4xh3AzglXgfish4EWkQkYampiYXi0VERJE8DQwRmQigDUBdvGlUdbKqVqlqVf/+/d0rHBERHabIqxWLyI0AxgL4mKqqV+UgIiIzngSGiIwB8AMAl6jqAS/KQEREqXHjstqnAcwCMFJENorIlwH8HkBPAG+IyHwR+bPT5SAi8lxdHVBZCRQUWP/Wxe2N9yXHWxiqOi7G2485vV4iIl+pqwPGjwcOhDtV1q2zfgeAmhrvypUCP1wlRUSUO+K1IiZO/DAsOhw4YL2fJTwb9CYiyjmJWhHr18eeJ977PsQWBhGRXRK1IsrLY88T730fYmAQEdklUSti0iSgtPTw90tLrfezBAODiMguiVoRNTXA5MlARQUgYv07eXLWDHgDDAwiIvska0XU1ACNjUAoZP2bRWEBMDCIiOyTA62IRHiVFBGRnWpqciYgorGFQURERhgYRERkhIFBRERGGBhERGSEgUFEREYYGEREucCFR6fzsloiomzn0qPT2cIgIsp2Lj06nYFBRJTtXHp0OgODiCjbufTodAYGEVG2c+nR6QwMIspt6Vw95MIVR7Zy6aGHoqq2LvCIFYg8DmAsgO2qenL4vb4A/gGgEkAjgM+r6u5ky6qqqtKGhgbnCktEuSX66iHAOvNOVJmmM4/Picg8Va3KdDlutDCeADAm6r07AbylqiMAvBX+nYjIXulcPeTSFUfZyPHAUNWpAHZFvX01gCnhn6cA+JTT5SCiPJTO1UMuXXGUjbwawxigqlsAIPzv0fEmFJHxItIgIg1NTU2uFZCIckA6Vw+5dMVRNvL9oLeqTlbVKlWt6t+/v9fFIaJUeD14nM7VQ05fcZTuPkljPg0pVr62NoPCRi9Q1fEXrMHtxRG/rwAwKPzzIAArTJZz1llnKRFlidpa1dJSVeDDV2mp9b7b5aioUBWx/jVZfzrzmJYlnX1iOF+oPaQrX1+rf6mp13EV03VQwZbw5GhQG+pyx6+SAgARqQTwon54ldTDAHaq6k9F5E4AfVX1jmTL4VVSRFmkstJ6plG0igqgsdHt0vhDuvskznxaXo7Vj09F8Mn1CE4tQHD9cGwKDQIADCzYhtFDV6H6wnZ8re4SW66ScuOy2qcBVAMoA7ANwL0AngfwDIByAOsBXKOq0QPjR2BgEGWRgoLwyW0UESAUcr88dqurs66cWr/eGt+YNCn5Zbfp7pM484UgKIQ134CC7agevAqjL2pD9fVDcNwnh0EKJLx4ey6rdfxptao6Ls5HH3N63UTkofLy2GfTuTB4HOvpsDfcAFx/vdVaiBceaeyTtVM3oH/p0ejRvO2Iz3YWHo0/XjMVo28YgpFjhkEK4l4/ZAvfD3oTURq8HmwGXHtchSdi3avR0QLoeLR4rH1usE/WzdiIKV+djpuOnY7Koo0YfslQjG/+BQ6g2+GrKy1F/ym/wISnL8bxlw/vbE04yo6BELdeHPQmMuDUYLOfBo+dZFJmkcP3b6xXRYXR8pvu/Z1O+eo0/dKIqVpZtL5z9n6yQz87eKb+7nNBXfz8exp6Kv19iWwa9LYLxzCIDDgx2JyDj8uIyXQ74+3jSHHGJTbO3YLg42sQeCuE4NpyrGmrAAD0lV24ZOBKjD6/BdXjBuGkq49FQZE9nUB2jWEwMIhyjRODzflyxZPpdsYKljjzbGqwAiL4v3YEVldgdTgg+shuXDJwBarPbcHo6wbh5E/ZFxDRsulZUkTkpnTuVE425pENj8uwY9zGdDsjnw4LWGEcoa2oCx5tvQHHlazFkLMH4fo/XYB/rTwVJ5Vtxa8+VY93/74CO1p747nN5+Lbz16CUz93nGNhYSs7+rXcenEMg8hAqmMYJtNXVKTWT+82u8Zt0tjOLQu26YxP3Ks7igZoO0TXokLHoVZ74329csBsffHMidpSdoyGnBrHMRhzgU1jGJ6HQCovBgaRoVQGm00qSb/ctR2PXYFmsJ1bF23Xf9w2Q285sV6PL1nVOVlP7NErjn5bfz42oA1PLdW2ljbn95vh8hkYRNnOL1cQxbviR+Tw6fxS3lhMt8FE1Ha+/9M/6TPfmalfPzmoJ0QFxOX95+jPLg/o3ClL9NDBQ0cuy+mWmeHy7QoMDnoTecFPVx3lwoB2WRmwc+eR7xcWAlOmpLRPm5btwNTH30PgtVYEVx6DJS0jAADdsR8XlS3D6LObUX1Nf5w5biSKuia599npu90Nl581d3oTUQyJvqTH7cCYNCl2eGXLDXZ1dcDevbE/a2+3tg2Iu193rNiJqY+9h+BrLQisOAaLW0YAKEN37MeF/Zbj+uogqj9XhrOuG4ni0rNTK5vTd7u7fTe9Hc0Ut17skqKcYWcXih383N2USG2tamFh7H0Zp4tm56pd+tyds/XW04J6atflH3b9Y79e2rdBf/yJgM78y0JtbW61p3wcw2BgEGXE71cdZYNYlWWcVwiit50R1NO6LldBuwKq3dCsH+/boA9+PKDT/7hAW/a1OFdOJ8PYxaukOIZB5IVYYxglJUDPnsCuXeZPP81nJndbhzWiAidgGc7vswyjz9yL6s/0xdnXj0SXXl2cLaNPcAyDKJt1BEHH47H79rX64TsGbjseYBc5LR3O8KbB1oIuOFjzNbz/+wJ06XWmw4XKbVlwayFRjqqpsa5CCoWAHj2AQ4cO/7xjEDwePzyR1gN7N+7Fy/fPxe7i2I/yDkHQ2v0oqAhQUYGSJx/DCU/elTetCScxMIj8INVHb3R0aa1bZ/XSd7RIvv71nAuRfZv34ZUfzcUPRgVxTo8l6DO0O66472x8u/UhfICuh09cWoqC2qdQsn83JBSyAtnPLbQsC32OYRD5Qar3QsSbXuTw6/Kz8Imy+7fux4zHliPwwn4EF5ehofl4tKMIxWjFqF7LMPq03ai+ujfOvXEkSl97LvVvvfMLF+/F4dNqiXJJqpVHvBu2YnHqBrx0vqI0hubtzZjx2HIEX9iHwKJ+aNh/PNpQjCIcwqiey1B96i5UX9UL5998PErLSpMvMFu4eMOkXYHh+aWyqbx4WS3lNDue/+TWvR3xrv+fMCHpNjQ3NesbD83Tu88P6Pk9F2gRWhVQLUKrntdjod59fkBf/0mD7t+23/5yOyWdS2ddvBcHvA+DyFC23pSWSKwKO14F5MS9HfECK7oMpaXa8pfH9c2fzdMfXhjQC3ou0GK0KKBaiEN6bo+Feue5AX3txw26b8s++8vphnRvznPxXpycCAwA3wGwBMBiAE8D6JpoegYGpczvT1nNRHQQTpjg3raafEVp+NWI8s6AOKf7Iv3BqIC+8sBc3btpr/3l8kK6Fb+Lx2bWBwaAwQDWAugW/v0ZADclmoeBQSlz+izOb60XO8uTaFkpdImFIPry/XN0z4Y9GW2ab2XSteTS8ZMrgbEBQF9YNxC+COATieZhYFDKnOwnzvXWS5xt+2DPB7r0hh9rS0GXqGCIExq5/rgTO09KHAqQrA8MaxvwbQD7ATQBqIszzXgADQAaysvLbdl5lEecbGF4/TwoJ89O42zbbumtXXFAAdXr8JRukkEaShQWbgaoV609u04cHDwByfrAANAHwP8A9AdQDOB5ANcnmoctDEqZk62AZK0XJyswB7erZV+LhhB720KAPlUxUf9z92zdtWa3NUO84CwszL5KO5P1Z/p/7eAJSC4ExjUAHov4/YsA/phoHgYGpSWdP2aTeRL9gSerwDKtYGysXFqbW3XGnxfqpEsD+vG+DdoNzboWcZYfax1+eFS71609Ozi4H3MhMEaFr5AqBSAApgD4VqJ5GBjkCtOz1UTTZRImJjKoXFqbW3XWI4v0x58I6Cf6zdVS7O+c/ZSuK/TW04I6Z+x98buZotfhh8raD6GVKbYwkobG/QCWhy+rfQpAl0TTMzDIFan84cZrKSSqwPr1y7xiSKGMhw4e0tmPLtKfjgnoJ/vN1e7Y1zn5yV1W6jdPCeq/vz9Lm5bvOHxG03J63R2k6o/QyhTHMOx9MTDIFXacrcarwLp3j/1+qstPULkcOnhI5zyxRB+6LKCX9Z+jPbC3c5ITu7yn3zglqP/87kzdtnh72uuIOa2Xlxf7IbTskO1XSQF4E8Bpdqws0xcDg1xhx9lqrAqsuDh+WKRzNhyuXEIi+kG/QfrC6RP1iqPf1l54v3ORJ5Ss0gkn1esz35mpWxclCYgE6/DNfSaJZFNZXeZmYJwZvprprwAG2bHSdF8MDIrL7hvW7LpMMrJM8bp4Ol6Gy29radN5tUv152MDOjYqIEaWrNZbTqzXv986Q7cs2JbyplNucr1LCsBnASwAcG/H3dluvxgYyrOoWFIZpDbdd07s50SP0+jXL+5s7Yfa9Z2/LdNfXh3QqwbO1qNkd+dsxxWv0fHH1+vT35qhm9/dmnkZY+Exl/VcDYzwVUwnA7gFwA4AGwHcYEcBUnnlfWDkSj+t3Uy6kPyw7xI9sC+iHO2H2nX+P5brrz4V1KsHztY+sqtz0mOL1+pXj6/Xuq9P141zNztfZj/sN7vkcfC52SU1HcBmAG8AeADAWADHAvgdgMl2FML0lfeBkQtXgqTC9A/cZJDaD/suzhNm22+5RRf8c4X+5jNB/dSgWYcFxEeKGvXLx03V2gnTdcMcFwIimh/2mx1yKfjS4GZgnIzwFy3F+GyZHYUwfeV9YOTCteamUvkDT/QgvI6g8cu+q63VUHm5hkR0T7cB+suj7tN+sqOzOMOK1unNI6bqk1+bputnb3K3bLGks9/8eCafK8GXJl9cVgtguB2FMH3lfWC4fdB7+Yef6r0Q0eESHTR23PuQplB7SBc//57+/pqgfnbwTC2Tps7VVxat15uOnapTvjpNG6dviL2AbPl/6CirH8/k/XLC4BFfBIbbr7wPDDf/GON9Qc+ECfavK5ZEA8TxypuopdGvn2v7LtQe0qUvrNI/fCGo1wyZqf1le+cqyws36I0fmaZ//fI0XTstTkBEb5fXz0hKZf1+PZP3a7lcwsDIV26dbRoO0Dom3fUnuxIp8mebtiPUHtJlL63WP42r188PnaFHRwTEkMJNesPwafrYTVN1Tf16DbWHUlu4Hyq6VI45v57Jex28HmNgkLMSVbxuVFaJxh0SrT9R0NhUWYTaQ7ri1TX65+vq9dryGTqwYGvnYgcXbNbrh03TR2+cqqveakw9IKL5tQKOJ50uLLe62/w4tuISBgY5K1H3jluVVTrrj9eVlkHwhdpDuvL1tTr5hnodVzFdBxVs6VzEoIItel3FdJ18Q72+96YNAREtWQXst0ow1ceJ5PFZv5sYGOSsRGf4bn3PQbrdMdGVaIrBE2oP6aq3GvWRL07VmsrpOrhgc+csAwu26riK6fqXmnpd8eoa+wMi1rbEq1T9WuGahpgfutvyBAODnDdhQvzQcKNisqtCTFIxhdpDujqwTh+7aapeP2yaDinc1DnJgIJt+oWhM/RP4+p1+curnQ+IWOJVwNle4WZbd1sWY2CQO2prrRaFl2MZmXa5xAie9i7dNHDx/+kXh0/T8sINOg61uhYV2g7R7YUD9PVzf6hLX1jlTUCYyvYKN9sDL4swMMg92V4xqer2e36r+3oM0BBEN2CwjkOtAqpl0qS/PupebS3scvi2+aFrJ5lsr3BNWpB+G6PJUgwMck8WVkzrZ2/SKV+dpl8aMVWHFa3rLHI/2aGfHTxTf/e5oC56dqW2H2rPyu1TVf+OYaQiUSDkwvb5BAOD3JMFf7gb5mzWp26ZrjePmKrDixo7i9lXduqnB83S33wmqAv/tcIKiGh+bUGZnF3n8hl4tga5D9kVGGItKztUVVVpQ0OD18XIT3V1wMSJwPr1QHk5MGkSUFPjWXE2NWxB8PE1CP6vHcE15Vh1qBIA0Ed245KBK1B9bguqrx2IUz4zAgVFBYkXVlkJrFt35PsVFUBjo91FN1NXB4wfDxw48OF7paXA5Mme7ndXFRRYERFNBAiF3C9PFhOReapalfFyGBiUDbbM34bgY6sReLMNwdVD8d6hYQCA3tiDSwYuR/Wogxg9bhBO+fSxKCwpTG3hfqyc/RhibuM+sI1dgVFkR2GI7LZ14XYEH12F4JttCK4eghWtwwEMQG/swcUDluOWUetQ/fkBOO2a41BYMiqzlXWEgo9aUFi/PrX3I/msNZi2SZNiB/mkSd6VKd/Z0a+V7gvAUQD+BWA5gGUAzks0PccwclRtrbYNGqIhiO4oGqC3F/6ys7u6J/boFUe/rQ9fEdCGp5ZqW0ub16V1RyY3Lfp8vCkluTxG4yLkwqA3gCkAvhL+uQTAUYmmZ2Dkju1Lm/SZ78zUx4dM1APoelgFdxBd9b+nTdQ5TyzRQwcPmS0w1yqWdCt+DhRTDHYFhmdjGCLSC9Z3hA9Xw0JwDCN77VixE/WPrkTgtVYEVx6DJS0jAADrUI5ybDhyhlT6qf04BmGHdLqWOFBMMdg1hpHk8hFHDQfQBOCvIvKuiDwqIt2jJxKR8SLSICINTU1N7pcy29TVWYOFBQXWv3V1nhRj53u78Owds3HrafU4tdtK9D++Hz738/Pw10VVGNxjD37yySBmPbIYQ2Vj7AWY9NV3mDjx8LAArN8nTkx/A+Jxc//W1FihGQpZ/5qEX3l5au8TpcKOZko6LwBVANoAjAr//hsADySax9ddUn7oEvGw/3rnql363J2z9dbTgnpq1+Ufrh779dK+DTrp0oDO/MtCbW1uPXxGO7pQ3LqPIhvGB7KhjOQ6ZPsYBoCBABojfr8IwEuJ5kk5MNyqxL38I43cRhef+bRrzW59/q7ZetsZQT292zIVtCug2g3N+rE+8/TBjwd0+h8XaMu+luTlz3TfudVvn2g9fjhh6OCnspAvZH1gWNuAaQBGhn++D8DDiaZPKTDcrMRNKiwn/oiTfZe1jWfauxvf1//+8G39zpkBPaPb0s6A6IoD+tE+8/RHHw3otD8s0A/2fJDedqSyb6KnnzDBnf/rRF8qxbN68rFcCYzTATQAWAjgeQB9Ek2fUmC4ebVIsi4Rp8Ir0Xc9ZLjN7697X1/4v7f19rMCelbpEi1AmwKqXXBQq496R+8fHdD6385PLyAyEW9fTpiQeiCnGlSRX/Ea+fLyab5EBnIiMFJ9pRQYbj4fKFk4ORVeic54UwymPRv26Ev3zdHvnx3QqoiAuAFP6BYZpCGIHuwzSFsm/zWzMmfKrn2ZaojX1qoWFx+53pKS+Pve62dREYUxMJKJV7H06+dO11Bk5eNUeMXbxsLCpNu3d9Neffn+OXrHOQE9p/siLcQhBVSL0aIX9Zqv91wc0MXXPaihbt3SCiDHZLIvMxnvSXY8sYVBPsbASCZWJV5cfOQZoV0VYKLuDacqlBTOkvdt2aevPjhX7zw3oKOiAuLCXvP1hxcG9K2fv6PNTc3OlzsTdt4BnUrwJAqqbLkyiYPheYuBYSL6DyReH7QdFXeiP0QnK5Q4696/bb++9uMGveu8gJ7XY6EWoVUB1SK06vk9F+jECwL6xkPzDg+IaH587Lfdd0Bn2sLomN7vlXG2hBo5goGRDicqwFh928XFsUPDwQqlualZ33honk68IKDn91xwWECc12Oh3nVeQF//SYPu37bffKF2tzAyvRqqY/p09mWm4z3ZXuH6sbVIrmFgpCPVPxqTiileq6VfP0dD4sDOA/rmz+bpDy8M6IW95msxWhRQLcQhHdV9kd55bkBffXCu7tuyL/2V2FlJpjPIbGcFncF4z2Fl8nMrIhE/thbJNQyMdKRSCZlOm+yM1aYK78DOA/q/X7yj91wc0It6zdcSfKCAagHa9Jzui/SOcwL6ygNzde+mvWktPy67KslUwzpRBe9GYOUatjDyGgMjXaYVoOkfmEm/eBp/oAd3H9TAr97Vey8J6CW939UuONgZEFWlS/T7Zwf0pfvm6J4Ne9LfF26prY2/P1IdZM60lZOtLYRM5Xtg5jm7AoPfuBet4wmhsb7pCzjyqZ9lZcDOnebLj/PU0Ja9LXh7ynIEn9uNwLu9Mev9E9CCrihAO84oXYHqE7Zj9NjuuPDm49C7vHeKG+WhWE+SjRTvqbTxvm0t2XwUX658sRKljF/R6oRklRtwZEVVVwfcfDPQ2vrheyUlQM+esYMkPH/L3hbMedIKiOC7vTBz9wn4AN0gCOH0bisw+sRtqL6sFBd9ZSSOqsiigIiWqOJP9AjyZP8XfFw3kTG7AsPzbqZUXo4/rTbZpZeJxjuiuzpidAG0lXTVZ0+aqB/rM0+7odnqkUG7nt5tmd52RlD/c/ds3bVmt7Pb6LZEXUsmg8wmN9jlc1cTkQFwDMMBiSq3FCui1uZWXfGln+j7XQdoO0TXYaiOQ60Cqqd2Xa5Tyidqc6+BGvKqknOrks10sDVZ3zv75omSys/ASKPiTqlizKBya21u1VmPLNKffDKgn+g3V7tjX+fsp3Rdod86NajP3jFLd6zc6X0l5+b67ViXF3fRE+WQ/A2MVCocB6/9P3TwkM5+dJH+dExAx5TN0R7Y2znLSV1W6jdPCeq/vjdTty9tOnI9Xldybq/fydYM7y8gSsquwMiuQW8R7RzyNrlKJt6Aa6J541xJ0vZBG979x0oEn9mOwNzumNZ0PPajJ8ahDg/jDhyDLTjQ82i0TbwPvX9wS+Jyef29y16v307p/B8T5Zn8HPRO9Qwy2QPjEpz1trW0acNTS/XhKwJ6xdFvay+83zn78SWrdMJJ9TpzzL3a3jWNp7nmWwvDSV537xFlAeR1l5Rp5ZbokdRRlUyotFTXTPiZ/uKqgI6NCoiRJav1ayfU699vnaFbFmxLvvxkdyN7Xcl5vX678SopooTyOzAyHcOI8/yntahQQHVE8Rodf3y9/u2bM3TTvC3xl5/J3cheV3Jer5+IXJO/gZHmVVIhEW0pO0Zfrpqo7Yhd0YcgunHuZvNlJ7tvIxu7eOzCQCLyDbsCI7sGvVO40zvUFsKS/6xC4G9bEJzVBfVbj8Mu7QsA2IghGIxNR86U6kAp70aOLdZ+SXRXNxE5yq5B7yI7CuMHobYQlr6wGsG/bUZgZhfUbxmBnXocgOMwrGg9rj52CUZ/VHDJjcMweM1DsSu0SZNSW2lH5XfjjUB7+5Gfl5envT1ZbeLEI0P0wAHrfQYGUdbyPDBEpBBAA4BNqjrWdD4NKZa9uBqB2k0IzixBcPMI7NARAEagonAjrvzIUlRXC6pvqkTFBeUAIirv88KVlh0PYuuYx44AcpKbD55bvz6194koK3jeJSUi3wVQBaBXssA4ecQp+o2qPyAwvRjBTceiSfsDAIYWbsLoirUYPRqovqkSlRcOcaHkUfz8JFC3u4h4bwSRr+TE02pFZAiAKQAmAfhussAQqVKgAUMKN2N0xRpUX6wY/SUrIKRAXClzVnK7AucYBpGv5MoYxq8B3AGgZ7wJRGQ8gPEAMKB7BWb8dx2GV5dDCo5xp4S5wO0uohobu/yIyDc8a2GIyFgAl6vq10WkGsD3krUwXPkCpVzELiKivGZXC6PAjsKk6QIAV4lII4C/A/ioiNR6WJ7cNWmS1SUUyW+D8kTke54FhqrepapDVLUSwLUA/qeq13tVnpxWU2ONH1RUWPeGVFRwPIGIUub1GAa5paaGAUFEGfFFYKhqEEDQ42IQEVECXo5hEBFRFmFgEBGREQZGXZ112WlBgfVvXZ3XJSIi8iVfjGF4JvqO5HXrrN8BDhATEUXJ7xZGoqeq5hK2oojIBvndwsiHp6qyFUVENsnvFka876vIpe+xcLoVxdYLUd7I78DIh0dmONmK6mi9rFtnfSltR+uFoUGUk/I7MPLhkRlOtqLyZQyIiADke2AAVjg0Nlrfvd3YmFthATjbisqHMSAi6sTAyHVOtqLyYQyIiDoxMPKBU62ofBgDIqJODAxKXz6MARFRp/y+D4Myx8emE+UNtjCIiMhI9gUGbxQjIvJEdnVJ7drFx1wQEXkku1oYmzbxRjEiIo9kV2C0tsZ+nzeKERE5LrsCo6Qk9vu8UYyIyHGeBYaIDBWRgIgsE5ElIvLtpDMNHswbxYiIPOJlC6MNwO2qegKAcwF8Q0ROTDhH3768UYyIyCOeXSWlqlsAbAn/vE9ElgEYDGBpwhl5oxgRkSd8MYYhIpUAzgDwdozPxotIg4g0NDU1uV42IiKyeB4YItIDwL8B3Kaqe6M/V9XJqlqlqlX9+/d3v4BERATA48AQkWJYYVGnqs96WRYiIkrMy6ukBMBjAJap6i+9KgcREZnxsoVxAYAbAHxUROaHX5d7WB4iIkrAy6ukpgMQr9ZPRESp8XzQm4iIsgMDg4iIjDAwiIjICAODiIiMMDCIiMgIA4OIiIwwMIiIyAgDg4iIjDAwiIjICAODiIiMMDCIiMgIA4OIiIwwMIiIyAgDg4iIjDAwiIjICAODiIiMMDCIiMgIA4OIiIwwMIiIyAgDg4iIjHgaGCIyRkRWiMgqEbnTy7IQEVFingWGiBQC+AOAywCcCGCciJzoVXmIiCgxL1sY5wBYpaprVLUVwN8BXO1heYiIKIEiD9c9GMCGiN83AhgVPZGIjAcwPvxri4gsdqFsmSoDsMPrQhhgOe2TDWUEWE67ZUs5R9qxEC8DQ2K8p0e8oToZwGQAEJEGVa1yumCZYjntlQ3lzIYyAiyn3bKpnHYsx8suqY0Ahkb8PgTAZo/KQkRESXgZGHMBjBCRYSJSAuBaAP/1sDxERJSAZ11SqtomIt8E8BqAQgCPq+qSJLNNdr5ktmA57ZUN5cyGMgIsp93yqpyiesSwARER0RF4pzcRERlhYBARkRFfBEayR4SI5bfhzxeKyJmm87pczppw+RaKyEwROS3is0YRWSQi8+26xC2DclaLyJ5wWeaLyD2m87pczu9HlHGxiLSLSN/wZ67sTxF5XES2x7v/x0fHZrJy+uXYTFZOvxybycrph2NzqIgERGSZiCwRkW/HmMbe41NVPX3BGvBeDWA4gBIACwCcGDXN5QBegXXvxrkA3jad1+Vyng+gT/jnyzrKGf69EUCZT/ZnNYAX05nXzXJGTX8lgP95sD8vBnAmgMVxPvf82DQsp+fHpmE5PT82Tcrpk2NzEIAzwz/3BLDS6brTDy0Mk0eEXA3gSbXMBnCUiAwynNe1cqrqTFXdHf51Nqx7S9yWyT7x1f6MMg7A0w6VJS5VnQpgV4JJ/HBsJi2nT45Nk/0Zj6/2ZxSvjs0tqvpO+Od9AJbBeoJGJFuPTz8ERqxHhERvdLxpTOa1S6rr+jKsZO+gAF4XkXliPe7EKablPE9EFojIKyJyUorz2sF4XSJSCmAMgH9HvO3W/kzGD8dmqrw6Nk15fWwa88uxKSKVAM4A8HbUR7Yen14+GqSDySNC4k1j9HgRmxivS0RGw/qjvDDi7QtUdbOIHA3gDRFZHj6L8aKc7wCoUNX9InI5gOcBjDCc1y6prOtKADNUNfKMz639mYwfjk1jHh+bJvxwbKbC82NTRHrACqzbVHVv9McxZkn7+PRDC8PkESHxpnHz8SJG6xKRUwE8CuBqVd3Z8b6qbg7/ux3Ac7CahJ6UU1X3qur+8M8vAygWkTKTed0sZ4RrEdXkd3F/JuOHY9OID47NpHxybKbC02NTRIphhUWdqj4bYxJ7j0+nB2YMBm6KAKwBMAwfDr6cFDXNFTh84GaO6bwul7McwCoA50e93x1Az4ifZwIY42E5B+LDmzbPAbA+vG99tT/D0/WG1Zfc3Yv9GV5HJeIP0np+bBqW0/Nj07Ccnh+bJuX0w7EZ3i9PAvh1gmlsPT4975LSOI8IEZFbwp//GcDLsEb7VwE4AOBLieb1sJz3AOgH4I8iAgBtaj3JcgCA58LvFQH4m6q+6mE5Pwdggoi0ATgI4Fq1jiK/7U8A+DSA11W1OWJ21/aniDwN68qdMhHZCOBeAMURZfT82DQsp+fHpmE5PT82DcsJeHxsArgAwA0AFonI/PB7d8M6OXDk+OSjQYiIyIgfxjCIiCgLMDCIiMgIA4OIiIwwMIiIyAgDg4iIjDAwiIjICAODiIiMMDCIMhD+PoJLwz8/KCK/9bpMRE7x/E5voix3L4AfhR80dwaAqzwuD5FjeKc3UYZEpB5ADwDVan0vAVFOYpcUUQZE5BRY33zWwrCgXMfAIEpT+JvL6mB9U1mziHzS4yIROYqBQZSG8DetPQvgdlVdBuABAPd5Wigih3EMg4iIjLCFQURERhgYRERkhIFBRERGGBhERGSEgUFEREYYGEREZISBQURERv4fXdmfwX9noj8AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "theta from own sdg\n", + "[[3.96308056]\n", + " [2.85515493]]\n" + ] + } + ], "source": [ "# Using Autograd to calculate gradients using SGD\n", "# OLS example\n", @@ -4435,22 +4093,34 @@ { "cell_type": "markdown", "id": "8daddbbb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Same code but now with momentum gradient descent" ] }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 23, "id": "f98d1a25", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.20627898]\n", + " [2.80034621]]\n", + "Eigenvalues of Hessian Matrix:[0.30328735 4.3292589 ]\n", + "theta from own gd\n", + "[[4.20559721]\n", + " [2.80092809]]\n", + "theta from own sdg with momentum\n", + "[[4.22593619]\n", + " [2.75740806]]\n" + ] + } + ], "source": [ "# Using Autograd to calculate gradients using SGD\n", "# OLS example\n", @@ -4524,22 +4194,32 @@ { "cell_type": "markdown", "id": "c679233f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Similar (second order function now) problem but now with AdaGrad" ] }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 16, "id": "94c2d690", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[2.]\n", + " [3.]\n", + " [4.]]\n", + "theta from own AdaGrad\n", + "[[2.00000024]\n", + " [2.99999844]\n", + " [4.00000161]]\n" + ] + } + ], "source": [ "# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n", "# OLS example\n", @@ -4599,9 +4279,7 @@ { "cell_type": "markdown", "id": "df4e327b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Running this code we note an almost perfect agreement with the results from matrix inversion." ] @@ -4609,22 +4287,32 @@ { "cell_type": "markdown", "id": "d132564e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## RMSprop for adaptive learning rate with Stochastic Gradient Descent" ] }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 17, "id": "f2143c85", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[2.]\n", + " [3.]\n", + " [4.]]\n", + "theta from own RMSprop\n", + "[[1.99999936]\n", + " [3.00000306]\n", + " [3.99999702]]\n" + ] + } + ], "source": [ "# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n", "# OLS example\n", @@ -4689,9 +4377,7 @@ { "cell_type": "markdown", "id": "3a32a369", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And Logistic Regression" ] @@ -4700,10 +4386,7 @@ "cell_type": "code", "execution_count": 35, "id": "401cfd7e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4744,9 +4427,7 @@ { "cell_type": "markdown", "id": "c1ebe2a0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n", "\n", @@ -4763,10 +4444,7 @@ "cell_type": "code", "execution_count": 36, "id": "f0f9a51c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import jax.numpy as jnp\n", @@ -4783,9 +4461,7 @@ { "cell_type": "markdown", "id": "cb8bbc4e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Weekend challenge\n", "\n", @@ -4797,7 +4473,25 @@ ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + } + }, "nbformat": 4, "nbformat_minor": 5 }