diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb index 59aea6e97..2ad653f46 100644 --- a/doc/pub/week39/ipynb/week39.ipynb +++ b/doc/pub/week39/ipynb/week39.ipynb @@ -3,9 +3,7 @@ { "cell_type": "markdown", "id": "624309b8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", @@ -15,9 +13,7 @@ { "cell_type": "markdown", "id": "047a991a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "# Week 39: Optimization and Gradient Methods\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", @@ -30,9 +26,7 @@ { "cell_type": "markdown", "id": "3001fadb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Plan for week 39\n", "\n", @@ -54,9 +48,7 @@ { "cell_type": "markdown", "id": "b30c33fc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Thursday September 29\n", "\n", @@ -66,9 +58,7 @@ { "cell_type": "markdown", "id": "73508bc3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Optimization, the central part of any Machine Learning algortithm\n", "\n", @@ -87,9 +77,7 @@ { "cell_type": "markdown", "id": "db45f384", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our Logistic Regression case\n", "\n", @@ -104,9 +92,7 @@ { "cell_type": "markdown", "id": "1af235be", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{align*}\n", @@ -119,9 +105,7 @@ { "cell_type": "markdown", "id": "9adc53cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$." ] @@ -129,9 +113,7 @@ { "cell_type": "markdown", "id": "77a0472d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations to solve\n", "\n", @@ -145,9 +127,7 @@ { "cell_type": "markdown", "id": "7feaa413", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n", @@ -157,9 +137,7 @@ { "cell_type": "markdown", "id": "e0e070a9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n", "$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as" @@ -168,9 +146,7 @@ { "cell_type": "markdown", "id": "f490b0e0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n", @@ -180,9 +156,7 @@ { "cell_type": "markdown", "id": "02387ee2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This defines what is called the Hessian matrix." ] @@ -190,9 +164,7 @@ { "cell_type": "markdown", "id": "a8fe4a21", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Solving using Newton-Raphson's method\n", "\n", @@ -204,9 +176,7 @@ { "cell_type": "markdown", "id": "eb736dda", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n", @@ -216,9 +186,7 @@ { "cell_type": "markdown", "id": "df8cb898", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or in matrix form as" ] @@ -226,9 +194,7 @@ { "cell_type": "markdown", "id": "fd44dff8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n", @@ -238,9 +204,7 @@ { "cell_type": "markdown", "id": "4e1ebc91", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The right-hand side is computed with the old values of $\\beta$. \n", "\n", @@ -250,9 +214,7 @@ { "cell_type": "markdown", "id": "90ad7450", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Brief reminder on Newton-Raphson's method\n", "\n", @@ -270,9 +232,7 @@ { "cell_type": "markdown", "id": "aee0363b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The equations\n", "\n", @@ -286,9 +246,7 @@ { "cell_type": "markdown", "id": "a51e8a41", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -302,9 +260,7 @@ { "cell_type": "markdown", "id": "c8d8d8e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "For small enough values of the function and for well-behaved\n", "functions, the terms beyond linear are unimportant, hence we obtain" @@ -313,9 +269,7 @@ { "cell_type": "markdown", "id": "5640b0b9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(x)+(s-x)f'(x)\\approx 0,\n", @@ -325,9 +279,7 @@ { "cell_type": "markdown", "id": "29b131b5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "yielding" ] @@ -335,9 +287,7 @@ { "cell_type": "markdown", "id": "85009a37", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "s\\approx x-\\frac{f(x)}{f'(x)}.\n", @@ -347,9 +297,7 @@ { "cell_type": "markdown", "id": "f831bc88", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Having in mind an iterative procedure, it is natural to start iterating with" ] @@ -357,9 +305,7 @@ { "cell_type": "markdown", "id": "0fc71a7f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n", @@ -369,9 +315,7 @@ { "cell_type": "markdown", "id": "51ffa803", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple geometric interpretation\n", "\n", @@ -391,9 +335,7 @@ { "cell_type": "markdown", "id": "a646a713", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Extending to more than one variable\n", "\n", @@ -404,9 +346,7 @@ { "cell_type": "markdown", "id": "ad162944", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n", @@ -417,9 +357,7 @@ { "cell_type": "markdown", "id": "bf72a75b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which we Taylor expand to obtain" ] @@ -427,9 +365,7 @@ { "cell_type": "markdown", "id": "c3e6fdc5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n", @@ -445,9 +381,7 @@ { "cell_type": "markdown", "id": "29a7c03b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have" ] @@ -455,9 +389,7 @@ { "cell_type": "markdown", "id": "60fa3964", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n", @@ -470,9 +402,7 @@ { "cell_type": "markdown", "id": "02bd34d4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "we can rephrase Newton's method as" ] @@ -480,9 +410,7 @@ { "cell_type": "markdown", "id": "5c2e7766", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n", @@ -494,9 +422,7 @@ { "cell_type": "markdown", "id": "ffd135f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined" ] @@ -504,9 +430,7 @@ { "cell_type": "markdown", "id": "098c35e0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n", @@ -518,9 +442,7 @@ { "cell_type": "markdown", "id": "38bbd32e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We need thus to compute the inverse of the Jacobian matrix and it\n", "is to understand that difficulties may\n", @@ -533,9 +455,7 @@ { "cell_type": "markdown", "id": "db7d1c20", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent\n", "\n", @@ -550,9 +470,7 @@ { "cell_type": "markdown", "id": "ec78a232", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n", @@ -562,9 +480,7 @@ { "cell_type": "markdown", "id": "9d79aa59", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $\\gamma_k > 0$.\n", "\n", @@ -576,9 +492,7 @@ { "cell_type": "markdown", "id": "d352745c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on Steepest descent\n", "\n", @@ -591,9 +505,7 @@ { "cell_type": "markdown", "id": "df488a82", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n", @@ -603,9 +515,7 @@ { "cell_type": "markdown", "id": "22875ea7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The parameter $\\gamma_k$ is often referred to as the step length or\n", "the learning rate within the context of Machine Learning." @@ -614,9 +524,7 @@ { "cell_type": "markdown", "id": "d19f30cb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The ideal\n", "\n", @@ -642,9 +550,7 @@ { "cell_type": "markdown", "id": "f1b4a5e4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The sensitiveness of the gradient descent\n", "\n", @@ -664,9 +570,7 @@ { "cell_type": "markdown", "id": "64ee4acd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex functions\n", "\n", @@ -686,9 +590,7 @@ { "cell_type": "markdown", "id": "7939950d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Convex function\n", "\n", @@ -698,9 +600,7 @@ { "cell_type": "markdown", "id": "9ebd13ec", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conditions on convex functions\n", "\n", @@ -735,9 +635,7 @@ { "cell_type": "markdown", "id": "e1e93067", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on convex functions\n", "\n", @@ -763,9 +661,7 @@ { "cell_type": "markdown", "id": "67e2eeb8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Some simple problems\n", "\n", @@ -793,9 +689,7 @@ { "cell_type": "markdown", "id": "cdce400d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Standard steepest descent\n", "\n", @@ -813,9 +707,7 @@ { "cell_type": "markdown", "id": "00f0b878", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n", @@ -825,9 +717,7 @@ { "cell_type": "markdown", "id": "b067183e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In the iterative process we end up with a problem like" ] @@ -835,9 +725,7 @@ { "cell_type": "markdown", "id": "023447f2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n", @@ -847,9 +735,7 @@ { "cell_type": "markdown", "id": "1a29b74d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n", "\n", @@ -859,9 +745,7 @@ { "cell_type": "markdown", "id": "c4088348", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient method\n", "\n", @@ -871,9 +755,7 @@ { "cell_type": "markdown", "id": "3f5cd556", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n", @@ -883,9 +765,7 @@ { "cell_type": "markdown", "id": "49eb9918", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n", "symmetric. This defines also the Hessian and we want it to be positive definite." @@ -894,9 +774,7 @@ { "cell_type": "markdown", "id": "294579e5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent method\n", "\n", @@ -907,9 +785,7 @@ { "cell_type": "markdown", "id": "4e35f70e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -919,9 +795,7 @@ { "cell_type": "markdown", "id": "b6055cf1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or consider the system" ] @@ -929,9 +803,7 @@ { "cell_type": "markdown", "id": "e4bf26af", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -941,9 +813,7 @@ { "cell_type": "markdown", "id": "08110157", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "instead." ] @@ -951,9 +821,7 @@ { "cell_type": "markdown", "id": "d0699ac3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -962,9 +830,7 @@ { "cell_type": "markdown", "id": "19e28e76", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -974,9 +840,7 @@ { "cell_type": "markdown", "id": "61503485", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -986,9 +850,7 @@ { "cell_type": "markdown", "id": "d54ae35a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -998,9 +860,7 @@ { "cell_type": "markdown", "id": "2a75f182", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$." @@ -1009,9 +869,7 @@ { "cell_type": "markdown", "id": "e76669ca", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Final expressions\n", "We can compute the residual iteratively as" @@ -1020,9 +878,7 @@ { "cell_type": "markdown", "id": "ddbe6b99", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1032,9 +888,7 @@ { "cell_type": "markdown", "id": "0ab65253", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which equals" ] @@ -1042,9 +896,7 @@ { "cell_type": "markdown", "id": "343d865c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n", @@ -1054,9 +906,7 @@ { "cell_type": "markdown", "id": "69e8fa9d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or" ] @@ -1064,9 +914,7 @@ { "cell_type": "markdown", "id": "260f0bd1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n", @@ -1076,9 +924,7 @@ { "cell_type": "markdown", "id": "28d654ef", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives" ] @@ -1086,9 +932,7 @@ { "cell_type": "markdown", "id": "f595dbe2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n", @@ -1098,9 +942,7 @@ { "cell_type": "markdown", "id": "c84110d2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "leading to the iterative scheme" ] @@ -1108,9 +950,7 @@ { "cell_type": "markdown", "id": "738d0376", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k-\\alpha_k\\boldsymbol{r}_{k},\n", @@ -1120,9 +960,7 @@ { "cell_type": "markdown", "id": "697bcfaf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Steepest descent example" ] @@ -1131,10 +969,7 @@ "cell_type": "code", "execution_count": 1, "id": "e714b8eb", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -1164,9 +999,7 @@ { "cell_type": "markdown", "id": "cd9916f9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "And then as countor plot" ] @@ -1175,10 +1008,7 @@ "cell_type": "code", "execution_count": 2, "id": "b8644e9d", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1189,9 +1019,7 @@ { "cell_type": "markdown", "id": "99c41a8f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Find guesses" ] @@ -1200,10 +1028,7 @@ "cell_type": "code", "execution_count": 3, "id": "ab68472c", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = guesses[-1]\n", @@ -1213,9 +1038,7 @@ { "cell_type": "markdown", "id": "71ac5c37", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Run it!" ] @@ -1224,10 +1047,7 @@ "cell_type": "code", "execution_count": 4, "id": "8454f18e", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "def f1d(alpha):\n", @@ -1242,9 +1062,7 @@ { "cell_type": "markdown", "id": "429802dd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "What happened?" ] @@ -1253,10 +1071,7 @@ "cell_type": "code", "execution_count": 5, "id": "974e38db", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "pt.axis(\"equal\")\n", @@ -1268,9 +1083,7 @@ { "cell_type": "markdown", "id": "bd44034d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that we did only one iteration here. We can easily add more using our previous guesses." ] @@ -1278,9 +1091,7 @@ { "cell_type": "markdown", "id": "6aa2a802", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "In the CG method we define so-called conjugate directions and two vectors \n", @@ -1292,9 +1103,7 @@ { "cell_type": "markdown", "id": "b21a338d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n", @@ -1304,9 +1113,7 @@ { "cell_type": "markdown", "id": "36de2035", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The philosophy of the CG method is to perform searches in various conjugate directions\n", "of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely" @@ -1315,9 +1122,7 @@ { "cell_type": "markdown", "id": "96460d24", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n", @@ -1327,9 +1132,7 @@ { "cell_type": "markdown", "id": "bcf8f093", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Two vectors are conjugate if they are orthogonal with respect to \n", "this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$." @@ -1338,9 +1141,7 @@ { "cell_type": "markdown", "id": "f25ef4c8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "An example is given by the eigenvectors of the matrix" @@ -1349,9 +1150,7 @@ { "cell_type": "markdown", "id": "081d1da3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n", @@ -1361,9 +1160,7 @@ { "cell_type": "markdown", "id": "aa6122e9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which is zero unless $i=j$." ] @@ -1371,9 +1168,7 @@ { "cell_type": "markdown", "id": "ddd0620e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n", @@ -1383,9 +1178,7 @@ { "cell_type": "markdown", "id": "53d704bd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n", @@ -1395,9 +1188,7 @@ { "cell_type": "markdown", "id": "53101dcb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n", "Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n", @@ -1407,9 +1198,7 @@ { "cell_type": "markdown", "id": "803ccc03", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n", @@ -1419,9 +1208,7 @@ { "cell_type": "markdown", "id": "abe3b5b0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "The coefficients are given by" @@ -1430,9 +1217,7 @@ { "cell_type": "markdown", "id": "6bf05de5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n", @@ -1442,9 +1227,7 @@ { "cell_type": "markdown", "id": "1253bcaf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Multiplying with $\\boldsymbol{p}_k^T$ from the left gives" ] @@ -1452,9 +1235,7 @@ { "cell_type": "markdown", "id": "f7671b5f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n", @@ -1464,9 +1245,7 @@ { "cell_type": "markdown", "id": "087861cd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and we can define the coefficients $\\alpha_k$ as" ] @@ -1474,9 +1253,7 @@ { "cell_type": "markdown", "id": "3859c9e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n", @@ -1486,9 +1263,7 @@ { "cell_type": "markdown", "id": "c36443ac", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method and iterations\n", "\n", @@ -1506,9 +1281,7 @@ { "cell_type": "markdown", "id": "4b135acf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{x}_0=0,\n", @@ -1518,9 +1291,7 @@ { "cell_type": "markdown", "id": "00556f1d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or consider the system" ] @@ -1528,9 +1299,7 @@ { "cell_type": "markdown", "id": "153a1d22", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n", @@ -1540,9 +1309,7 @@ { "cell_type": "markdown", "id": "a3c51787", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "instead." ] @@ -1550,9 +1317,7 @@ { "cell_type": "markdown", "id": "7bc48291", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form" @@ -1561,9 +1326,7 @@ { "cell_type": "markdown", "id": "a431587b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n", @@ -1573,9 +1336,7 @@ { "cell_type": "markdown", "id": "0d322f94", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n", "to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n", @@ -1585,9 +1346,7 @@ { "cell_type": "markdown", "id": "9382ecc2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n", @@ -1597,9 +1356,7 @@ { "cell_type": "markdown", "id": "66d7625d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and \n", "$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n", @@ -1610,9 +1367,7 @@ { "cell_type": "markdown", "id": "e82b8692", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:" @@ -1621,9 +1376,7 @@ { "cell_type": "markdown", "id": "05180d7b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n", @@ -1633,9 +1386,7 @@ { "cell_type": "markdown", "id": "52a090f3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n", "$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n", @@ -1649,9 +1400,7 @@ { "cell_type": "markdown", "id": "ded09ab0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n", @@ -1661,9 +1410,7 @@ { "cell_type": "markdown", "id": "d92bad7b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Conjugate gradient method\n", "We can also compute the residual iteratively as" @@ -1672,9 +1419,7 @@ { "cell_type": "markdown", "id": "a25cad33", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n", @@ -1684,9 +1429,7 @@ { "cell_type": "markdown", "id": "51681d94", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which equals" ] @@ -1694,9 +1437,7 @@ { "cell_type": "markdown", "id": "d0d7c11f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n", @@ -1706,9 +1447,7 @@ { "cell_type": "markdown", "id": "470b62a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "or" ] @@ -1716,9 +1455,7 @@ { "cell_type": "markdown", "id": "597cd7c3", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n", @@ -1728,9 +1465,7 @@ { "cell_type": "markdown", "id": "2d289b36", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which gives" ] @@ -1738,9 +1473,7 @@ { "cell_type": "markdown", "id": "676a6a8e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n", @@ -1750,9 +1483,7 @@ { "cell_type": "markdown", "id": "fb6b2368", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Revisiting our first homework\n", "\n", @@ -1774,10 +1505,7 @@ "cell_type": "code", "execution_count": 6, "id": "c8ff1895", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "x = 2*np.random.rand(m,1)\n", @@ -1787,9 +1515,7 @@ { "cell_type": "markdown", "id": "ba8abbc0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n", "The linear regression model is given by" @@ -1798,9 +1524,7 @@ { "cell_type": "markdown", "id": "681fb1ee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n", @@ -1810,9 +1534,7 @@ { "cell_type": "markdown", "id": "3b4b2655", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "such that" ] @@ -1820,9 +1542,7 @@ { "cell_type": "markdown", "id": "3dd7cb44", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n", @@ -1832,9 +1552,7 @@ { "cell_type": "markdown", "id": "963c2869", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent example\n", "\n", @@ -1846,9 +1564,7 @@ { "cell_type": "markdown", "id": "3faae97a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "X \\equiv \\begin{bmatrix}\n", @@ -1862,9 +1578,7 @@ { "cell_type": "markdown", "id": "f4be1f8f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "The cost/loss/risk function is given by (" ] @@ -1872,9 +1586,7 @@ { "cell_type": "markdown", "id": "a62c2f65", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n", @@ -1884,9 +1596,7 @@ { "cell_type": "markdown", "id": "bd482d3d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "and we want to find $\\beta$ such that $C(\\beta)$ is minimized." ] @@ -1894,9 +1604,7 @@ { "cell_type": "markdown", "id": "e03f97ee", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The derivative of the cost/loss function\n", "\n", @@ -1906,9 +1614,7 @@ { "cell_type": "markdown", "id": "59c746ff", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -1920,9 +1626,7 @@ { "cell_type": "markdown", "id": "ff621a81", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $X$ is the design matrix defined above." ] @@ -1930,9 +1634,7 @@ { "cell_type": "markdown", "id": "229284c8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix\n", "The Hessian matrix of $C(\\beta)$ is given by" @@ -1941,9 +1643,7 @@ { "cell_type": "markdown", "id": "a5632b50", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -1956,9 +1656,7 @@ { "cell_type": "markdown", "id": "44f5596d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite." ] @@ -1966,9 +1664,7 @@ { "cell_type": "markdown", "id": "92164631", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple program\n", "\n", @@ -1978,9 +1674,7 @@ { "cell_type": "markdown", "id": "a48fce93", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n", @@ -1990,9 +1684,7 @@ { "cell_type": "markdown", "id": "8bbca853", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can use the expression we computed for the gradient and let use a\n", "$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n", @@ -2005,9 +1697,7 @@ { "cell_type": "markdown", "id": "3c9fd447", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient Descent Example\n", "\n", @@ -2018,10 +1708,7 @@ "cell_type": "code", "execution_count": 7, "id": "65e8d8f5", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "\n", @@ -2075,9 +1762,7 @@ { "cell_type": "markdown", "id": "321fd022", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And a corresponding example using **scikit-learn**" ] @@ -2086,10 +1771,7 @@ "cell_type": "code", "execution_count": 8, "id": "5847a934", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "# Importing various packages\n", @@ -2113,9 +1795,7 @@ { "cell_type": "markdown", "id": "68c9136e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Gradient descent and Ridge\n", "\n", @@ -2125,9 +1805,7 @@ { "cell_type": "markdown", "id": "0c4580c8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n", @@ -2137,9 +1815,7 @@ { "cell_type": "markdown", "id": "5e578724", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows" ] @@ -2147,9 +1823,7 @@ { "cell_type": "markdown", "id": "3bacb190", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n", @@ -2161,9 +1835,7 @@ { "cell_type": "markdown", "id": "028de774", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by" ] @@ -2171,9 +1843,7 @@ { "cell_type": "markdown", "id": "5f75aef9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n", @@ -2183,9 +1853,7 @@ { "cell_type": "markdown", "id": "be235465", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The Hessian matrix for Ridge Regression\n", "The Hessian matrix of Ridge Regression for our simple example is given by" @@ -2194,9 +1862,7 @@ { "cell_type": "markdown", "id": "cee3e15d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{H} \\equiv \\begin{bmatrix}\n", @@ -2209,9 +1875,7 @@ { "cell_type": "markdown", "id": "06592b0e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "This implies that the Hessian matrix is positive definite, hence the stationary point is a\n", "minimum.\n", @@ -2223,22 +1887,41 @@ { "cell_type": "markdown", "id": "fbaae00b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Program example for gradient descent with Ridge Regression" ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 1, "id": "c85edc81", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Eigenvalues of Hessian Matrix:[0.24210446 5.0349011 ]\n", + "[[3.62869484]\n", + " [3.21681784]]\n", + "[[3.62683186]\n", + " [3.21823697]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoXklEQVR4nO3de5wcZZ3v8c9vJgmQEEGSINfMcBFcRRcxKqJH8cKKiJd19+wSAgddNYdwXPHoegOPKBpXd91dWM8qm1UWdAIeX3hZ5XhBRWRFQBIuCsYLkGQSuU0CAZJwy8xv/6jqpKanqvvp7uqq6p7v+/XqV6a7bk9XV55fPdcyd0dERKSZgbITICIivUEBQ0REgihgiIhIEAUMEREJooAhIiJBFDBERCSIAsY0Z2brzOw18d/nmNkXS0rH8Wa2sYxjTxfJ37qNbZeZ2f1mttXM5uWdtpTjXWRm/6fBcjezw7udDplMAaPCzOwUM7vRzLaZ2QPx32eZmXXjeO7+KXd/R6f7MbPh+D/0jDzSVTYzu8TMPll2OspiZjOBfwT+xN33dPfNOexznZk9Fgeg++JzvGdtubuf6e6f6PQ4ki8FjIoys/cBFwJ/D+wHPAM4E3gpMCtjm8HCEijTyTOA3YE7Wt3QIln5zBvcfU/gaOD5wIfbTqEUQgGjgsxsL+B84Cx3v8LdH/XILe6+xN2fiNe7xMy+YGbfNbNtwCvN7PVmdouZPWJmG8zsY3X7Pt3M1pvZZjM7t27Zx8xsJPH+WDP7uZltMbPbzOz4xLJrzOwTZnadmT1qZleZ2fx48bXxv1viO8iXpHzHPeL0P2RmvwZeWLf8ADP7upmNmdlaM3t3YtmLzGxV/B3vN7N/TCx7WSLNG8zsrfHnu5nZZ81sNN7mIjPbI152vJltNLP3xSW5e83sbfGypcAS4APxd/lOxm/2LDP7oZk9aGa/NbO/iD8/LP7smMT32lQ7l2b2NjNbE5/Du83sfyb2WUvXBxLperOZnWRmv4v3e07d73eFmf2/eH83m9kfZ6R3wMw+ZGZ3xdfC18xsn5T1jgB+m/g9r44/P87MbjKzh+N/j0tsc42ZLTez64DtwKFpaahx9/uAHxAFjto+JpXqzOz98fe/x8z+qi6N88zsO/H1cJOZfdLMftbst5E2uLteFXsBJwI7gBlN1rsEeJio1DFAdBd4PPDc+P3zgPuBN8frPxvYCrwc2I2ommEH8Jp4+ceAkfjvA4HNwEnxvk6I3y+Il18D3AUcAewRv/90vGwY8EbpBz4N/CewD3AwcDuwMV42AKwGPkpUmjoUuBt4bbz8euD0+O89gWPjvxcCjwKLgZnAPODoeNkFwLfj480FvgP8bbzs+Pg8nB9vdxJRRvf0xHn+ZIPvMgfYALwNmAEcA2wCnhMvfyewBphNlDF+NrHt64HDAANeER/3mLp0fTRO1zuBMeCy+Ds8B3gcODTx+z0F/Hm8/t8Aa4GZ8fJ1id/6PcANwEFE18K/ApdnfL9Jv2d8Dh8CTo+/7+L4/bzEtTEap29G7fh1+0ym5SDgV8CFddf2JxP/H+4HjorP9WVxeg6Pl381fs0musY3AD8L+W30ajFvKjsBeqX8KHAacF/dZz8HtgCPAS+PP7sE+HKTfV0A/FP890eBryaWzQGeJD1gfBD4St2+fgCcEf99DfCRxLKzgO/Hf0/KYDLSdTdwYuL9UnYFjBcDo3Xrfxj49/jva4GPA/NT1vlmyrEM2AYclvjsJcDa+O/j4/M6I7H8AXYFop2ZV8Z3+UvgP+s++1fgvMT7b8eZ4i+B3Rrs61vA2XXpGozfz43P64sT669m1w3Bx4AbEssGgHuB/xa/X5f4rdcAr06suz9RsJnym9X/nkSB4hd161wPvDVxbZzf5LpcR3Tz8mi87x8DeyeW7zznwMXENyPx+yPibQ4HBuN0H5lY/kl2BYymv41e4S9VSVXTZmC+JRqN3f04d987Xpb83TYkNzSzF5vZT+KqnIeJ2j1qVUUHJNd3923x/tIMAf89rtrZYmZbgJcRZSw19yX+3k50tx9qUlqA9XXHPqDu2OcQ1aUDvJ0o0/hNXAVxcvz5wUSlnnoLiO4+Vyf29/3485rN7r6jze8zBLy4Lr1LiNqeav6N6A75cx5XKQKY2evM7Ia4umQLUelmfmK7ze4+Hv/9WPzv/Ynlj9WlM/n7TgAbic51Wpq/mUjvGmCcXee4kQOY/HsRvz8wLR0NvNnd5xIFxmcx+XvXHy/rWllAVHJILk/+HfLbSCAFjGq6HngCeFPAuvXTDV9GdDd7sLvvBVxEdIcN0d3mwbUVzWw2UbVNmg1EJYy9E6857v7pNtKUZlJaiKqTksdeW3fsue5+EoC7/97dFwP7Ap8BrjCzWtXDYSnH2kSUsT4nsb+9PGpwDdHs+2wAflqX3j3dfRmARb1/LgC+BHys1lZgZrsBXwc+CzwjviH4Lrt+r3Ykf98BouqeezLS/Lq6NO/u7n8IOMY9RBlx0kIguW3wNNju/lOiEsVnM1ZpdK2MEVXbHZT4LLluw99GWqOAUUHuvoWoyuXzZvbnZrZn3Eh5NFE1UiNzgQfd/XEzexFwamLZFcDJccPwLKI6+6xrYAR4g5m91swGzWz3uBH2oIz1k8aACRo3dn4N+LCZPT3e518nlv0CeMTMPmhR4/igmR1lZi8EMLPTzGxBfAe9Jd5mHFgJvMbM/sLMZsSNoUfH6/0b8E9mtm+8jwPN7LUB3wWiO/pG3+VK4AiLOhTMjF8vNLM/ipdfCKz2qMvy/ycK4hC1z+xGnOmZ2euAPwlMU5YXmNlb4tLpe4huPG5IWe8iYLmZDQGY2QIzC7lBgSioHWFmp8bn+S+J2g6u7CDdFwAnxNd4va8BbzWzZ8c3OefVFsSlr28QBeLZZvYs4H8ktm3220gLFDAqyt3/Dngv8AGi+vT7iepeP0jUnpHlLOB8M3uUqM3ia4l93gH8L6JSyL1EDZWpg+XcfQNRCeccogxtA/B+Aq4Zd98OLAeui6sBjk1Z7eNEVQtrgauAryS2HwfeQNRrZi1RCeGLwF7xKicCd5jZVqLM+BR3f9zdR4mqdN4HPAjcCtR6CX0QuBO4wcweAX4EHNnsu8S+BDw7/i7fSvm+jxJl9KcQ3X3fR1Ty2S3OhE8kqhqE6Dc9xsyWxNu9m+g3eogouH87ME1Z/oOo3r7WKP0Wd38qZb0L42NdFV8rNxC1HTXl0TiMk4nO82aia/Rkd9/UbqLdfQz4MjBlsJ67f48ooFxN9BteXbfKu4iujfuIrqPLiQJlw9+m3bROZ+auByiJ9AOLulAf7u6nlZ2WMpnZZ4D93P2MstPSb1TCEJGeFo+zeJ5FXkTUKeKbZaerH/XF1A0iMq3NJaqGOoCo+vYfiKrmJGeqkhIRkSCqkhIRkSA9VSU1f/58Hx4eLjsZIiI9ZfXq1ZvcfUHzNRvrqYAxPDzMqlWryk6GiEhPMbP6kfltUZWUiIgEUcAQEZEgChgiIhJEAUNERIIoYIiISBAFDBERCaKAISIiQRQwREQkiAKGiIgEUcAQEZEgXQ8YZnaxmT1gZrenLPsbM3Mzy3r4u4iIVEQRJYxLiB5ROYmZHQycAIwWkAYREelQ1wOGu19L9Hzlev9E9CxgPZBDRKQHlNKGYWZvBP7g7rcFrLvUzFaZ2aqxsbECUiciImkKDxhmNhs4F/hoyPruvsLdF7n7ogULOp7OXURE2lRGCeMw4BDgNjNbBxwE3Gxm+5WQFhERCVT4A5Tc/VfAvrX3cdBY5O6bik6LiIiEK6Jb7eXA9cCRZrbRzN7e7WOKiEj+ul7CcPfFTZYPdzsNIiLSOY30FhGRIAoYIiISRAFDRESCKGCIiEgQBQwREQmigCEiIkEUMEREJIgChoiIBFHAEBGRIAoYIiISRAFDRESCKGCIiEgQBQwREQmigCEiIkEUMEREJIgChoiIBFHAEBGRIAoYIiJ5WrkShodhYCD6d+XKslOUm64/olVEZNpYuRKWLoXt26P369dH7wGWLCkvXTlRCUNEJC/nnrsrWNRs3x593ge6HjDM7GIze8DMbk989vdm9hsz+6WZfdPM9u52OkREum50tLXPe0wRJYxLgBPrPvshcJS7Pw/4HfDhAtIhItJdCxe29nmP6XrAcPdrgQfrPrvK3XfEb28ADup2OkREum75cpg9e/Jns2dHn/eBKrRh/BXwvayFZrbUzFaZ2aqxsbECkyUi0qIlS2DFChgaArPo3xUr+qLBG8DcvfsHMRsGrnT3o+o+PxdYBLzFAxKyaNEiX7VqVXcSKSLSp8xstbsv6nQ/pXWrNbMzgJOBV4cECxERKVcpAcPMTgQ+CLzC3bc3W19ERMpXRLfay4HrgSPNbKOZvR34v8Bc4IdmdquZXdTtdIiISGeK6CW12N33d/eZ7n6Qu3/J3Q9394Pd/ej4dWa30yEi0pY+nuqjVZoaREQkS59P9dGqKnSrFRGppj6f6qNVChgiIln6fKqPVilgiIhk6fOpPlqlgCEikqXPp/polQKGiEiWoqb66JGeWOolJSLSyJIl3e0R1UM9sVTCEBEpUw/1xFLAEBFppptVRj3UE0sBQ0T6Rzcy9lqV0fr14L6ryii5706O20M9sRQwRKQ/hGTs7WhWZdTpcXuoJ1Yhz8PIi56HISKZhoejzLre0BCsW9f+fgcGokBQzwwmJvI57llnRb2vxsdhcDAKOJ//fPtpnpLUfJ6HoRKGiPSHbrUFNKsy6vS4K1fCpZdGwQKify+9tJJdaxUwRKQ/dKstoFmVUafHzaryOuOMyo3LUMAQkf7QrbaAZoP3Oj1uVklkfDzftpgcKGCISH/o5qjsJUui9oiJiejf5D47PW5ISaQi4zLU6C0i5Vu5MsoQR0ejDHT58sqNcu6a+pHeWWqN7G1Qo7eI9IdudYftFfUllMHB9PVqJZES551SCUNEytWt7rC9Kq3EMXt2FFQge1mDEplKGCLSH3poaoxCNGoTKXneqa4HDDO72MweMLPbE5/tY2Y/NLPfx/8+vdvpEJGK6qGpMQqT1checnAtooRxCXBi3WcfAn7s7s8Efhy/F5HpqIemxihdycG16wHD3a8FHqz7+E3ApfHflwJv7nY6RKSi2umW2iMPHGqq1e+RFlxnzoStW/EizoW7d/0FDAO3J95vqVv+UINtlwKrgFULFy50EakzMuI+NORuFv07MlJ2irprZMR99mz3qE9V9Jo9u/n3rtp56vB7TJj5A8zzx5nVdB/AKs8jL89jJ00P0kHASL5e8IIXNP8RRKaTdjOdXjY0NPn71l5DQ9nbdPM8tRuIWvweEzvGfR82TVp1LWH7yCtgFNKt1syGgSvd/aj4/W+B4939XjPbH7jG3Y9sth91qxWpMx27pDabPTZNt85Toy6wzQYeBnyPv3/V9/jAT16XuYtxBhig+bno9W613wbOiP8+A/iPktIh0tumY5fUdhp+u3Wezj67/W6uGeld5wsxi/L8tGDxinm37yxKDAwV2wheRLfay4HrgSPNbKOZvR34NHCCmf0eOCF+LyKtmo5dUtvpVdWN87RyJWzenL4sJBClfI9tzOYcpn6PDbds2hkkrtl0VMN9dLWHWR71WkW91IYhUmc6tmG4t95ukPd5GhlxHxxMbz9o0p7yLydduXO1xYz4WoZ8HPO1DPliRhzcnzPn7tbS0uRc0EuN3nm9FDBEUlSt909V5XWe0oJP/Su574mJhqvWXlvu3Z7Ht0yVV8DQXFIiIq3IakCvmTePQzffxFoOabibg2fdz+jFPypklt5eb/QWEclfJwP6Qrdt0D6xjdmcuvnCzGCRLFOMXvyjnpulVwFDRKovJDPvZJr0VrbNaCjfwSDvZAWXs6uEcOHpqyYFiUlKnkiwLXnUaxX1UhuGyDQU2mDdzoC+Frb925ddubOheiuT07OV2TsbrIOZpR/TrIWdhEFtGCIyLYQOumtnQF+TbScwBpm67WJW8inOZSGjjD/jAGb+w2dab3socNCl2jBEZHoIHXTX6liLZDXXQHpWOEr6tpf5EoZ9HQM+wcz7NrbXUN2Ds/QqYIhItYUGglYy4Po2i/HxKRNsJAfR7XjKs9si2tVslt4qzsibR71WUS+1YYhMQ1njHgYG3Jctm7pu00Fs2ZP2PcWgj2O+0Q4sdzxLzgMNyakNQyUMEamW+jtriO6858yZvN7EBHzhC3DWWbs+S3lS3fKXfW/n3Exm0WoLSa/mmmETDPgEB060Wc2Ul4r2oFLAEJHqyOreCvD44+nbrFgx5aNkgPjIdVMn8Mtqm2CffaIgZQYzZkT/llEdVNFJJRUwRKQ6Gt1Zj4+nbzM+PilA1EoRaWr1O8MjGU+ue/TRXT2XascrY0BdaLtNwe0cChgiUh2N7qwHB1MX7SD9c4CH73ssvbE6rcH5aU+DJ59M31FadVA3M+uQBvxOBiq2K4+GkKJeavQWKUErk/Z1OsFfgwF0n2OZT9R9PgH+OZZNWrVtWQPp0gbUFTFLcLNz2cJARTRbrYh0XSsZYx6ZaMo+kqOoP8cyf4pBn4h7NH2OZc33GSorA07LiBut20qg7CTAtjBSXAFDRLqvlek2OpmaI1abeiPtGRG116mH35jTl6vTaNry+sDXrDQSEig7DbAllDA0NYiIZGtluo02puZo1ECdVFg2tXJl1Faxfn3UZjI+HrVv1E873myKc2g+xUenU4O08DxxTQ0iIt3XynQbAev6+ERQb6Y1V22YdNtcmNo4DnfYsSP6Nx7PMUlao3S9Zl1gO+0622ykeBcoYIgkVXE6hjK1Mt1Gxrqnrl++M0AMzMjOcpIB4lknHJxD4rsomVlnafa88DyeM54yULGr8qjXKuqlNgzpqun6fOxmWuwltX3v/TLbH3LpzVQ17V43BV5v9EOjN/C/gTuA24HLgd0bra+AIV2VQ6PttBMHk5AAsd/g/WWndpe8n4Pe7v4Keh57zwcM4EBgLbBH/P5rwFsbbaOAIV1V4ANtKi1wAr9mDxKqbCliGpYk8woYZbdhzAD2MLMZwGzgnpLTI9NZHnXKva7B6OH6xupPcS5zmDyNxxy2c9nQuTtz4lLU2qGy5oOq6MR+vaC0gOHufwA+C4wC9wIPu/tV9euZ2VIzW2Vmq8bGxopOpkwnPfhAm9xlZKbrTpuamWbN+FrqBHnJgAfp80FVdGK/XlBawDCzpwNvAg4BDgDmmNlp9eu5+wp3X+TuixYsWFB0MmU6KaGbYpCCem7d/h93MbE+PdNMBoeJcccdBoYqWCJLC3g1tVKESpJtK7NK6jXAWncfc/engG8Ax5WYHpH8uyl2mtl3eYK5ZDXTc998WOa03wNDC3dWM9lAXCfVbomsmwEwZOxD1hiKrVvVjbqZPBpC2nkBLybqITUbMOBS4K8bbaNGb+kpaY2rtYb10B4xOffcOnburxrOaJHWkN2wQbjVXj7NGpy7NXlh/XkbGXGfN2/q8j5t/KbXe0lF34GPA78h6lb7FWC3RusrYEhPaZZ5hWROOfTcapSE2uvvTvrJrg3SMu28un82CoBZAbb+MayNtDIf1DTqRt0XAaPVlwKGFCKvzLHZBHUhmVMbmVpIgGipy2ue3VAbBcCs72rW+oy3tX0NDk4OSKFp6TOFBQzgR8Af53GwTl8KGNKxZsEgj2qkmmYljKzMKZnGefPcZ81qmFmP3bklKEBsu//R1s9Xo+/Rzp14o301CrDduOtXCaMrAeMY4Grg34H98zhouy8FDOlIyJ1yHtVIjY7XLHNK22bmzChwJIJc7qWIRppl5K2UxBr9Bo3OfTfu+qfRAL7Cq6SAPwNuA84jHp1d9EsBQzoSckcZUo00b17rT6BLllSSmdOyZZP3ldYQC7551jOKCxDJdJvtqtZJy8TbyWyzSnkjI9nnv1t3/QVNzVG2QgNG3IvpKOBMYBOwETg9jwS08lLAkI6E1FmHVCO1W+qoz5yWLWteAolf41jqor884Kf5n6eQklG3MvZly9oPRJIpr4DR9AFKZvYz4NC4C+wNwI1xz6az415NS9vt0tsqPUBJOhLywJq0h9KECH3oTUh6Uowxj33ZBES5aFdlpWtwMBqfsnBhdrobPCwpWO0hRqOj0bHqH14kLcvrAUohpYujIAosKcvW5BG1Ql8qYUhHQuusG1Uj5VjHPpGx74m0/c+aVdwMqJ2UxPqwwbgfUNTkg+5+e3zANK/vOGKJFCV06o/kU9e+8pXJ68+bl77vwGklkiOr13v6NhNp/y2ffDJscrw8RoaHTJ2hebempzyiTlEvlTCkdC32rFmx9KbMQknmqOpOSjF53Pm3WhLr8wbjfoAG7kml9XNm0uS7hdRgncqXsweXdZLp5zUYrZ9/v2lIAUOqK+/+7RXPvEICBCQ2CJlPqd3zp7YFSaGAIdWVZ6ZVwcFVExNhAeI/P3Nd+g5Czk8nj/ys2PmS8ilgSHXlOUdPGXfMKZl1VhvEWiY/zzpIt+cwqvjzpaV4ChhSXXlm8kVPEJdyh17/nOqGDda9Wm2kkklfyytglP1Mb+lHeXa5LPDpaGZEjyLdPvU51Z9iV5dWd7hsaOrzrIOfC13FLql6zrUEUMCQ/OX5qNMuZq4v2ufOSeMiIPs51cM2uvPWG+jsudBVfBSsnnMtARQwshT0HOVcVDGteT3qNOfMNRkgbnro8CnLsx5ROqVE02nJJ+v8lPVb6jnXEiKPeq2iXoW1YfRSfW4vpbUEuXd5bXW9VpT5W+o66muo0buLqtgomaXR7Kpl9HQpuafNfXdtDQoQTz28LXsnod8h7+9a9nWnXlJ9K6+A0XS22iopbLbagYFEZXWdqp2vRmmFqL6/qPrxtJleGx0/p1lJa+0PzVTtp5si67fMYwZYmdbymq1WbRhpsuptzarRPpDUrI65yJ4urfS06WCSvPP/9OYpjdVp6m/VK0/tCFJxpQYMM9vbzK4ws9+Y2Roze0mZ6dlp+fL0nMi9et0M03oR1Suqp0vWcdavnxoIWuzGmQwQ533rmNR1Vpz87d4KEPWq2N22iqrYyWO6yKNeq90XcCnwjvjvWcDejdYvdOBeVuV3twaMdaLZ85CLqgNvlIb6BtQmA/JabqzuF2pHaEyN822h19swzOxpRM8IP9QDE1HoE/dCns5WNa22IRRx/KTkucs4v+sY4hDWZR7ikd/dx9xn7tdxUqVH9eL/ywrohzaMQ4Ex4N/N7BYz+6KZzalfycyWmtkqM1s1NjZWXOp6sXqg7AFhteNnSVZZLV/OE8yatPgJZnEOU89v8nZSwWKa0wDDUpUZMGYAxwBfcPfnA9uAD9Wv5O4r3H2Ruy9asGBBcakrO/NtV14D5jo5/tBQ6qLte+2/sx3i1NPAmVywrL3vucZqKY46BpSqzICxEdjo7jfG768gCiDVUXbm26tSSmfbmM07tvzdzvef4lx256lJ6+zOU1w2VLFOBVItvVjy7yOlBQx3vw/YYGZHxh+9Gvh1WemRfLz+0F9jpy3h1O0rWMcQExjrGOKdrOBydgXcYatw1UKnvXDUi6d7erXk3ydKHbhnZkcDXyTqIXU38DZ3fyhr/UIbvSWIe5QvNvOHH63hgFf/0a4Pqtp42WnHgbI7Hoik6IdGb9z91rh94nnu/uZGwUKqIzkmolGwSLZDTAoWUF7VQrO7/06n+dY04dLHps9Ib1UTNNbg/Pz62k1hI6snPLyxuoyqhZDR5Z32wlEvHulneQzmKOrV9sA9DfZpLPApc/Wvsxd+veyUhxsZcR8cbD6wsdMJAMueQFAkBXriXgtUTdBYyvmpf8pcTTIXvGD9W4pKYWclxFrJYnw8fXnd+JCOqsrUi0f62PQIGKomSFWrYppYn34eFjLK71beVP6YiA4mKgTSbxiSkn34O60qy9oeVCUqPW96TG9e1R45Bctqf1jLMMNU+Px0+vs1mgK+iB5M6jklJeuLXlKFmQ7VBClVNmMbHg9qrB665Pxqn59OS4hZo4AHB4vJtKtaJaqOINKqPBpCinp1NFttP88C2mKj9bsPvCJ9H1U9P502JC9bNnV23CI7PTSZmbcU6ggyraBHtIq7+5UX3ulrGUrNkNYy1B/TgHeSuaVtaxYFkaJUsedUFdMkXZNXwJgeVVK9rq7q4FRbubOa6eSzD2Mh6VUzwza6MyfIOw2FVl900hCdVh3kDt/9bnfSmqaKVaLqCCLtyCPqFPWaliWMkRF/YmD3SXeB9dVNWSWM3O4We7n6oirVQVWr8lMJY1pBJYwe0cad+ROP+84SxLrTzmXWxOOTltfGSDz14KO4w/BIl+9gq9poG6Iq02FXbebjKpZ6pPryiDpFvXquhNHCnflz91qXesM3TuAdcjfvYIu8S8/7e/Ry6ajbqlbqka5Bjd49oEGx/3c3bE5dVP+qRNVBUWnoVuaujFGmubwChqqkuimjAXFi/ShHHLtP6rLfrFw9KVcupOqgWbVZUdUX3ar6qlp1kEiP6u2A0SyjK3lg0ta56c+fHmVX/fmzWDMpQBx56gsmr9ztWV1Dpt0oamZZ9dwRqbY8iilFvSZVSTWrviih7np8fPLhFjPiW5k6oO6pf/nXrqWhZXlWN3Va9VOF6jeRPsS0b8NolrkUlPmceewtDdsgFjPiW2bOr279eV4N2nkEaDVQi3RFXgGjdycfzJpQziyqq262vE1jo4+x79AeTdfzCW88gVNV5DUxY177WbkyarMYHY26vi5frjYHkQ5p8sFm/etz7H+/x8ATO8dFZAWLX664ftItek8EC8ivQTuv9gc1UItUVu8GjGYZXQcZ4erv3DNpltfHfbcp6xzNLZMCxHPf+ZLsHVZ5VtC8GrSrMkBORLonj3qtTl7AIHALcGWzdaeMw2jWyBrYCDsxkd0GkXxtveu+rCrCbK3Uy/fyeAG1P4hUFv3S6A28F7isrYDRgYve9cumAeLiN36j8wOFNr73Q4bbywFPpI/lFTBKbfQ2s4OAS4HlwHvd/eRG67f9xD3g8e0T7DGneQ2cj8cN5u1Ia7A9/fSwxnc9FVBEuqRfGr0vAD4AZHZbMrOlZrbKzFaNjY21tPOvnvurne0QWcHiti/8fNKtf0fBIm0A3D7pI7qn1O1r0JqIVFxpAcPMTgYecPfVjdZz9xXuvsjdFy1YsKDhPrc++CS72a4eTYs/9dwp67x65rWTAsTzzjyuo++xU9a0FhDW+K5GYxGpuDJLGC8F3mhm64CvAq8ys5FWdzJ60/0cvefvMYO582bxJJN7NB3C3Wy9+4GdAeJHT748l8RPTUhGSeDBB8N6IWm6aRGpuNIChrt/2N0Pcvdh4BTganc/rdX9vP/P7uK2bc+c9Nl1n/nZzgBxtx/KnEP2zSfRjTQqIYSMLShqviYRkTaV3YbRsYt+fARXf+I6JnZM7AwSx33gZcUnJI8SggatiUiFVSJguPs1zXpIZXn6M+fzyo+8FBss+auohCAifa4SASMXVRhNrRKCiPSxGWUnIBe1Lq21Xkm1Lq2gTFtEJCf9UcLo1pPaqqgKJSkRmZb6I2DkPeitqplyyNPxRES6pD8CRp6D3qqcKU+nkpSIVE5/BIw8B71VOVPW9CEiUqL+CBh5dmmtcqas6UNEpET9ETAgvy6tVc6UNX2IiJSofwJGXqqcKWtwoIiUqD/GYeSplvnWP9eiKpnykiXVSYuITCsKGGmUKYuITKEqKRERCaKAISIiQRQwREQkiAKGiIgEUcAQEZEgChgiIhJEAUNERIIoYIiISBAFDBERCVJawDCzg83sJ2a2xszuMLOzy0qLiIg0V+bUIDuA97n7zWY2F1htZj9091+XmCYREclQWgnD3e9195vjvx8F1gAHlpUeERFprBJtGGY2DDwfuDFl2VIzW2Vmq8bGxgpPm4iIREoPGGa2J/B14D3u/kj9cndf4e6L3H3RggULik+giIgAJQcMM5tJFCxWuvs3ykyLiIg0VmYvKQO+BKxx938sKx0iIhKmzBLGS4HTgVeZ2a3x66QS0yMiIg2U1q3W3X8GWFnHFxGR1pTe6C0iIr1BAUNERIIoYIiISBAFDBERCaKAISIiQRQwREQkiAKGiIgEUcAQEZEgChgiIhJEAUNERIIoYIiISBAFDBERCaKAISIiQRQwREQkiAKGiIgEUcAQEZEgChgiIhJEAUNERIIoYIiISBAFDBERCVJqwDCzE83st2Z2p5l9qMy0iIhIY6UFDDMbBP4FeB3wbGCxmT27rPSIiEhjZZYwXgTc6e53u/uTwFeBN5WYHhERaWBGicc+ENiQeL8ReHH9Sma2FFgav33CzG4vIG2dmg9sKjsRAZTO/PRCGkHpzFuvpPPIPHZSZsCwlM98ygfuK4AVAGa2yt0XdTthnVI689UL6eyFNILSmbdeSmce+ymzSmojcHDi/UHAPSWlRUREmigzYNwEPNPMDjGzWcApwLdLTI+IiDRQWpWUu+8ws3cBPwAGgYvd/Y4mm63ofspyoXTmqxfS2QtpBKUzb9MqneY+pdlARERkCo30FhGRIAoYIiISpBIBo9kUIRb553j5L83smNBtC07nkjh9vzSzn5vZHyeWrTOzX5nZrXl1cesgnceb2cNxWm41s4+GbltwOt+fSOPtZjZuZvvEywo5n2Z2sZk9kDX+p0LXZrN0VuXabJbOqlybzdJZhWvzYDP7iZmtMbM7zOzslHXyvT7dvdQXUYP3XcChwCzgNuDZdeucBHyPaOzGscCNodsWnM7jgKfHf7+uls74/TpgfkXO5/HAle1sW2Q669Z/A3B1Cefz5cAxwO0Zy0u/NgPTWfq1GZjO0q/NkHRW5NrcHzgm/nsu8Ltu551VKGGETBHyJuDLHrkB2NvM9g/ctrB0uvvP3f2h+O0NRGNLitbJOanU+ayzGLi8S2nJ5O7XAg82WKUK12bTdFbk2gw5n1kqdT7rlHVt3uvuN8d/PwqsIZpBIynX67MKASNtipD6L521Tsi2eWn1WG8niuw1DlxlZqstmu6kW0LT+RIzu83Mvmdmz2lx2zwEH8vMZgMnAl9PfFzU+WymCtdmq8q6NkOVfW0Gq8q1aWbDwPOBG+sW5Xp9ljk1SE3IFCFZ6wRNL5KT4GOZ2SuJ/lO+LPHxS939HjPbF/ihmf0mvospI503A0PuvtXMTgK+BTwzcNu8tHKsNwDXuXvyjq+o89lMFa7NYCVfmyGqcG22ovRr08z2JApY73H3R+oXp2zS9vVZhRJGyBQhWesUOb1I0LHM7HnAF4E3ufvm2ufufk/87wPAN4mKhKWk090fcfet8d/fBWaa2fyQbYtMZ8Ip1BX5CzyfzVTh2gxSgWuzqYpcm60o9do0s5lEwWKlu38jZZV8r89uN8wENNzMAO4GDmFX48tz6tZ5PZMbbn4Rum3B6VwI3AkcV/f5HGBu4u+fAyeWmM792DVo80XAaHxuK3U+4/X2IqpLnlPG+YyPMUx2I23p12ZgOku/NgPTWfq1GZLOKlyb8Xn5MnBBg3VyvT5Lr5LyjClCzOzMePlFwHeJWvvvBLYDb2u0bYnp/CgwD/i8mQHs8Ggmy2cA34w/mwFc5u7fLzGdfw4sM7MdwGPAKR5dRVU7nwB/Clzl7tsSmxd2Ps3scqKeO/PNbCNwHjAzkcbSr83AdJZ+bQams/RrMzCdUPK1CbwUOB34lZndGn92DtHNQVeuT00NIiIiQarQhiEiIj1AAUNERIIoYIiISBAFDBERCaKAISIiQRQwREQkiAKGiIgEUcAQ6UD8PIIT4r8/aWb/XHaaRLql9JHeIj3uPOD8eKK55wNvLDk9Il2jkd4iHTKznwJ7Asd79FwCkb6kKimRDpjZc4mefPaEgoX0OwUMkTbFTy5bSfSksm1m9tqSkyTSVQoYIm2In7T2DeB97r4G+ATwsVITJdJlasMQEZEgKmGIiEgQBQwREQmigCEiIkEUMEREJIgChoiIBFHAEBGRIAoYIiIS5L8AOeP7jHjuh6oAAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "from random import random, seed\n", "import numpy as np\n", @@ -2295,9 +1978,7 @@ { "cell_type": "markdown", "id": "4d80fe88", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using gradient descent methods, limitations\n", "\n", @@ -2317,9 +1998,7 @@ { "cell_type": "markdown", "id": "cba6bc16", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Overview video on Stochastic Gradient Descent\n", "\n", @@ -2329,9 +2008,7 @@ { "cell_type": "markdown", "id": "314db380", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Batches and mini-batches\n", "\n", @@ -2351,9 +2028,7 @@ { "cell_type": "markdown", "id": "a3bd5dc9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Stochastic Gradient Descent (SGD)\n", "\n", @@ -2383,9 +2058,7 @@ { "cell_type": "markdown", "id": "a3a4ccdd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2400,9 +2073,7 @@ { "cell_type": "markdown", "id": "bcd2d9da", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2413,9 +2084,7 @@ { "cell_type": "markdown", "id": "e8e31492", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Computation of gradients\n", "\n", @@ -2426,9 +2095,7 @@ { "cell_type": "markdown", "id": "98a46f0a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2439,9 +2106,7 @@ { "cell_type": "markdown", "id": "842a65e9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2453,9 +2118,7 @@ { "cell_type": "markdown", "id": "6076c141", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Stochastic Gradient Descent\n", "\n", @@ -2470,9 +2133,7 @@ { "cell_type": "markdown", "id": "9059febd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n", @@ -2483,9 +2144,7 @@ { "cell_type": "markdown", "id": "da676e64", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Computation of gradients\n", "\n", @@ -2496,9 +2155,7 @@ { "cell_type": "markdown", "id": "8ec48f17", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2509,9 +2166,7 @@ { "cell_type": "markdown", "id": "57cafc9b", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Stochasticity/randomness is introduced by only taking the\n", "gradient on a subset of the data called minibatches. If there are $n$\n", @@ -2523,9 +2178,7 @@ { "cell_type": "markdown", "id": "44578e02", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## SGD example\n", "As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n", @@ -2545,9 +2198,7 @@ { "cell_type": "markdown", "id": "41e35cd9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2560,9 +2211,7 @@ { "cell_type": "markdown", "id": "26cfd6d9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The gradient step\n", "\n", @@ -2572,9 +2221,7 @@ { "cell_type": "markdown", "id": "fecc043f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2585,9 +2232,7 @@ { "cell_type": "markdown", "id": "f7dad62a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2599,21 +2244,16 @@ { "cell_type": "markdown", "id": "0a1122c5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example code" ] }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 2, "id": "e0b0f1e4", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -2635,9 +2275,7 @@ { "cell_type": "markdown", "id": "b03d1b70", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -2651,9 +2289,7 @@ { "cell_type": "markdown", "id": "81392e39", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## When do we stop?\n", "\n", @@ -2672,9 +2308,7 @@ { "cell_type": "markdown", "id": "a35c98fa", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Slightly different approach\n", "\n", @@ -2693,13 +2327,18 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 4, "id": "faa43d57", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gamma_j after 500 epochs: 9.97108e-05\n" + ] + } + ], "source": [ "import numpy as np \n", "\n", @@ -2730,22 +2369,47 @@ { "cell_type": "markdown", "id": "026cdc3d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Program for stochastic gradient" ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 7, "id": "36d689f3", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.19396159]\n", + " [2.83692461]]\n", + "sgdreg from scikit\n", + "[4.17273321] [2.85691671]\n", + "theta from own gd\n", + "[[4.19396159]\n", + " [2.83692461]]\n", + "theta from own sdg\n", + "[[4.18119832]\n", + " [2.81895504]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAp+klEQVR4nO3deZxcVZn/8c/TWSCdBEL2EOhu1iDLABJZJc2MoAiIDjOOxIioaDS+ZoQZHUXzU1DBGZcZFX86TGQTO+IuMyoq+LPTSSAsHQgQyAKEdBKydQhkXzpdz++Pqg6VSi23qm/de6v7+3696pWuW3d57q2b89xzzr2nzN0REREppS7uAEREpDYoYYiISCBKGCIiEogShoiIBKKEISIigShhiIhIIEoY0meZ2c1m1hJ3HGEys4vMbE3ccUj/pIQhkTKzlWa2y8y2m9l6M7vHzIbFHZeIlKaEIXF4l7sPA84AzgQ+H284/ZOZDYw7BqktShgSG3dfD/yJdOIAwMxuNLOXzGybmT1vZn+b9dmHzGy+mX3LzF4zs5fN7J1Znx9jZm2ZZR8CRmdvz8yuNLPnzOx1M5tjZm/K+mylmf2rmT1jZjvM7E4zG2dmf8is789mdkS+/ehpJjKzT5vZRjNbZ2Yfzvp8jpl9NHc/st67mX3SzF7IbOurZnacmS0ws61m9nMzG5yzzS+Y2aZM3NOyph+SOT6rzGyDmd1uZkNy4vycma0H7jaz0Wb2u8wx2Wxm88xM5YLkpRNDYmNmRwHvBF7MmvwScCFwOPBloMXMJmR9fg6wjHQy+AZwp5lZ5rOfAAszn30VuDZrWycC9wE3AGOAB4Df5hTEfwdcApwIvAv4A/CFzPrqgE8V2Z3xmZgnAtcB3y+UYAq4FDgLOBf4LDALmAYcDZwKTM3Z1ujMtq4FZpnZpMxnX8/EfwZwfGaeL+UsOxJoBKYDnwbWkD4m4zL7q/GCJC8lDInD/Wa2DVgNbARu6vnA3X/h7mvdPeXuPwNeAM7OWrbD3X/o7t3Aj4AJwDgzawDeAnzR3fe4+1zgt1nLvQ/4vbs/5O5dwLeAIcD5WfN8z903uPsrwDzgMXd/yt33AL8h3XxWSBfwFXfvcvcHgO3ApCLz5/q6u2919+eAxcCD7r7C3beQTly52+7Zzzbg98A/ZBLnx4B/dvfN7r4N+BpwddZyKeCmzLK7MnFPABozsc9zDTAnBShhSBze4+7DgYuAk8hqOjKzD5rZokwTyeukr66zm5bW9/zh7jszfw4DjgRec/cdWfN2ZP19ZPZ7d0+RTlgTs+bZkPX3rjzvi3XOv+ru+7Le7ywxf65ytp1vP48kXUuoBxZmHb8/Zqb36HT33Vnvv0m6hvegma0wsxvLiFn6GSUMiU3m6vge0lf7mFkj8EPgH4FR7j6C9NW2FVhFtnXAEWY2NGtaQ9bfa0k3w5DZlpFu7nml8j0IbAfpgrzH+F6uL99+rgU2kU4up7j7iMzr8MwNBj0OqD24+zZ3/7S7H0u6Ge5fzOxtvYxP+iglDInbd4BLzOwMYCjpAq0TINNxfGqQlbh7B9AOfNnMBpvZW0kXgD1+DlxuZm8zs0Gk2+73AI+EtB/FLAKuMrN6MzuedB9Hb/Xs54XAFcAvMrWmHwLfNrOxAGY20czeUWglZnaFmR2fSaBbge7MS+QgShgSK3fvBO4l3Sb/PPAfwALSTTKnAQ+Xsbr3k+4U30y6X+TerO0sAz4AfI/0lfi7SN/euzeE3Sjl28Be0vv0I2B2L9e3HniNdK1iNvAJd1+a+exzpJuYHjWzrcCfKd6XckJmnu2kj/sP3H1OL+OTPsrUvyUiIkGohiEiIoEoYYiISCBKGCIiEogShoiIBFJTg4+NHj3am5qa4g5DRKSmLFy4cJO7jyk9Z3E1lTCamppob2+POwwRkZpiZh2l5ypNTVIiIhKIEoaIiASihCEiIoEoYYiISCBKGCIiEogShoiIBKKEISIigShhiIhIIEoYIiISiBKGiIgEUvWEYWZ3mdlGM1uc57PPmJmb2ehqxyEiIr0TRQ3jHuDS3IlmdjRwCbAqghhERKSXqp4w3H0u6d9YzvVt4LOAfiNWRKQGxNKHYWZXAq+4+9MB5p1uZu1m1t7Z2RlBdCIikk/kCcPM6oGZwJeCzO/us9x9srtPHjOm18O5i4hIheKoYRwHHAM8bWYrgaOAJ81sfAyxiIhIQJH/gJK7PwuM7XmfSRqT3X1T1LGIiEhwUdxWex+wAJhkZmvM7Lpqb1NERMJX9RqGu08t8XlTtWMQEZHe05PeIiISiBKGiIgEooQhIiKBKGGIiEggShgiIhKIEoaIiASihCEiIoEoYYiISCBKGCIiEogShoiIBKKEISIigShhiIhIIEoYIiISiBKGiIgEooQhIiKBKGGIiEggShgiIhKIEoaIiASihCEiIoEoYYiISCBVTxhmdpeZbTSzxVnTvmlmS83sGTP7jZmNqHYcIiLSO1HUMO4BLs2Z9hBwqrv/FbAc+HwEcYiISC9UPWG4+1xgc860B919X+bto8BR1Y5DRER6Jwl9GB8B/lDoQzObbmbtZtbe2dkZYVgiIpIt1oRhZjOBfcDsQvO4+yx3n+zuk8eMGRNdcCIicoCBcW3YzK4FrgDe5u4eVxwiIhJMLAnDzC4FPgc0u/vOOGIQEZHyRHFb7X3AAmCSma0xs+uA/wsMBx4ys0Vmdnu14xARkd6peg3D3afmmXxntbcrIiLhSsJdUiIiEobZs6GpCerq0v/OLng/UUVi6/QWEZEQzZ4N06fDzky3cEdH+n2IVMMQEekLZs58I1n02LkzPT0kShgiIn3BqlXlTa+AEoaISF/Q0FDe9AooYYiI9AW33gr19QdOq69PTw+JEoaISF8wbRrMmgWNjWCW/nfWrPT0kOguKRGRvmLatFATRC7VMEREJBAlDBGRpKjyg3e9pSYpEZEkKPbgXRWbmcqhGoaISDmqVQuI4MG73lINQ0QkqGrWAiJ48K63VMMQEQmqmrWACB686y0lDBGRoKpVC5g9G7ZvP3h6yA/e9ZYShohIUNWoBfQ0c7366oHTR40K/uBdRHdXKWGIiAQVZPiNcgvvfM1cAMOGBU8W06en+1Pc3+hXqULSUMIQkeIS/mxApEoNv1FJ4d3bZq4I764ydw99pdUyefJkb29vjzsMkf4j964gSF9RhzxGUZ/R1JROErkaG2HlyvCWyVZXl05Oucwglcr8aQvdfXLplZXYVG9XICJ9WA08G3CAuGtD+Qp+KF5b6O0osxHeXVX1hGFmd5nZRjNbnDVtpJk9ZGYvZP49otpxiEgFauDZgP0ibMsvuH2z/J8VK7zLHWU2NyledlnVhzXvUfUmKTObAmwH7nX3UzPTvgFsdvd/N7MbgSPc/XOl1qUmKZGI9ba5JEpxx1po+2bw4x+H04RXqInw2mvhgQfSibyhIZ0ssrYXVpNUJH0YZtYE/C4rYSwDLnL3dWY2AZjj7pNKrUcJQyRitdSHEaAtP5btQ+Hp5aowKdZ6H8Y4d18HkPl3bKEZzWy6mbWbWXtnZ2dkAYoIkfwoT2jiflK60HYaG8PbRgVNhPt27wtt84nv9Hb3We4+2d0njxkzJu5wRPqfadPSV6+pVPrfJCYLiOQnSmPffoCk2LWziwWznuXfL53DO8c8wcgheZ7xqFBcCWNDpimKzL8bY4pDRPqKqGpDhe7EimL7eZKSD6lnafPHueXiOVwyaiEjhu7l/I+fxuf/dBEdW0cy7eRF4W3f3av+ApqAxVnvvwncmPn7RuAbQdZz1llnuYiEoKXFvbHR3Sz9b0tL3BHVhpYW9/p693SvRPpVXx/p8dv9X3f5zhETPIX5OpvgH+Tu/aGcdugy/8fT5vgv/uUR37B44/5lgHYPoSyP4i6p+4CLgNHABuAm4H7g50ADsAp4r7tvLrUudXqLhKCWOrKTJoY7sbav384jdy+j7XfbaHt2JI9vO4kuBlNHN2cMWU7zmzYw5e1DuPAjJzDqhJF511FTd0mFRQlDJARx335ayyK4E2vLqi3Mv2s5bQ/sYO7zo2nfcRLdDGQA+zhr6FKuH/tT3rP5ToZs3YDluYU2n1q/S0qk/4j76eNctfQwXtJU4U6szS+9xv984TH+5aw5nFW/hJGNw7jiy2/hO0+cz8C6FJ89bz5/vKWd117ZxWP//TTv3/Bt6resx2J4OFE1DJFqSmLzj2oY5Zs9Oz0cSkdHujaRXW6W+X1ufK6Tefe8SNuDe2lbPp5nd5+AU8ch7Obcw5fSfPrrNF95OOdeO4n60Tl3XcX8HEYknd5hvdTpLTWnsfHADtKeV2NjfDEloON2fxy10PGe73iZvfE9loh77VPr/b5/etg/cXKbv2nwi28ccrb7xSPb/St/0+ptty3yXa/tKh1Lz3ZzX2ZFFyOkTm/9prdINSWx+afnSnjmzIJDSVRdNX8bO2z5BmB0L3hVv2rBK7T96GXmzknRtuJoXug6BhjHMLbx1tFL+eBZq2m+ahRnvX8Sg4edVV4sDQ35axgRPZyoJimRalLzT361dFyKdHT7vm5enruatns7aJtrtHU0snLf0QAczhYuHLuM5rfspPm9YznzfScy8NBeXqNX2MQZVpOUahgi1XTrrfn/gyfod5pjkcSaVyEFruo768by5sHrWNPdADQwyl5lyvgXuOHcFTS/bzyn/e3xDBh8drixxFw7VMIQqaYkNP8kUcxNK0F5yllz1acYf9sXGNS9Z//0HdTzxe6bOe+oDprPe4HmqUdy8ruOo27gudUPatq0+M6fMDpConqp01ukj0hKx3uO7q5uX/Szpf7dq+b4VUcu8NHW6eA+lRZfzUTvxnxr/Thf85n/9FR3KtZYD1LkJgLU6S0iNSshNa99u/ex6OfLafvFRtqeGMK8jZN43ScBk2gcsIbLjllK8xSn+Zq3MvGi1VidMRwYHmmUAUR0E4E6vUWkunqeYUhAk1zXzi4W/mQZbb/aRNvCoczvnMQ2DgPg+EEraT5mNVOajeZrm2i84KhYYqxIiZsI1Okt0tclqKCtWMy3z+7ZuofH711K2/2vMfep4TyyeRI7OBWAkwa/xPvftIjmtw1kyrXHMHFyE+lxUmtQRDcRqIYhkkRJfEK8EhHfPrtr8y4evWcpbf+7hbanD+fR109iN0MAOO3Q5Uw5YR3NFw9mykeOZ9ypCfp9nd5eHERUw4i9I7uclzq9pd9I4hPilajwyWR3D/Qk+LZ12/zBf2v3mRe0+lsPW+SD2Z1ePd1+5pDn/YYz5/hvbnzUNy1/NXjMUT+BHsYNACXWQUid3rEngXJeShiSWGEXMr0paMtR7cKx0sRXoADccdsP/fc3P+6fPbvVzxn6rA9kr4P7ALr8LUMX+2cmt/pvv/iYv7by9crijePuraDHqNR3FcFdUrEngXJeShiSSNUoZKKoYURROFa6jQL7v5IGB/dB7PHzhz/tnz+v1f94yxO+9ZWt4cQbR80uyMVBL78rJQyRpKhGIRNFYR5V4VhGLWbj853+y8884inyF6IpzP/8jYW+o3NHuDH2iKpmly3I99DL70oJQyQpqlXIhNVcVGg9cRSOOdY+td5/+qmHfcYpbX7yIS/sD6GDowsXkNVsRoujhhHk4qCX35UShkhSJLmDulhhFHXcLS3eNX6ipzDvHDjO/7nu2/s3OYyt/o5RT/jX3t7qD9/+jHfdeU/+uGfMqG7Nq9KaXW+TWKnlVcNQwpA+IqHDXLh78YKmynGnulO+om2V333dPL997Bd9B0MO2NYuDvXfnjHTH7trsXft6jp4BfkK0aj6dsop/JPcF5ShhCGSJEn9MaBSTRkhxp3qTvmyP67wWde0+bSm+X70gDX7N7eKo8Ip6BPQjHaQBPYF5QorYcT64J6Z/TPwUcCBZ4EPu/vuQvPrwT2RMpX74FwZD5B5ylnyu5dou28tbQ8PZO4rx7IuNR6AsdZJ88QXmXLuXpqvnsCp7z0p3Y2dywxSqertTxSK/F5GWftWRTX/4B4wEXgZGJJ5/3PgQ8WWUQ1DakoSah3lNGWUmDd7JNe/m/iIj7GN+2c7sm6tT22c77e/v82X/P6lg0dyDesqPInNf0nuw8qg1pukMgljNTCS9JhWvwPeXmwZJQypGUkq2IImrgIF35Yh4/xbh93sHRzl3Zi/TKP/k93m1xw7z++4dq6/8OeVpYf6DvN4VJKIq5m8k/RdF1DzCSO9D1wPbAc6gdkF5pkOtAPtDQ0NoR5EqSG9eMo1FjVw1ZkrVaB/oBsO6rCuqECM6zuKqlM6SedfjppPGMARwF+AMcAg4H7gA8WWUQ2jnyr1H76SAqHa/8GT2DmbY/eW3T7v+0/7LRe3+iUj2ws++5CqG5B/X0aNinsXgqnB5B22vpAw3gvcmfX+g8APii2jhNFPlfoPX26B0Jeeoi7Dzld3euu3n/KbL2r1vx7xpB/Kzv1hnXrIcr/7qJneNfCQg49Lvv3oeSXsSjqvGkje1dYXEsY5wHNAPWDAj4B/KraMEkY/Veo/fLkFQq2P0xSwdrR9w/aCI7meMWSJX3/GHP/1Zxd459JNxddd6HjVylV6ApN31Go+YaT3gS8DS4HFwI+BQ4rNr4TRT4Vdw6jlkWBbWtwHDz4w7sGD3VtafMvqLf7Alx/3z53T6ucOeya8kVxbWgonjKQMf1JqGwnvlK62PpEwyn0pYfRTYfdh1PIV56hReWPfzAivY5+D+0D2+nnDnvEbz231P3w1pJFcC2w38QMsZm8rwZ3S1RZZwgD+DJwexsZ6+1LC6MfCvEuqRq84O5du8lSBK/0U+BcvbPWHvr7Qt2/YHv7Ga3UId3H3aBPGmzN3M90NTAhjo5W+lDAkNHHe4hlwu+ue3rB/JNdTDlnu4AUTRvq/cnJiD0Sd0ZGJvEkK+DvgaeCmnqezo34pYch+URb4YQ4zXuQqffXja71lxnz/2EltfuKgFftn6RnJ9dZLWn3P0MPzF7JR3uIa1vFQDSMykSaMzF1MpwKfADYBa4BrwgignJcShrh79G3f+W4tHTWq/O0VKCA7B47zYweu3D/pMF73y8c+5t+4rPXgkVxbWtwHDTpwHYMGhTu8d6mmvzCf2K7BpsFaFGWT1HxgLfAQ8FXgCuB44HvArDCCCPpSwhB3j/bKtNgtpWUUbqnuVJEnqc3fPf5R/893t/rClud93559xVdWrdpVkAI87GPfzzujoxJWwig5Wq2ZnQo853lmNLMl7v6moisIkUarFSDa0UELbatHgVFSPeUsfWAFbT95hbaHB9K25lgeSZ1LEwePtOoNDVi+EVijFmQk2BoYmVUOFtZotQNLzeDui4t8fHlvAxApW0ND/oKtoSG6bfVYtQqA1L4Ui+9/kbafrqPt0cHMXXs8nX4ccBwT6tbTfNQKXpx4DQ1Pfou6PVkj+NfXY1/7WvhxVyKzL0WnR3nsJXHqerOwu68IKxApYvbs9NVfXV3639mz444oXrfeCvX1B06rr09Pj2JbWbYeOpb3THiMMYNf5/T3nsinftXME+uP5tJjlnHHtfNY/uBKXukax30d53PxI1+l7s470lfsZul/Z80q+HsTkStU6GdPj/LYS/KE0a4V1atf9mHUQsdgHO3QEd8llRo16qBbWrdT71Np8WMHrvQPnzDX7/noPH953urkxF2uoOda9nAhAwa80YeRpH2RA6AnvfuJpN96WAsJLYicgrzrznt8/g+e9lsvafW3j3rCh7LNp9LiL9Po3ZhvGjjO51/8JV/9+NrytlHqifW4k0nQGPrK995PKGH0F0l9uKkvDErXo6XFU0MOLPy2M8Sn0uLgfsohy/2Tp87xn93wsK97ekPl2ymW/GulAO5L33s/ooTRXySxhlHo2YSkJLQAV8nbN2z3h76+0P/PW1t9nU3Iuw/bDxt/4EiuvVUs+Sfxe85V6feehJpTP6eE0V8k8cqz2BVmXAVd9pVvbsFcX+87bvvh/pFcz8saybWOfd5NRLW4YkmhWDJJSoFbyfeexPO3H1LC6E+SUmD0KFS4ZRcIM2ZEO3RHiSvflTQ4vDGS6+fOafUHvvy4b1m9Jbqr+2KFZ6EYRo1KToEb5HvPjasWak79gBJGXJJWeMehVBv2jBnRFnIBrnxTWOGRXJMwzHahGKoxrHilSn3v+Y5XUvvg+hkljDioep2W7zj0FAyNjZUVcmUm4vXPbvSf3fCwf/LUOYWblMopYJNwIZAvhiQVuJWc/6phJIISRhx08r+hWJ9BoVehQi5AQbTmibU++5PzffpJbT5p8Ev7ZxvKNl9fN774dms5qSftnCs3seoiKxGUMOKQpKu9pAjSEVqqkCuwjm3DxvmHT5h70Eiul4153L/+zlZ/9I5nfe+OvaVrPC0tBxd0Ufax9EZfKHCTUHvr55Qw4pC0q70kCFq7KFLIFRvF9QjbvH8k1/YfFxnJtVihFOR20Nz4klTIJSkWqUlKGHHoC1d7YSt2d0+BQi7VnfLnf/ui/9fUNr+64WFfzcS869gz+kjv7uquXoyFEr++Z+lj+kTCAEYAvwSWAkuA84rNH3rCqOTKra9c7WXvx6hR6Vcl+xSgcO3u6vZnfrnMv/f3c/zvJz7iY23j/lnH163320be5HsHHFK9ArrcPpZaqkn2lfNRqqqvJIwfAR/N/D0YGFFs/lATRn++iizVRFPuccgptLrvudcXtjzv337PHH/PhAU+0l7dv+qjB6zxDxwzz3/4wbm+/MGXPdWdyruOin/BLd86yq1h1EpfVX8+h6UsNZ8wgMOAlyH9I05BXqEmjEquIuO+mgtr+yE/qd21q8sfu2uxf+OyVr987GN+OK/vX03PSK53XzfPV7SteiNBhK1Y4VluH0at1DAKxTlggGoccoC+kDDOAB4H7gGeAu4AhuaZbzrQDrQ3NDSEdwTLvYqM+2ouzO0HaaIpcjW9Z9ue/SO5vmPUEz6MrfsXO3HQCv/YSW3eMmO+r3r0lV7scJlKFfLl3CUV93cdVJDvMYlxS+T6QsKYDOwDzsm8/y7w1WLLxFrDiPuqM8ztl1nD2PXaLp/znaf8y3/d6n9zxEIfwo79s51yyHKfcUqb//RTD/vap9YH235Ly4EP940a1ftCLexmpLhrk0GU29Qm/VZfSBjjgZVZ7y8Efl9smVj7MOJu1y52NVlJx32RJprUkCH+zPtu8S9e2OpTDn/KD2FXelfp9tMPXeqfOn2O/+pfF/jG5zvL34+WFvfBgw/e7qBBvSuU407ocQjS1BblOSqJVfMJI70PzAMmZf6+GfhmsfljvUuq2gVSqVgKbT/PyKzl3u3VPWKk76k/3FOYr7UJ/gF+5JAeyfWs+uf8X85q9f/5wqO+ecVrvd/PUuMRVapWmpHCln3e9Pz6XX9KmhJIX0kYZ2T6J54B7geOKDZ/rM9hVLNACrLuYk8zl1lAvLbydf/tFx/zz0xu9bcMXewD6HJIj+R6btZIrq93vN77fctVrKbU2+YjKP6ToTNmvPH5gAHp931Jf02aUlKfSBjlvhLx4F412rWD1l5ytx+w4N20/FX/9WcX+PVnzPEzhixxo9vBfTC7/a2HLfKZF7T6g//W7tvWbQtnfyrZ10qvhIMWkjNm5N9mX0waSe97kciFlTAsva7aMHnyZG9vb487jPDV1aWLr1xmkEoVXq6pCTo6DprcPeEofn31z2n7817alh/J4j0nAHAouzhvxFKaz9jClHcdzrkfOokhI4eEtBMBzZ4NH/kI7N174PRBg+Duu2HatPLWV+AY0NgIK1e+8X7gQOjuPni+AQNg377ytilSY8xsobtP7vWKwsg6Ub1ir2GEJbdpZOjQyq6481xd7+TQ/b9FXc92v2Rku99ycavP+/7TvnvL7kh2r6RSd0mVc5Uc9GaEYp3CIn0capKqUYWaRnI7LIu0Pa+cv9p/9LF5/pET5voN9h1/mUbvxryDo/3rw7/i/35pqy/4YWYk16TL93xEOe3wQZvzCnUIDxhQfoxq5pEao4RRqwoVXHV1eQulVHfKX/jzSr/zQ3P9g8fO88YBq/cvcoRt9ivHP+r/cWV6JNeuXV1x7ln5wujIr3YfhjqSpQ9QwqhU3FeLJZpGUt0pX/L7l/z297f51Mb5PrFu7f6PR1unX3XkAv/uVXN80c+WhjOSa5zK+S2NYndQBf1OK7lLqj8+3yF9TlgJo/90es+eDddfD6++euD0+nqYNav8ztZKFeh8TVkdV0+cz9y1x7EhNRaA8XUbaJ74Es0X7KN56pG86YrjsDqLJs4oFOrszye3Ezsqld6QIJIgYXV614URTOLNng3Tpx+cLAB27oSZM9PzNDWlC4impvT7Kkhd9zFyix8HfuAfZ8G6Ri5ueIFZ18xl2R9fZm3XWH666nxm3DeFk688vm8lC4CGhvzTLWc/6+vh1lurH08+hWIsNF2kLwujmhLVq+ImqSBNH1Vqp+4ZyfWbl7f6FZmRXL/HDO9igKfA91Hnz530nuIjucbdjFYthfoHkvTzqerDkD4A9WGUodSoniEOqbBn2x5/+PZn/GtvP3gk1xMGrfCPTmrzH3+ijJFc+3qBVQvJsBZiFCkirITRP/owCj3cBenmjp07838WoJ169+u7efzHy2i7/zXanjqMR147iV3UA3DyIS/SfPxamt82kCkfPo4JZ4wLL/a42vRFpOaoD6Mct96aTgy5Ro1Kd3g3NuZfLk879c5NO/l/33ySL02ZQ/OIRYw4Apo/dTo3/WUKnbuG8tG/eoJffmYBG5/fxHO7j+cHi6fwvu+eX1myAFi1qrzpYYioP6dm6fhIfxVGNSWqV69uqy3WrFCk2WfrK1v9j7c84Z8/r9XPH/60D2JP+rGJrJFc7//8o/7qi5srj62YqG/r7OtNYL2l4yM1CPVhlCFIG3RmnpSZ7zh8vM8+ZqafPfTZ/SO5DqDLzxn6rH/27Fb//c1VGsm1UOxRFlB67qA4HR+pQWEljL7fh9FzS212P0XWsxevvrCZeXe9QNufdtO2dCyLdk3CqWMQezl7+FKaT9tM8xXDOf/Dkxg2fli4O1TOPsycmW6GamhIN7FV67kRPXdQnI6P1KCw+jD6fsIo0Gm8edA4Lhowl2d3nwikR3I9d8RSmk/fQvOVIYzkGmUh3xu5cW7fnv95FXWyp+kmBKlBYSWMgWEEk2S+ahX5Hncb0bWRccO38Q8XzKH5PUdw9gdP4pDDzgxno7m1mo6O9HtITtLI9+R7R0d6mPHBgw8cfjzOB+eS5tZb89dYdXykPwijXSuqV5A+jI5H1vi9H5/n1504148f9LK/TGPeNufU0Q2lG/5yBb0fP+nt3KV+C3rUKD13UIyey5Aag/owwFPOijmraPvxKtrmGm0dTXR0HwXACHudC8cu4+MTfsulz/8HA/bufmNFlYwfVaIv5ABJb+cu9lwKJCdOEQlFv30OY9kfVjDrA3OZ1vQwRw9az/Fva+S6ey7kgZdP4qxxa/juVW089dNlbNo9nP9dfw6XP3ULA+66I93GbJb+d9as9MrKuZd+5syDH/DrGYcqV9LHHyr1DEdS4hSRRKmpGsYgO8P3sQiAcXUbaZ74Is3nd9E8dSJvuvxY6gYGzH/l1BZ6lFNrqGT9USr15HtS4hSRUPSZn2gFBgBPAb8rNe/IwZP8v6e1+dIHXio8UF8QlfQxlLtMktu5C/Vh5P5Uqoj0CYTUh5GEJqnrgSVBZjzmtGFMb5nCpHcee+BQ3+UO1VDJcBv5hhcpdnfMtGnp2yxTqfS/SbpinzbtjSFReprpWlpg06ZkxSkiiRJrwjCzo4DLgTsqXklP809HR/o6uecW1mJJo5I+hnyFbC033SQ5oYlIIsXah2FmvwT+DRgOfMbdr8gzz3RgOkBDQ8NZHblt75U8SJX0PgYRkRDV/F1SZnYFsNHdFxabz91nuftkd588ZsyYg2eopHmpFmoLGhFVRBImzie9LwCuNLPLgEOBw8ysxd0/UNZaGhry1zBK3Ro6bVqyEkS2WnhSXET6ndhqGO7+eXc/yt2bgKuBv5SdLAAuu6y86bWgnGc+REQikoS7pHrngQfKm14L4vjRJBGREhKRMNx9Tr4O70D6YuGa9CfFRaRfSkTC6JW+WLiW+8yHiEgEaj9h9MXCtRbu4hKRfqf2fw+jpxCthR8rKkeS7+ISkX6p9hMGqHAVEYlA7TdJiYhIJJQwREQkECUMEREJRAlDREQCUcIQEZFAlDBERCQQJQwREQlECUNERAJRwhARkUCUMEREJBAlDBERCUQJQ0REAlHCEBGRQJQwREQkECUMEREJRAlDREQCiS1hmNnRZtZqZkvM7Dkzuz6uWEREpLQ4f3FvH/Bpd3/SzIYDC83sIXd/PsaYRESkgNhqGO6+zt2fzPy9DVgCTIwrHhERKS4RfRhm1gScCTyW57PpZtZuZu2dnZ2RxyYiImmxJwwzGwb8CrjB3bfmfu7us9x9srtPHjNmTPQBiogIEHPCMLNBpJPFbHf/dZyxiIhIcXHeJWXAncASd//PuOIQEZFg4qxhXABcA/yNmS3KvC6LMR4RESkitttq3X0+YHFtX0REyhN7p7eIiNQGJQwREQlECUNERAJRwhARkUCUMEREJBAlDBERCUQJQ0REAlHCEBGRQJQwREQkECUMEREJRAlDREQCUcIQEZFAlDBERCQQJQwREQlECUNERAJRwhARkUCUMEREJBAlDBERCUQJQ0REAlHCEBGRQGJNGGZ2qZktM7MXzezGOGMREZHiYksYZjYA+D7wTuBkYKqZnRxXPCIiUlycNYyzgRfdfYW77wV+Crw7xnhERKSIgTFueyKwOuv9GuCc3JnMbDowPfN2j5ktjiC23hoNbIo7iAAUZ3hqIUZQnGGrlTgnhbGSOBOG5ZnmB01wnwXMAjCzdnefXO3AektxhqsW4qyFGEFxhq2W4gxjPXE2Sa0Bjs56fxSwNqZYRESkhDgTxhPACWZ2jJkNBq4G/jfGeEREpIjYmqTcfZ+Z/SPwJ2AAcJe7P1disVnVjywUijNctRBnLcQIijNs/SpOcz+o20BEROQgetJbREQCUcIQEZFAEpEwSg0RYmm3ZT5/xszeHHTZiOOclonvGTN7xMxOz/pspZk9a2aLwrrFrRdxXmRmWzKxLDKzLwVdNuI4/zUrxsVm1m1mIzOfRXI8zewuM9tY6PmfBJ2bpeJMyrlZKs6knJul4kzCuXm0mbWa2RIze87Mrs8zT7jnp7vH+iLd4f0ScCwwGHgaODlnnsuAP5B+duNc4LGgy0Yc5/nAEZm/39kTZ+b9SmB0Qo7nRcDvKlk2yjhz5n8X8JcYjucU4M3A4gKfx35uBowz9nMzYJyxn5tB4kzIuTkBeHPm7+HA8mqXnUmoYQQZIuTdwL2e9igwwswmBFw2sjjd/RF3fy3z9lHSz5ZErTfHJFHHM8dU4L4qxVKQu88FNheZJQnnZsk4E3JuBjmehSTqeOaI69xc5+5PZv7eBiwhPYJGtlDPzyQkjHxDhOTudKF5giwblnK3dR3pzN7DgQfNbKGlhzuplqBxnmdmT5vZH8zslDKXDUPgbZlZPXAp8KusyVEdz1KScG6WK65zM6i4z83AknJumlkTcCbwWM5HoZ6fcQ4N0iPIECGF5gk0vEhIAm/LzP6a9H/Kt2ZNvsDd15rZWOAhM1uauYqJI84ngUZ3325mlwH3AycEXDYs5WzrXcDD7p59xRfV8SwlCedmYDGfm0Ek4dwsR+znppkNI52wbnD3rbkf51mk4vMzCTWMIEOEFJonyuFFAm3LzP4KuAN4t7u/2jPd3ddm/t0I/IZ0lTCWON19q7tvz/z9ADDIzEYHWTbKOLNcTU6VP8LjWUoSzs1AEnBulpSQc7McsZ6bZjaIdLKY7e6/zjNLuOdntTtmAnTcDARWAMfwRufLKTnzXM6BHTePB1024jgbgBeB83OmDwWGZ/39CHBpjHGO542HNs8GVmWObaKOZ2a+w0m3JQ+N43hmttFE4U7a2M/NgHHGfm4GjDP2czNInEk4NzPH5V7gO0XmCfX8jL1JygsMEWJmn8h8fjvwAOne/heBncCHiy0bY5xfAkYBPzAzgH2eHslyHPCbzLSBwE/c/Y8xxvn3wAwz2wfsAq729FmUtOMJ8LfAg+6+I2vxyI6nmd1H+s6d0Wa2BrgJGJQVY+znZsA4Yz83A8YZ+7kZME6I+dwELgCuAZ41s0WZaV8gfXFQlfNTQ4OIiEggSejDEBGRGqCEISIigShhiIhIIEoYIiISiBKGiIgEooQhIiKBKGGIiEggShgivZD5PYJLMn/fYma3xR2TSLXE/qS3SI27CfhKZqC5M4ErY45HpGr0pLdIL5lZGzAMuMjTv0sg0iepSUqkF8zsNNK/fLZHyUL6OiUMkQplfrlsNulfKtthZu+IOSSRqlLCEKlA5pfWfg182t2XAF8Fbo41KJEqUx+GiIgEohqGiIgEooQhIiKBKGGIiEggShgiIhKIEoaIiASihCEiIoEoYYiISCD/H/EJOrR/1t+fAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -2798,7 +2462,7 @@ " random_index = np.random.randint(m)\n", " xi = X[random_index:random_index+1]\n", " yi = y[random_index:random_index+1]\n", - " gradients = 2 * xi.T @ ((xi @ theta)-yi)\n", + " gradients = 2.0 * xi.T @ ((xi @ theta)-yi)\n", " eta = learning_schedule(epoch*m+i)\n", " theta = theta - eta*gradients\n", "print(\"theta from own sdg\")\n", @@ -2817,9 +2481,7 @@ { "cell_type": "markdown", "id": "e352b052", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "**Challenge**: try to write a similar code for a Logistic Regression case." ] @@ -2827,9 +2489,7 @@ { "cell_type": "markdown", "id": "6df2d658", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Code with a Number of Minibatches which varies\n", "\n", @@ -2838,13 +2498,39 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 8, "id": "ef94f697", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.11411856]\n", + " [2.8912056 ]]\n", + "Eigenvalues of Hessian Matrix:[0.30049071 4.27622313]\n", + "theta from own gd\n", + "[[4.11411856]\n", + " [2.8912056 ]]\n", + "theta from own sdg\n", + "[[4.13041082]\n", + " [2.89643997]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlmUlEQVR4nO3deZwU1b338c9vWB1AdgGBmTFRiYo7Cu6YxLhm9SbRS9Zrwo15spjNqDxGDdFrYvIk1zzJzUPcw8Tsq9GoiTNo3AcE3MAFAdkEBUUW2eb3/FE92jS9VE9XdVU33/fr1a+Zrq7uOt1TU98659Q5be6OiIhIKQ1JF0BERGqDAkNEREJRYIiISCgKDBERCUWBISIioSgwREQkFAWG1C0zu9zMZiZdjiiZ2WQzW5Z0OWT3pMCQqjKzxWa22cw2mNkqM7vJzPonXS4RKU2BIUl4r7v3Bw4DDgcuTrY4uycz65l0GaS2KDAkMe6+CriTIDgAMLOLzOx5M3vdzJ4ysw9mPfYpM/uXmX3fzNaZ2QtmdnrW4/uY2azMc+8GhmVvz8zeZ2ZPmtmrZtZuZgdkPbbYzL5hZvPNbKOZXW9mI8zsjszr/cPMBud7H13NRGb2NTNbbWYrzezTWY+3m9lnct9H1n03s8+b2bOZbU03s7eb2YNmtt7MfmNmvXO2eYmZvZwp95Ss5X0yn89SM3vJzH5mZnvklPObZrYKuNHMhpnZbZnPZK2Z3WdmOi5IXtoxJDFmNgY4HXgua/HzwAnAQOAKYKaZjcp6fCKwkCAMvgdcb2aWeeyXwOzMY9OBT2Zta3/gVuACYDhwO/DXnAPx2cApwP7Ae4E7gEsyr9cAfKnI2xmZKfNo4DzgJ4UCpoDTgCOBScCFwAxgCjAWGA+cm7OtYZltfRKYYWbjMo99N1P+w4B9M+t8K+e5Q4BmYCrwNWAZwWcyIvN+NV+Q5KXAkCT8ycxeB14EVgOXdT3g7r919xXu3unuvwaeBY7Oeu4Sd/+5u+8AbgZGASPMrAk4CrjU3be4+73AX7Oe91Hgb+5+t7tvA74P7AEcm7XOj939JXdfDtwHPOzuj7n7FuCPBM1nhWwDvu3u29z9dmADMK7I+rm+6+7r3f1J4AngLndf5O6vEQRX7ra73ucs4G/ARzLB+VngK+6+1t1fB64Czsl6XidwWea5mzPlHgU0Z8p+n2uCOSlAgSFJ+IC7DwAmA+8gq+nIzD5hZnMzTSSvEpxdZzctrer6xd03ZX7tD+wNrHP3jVnrLsn6fe/s++7eSRBYo7PWeSnr98157hfrnH/F3bdn3d9UYv1c5Ww73/vcm6CW0AjMzvr8/p5Z3mWNu7+Rdf8aghreXWa2yMwuKqPMsptRYEhiMmfHNxGc7WNmzcDPgS8AQ919EMHZthV4iWwrgcFm1i9rWVPW7ysImmHIbMsImnuWd/8dhLaR4EDeZWSFr5fvfa4AXiYIl4PcfVDmNjBzgUGXnWoP7v66u3/N3d9G0Az3VTN7V4XlkzqlwJCk/Qg4xcwOA/oRHNDWAGQ6jseHeRF3XwJ0AFeYWW8zO57gANjlN8CZZvYuM+tF0Ha/BXggovdRzFzgQ2bWaGb7EvRxVKrrfZ4AnAX8NlNr+jnwQzPbC8DMRpvZqYVexMzOMrN9MwG6HtiRuYnsQoEhiXL3NcAtBG3yTwE/AB4kaJI5GLi/jJf7d4JO8bUE/SK3ZG1nIfAx4McEZ+LvJbi8d2sEb6OUHwJbCd7TzUBrha+3ClhHUKtoBT7n7gsyj32ToInpITNbD/yD4n0p+2XW2UDwuf/U3dsrLJ/UKVP/loiIhKEahoiIhKLAEBGRUBQYIiISigJDRERCqanJx4YNG+YtLS1JF0NEpKbMnj37ZXcfXnrN4moqMFpaWujo6Ei6GCIiNcXMlpReqzQ1SYmISCgKDBERCUWBISIioSgwREQkFAWGiIiEosAQEZFQFBgiIhKKAkNEREJRYIiISCgKDBERCSX2wDCzG8xstZk9keexr5uZm9mwuMshIiKVqUYN4ybgtNyFZjYWOAVYWoUyiIhIhWIPDHe/l+A7lnP9ELgQ0HfEiojUgET6MMzsfcByd58XYt2pZtZhZh1r1qypQulERCSfqgeGmTUC04BvhVnf3We4+wR3nzB8eMXTuYuISDclUcN4O7APMM/MFgNjgDlmNjKBsoiISEhV/wIld38c2KvrfiY0Jrj7y9Uui4iIhFeNy2pvBR4ExpnZMjM7L+5tiohI9GKvYbj7uSUeb4m7DCIiUjmN9BYRkVAUGCIiEooCQ0REQlFgiIhIKAoMEREJRYEhIiKhKDBERCQUBYaIiISiwBARkVAUGCIiEooCQ0REQlFgiIhIKAoMEREJRYEhIiKhKDBERCQUBYaIiISiwBARkVAUGCIiEooCQ0REQlFgiIhEqbUVWlqgoSH42dqadIkiE3tgmNkNZrbazJ7IWnaNmS0ws/lm9kczGxR3OUREYtfaClOnwpIl4B78nDq1bkKjGjWMm4DTcpbdDYx390OAZ4CLq1AOEZF4TZsGmzbtvGzTpmB5HYg9MNz9XmBtzrK73H175u5DwJi4yyEiErulS8tbXmPS0IfxH8AdhR40s6lm1mFmHWvWrKlisUREytTUVN7yGpNoYJjZNGA7ULCBz91nuPsEd58wfPjw6hVORKRcV14JjY07L2tsDJbXgcQCw8w+CZwFTHF3T6ocIiKRmTIFZsyA5mYwC37OmBEsrwM9k9iomZ0GfBM4yd03lVpfRKRmTJlSNwGRqxqX1d4KPAiMM7NlZnYe8H+BAcDdZjbXzH4WdzlERKQysdcw3P3cPIuvj3u7IiISrTRcJSUi8pY6Hild6xLpwxARyatrpHTX4LeukdJQt/0CtUQ1DBFJjzofKV3rFBgikh51PlK61ikwRCQ96nykdFE10HejwBCR9KjzkdIF1cgstwoMEUmPOh8pXVCN9N1YLc3KMWHCBO/o6Ei6GCIi0WpoCGoWucygs7Pilzez2e4+odLXUQ1DRCRpNdJ3o8AQEUlajfTdKDBERJJWI303GuktIpIGNTDLrWoYIiISigJDRERCUWCIiEgoCgwRkSSVmhIkRVOGKDBERCpRyQG91JQgYaYMqWKgaKS3iEh35X5/BwTjJ8JeEtvSEoRAruZmWLy49OMhtx/VSG8FhohId5U6oJdSakqQUo+H3L6mBhERSVql399RakqQUo9X+ftDYg8MM7vBzFab2RNZy4aY2d1m9mzm5+C4yyEiErlK54AqNSXIGWcEtYncx884I6hdFGohimkOqmrUMG4CTstZdhHwT3ffD/hn5r6ISG2pdA6oYlOCtLbCzTfvHApmcMwxwfJ8TVHlbr9MVenDMLMW4DZ3H5+5vxCY7O4rzWwU0O7u40q9jvowRCR1WluD761YujQ4s7/yymim+CjUP9GjB+zYkf85zc15tx9VH0ZSc0mNcPeVAJnQ2KvQimY2FZgK0JSyqX5FRGKbA6pQP0ShsDDbpaN98b+WMevmxXlX747UTz7o7jOAGRDUMBIujohIfLJrKw0N+cOhUA2jqYkl9y+j/abFtLc77YtbWLx9LDAmsuIldZXUS5mmKDI/VydUDhGRwqo5yjp3kF6+UGhsDNbJ6TfZYn348tKv0nL8GD513fH89fkDOWKv5Vx79izm/+6ZyIqYVA3jL8AngaszP/+cUDlERPLLHRTXNcoa4mmCyve93hDUKDo7oamJlz/1dW5fcRib+w/jzE03sDcrWEoTV/nFvLL3oVx7zCwmnzuKg96/Lw09J0VfRneP9QbcCqwEtgHLgPOAoQRXRz2b+TkkzGsdeeSRLiJ1ZuZM9+Zmd7Pg58yZ6dhuc7N7cK6/8625OZ7ymOXdXifmn97vXt+n55I3Fw+xV/xDez/o157d7vN/t9B3bNtR9KWBDo/geB57DcPdzy3w0Lvi3raIpFy1z+LL2W6VB8XR1JT3qqgdNNDz2ac5bFQvLjjmBSafO4rxH4ipBlGCpgYRkeRUOrVGnNstdllrpomo0ktoX3x4Be03LqL9HmfAc3O40i+hH7s2S3ljI1bBV7ZqLikRqX2l5kpKcrv5JvbLVc5Eg+wcEO0vNLFoezMAQ2wtJ418hqmjb+PUOVdjnXk6vCsIUQWGiNS+NNcwINxlrkXKWiggBts6Thq5kMmTtjD5nJEc/KH9aOiZuWg1hhCt9YF7IiJBk06+6bljmtqi7O1mD8prKDAKIatPY9mjK2m/YRHt93TSvqiJ57c3A3u/GRBfmrQ4KyAK9EEU6MuIa36ocigwRCQ5XQfjOKbWiHq7BQ7kG/rtxQXj7ssKiFEMslc5acRCvjBpMSefWyIgciUVoiGoSUpEJIzWVjo/81ka3tj85qKNNPJZZnCHnclJIxYyedIbQQ3ig/vSo3ePirYVZYiqD0NEJGbLO7qamHbQvqiJo7bdz1VMo4mlvNwwgtmHn8eob3y88oCImfowREQilhsQz21roauJ6cQRCzlq4mjWffTvjD17P/bq3YPTky5wlSkwRGS3tbxjJbNuXETbP3cOiIG8xkkjF/D5iYuZ/NGRHHL2fvToPTHp4iZOgSEiu42ugGi/Zwftz4/l2W37oIAIT4EhInVrxZxVtF//fN6AOHHEAj43cQmTPzKCQz+8vwIihKSmNxepD9Wc/jqN20+ZFXNWcesXH+A/D7iXcb1fYPSRI5ny0+P4zYJDeceQ1fzgfe3Mnvk0r2zpz19WTeSrf57MEVMOSHWHddni3CeimMGwWjfNViupMnOme2PjzrOLNjZWd7bVJLefAstnr/RffuF+n/qOWb5/r0Vvfgx78qqftdfD/v2z2nz2zKd8+5btSRe1fN2ZxbfAPjEMFnkUs49H8SLVuikwJFWqPf11nNsv5+CU1HTk7r7isVUlA6LjFzUaENm6ezJQYJ84BLZ4BMdgjcMQ6a6kJs6Levv5JtkrNKleOetGYOXcl5h1w/O0/2M77c+PYeHWtwGwJ69x4l4LmXz0JiZ/dASHfWT/+mpW6u4cWwX2iQlAh7tVXK4oUqdaN9UwJFWSOsOPevvlvE7MtaoVj63yW794v//nAbN8XO/nd6pBnLnXw37NmW3+6M1P+rbN2yLZXlmiqlmFeZ0CX6bkZsVfO+YaRuIhUM5NgSGJyfdPHlUfQndfJ6rtl3Nw6u6BrICV817yX31p14AYwGvJB0S2av+tuxvM6sNQYEjCiv2TR3HWWclZe7W3X2ENoysgPnfgLH9H7+fSGxC5qlmbmznTfejQXdcJG1B59gki+orWxEOgnJsCQxIRd+d2xGftZSvn7LnMM+1iAXHG8Ef8e2e0+SM3pTAgckX1Nyr1Ovk+XwgCpIKLCxQYItUS9wE96aut3CO7SmrV46v91xfc7+cfNMsPqOWAyFWtGkZM+0JdBAbwFeBJ4AngVqBvsfUVGJKIuA/oMZ1VlrX9bjZr1W1A5KpWH0ZMJyc1HxjAaOAFYI/M/d8Anyr2HAWGJKIaA+QqbbeuZLtlvLdCAdGf9X768Ef8u6e3+cM3PFH7AZFPNa6SUg2jaGC8CAwhmNPqNuA9xZ6jwJDEVGOwWhJNUyW2+dITq/03X3nAPz++3Q/s8+zuFRBJiOnkJKrASGzyQXdfbmbfB5YCm4G73P2u3PXMbCowFaApBd9pK7uR7G89GzIk/u1lfTd0qOUxbtOXLGV83+d4asu+wHD68zrHD1vAJ05u5+SPDOeIc8fRs+9R8ZVrd5XUV9aGFUXqdOcGDAbuAYYDvYA/AR8r9hzVMMTdq3O2X6hfIcqmotz3ka9JKuYaxvZRY/Jucwlj/bRhj/jVp7X5Q9c97ls3bo2tDBI/6qBJ6sPA9Vn3PwH8tNhzFBhStQn3CjXVRHUgz/c+evVy79071ve2+qk1/tuvPuD/6+B2P6jPM34uM30DO5dje+++vu2Gm0uXP6H5pFJZjpSrh8CYSHCFVCNgwM3AF4s9R4EhVWvnL3S1SlRXrhR6H0OHdm+G0gLPyQ2Irs3043U/deij/l+ntvkz513tnWObwm+zkpHpUR7cNVtvaDUfGMF74ApgAcFltb8A+hRbX4EhVRvkFncNI6r3keegua1nH79xzDQfXyAgHvx5hU1M3QntOA7uaRi/UiPqIjDKvSkwpGoHibj7MCJ6H8X6ICILiFzdCbtS77c7tY+kR8jXkKoFBvAP4NAoNlbpTYEhsc/rlLutrtcbOjS4JdycsmbBy/67rz/gXzi43cf3ecZ3kP+g2RnnQbM7YVfs4B7xdz+ohrGragbGEZmrmW4ERkWx0e7eFBji7vHOHBtVeSJ63poFL/vvv/Ggf/GQdj+478K33hob/D1DH/VX+47If9Ds0SO+juDufNbFDu4Rz8yqPoxdVb1JCjgbmAdcRmZ0drVvCowqqrWrT5I624z4oFUsIE4Z0uFXntLmD/y/+b7l9S2Ftx9181k+5e4fxT6nSpqWam0/TUhVAyNzFdN44HPAy8Ay4ONRFKCcmwKjSmrxzC2p9uwKgypMQNz/s6yAyCf7oNmjRzLBGUahg7ualmJXzSapfwErgLuB6cBZwL7Aj4EZURQi7E2BUSW18A+cgkFv7l52UL38zCv+hwsf9C8d2u6H9F3w5up7sDF8QERYntiUO/ttrZ2g1JhqBsZ4CL77O89jT0dRiLA3BUaVpOWgU0iYZphqHXQKBVVmPEWnmW/cc6TfPHbaLgHx7iEd/p13t/m/fjqv+wGRKw1h350LE9S0FKtUXFYLvC2KQoS9KTCqJA0HnWKKjZHoCrtqHHRmztx1ZDb4Dsy3Wc+dlm1gD7+y3/Q3A2Lb9TfFc4BMw9l6sUGJSZetO+ogzFIRGNW+KTCqJA0HnWJKjcKuVrAVODBup6F4ueL+fJM+wIUZJZ/GE5F80v6/EJICQ+KV9EGnmFKjsGNsOnvlubX+x4se8i8f1l54DESpcqW9BlepMKPkS/290rL/1cnfSoEh6VHtf+5SfRgR/jO/8txa/9PFD/kFh7f7YXs87cYOB/e+bPJVDSPzb7/UlUpp7yOqVKGz8rAXJqTprL5O/lYKDEmHpP65Y/qGurWL1hUMiHcNnu3T39Xm9/1knr/x2huF3/v55xf/TJI4a00i1Ls7uDJNZ/VpKksFFBiSDkn/Q1V4ICwWEO8cPNu//c42v/fHc4OACLP9889/6zPpqmnklqvaIZumM/Ywf680ndWn6bOrgAKjnqWl/TaMNP1zh7B20Tr/8yUP+VeOaPPD93iq/IAoppyDS3dGSnd3n0g61MuVtvLW0v9jAQqMelVrZzRxzEIaoUIB0YfNfvKgOX7FyRUERK64DnSV7hM1Fuo19z9QAxQY9SptZ1eFFOpDyP7nruY/fiaYOs1848CR/ovmaQUDYta1c33zus07PW+XJqXuBFxcB+ZK94la2aey1cFZfZooMOpVLZwNzpwZfJ1ovnIOHRpP526BA8i6xa96xweu8C3WZ6dtbKDRp+/xHf/dAdN88+BRwXTfYfoSCoVfGHEdmCvdJ3TGvttTYNSrWjgbLHadfXY5Y/xWuS3Wxy/p9T03dvgLFChPqZHFYccLhP3s4zowR7FP6Ix9t6bAqFe1cDZYbCRvdhgUmyIi5MFr3eJXfePA/OMdVtoov3xym3cWGEBXMgDCjkguJ+DiODDXwj4hqabAqGdpPxsMW8PId6Dr1WvX+Zd6937z2+x2DBrib+wx0DsxX2Gj/N+5peCI6pIjp0sFQNQ1jDilfZ+QVFNgSHIK9WH07r3rgSzsNOQFblsa+viWxkHFD+T5ginPpIChnpd7C3Mmr4O5pFxdBAYwCPgdsAB4Gjim2PoKjBTJvUoqu7O7iM5yJ6breu1STTJhg6nrO6QLPa/cq6TUXCQ1oF4C42bgM5nfewODiq1fl4FR6uy0xs9eX13yqt922SP+9QltPqHxSV9MU/mB0XWQL+dzKBZMlcouS5q/4U4ko+YDA9gTeKHQlzPlu9VdYJQ6O63Bs9fXXnzN/3b5WwHRwPaghYg3/MSBj/lvx03z7b36lhcY3Tn4VnMQXbF+EpEUqIfAOAx4BLgJeAy4DuiXZ72pQAfQ0dTUFPHHmLBSB7UauMS2KyC+cVSbH9XviV0C4lsntvk9P5jjm17Z9NaTus7Qw4RFdwOy2pe45mtGi0KN1zAlHeohMCYA24GJmfv/DUwv9py6q2GUGqeQwkF8hQKiF1v8hD3n+qUntPk/vz/HN67ZWPrFCp2t9+8fzQEyjoNt2D6YXr0q314N1jAlneohMEYCi7PunwD8rdhzyg6MtJ+d1UAN47UXX/Pbr4goIPJJ+98oVzmX8Fb6d0rB31/qQ80HRvAeuA8Yl/n9cuCaYuuXFRi1cHbWnT6MruaOmN7H+uXr/fYrHvELj27zo/s97j3Y9mZAXNH3Kl/XZ4R3Yr5jzNhwZai1QCglbB9GFDXBFNYwpTbVS2AclumfmA/8CRhcbP2yAqNWzs7CXCUVwxcFdSkWEMfvOdf/9/Ft/o/vzfY3/ueG8gO4FkK7O8JewqsahqREXQRGubeyAqOezs66e+DIE0brl6/3O6Y/6t+c2OYTiwTELk1M3SlDnFcqpanWElcw1mvgStUpMEqpp7Oz7oRfnoPNJvr6FG55MyCOGzDPpx1XICCiKEMcoZ3Wg2hcIZa2cJSapMAoJa0Hlu4oI/y6ahBre4/I+5y1fUb43d+d7Rte2hBbGSp6ThzlENnNRRUYDdSrKVNgxgxobgaz4OeMGcHyWnPlldDYuPOyxka48ko2rNrAvA9PZ12fkXRaA2tHj+eWSxcycOvqvC81eOtq3n3hEfTbq19kZYj0OaUsXVrechGJThSpU61b3Y3DyFWs+SHrG+U2DRzpt759mk/qP9+ncItvYOea1PZefX3HkBg6YsttHunmfFNFqYYhUjbUJFVCrbX9FmhC2/yT6/3Oqzr8okltPqn/fO/JVgf3nmz1YwfM87V98jc9hZqwL4H3o45gkepTYBQzc2b+71yo5sGy3LAqcOa8mKadAuKSY9v8rv/qeKsPoljHcpKhGVVNIN97qLWTAZGEKTCKKXRdfFTz+xRT5hnwhpc2+J1XdRT81rhOzO+8qsNfX/l6/u2ltYkmiiukVJsQiURUgVGfnd6vvFLe8ihNmwabNu28bNOmYDmwcfVG7r56NtOOa+e4PeczaERvTr3kSJYyNu/LWXMT77n4SPqP7J9/e6U6lltboaUFGhqCn62t3X9v5WhqKm95PiU+SxGpsihSp1q30DWMfGe2Xbe4FTiz7sT82AHzduqDOKb/fL/4mDa/86oO3/yT67t/Nl2oiSbJM/Qotl1Pgy9FEoSapIqotEmqgjbyHWPG5t32Ypp8Uv/5ftGkNv/7dx7N38QUddt80s1Vlb6fpMsvUicUGMXk+87psNNN5zsz7tUrCJs8Ha+dZr5p0Ej/9X7T/LgB8/xj3JT3MtdNP74uXNmjVOkZetKdy+rDEImEAqOUEGMa8j4WYvrqHT16+TZ2/mrODTT6pb2v9m9ObPO5//Zt3zF6bPJX8VRyhp6Wg3XSoSVSB6IKDAteqzZMmDDBOzo6KnuR1laYOnXnztTGxrdGgTc0BIfH7mhuhsWLKytflEq912JaWmDJkl2Xp+09ikhJZjbb3SdU+jr1eZVUMSWuvOkcPab7r5226SkqmR5FU3CISI7dLzAKHPB8yVJOGDiPTy6bzkYa865TUjmXjFbLlClBjaCzM/gZdi6tKC6LFZG6stsFRqEaxFLGsnVHD/Y+uplnP3QRnaPHBmflYVU6qV7axDFxoIjUtJ5JFyBum9du5sEbF9D+59donzeYlvXT+R8+Tz/eapba3qsvw39wKQ9/cXxmyWTg0uDXQm35Q4dC//5BjaWpKTiQ1uJMuIV0vZdp0+r3PYpIWequ0zs3IB5e/w620ocGdnBkvwVMPnAN5469j0Mf/jkNK5aVPhBW0nEsIpICUXV613wNY/PazTx0U1dADOKh1w5gK4e/GRBfPupBJp/Zj+PPG8eeYw7KPGsyb9YgStGZtogIUIM1jPvuui9PQAQ1iCMaFzL5wNWcfFY/jvv0/gxsGphsgVtbFTQikrioahiJB4aZ9QA6gOXuflaxdQf0PNC37nhsl4CYfGY/jv+PFARENjVliUhK1NM4jC8DT4dZsdMb+NKEB7ntskdZu2QDj248kGsuWM6ZN32YgS2DqzsbaymaaVVE6kyigWFmY4AzgevCrH/A4X255tHJnHn5UUFtoussfsmSYHT2kiXB/TSERlQD35KanlxEJEfSNYwfARcCnYVWMLOpZtZhZh1r1qzZ+cFyzuKrfeCNYuBbmgNRRHY7iQWGmZ0FrHb32cXWc/cZ7j7B3ScMHz585wfDnsUnceCNYuCbmrVEJEWSrGEcB7zPzBYDvwLeaWYzy3qFsGfxSRx4K5nHqYvmcxKRFEksMNz9Yncf4+4twDnAPe7+sbJeJOxZfFIH3u7O49RF8zmJSIok3YdRmbBn8bV64NV8TiKSIqkIDHdvLzUGo6AwZ/G1euCNollLRCQiNT81SCi1PL3HlCm1UU4RqXu7R2CADrwiIhVKRZOUiIikX/0HhkZKi4hEor6bpHInAOwasAdqnhIRKVN91zA0UlpEJDL1HRgaKS0iEpn6DoxaHbAnIpJC9R0YtTpgT0Qkheo7MDRSWkQkMvV9lRRowJ6ISETqu4YhIiKRUWCIiEgotRUYs2drtLaISEJqKzBA32stIpKQ2gsM0GhtEZEE1GZggEZri4hUWe0GhkZri4hUVW0GhkZri4hUXe0FhkZri4gkIrGR3mY2FrgFGAl0AjPc/b+LPunII6GjowqlExGRXElODbId+Jq7zzGzAcBsM7vb3Z9KsEwiIlJAYk1S7r7S3edkfn8deBoYnVR5RESkuFT0YZhZC3A48HCex6aaWYeZdaxZs6bqZRMRkUDigWFm/YHfAxe4+/rcx919hrtPcPcJw4cPr34BRUQESDgwzKwXQVi0uvsfkiyLiIgUl1hgmJkB1wNPu/v/SaocIiISTpI1jOOAjwPvNLO5mdsZCZZHRESKSOyyWnf/F2BJbV9ERMqTeKe3iIjUBgWGiIiEosAQEZFQFBgiIhKKAkNEREJRYIiISCgKDBERCUWBISIioSgwREQkFAWGiIiEosAQEZFQFBgiIhKKAkNEREJRYIiISCgKDBERCUWBISIioSgwREQkFAWGiIiEosAQEZFQFBgiIhJKooFhZqeZ2UIze87MLkqyLCIiUlxigWFmPYCfAKcDBwLnmtmBSZVHRESKS7KGcTTwnLsvcvetwK+A9ydYHhERKaJngtseDbyYdX8ZMDF3JTObCkzN3N1iZk9UoWyVGga8nHQhQlA5o1MLZQSVM2q1Us5xUbxIkoFheZb5LgvcZwAzAMysw90nxF2wSqmc0aqFctZCGUHljFotlTOK10mySWoZMDbr/hhgRUJlERGREpIMjEeB/cxsHzPrDZwD/CXB8oiISBGJNUm5+3Yz+wJwJ9ADuMHdnyzxtBnxlywSKme0aqGctVBGUDmjtluV09x36TYQERHZhUZ6i4hIKAoMEREJJRWBUWqKEAtcm3l8vpkdEfa5VS7nlEz55pvZA2Z2aNZji83scTObG9UlbhWUc7KZvZYpy1wz+1bY51a5nN/IKuMTZrbDzIZkHqvK52lmN5jZ6kLjf1K0b5YqZ1r2zVLlTMu+Waqcadg3x5pZm5k9bWZPmtmX86wT7f7p7oneCDq8nwfeBvQG5gEH5qxzBnAHwdiNScDDYZ9b5XIeCwzO/H56Vzkz9xcDw1LyeU4GbuvOc6tZzpz13wvck8DneSJwBPBEgccT3zdDljPxfTNkORPfN8OUMyX75ijgiMzvA4Bn4j52pqGGEWaKkPcDt3jgIWCQmY0K+dyqldPdH3D3dZm7DxGMLam2Sj6TVH2eOc4Fbo2pLAW5+73A2iKrpGHfLFnOlOybYT7PQlL1eeZIat9c6e5zMr+/DjxNMINGtkj3zzQERr4pQnLfdKF1wjw3KuVu6zyCZO/iwF1mNtuC6U7iEracx5jZPDO7w8wOKvO5UQi9LTNrBE4Dfp+1uFqfZylp2DfLldS+GVbS+2Zoadk3zawFOBx4OOehSPfPJKcG6RJmipBC64SaXiQiobdlZicT/FMen7X4OHdfYWZ7AXeb2YLMWUwS5ZwDNLv7BjM7A/gTsF/I50alnG29F7jf3bPP+Kr1eZaShn0ztIT3zTDSsG+WI/F908z6EwTWBe6+PvfhPE/p9v6ZhhpGmClCCq1TzelFQm3LzA4BrgPe7+6vdC139xWZn6uBPxJUCRMpp7uvd/cNmd9vB3qZ2bAwz61mObOcQ06Vv4qfZylp2DdDScG+WVJK9s1yJLpvmlkvgrBodfc/5Fkl2v0z7o6ZEB03PYFFwD681flyUM46Z7Jzx80jYZ9b5XI2Ac8Bx+Ys7wcMyPr9AeC0BMs5krcGbR4NLM18tqn6PDPrDSRoS+6XxOeZ2UYLhTtpE983Q5Yz8X0zZDkT3zfDlDMN+2bmc7kF+FGRdSLdPxNvkvICU4SY2ecyj/8MuJ2gt/85YBPw6WLPTbCc3wKGAj81M4DtHsxkOQL4Y2ZZT+CX7v73BMv5b8D5ZrYd2Ayc48FelLbPE+CDwF3uvjHr6VX7PM3sVoIrd4aZ2TLgMqBXVhkT3zdDljPxfTNkORPfN0OWExLeN4HjgI8Dj5vZ3MyySwhODmLZPzU1iIiIhJKGPgwREakBCgwREQlFgSEiIqEoMEREJBQFhoiIhKLAEBGRUBQYIiISigJDpAKZ7yM4JfP7d8zs2qTLJBKXxEd6i9S4y4BvZyaaOxx4X8LlEYmNRnqLVMjMZgH9gckefC+BSF1Sk5RIBczsYIJvPtuisJB6p8AQ6abMN5e1EnxT2UYzOzXhIonESoEh0g2Zb1r7A/A1d38amA5cnmihRGKmPgwREQlFNQwREQlFgSEiIqEoMEREJBQFhoiIhKLAEBGRUBQYIiISigJDRERC+f+rwyKX4w8fAgAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Importing various packages\n", "from math import exp, sqrt\n", @@ -2916,9 +2602,7 @@ { "cell_type": "markdown", "id": "1a493e7e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## SGD example\n", "\n", @@ -2939,9 +2623,7 @@ { "cell_type": "markdown", "id": "df7909bc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\nabla_{\\beta}\n", @@ -2954,9 +2636,7 @@ { "cell_type": "markdown", "id": "afdf2e10", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The gradient step\n", "\n", @@ -2966,9 +2646,7 @@ { "cell_type": "markdown", "id": "c2e37d69", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n", @@ -2979,9 +2657,7 @@ { "cell_type": "markdown", "id": "6a125356", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $k$ is picked at random with equal\n", "probability from $[1,n/M]$. An iteration over the number of\n", @@ -2993,9 +2669,7 @@ { "cell_type": "markdown", "id": "abaf8e3d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Simple example code" ] @@ -3004,10 +2678,7 @@ "cell_type": "code", "execution_count": 14, "id": "4628460a", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -3029,9 +2700,7 @@ { "cell_type": "markdown", "id": "ec7e3114", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Taking the gradient only on a subset of the data has two important\n", "benefits. First, it introduces randomness which decreases the chance\n", @@ -3045,9 +2714,7 @@ { "cell_type": "markdown", "id": "8edd25f6", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## When do we stop?\n", "\n", @@ -3066,9 +2733,7 @@ { "cell_type": "markdown", "id": "4a05e627", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Slightly different approach\n", "\n", @@ -3089,10 +2754,7 @@ "cell_type": "code", "execution_count": 15, "id": "c016a06b", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "import numpy as np \n", @@ -3124,9 +2786,7 @@ { "cell_type": "markdown", "id": "547b28ad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We note that we have defined several hyperparameters. These are now the number of epochs, the number of mini-batches and the parameters $t_0$ and $t_1$." ] @@ -3134,22 +2794,46 @@ { "cell_type": "markdown", "id": "ca68cd6e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Program for stochastic gradient" ] }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 9, "id": "1890a370", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.16244911]\n", + " [2.83947224]]\n", + "Eigenvalues of Hessian Matrix:[0.29347357 4.43832713]\n", + "theta from own gd\n", + "[[4.16244911]\n", + " [2.83947224]]\n", + "theta from own sdg\n", + "[[4.18064792]\n", + " [2.8068358 ]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAprklEQVR4nO3de5wcVZn/8c8zkwy5EZJMQm4wEyAhgKAIUYhcJl5YgwIu7G8FjArobpRd94cuq6JREJFd3dVV3JXXGi4LkhGvoOKigm7SIZAAAwZIIIEEMrlB7uR+m5ln/6iepDPpS3V3dVd3z/f9evVrZqqqq56qVOqpc06dU+buiIiI5FIXdwAiIlIdlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDKlZZvY1M5sVdxxRMrMpZrY67jikd1LCkLIysxVmttvMdpjZG2Z2j5kNijsuEclNCUPicLG7DwJOB94OfCnecHonM+sTdwxSXZQwJDbu/gbwB4LEAYCZ3WBmy81su5m9aGaXpsy72szmmdm3zWyLmb1mZhemzD/OzBLJ7z4KDE/dnpldYmaLzexNM5tjZienzFthZp83s+fNbKeZ3WVmI83sd8n1/dHMhqbbj+5qIjO73szWm9nrZnZNyvw5ZvY3Pfcj5W83s78zs1eS27rFzE4ws/lmts3MfmZmDT22+WUz25iMe1rK9COSx2elma0zs/8ys/494vyimb0B/LeZDTez3yaPyWYze8zMdF2QtHRiSGzM7BjgQmBZyuTlwHnAUcDNwCwzG50y/yxgKUEy+FfgLjOz5LwfA88k590CXJWyrROB+4HPAiOAh4GHelyI/wq4ADgRuBj4HfDl5PrqgP+fZXdGJWMeC3wS+EGmBJPBVOBM4GzgC8BMYBpwLHAqcGWPbQ1PbusqYKaZTUzO+1Yy/tOB8cllbuzx3WFAMzAduB5YTXBMRib3V+MFSVpKGBKHX5nZdmAVsB64qXuGu//c3de6e5e7/xR4BXhnynfb3f0Od+8E7gVGAyPNrAl4B/BVd9/r7nOBh1K+dznwP+7+qLvvB74N9AfelbLMf7j7OndfAzwGPOnuf3b3vcCDBNVnmewHvu7u+939YWAHMDHL8j19y923uftiYBHwiLu/6u5bCRJXz21372cC+B/gw8nE+bfA59x9s7tvB/4ZuCLle13ATcnv7k7GPRpoTsb+mGuAOclACUPi8JfufiQwBTiJlKojM/u4mS1MVpG8SXB3nVq19Eb3L+6+K/nrIGAMsMXdd6Ys257y+5jUv929iyBhjU1ZZl3K77vT/J2tcX6Tu3ek/L0rx/I95bPtdPs5hqCUMAB4JuX4/T45vdsGd9+T8ve/EZTwHjGzV83shjxill5GCUNik7w7vofgbh8zawbuAD4DNLr7EIK7bcuwilSvA0PNbGDKtKaU39cSVMOQ3JYRVPesKXwPQttJcCHvNqrI9aXbz7XARoLk8hZ3H5L8HJV8wKDbIaUHd9/u7te7+/EE1XD/aGbvLTI+qVFKGBK37wEXmNnpwECCC9oGgGTD8alhVuLu7UAbcLOZNZjZuQQXwG4/Az5oZu81s74Edfd7gSci2o9sFgKXmdkAMxtP0MZRrO79PA+4CPh5stR0B/BdMzsawMzGmtn7M63EzC4ys/HJBLoN6Ex+RA6jhCGxcvcNwI8I6uRfBL4DzCeokjkNeDyP1X2EoFF8M0G7yI9StrMU+CjwHwR34hcTPN67L4LdyOW7wD6CfboXaC1yfW8AWwhKFa3Ap919SXLeFwmqmBaY2Tbgj2RvS5mQXGYHwXG/3d3nFBmf1ChT+5aIiIShEoaIiISihCEiIqEoYYiISChKGCIiEkpVDT42fPhwHzduXNxhiIhUlWeeeWaju4/IvWR2VZUwxo0bR1tbW9xhiIhUFTNrz71UbqqSEhGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJJSSJwwzu9vM1pvZojTz/snM3MyGlzoOEREpTjlKGPcAU3tONLNjgQuAlWWIQUREilTyhOHucwnesdzTd4EvAHpHrIhIFYilDcPMLgHWuPtzIZadbmZtZta2YcOGMkQnIiLplD1hmNkAYAZwY5jl3X2mu09y90kjRhQ9nLuIiBQojhLGCcBxwHNmtgI4BnjWzEbFEIuIiIRU9hcoufsLwNHdfyeTxiR331juWEREJLxyPFZ7PzAfmGhmq83sk6XepoiIRK/kJQx3vzLH/HGljkFERIqnnt4iIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKEoYIiISihKGiIiEooQhIiKhKGGIiEgoShgiIhKKEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKCVPGGZ2t5mtN7NFKdP+zcyWmNnzZvagmQ0pdRwiIlKccpQw7gGm9pj2KHCqu78VeBn4UhniEBGRIpQ8Ybj7XGBzj2mPuHtH8s8FwDGljkNERIpTCW0YnwB+l2mmmU03szYza9uwYUMZwxIRkVSxJgwzmwF0AK2ZlnH3me4+yd0njRgxonzBiYjIIfrEtWEzuwq4CHivu3tccYiISDixJAwzmwp8EWhx911xxCAiIvkpx2O19wPzgYlmttrMPgn8J3Ak8KiZLTSz/yp1HCIiUpySlzDc/co0k+8q9XZFRCRalfCUlIiIVAElDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwRkbi1tsK4cVBXF/xszTiAd6xiG61WREQIksP06bArOQ5re3vwN8C0afHFlYZKGCIicZox42Cy6LZrVzA9XyUuqaiEISISp5Ur85ueSbaSSkRUwhARiVNTU37TM4mypJKBEoaISJxuvRUGDDh02oABwfR8RFVSyUIJQ0QkTtOmwcyZ0NwMZsHPmTPzb/COqqSShRKGiEg25Xjkddo0WLECurqCn4U8HRVVSSULJQwRkUy6G5Lb28H9YENyvkmjXEknipJKFubuka2s1CZNmuRtbW1xhyEivcW4cUGS6Km5OSgJhNHz6SUI7vwjvphnY2bPuPukYtejEoaISCZRNCSX4emlcil5wjCzu81svZktSpk2zMweNbNXkj+HljoOEZG8RdGQHPXTSzEOI1KOEsY9wNQe024A/uTuE4A/Jf8WEaks2RqSs124U+fVZbjMFvL0UlRtKgUqSxuGmY0Dfuvupyb/XgpMcffXzWw0MMfdJ+Zaj9owRKTsWluD6qOVK4OLfPdTR5naJdLN66nQNowC21SiasOIK2G86e5DUuZvcfe01VJmNh2YDtDU1HRme7qDJSJSTtku3JB+Xn198Nhsd9IppMG7ri4oWfRkFqw7g6gSRsWPJeXuM4GZEJQwYg5HRKSwdomurqwX9VCamtInozTVW+sXbyBx1zLm/nFfcdtMEVfCWGdmo1OqpNbHFIeISP5yXbhDXtTzduut6avCbr2Vtc++QeK/XyXxpw4Sy8eyZN8JwAgGsLP47SbF9Vjtb4Crkr9fBfw6pjhERMLrbsxubw+qgVJ1N4aXssd1Suc8N2PHoJHcMfh6JlxzDmPPHMVH/vNd/Pil0znuqM18c+oc5t+xiDd3NhS/3aSSlzDM7H5gCjDczFYDNwHfBH5mZp8EVgJ/Xeo4REQK1toK110HmzYdnOYeJA33oO2iZ7tEz4byIjrpeZfz2txVJH7UTmJuM4k1j7HCj4UdMGTnm5x39FI+/Y4VtPz10Zz+4RPp0+8dRexsZurpLSKSTbqe2qny6fUdknc5L//hNRKtq0nMq2fu6uNY3TkGgEbbxPmjXqHl7L20XD6K0y4dT31Dfdb19ZpGbxGRWKXrqZ0qguHDvct58aHlJO5fQ+KJvsxdcwJvdB0PHM/IuvW0jF3G+We/QsuVYzjl4hOo63N20dsshBKGiEg2uRJCAY3ZXR1dvPDAKyR++gaJBUcw9/XxbPTxwHjG1r3Oe45dRsu5S2mZdgwnvv84rO7owmKPmBKGiEg2mZ6IgtCN2Z37Oln4s5dJ/Hwdiaf689i6E9niE4GJNNev5gPHLaHlfKflY00cP6UJqxsd7T5ERIMPikh1K/XYSumeegJobMzYW3v/rv08edci/vUDc/jg0U8z7IgdTPrYyVz/myks3jiKS8cv4t6/nceKeatZ0XEM9y4/l0/893mc8J5mrM4O31aq1P0dPjz4lGtcKXevms+ZZ57pIiIHzJrlPmCAe/CsUvAZMCCYnm7Z5mZ3s+BnumWybSfdd5PTu8x899DR/sBbZvgFw9p8INsPhDOxYblPPynhrX83z1c/vTb6/Q2x70CbR3AN1lNSIlK9wo6tFPE7Kfa8uYdXP/PvjL//6zR07T0wfScD+Hr919lx8iRa3teX868Zz6i3Rtj+kGl/U6V5aquqxpKKihKGiBwi7NhKRb4Iaef6ncy/ZymJh7aReH4oT247iaVMZBxFvlwpX5n2N1WacaX0AiURqQ3FtEGEfV9FnmM/bV+7nd9/o40vTZ7Du458gSEjG7jgi2fwz/POY3dHX/7hzPk0E/F7LsII80RWFEOQZKCEIVLpYnxhTskV+36HsMNw5Egsb7Zv5bc3PsXn3zGHdw5azNCx/bnwq5P49oJzcOD6sx7n4ZufZsuqnTy98xS+3TYFa47g5Ur5ytQA3y2qIUgyiaIhpFwfNXpLr5NPo2667xbayFsuzc3pG2+bm8OvI8x+pjmO+/v083ubZvjp/V9yo9PBvYE9ft7ghT7jnNn+yL+0+Y51O7Jvt9B/m2Kk7m9jY/DJ8W9MRI3esSeBfD5KGNLrFHpBjetili+z9PtnFulm3nhhvc+fepNv7DvSOzF/jWa/klnej13+7iHP+temzPbZ3/2z79q0K78Vp0tWFZioo0oYavQWqWTZGjmz/d8tspG3bKKIM80b8dae/N40Q33DAHZyzrAltJyxnZZLh/GOj07kiMFHRLY7UT+NFZWoGr1jLzXk81EJQ3qdTCUMs0PvXHve1WZ6Tj/iO/e89axOGTjw8BjzKQmlKUntpL9fySwH9yPZ6heOeMq/OXW2z7/jBd+3c19Jdy+SKrYSQFVSIr3ArFmZq226L0Lpqp9yfSfXNqOqUumZIBoaMiczCJbJsb2uzi5f9qcVftfVc31Dn5Fp1/Nm/5H+9L2Lff/u/YXHXogyVbHlK6qEoSopkUrX80U9qdO7ujJX63S/q6FbmKqRKKtUcg0Lnk6aqqieQ30nVh3Pmq5grKVO6qgjRD+McqnQqkD1wxDpLZqb00/vfnwz03P/nnyxj1nwM8xFP91Q3rt2BdPzlWtY8HRWrsS7nMW/XsbtVyS4vOkJxvRdz0kfOJ5PtZ7Pn1ZN4Jyxr/GDyxMs+tUyrHFY+vUU+mhrsY8wl/Jte5UgimJKuT6qkpJeKdcTT1HWmxdapZKuGivTurJ81tWP8uG24cCksXVr/T8bb/RtA0Z6l5l3NTUdOo5T376Hr6e+PtSjpnkf57Bq+Cmp2JNAPh8lDKlp2S40ueZlu9DluoClzq+vzz/5ZNp+Y2NeyWIHA/wf7Pv+8eMf87uunuvL/rTCu+7Lsm/ZGvcLuehXaIN1FJQwRGpJsXe32UZTzZVMso1+GiaOTBfaxsbDO8th3tVjuS7w3UcM9vU3fj/8urv3NWxCCnPRr9AG6ygoYYjUklLd3eZab6b59fXhq1QyXGi7MP/ngbd4O8ce6Cy3iaH57We2i3jYEkbYi34pSxgxV1NFlTBibfQ2s8+Z2WIzW2Rm95tZvzjjEYlNnoPjRbbeTPO7uoLPihU5G8q7hgxNO30jw/hxx+X862n38cvPPU7/F55mmL2ZfiXt7ekbmLONAZWugTnTE2VhGsFL1WBd7HhZlSSKrFPIBxgLvAb0T/79M+DqbN9RCUNqVqWVMLJsd8e6Hf7ot57xr5w7288bvNC3kqbzHXjnkGHh48lU9ZVv+8y115amaq8YFdA2QrVXSSUTxipgGMG7xX8L/EW27yhhSElUwlMtpRr7qZA2jB7b3bpqqz9881N+w9mzffKg570P+xzc6+jwGX2/dVibRNZqoFxtJukuovn++xQwOF9JVUDbSNUnjGAfuA7YAWwAWjMsMx1oA9qampoiPYgiFTVIX6kSVz5PSTU3+47v3eG/+cqTfv2Zs33SgMVeR4eDex/2+eRBz/sNZ8/2h29+yreu2pq9xJDpDnrWrMzfifIiWin/tjVUwoitp7eZDQV+CVwOvAn8HPiFu8/K9B319JbIVWjP3HLa9Mpm5t75MolH9pJYOpLndp+IU0cDezlr8BJa3rqFlosHM/nqiQw8euChX842OOKsWZnbP8px3Cvl37YCBiSMqqd3nyiCKdD7gNfcfQOAmT0AvAvImDBEIpep0TfXe5Or2LpFG5h79zISf9xH4uUxLNo7ATibfuxm8pAl3HTWXFo+NISzPj6R/sPeln1lTU3pj1VjY/aL4a23pr+IRtkjulQPEuSr+zj0GFE3ztFrCxZFMaWQD3AWsBgYABhwL/AP2b6jNgyJXNjRYKvY6qfXeuvfzfNPnZzwkxqWHaydYYdfMKzNv/G+2f7YD57zPVv35L/ySn7BU7b+IZXQZlXGGKiRNoybgSXAIuA+4Ihsyyth9ALpnnop5j9WmPr7YkZ2rUAr5q3ye//2Mf/EhLl+Qp8VB3anZEN9V8JDA+mkS2Z9+x4+Ym652zViaFupiYSR70cJo4Qq4T99FL2Oc60v3ffL0QBbIqlDfX/8+Me8uX7VgfCH2ma/ZNQC/84ls+MZ6jsf5WrwzzRcSTluDnINZ1LCGMqWMIA/Am+LYmPFfpQwSqTSnyYp9D9WrqdTcv0HroSqix66Ort8ycPL/YfTEv6R5nk+tm7tgXCH2wa/bMx8v+2yOb7wp0u8c39n3OGGU87zL65HXMPcDJUwhqgSRs6npMzsDODbQDvwZXd/Pep2lLD0lFSJVMrTJNmeuEkV9l0HmdZnBvfdl/1dDX37Bsvt23dwWgyv2vQu58WHlpO4fw2JJ/oyd80JvNE1EoCRdetpGbuMlnftp+XKsZz8weOp61OFbywo5/kX17meabtliqHsr2gF/gp4DriJZO/scn9UwiiRuDsW5TPyaFQljFxVAzFVXXTu7/Q//2SJf+/SOX7ZmPmHDfX9keZ5/sNpCV/y8HLv6uw6fAWVULWYr3Kef3GVpnMNlFhLbRgETzGdCnwa2AisBj4WRQD5fJQwSiTOjkVhiuqF/sfKdnHIdZEq00Vs/+79/vS9i/3bF832i0cu8CG25eDhr1916FDf6RJE2P2tZOU+/+JIqrluUGrlKSlgHrAWeBS4BbgIGA/8BzAziiDCfpQwSqSUw1Lk+o+Z6z9SqZ6SKsEYS2Hs27nP59/xgn9z6my/cMRTfiRbD6x6fN/X/JMnzvUffeoxXzFvVf4rr4AexQWp1kSXj5j3sZwJ41QI2jrSzHspiiDCfpQwChTmwh31XVfY/yBR3MkXEnu6+LpjiWIQu6Q9W/f4Yz94zr/xvtl+wbA2H8j2A6s7qWGZf+rkhP/4M4/76qfX5rXetOKuWixGNVal5SvGfSxrlVTGL8PxUQQR9qOEUYC47mzC3u0We1ccRcex1GSRuo6epZsQpZ1dm3b57O/+2b82Zba/e8iz3o9dB1Z56hEv+9+fNsfnT73JO0YfE/2Fo1pLGFJyFZEwyv1RwihAXBeRMHe7s2alb1zOJ6FFsX9hGrgzJKbdt9/lj/xLm884Z7afO3ihN7An2E06/fT+L/l1p8/xB74w3zcs2Zh1PZEkjd5QtSMFUcKQcOKqpgjTByJdY3djY34XuGL3L+zIqRn2ZwVNDu717Pd3DFzk/zRptj/01Sd986tbCjsuxeoNVTuSNyUMCSeuEkauu92o4ip2PTka3beseNN/85UnvYvMryH93S1P+7Y128Jtr5rbGaRqRZUwqrCXj+SlVK+dzGXatKCTW3Nz0AGuufnQTm9RjSRa7P5l2J4DX1r79wwbdySXfOOdrOTYtMtZcxNTvzKJI8ccGW572V45KlLhlDBqXa4Ld6m3vWJF+ndDR3XhLHL/OkeNTTt9I8N4cuD7uGnKXGZ/dyGjf/i1aBJvXAlcJApRFFPK9VGVVA2JqYG2e6jv6SclfGLDcr+SWb6DQ+Po6NvP991xT/qYo2gfUDuDlBnV/sa9QmgsqSxaW6vvBS1liLn98dUk7l1BYo6TeK2J5R3NAAxmK+eOeJmWSTu5bOyTnPCH27HVq6rn2InkoexjSVXCRyWMDGrhccoI7rrDDvXddt+L3rG3I+o9qCwqxUgK9JRUL5XuQlDtHbZyJbwMF7+aHOo7Ctdem74jopJGr6WE0RtlurCG6UdQybIlvDT7vK/+CL9t2E0+sm7dgckj69b5h4993H9w+Rxf9KtXvOu+LHfYtXz3XYNvEJTiRZUw1IZRTTKNqV9fD52dh08v9/ssCpXhvRVuxv7G0TRsXHvYvNWM5YZxP+X8czppmXYMJ77/OKzOgpmtrYe/66L7XRaQeV4ttFtke+9C2PeISM1RG0ZvlG1M/XK1YZTi7jxDCaOdY7wzU4e5bKWnQt6Dke7uuxpLItnOEZUwei3Uca8XytRHobvvQan7WnTfube3B5eg9vbg79bWgla3f9d+Fty5iN8c9VH20O+Qebvozx9GXcOuQUen/a4NGxbcTdfVBT9TY8jWKTBsh8GI97VsMp0jZurrIcWLIusU+gGGAL8AlgAvAZOzLd/rSxhxPw1VZON6tqG+r6//d9/YZ6R3Yb5/1NhDG7x77nPfvu4NDZmPQxQljGp9kCDTsO3XXht3ZBIjaqHRG7gX+Jvk7w3AkGzL98qE0bNapNgXChUjz3GQdm3a5f/7nWf9ppbZPqXHUN+n9Vvqf3/aHP/Z557wN15Yn327PY9BrtFlsyXWcr6nIy7VWJUmJVX1CQMYDLxGhpczpfv0uoQRVYkiqgtIjrvuHet2ZBzq++39X/TPvn2OP3jDAt/48qbCtt8t7NDpxTwlVa0ljFJREqpqtZAwTgeeAu4B/gzcCQxMs9x0oA1oa2pqivgwVrgoLlpRVmOlWdf+vv38/hNm+NmDnvc+7HM4fKjvLSvezH9b2ZTjYh539V8l0bGoerWQMCYBHcBZyb9vA27J9p1eV8KIolokwovr5le3eNtf3uybGkZ6J+YraPIrmeV92OeTBz3vN5w9O7+hvgtVrguY7qoDKm1VvagSRp9SNaaHsBpY7e5PJv/+BXBDjPFUnqam9M/U5zOiaxHDiG9cuom5d71C4pG9JJaO4vk9E3BupIEvctbgJbS8dQufuHgwd1y9j4FHnxY+pij073+wL0VjI9x2W/RPhU2bVht9M4oV1VD0UvViSxju/oaZrTKzie6+FHgv8GJc8RSk1IPn3Xpr+k5m+TwemUfSWbdoA4m7XiHxx/0kXhnD4r0TgEb6sZvJQ5Zw09lzafnQEM76+ET6D3tb/vsThXSd8nbvjieW3iKKGxepDVEUUwr9ELRjtAHPA78ChmZbvqKqpKqlWiRLnD2H+u6ePZDt/heNT/utF8z2ebc/53u27ol2n4qh6pHyUxtG1aPa2zAK+VRUwqikC1eupJKc32Xm2weN9B+O/Kqf0GfFgZAH86Z/YMRT/q0LZ/uCO1/wfTv3lX8fwqrmx12rmdpzqlpUCUNjSRUqw/hHZR+vJ8O4Sf7Dmbw65lwS960kMddItI+jvfMYAIbaFs4b+TIt79xNy1+P5PQPn0h9Q335Yi5GprGSqmXcLJEYRDWWVJyN3tWtUup1Z8w4NFkA7NrFmo99kfGsBpoZbhs5f/Qy/vHs5bRcPorTLptAXZ+zyhtnVKJo1xGRgihhFCrmC1dXRxcvPrSct7SvxNLMH8Nabr9iLudfMYZTLj4BqxtelrhKrvuhgmp7u6BIDdDgg4WaNq08A/4ldXV0sfCnS7ntsgSXjVnA0Q1bOO2yCbSTvkRT19zEtfefz1s+NB67/8eZB+qrRtOmBUmiqSlIGjNmVP8+iVSDKBpCyvWpqEbvEtu/e78/fe9i//ZFs/3ikQt8iG050L47rs9Kv+qEx/zua+b6GzNu865cb6urtSdcotgnNeJKL4IavWvL/l37aZu1hMQDm5j77EDmbZjIdgYDML7vClqOW0XLu42Wq46jafLYg19sbYXrroNNm4K/e3Ziq8VG4mL3KdsLllS1JTVIjd7FKHWHuxD2btvLUz9aQuJXW0j8+Uie2HwSuwh6S5/csJxppyzk/Pf0oeWa4xlzxjhg3OErCdOJrRZ76Ra7TxkeFGDGDCUMkSx6XwkjprvL3Zt3s+CeJSR+s5XEc0ex4M2T2EN/AE7r9zItE16n5YIGzv/EeI5+y4hwKw1zp60SxuEq5ZFokTKJqoTR+xJGmS6gO9fv5Im7l5D47XYSLwzlqW0nsY8jMLo4vf9SWk5aT8v7+3HeJybQOGFYYRvJdOGDg9Nrsfql2H2qxSQqkoXe6Z2v7kbOdL2EI+gpvHXVVn/45qf8i2fNzm+o72IaXzPtj1n+73+oNsXsUy0+CCCSBRoaJA/pLhBFDumx+dUt/usvL/B/PHO2nzlgsdfR4eD5DfVd7IVr1qzMQ2VobKXsajGJimQQVcLoHVVSmaogUuUYIjv9UN91NLD3wFDfLRcPZvLVExl49MDi4sqnasTSddujPPXxFfDwgIjkpqek8hHm6ZlNm4J68aTOz99A3etr2Nz3aL7V9Xn+rfN6oJH+7GLy0CV8bXJyqO+rTqLfkAKH+o7iCabm5niGKOnZjtDefvD4KWmI1KYoiinl+hRcJZWt7aLHZ1vdUb6L/odM200/f/DUGT7v9ud87/a9+W8/U/VHnK9gLbZKppJG6xWRrFAbRh7SXFS7MiSMTNMLvhBmu6BH1fia78U/iu1qmHGRqhFVwugVbRje5az7yvcZ+P1/YdDO9axhDEewhxFsOnxZSDuYXzCzgGOVq50ijnaAKNpO9GiqSNVQP4wsvMtZ+vvXSLSuJvF4PXNXHc+artEADLeNtIx5hb8Z8z/8xXPfoW7fnoNfHDAg6Cmd7pjU10NHR/5BV2InsShiqsX+HSI1KqqEUROj1XZ1dLHowVd45KyvsKHPKLy+nn4ffA+JH69i9qrxnDv2NW6/Yi6Lf72M9R2N/GL1ZKY+9Q3q7r7z8NFmMyXQzs7CgsvU+Bzn+5CjiKnMo/WKSAWIol6rXJ/uNoyOvR3+558s8e9dOscvHT3fG22jX8ks38Gh9fKdR/T3rvvy6NOQrXG8FG0YcanEmESkZOiNjd7HHHVixqG+tw8cWfiFPlfHvmIvppXYSawSYxKRkogqYcTehmFm9UAbsMbdL8q+7CSf0PfntBy/ipZ313P+x8cdHOq7mHr5bB37GhuDn5s3q3OaiFSlWuq4dx3wEiRf/pDFW0/ez3MvHgccd/jMYt6xna2j3O7d6pwmIkLMjd5mdgzwQeDOMMv3HdA388xbbw2e0kkV9h3bmZJKXV3m9yaIiPQycT8l9T3gC0DGOiMzm25mbWbWtmHDhsxrKuapnXTJpqEhc1VWNb98SESkQLElDDO7CFjv7s9kW87dZ7r7JHefNGJEjhcLTZsWdBrr6gp+hq02Spdsjjwy8/JxPhIrIhKTOEsY5wCXmNkK4CfAe8xsVmzR9Ew2mzdnXja1mqu1NWg0N4M+fYKf48YF00VEakhsCcPdv+Tux7j7OOAK4H/d/aMl22D3hb2uLtwFPVMporHxYMmlu7dzd2N7d+e+7sZxJQ0RqSFxt2GUR+qF3T3cBT1TI/pttx38e8aMwxvFu6lxXERqTOz9MPIR+QuUcg2Ul2tgwGzv1IZ4x4sSEUmqpX4YpVfoi4qmTcvecJ6p70fqfBGRGtE7qqRKNQBgumqrbmH7gIiIVInekTCK6dSXTerjuBAMgQ4auVVEalLvqJLqvnCX4kVFuaqtRERqRO9IGKALu4hIkXpHlZSIiBRNCUNEREKpvYSRb49uEREJpbbaMLp7dOv9FSIikautEka6oTo0RIeISCRqK2EU2qNbRERyqq2EUaoe3SIiUmMJo1Q9ukVEpMYSRjGvaRURkaxq6ykpUI9uEZESqa0ShoiIlIwShoiIhKKEISIioShhiIhIKEoYIiISSmwJw8yONbPZZvaSmS02s+viikVERHKL87HaDuB6d3/WzI4EnjGzR939xRhjEhGRDGIrYbj76+7+bPL37cBLwNi44hERkewqog3DzMYBbweeTDNvupm1mVnbhg0byh6biIgEYk8YZjYI+CXwWXff1nO+u89090nuPmnEiBHlD1BERICYE4aZ9SVIFq3u/kCcsYiISHZxPiVlwF3AS+7+73HFISIi4cRZwjgH+BjwHjNbmPx8IMZ4REQki9geq3X3eYDFtX0REclP7I3eIiJSHZQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwREQkFCUMEREJRQlDRERCUcIQEZFQlDBERCQUJQwREQlFCUNEREKJNWGY2VQzW2pmy8zshjhjERGR7GJLGGZWD/wAuBA4BbjSzE6JKx4REckuzhLGO4Fl7v6qu+8DfgJ8KMZ4REQkiz4xbnsssCrl79XAWT0XMrPpwPTkn3vNbFEZYivWcGBj3EGEoDijUw0xguKMWrXEOTGKlcSZMCzNND9sgvtMYCaAmbW5+6RSB1YsxRmtaoizGmIExRm1aoozivXEWSW1Gjg25e9jgLUxxSIiIjnEmTCeBiaY2XFm1gBcAfwmxnhERCSL2Kqk3L3DzD4D/AGoB+5298U5vjaz9JFFQnFGqxrirIYYQXFGrVfFae6HNRuIiIgcRj29RUQkFCUMEREJpSISRq4hQizw/eT8583sjLDfLXOc05LxPW9mT5jZ21LmrTCzF8xsYVSPuBUR5xQz25qMZaGZ3Rj2u2WO8/MpMS4ys04zG5acV5bjaWZ3m9n6TP1/KujczBVnpZybueKslHMzV5yVcG4ea2azzewlM1tsZtelWSba89PdY/0QNHgvB44HGoDngFN6LPMB4HcEfTfOBp4M+90yx/kuYGjy9wu740z+vQIYXiHHcwrw20K+W844eyx/MfC/MRzP84EzgEUZ5sd+boaMM/ZzM2ScsZ+bYeKskHNzNHBG8vcjgZdLfe2shBJGmCFCPgT8yAMLgCFmNjrkd8sWp7s/4e5bkn8uIOhbUm7FHJOKOp49XAncX6JYMnL3ucDmLItUwrmZM84KOTfDHM9MKup49hDXufm6uz+b/H078BLBCBqpIj0/KyFhpBsipOdOZ1omzHejku+2PkmQ2bs58IiZPWPBcCelEjbOyWb2nJn9zszekud3oxB6W2Y2AJgK/DJlcrmOZy6VcG7mK65zM6y4z83QKuXcNLNxwNuBJ3vMivT8jHNokG5hhgjJtEyo4UUiEnpbZvZugv+U56ZMPsfd15rZ0cCjZrYkeRcTR5zPAs3uvsPMPgD8CpgQ8rtRyWdbFwOPu3vqHV+5jmculXBuhhbzuRlGJZyb+Yj93DSzQQQJ67Puvq3n7DRfKfj8rIQSRpghQjItU87hRUJty8zeCtwJfMjdN3VPd/e1yZ/rgQcJioSxxOnu29x9R/L3h4G+ZjY8zHfLGWeKK+hR5C/j8cylEs7NUCrg3MypQs7NfMR6bppZX4Jk0eruD6RZJNrzs9QNMyEabvoArwLHcbDx5S09lvkghzbcPBX2u2WOswlYBryrx/SBwJEpvz8BTI0xzlEc7LT5TmBl8thW1PFMLncUQV3ywDiOZ3Ib48jcSBv7uRkyztjPzZBxxn5uhomzEs7N5HH5EfC9LMtEen7GXiXlGYYIMbNPJ+f/F/AwQWv/MmAXcE2278YY541AI3C7mQF0eDCS5UjgweS0PsCP3f33Mcb5/4BrzawD2A1c4cFZVGnHE+BS4BF335ny9bIdTzO7n+DJneFmthq4CeibEmPs52bIOGM/N0PGGfu5GTJOiPncBM4BPga8YGYLk9O+THBzUJLzU0ODiIhIKJXQhiEiIlVACUNEREJRwhARkVCUMEREJBQlDBERCUUJQ0REQlHCEBGRUJQwRIqQfB/BBcnfv2Fm3487JpFSib2nt0iVuwn4enKgubcDl8Qcj0jJqKe3SJHMLAEMAqZ48F4CkZqkKimRIpjZaQRvPturZCG1TglDpEDJN5e1ErypbKeZvT/mkERKSglDpADJN609AFzv7i8BtwBfizUokRJTG4aIiISiEoaIiISihCEiIqEoYYiISChKGCIiEooShoiIhKKEISIioShhiIhIKP8H4/QWs8CWcW4AAAAASUVORK5CYII=\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Importing various packages\n", "# Importing various packages\n", @@ -3225,9 +2909,7 @@ { "cell_type": "markdown", "id": "49371ea1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Replace or not\n", "\n", @@ -3240,9 +2922,7 @@ { "cell_type": "markdown", "id": "6f44130c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Momentum based GD\n", "\n", @@ -3255,9 +2935,7 @@ { "cell_type": "markdown", "id": "3dfe1344", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n", @@ -3267,9 +2945,7 @@ { "cell_type": "markdown", "id": "7438ccdb", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3285,9 +2961,7 @@ { "cell_type": "markdown", "id": "f6c3290d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have introduced a momentum parameter $\\gamma$, with\n", "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n", @@ -3304,9 +2978,7 @@ { "cell_type": "markdown", "id": "3e3bf9cd", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n", @@ -3316,9 +2988,7 @@ { "cell_type": "markdown", "id": "30fb1856", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$." ] @@ -3326,9 +2996,7 @@ { "cell_type": "markdown", "id": "59802bd4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More on momentum based approaches\n", "\n", @@ -3342,9 +3010,7 @@ { "cell_type": "markdown", "id": "4a93e7c9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n", @@ -3354,9 +3020,7 @@ { "cell_type": "markdown", "id": "2bd5f401", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "We can discretize this equation in the usual way to get" ] @@ -3364,9 +3028,7 @@ { "cell_type": "markdown", "id": "8a70082f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n", @@ -3376,9 +3038,7 @@ { "cell_type": "markdown", "id": "18808d48", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Rearranging this equation, we can rewrite this as" ] @@ -3386,9 +3046,7 @@ { "cell_type": "markdown", "id": "26406ac8", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n", @@ -3398,9 +3056,7 @@ { "cell_type": "markdown", "id": "c462f769", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Momentum parameter\n", "\n", @@ -3414,9 +3070,7 @@ { "cell_type": "markdown", "id": "6bf678e7", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n", @@ -3426,9 +3080,7 @@ { "cell_type": "markdown", "id": "4cdbd09d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Thus, as the name suggests, the momentum parameter is proportional to\n", "the mass of the particle and effectively provides inertia.\n", @@ -3459,9 +3111,7 @@ { "cell_type": "markdown", "id": "5d252515", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n", @@ -3471,9 +3121,7 @@ { "cell_type": "markdown", "id": "596d58a2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3489,9 +3137,7 @@ { "cell_type": "markdown", "id": "50e01c2e", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$." ] @@ -3499,9 +3145,7 @@ { "cell_type": "markdown", "id": "386f9167", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Second moment of the gradient\n", "\n", @@ -3530,9 +3174,7 @@ { "cell_type": "markdown", "id": "709b2114", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## RMS prop\n", "\n", @@ -3545,9 +3187,7 @@ { "cell_type": "markdown", "id": "116ee962", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3563,9 +3203,7 @@ { "cell_type": "markdown", "id": "fe810dcf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n", @@ -3575,9 +3213,7 @@ { "cell_type": "markdown", "id": "cd81ad67", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n", @@ -3587,9 +3223,7 @@ { "cell_type": "markdown", "id": "06498abc", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\beta$ controls the averaging time of the second moment and is\n", "typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n", @@ -3605,9 +3239,7 @@ { "cell_type": "markdown", "id": "8a3c0446", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## ADAM optimizer\n", "\n", @@ -3628,9 +3260,7 @@ { "cell_type": "markdown", "id": "82aeaa28", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3646,9 +3276,7 @@ { "cell_type": "markdown", "id": "8c71ca82", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n", @@ -3658,9 +3286,7 @@ { "cell_type": "markdown", "id": "f0b27db1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n", @@ -3670,9 +3296,7 @@ { "cell_type": "markdown", "id": "5e3da9e9", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n", @@ -3682,9 +3306,7 @@ { "cell_type": "markdown", "id": "6c5c06b4", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n", @@ -3694,9 +3316,7 @@ { "cell_type": "markdown", "id": "3b247467", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n", @@ -3706,9 +3326,7 @@ { "cell_type": "markdown", "id": "22079281", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "\n", "
\n", @@ -3723,9 +3341,7 @@ { "cell_type": "markdown", "id": "a24eece5", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n", "second moment and are typically taken to be $0.9$ and $0.99$\n", @@ -3742,9 +3358,7 @@ { "cell_type": "markdown", "id": "634c6f26", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n", @@ -3754,9 +3368,7 @@ { "cell_type": "markdown", "id": "71ef4fbf", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Practical tips\n", "\n", @@ -3774,9 +3386,7 @@ { "cell_type": "markdown", "id": "40720b38", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Automatic differentiation\n", "\n", @@ -3812,9 +3422,7 @@ { "cell_type": "markdown", "id": "5c07021c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f(x) = \\sin\\left(2\\pi x + x^2\\right)\n", @@ -3824,9 +3432,7 @@ { "cell_type": "markdown", "id": "ae7a6c62", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "which has the following derivative" ] @@ -3834,9 +3440,7 @@ { "cell_type": "markdown", "id": "5b9164e2", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "$$\n", "f'(x) = \\cos\\left(2\\pi x + x^2\\right)\\left(2\\pi + 2x\\right)\n", @@ -3846,22 +3450,37 @@ { "cell_type": "markdown", "id": "d56054c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Using **autograd** we have" ] }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 10, "id": "15d3d863", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAdwAAAEWCAYAAADM0CYnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABDhklEQVR4nO3dd3hUZdrH8e+dRggJIZDQO9IhBAgCFrBgwQIqFhQFK7ouln2ti2vbteuuXVGxN1QQBVFRVLDREgih9xZCCYFAKCEked4/zgHHmE4yz5T7c125MjPnzJnfeebMc582c8QYg1JKKaVqVojtAEoppVQw0IKrlFJKeYEWXKWUUsoLtOAqpZRSXqAFVymllPICLbhKKaWUF1gpuCJysoisrKFpjxWR8TUx7UAnIu+IyCNlDH9ERHaKyDZv5gpEInKKiGTYzmGTiBgROe4Yp7FURE4pY/hMEbn+GKb/kIh8UNXne8uxtKWIfCMio47x9a8WkV8rMf4GERlUxddqKSL7RCS0Ks8vZ9ojROS76p7uEeUWXLdhDopIrojkiMjvInKTiFS5WBtjfjHGdKzq8z2y/aXTMsY8Zoyp8gfMn1VHB1bGtFsAdwBdjDGNa+I1Kpjjanc+L63k88pcmVD+yRjT1RgzE469OAbDSlBJbWSMGWyMeddWpsoyxmwyxkQbYwqPZToi0trtS8I8pv2hMebMY09ZsooWzfONMTFAK+AJ4B7gzaq8oOfMKb/SCsg2xuwoaaAX39dRwC73v0/QZVrbQHmH3y9nxpgy/4ANwKBijx0PFAHd3Pu1gGeATcB2YBxQ2x12CpCBU6S3Ae8fecwdfi8wsdj0nwdecG9fAywHcoF1wI3u43WAg26Ofe5fU+Ah4AN3nG+BMcWmvQi4yL3dCfgepwNfCVxaRjvUB94GMoHdwBcew24A1rjTmQI09RhmgJuB1e48/AdoB8wG9gKfAhHF2mossNNt+xEe05oJXO9x/2rgV/f2z+5r7Xfb4jL38fOANCAH+B1I9Hh+T2CBm+sTYALwSAnzPqhYW78DtHZf7zr3ff8ZZwXuX8BGYAfwHhDrTuPI+NcAm902vAnoA6S7+V4qZ1ls5WYYBhQAjUpqi2JtfxwwGjgM5Lv5p7rDO7ttmgMsBYZ4PLcBMNV9j+YDj3hO35323933db3HcrvZfU4qcLLH+LXddtsNLAPuwv0MlDKvXflj2dwOjPX4rD2HsxxmurdrFVt+7nbbfytwAXAOsMqd1liP13gImOi+97nustDDY/i9wFp32DLgwmLt/RvwrDvdRyijH3Cfc5ebKRO49sj7U8K8nwos9rg/A5jncf9X4ALP/gk4231/D7vv8SKPz8x/3Ky5wHdAfAmvWVZ/8inOspzrLifJHs9rCkwCsoD1wK1lvKfnAgvd5WMz8JDHsNZue4xy228ncF+xPnc2zrK6FXgJt98otqz3cds+zGPYMJw+oKw28uxXbuCPPncZ0KuCy8OvZcz7VTj9QjZwHx51BaffODLtbLe96xdrF89+5shjYcBwIKXYa/0DmFKBNt/kTufI+92fP/ep44Bnik37S+D/KvveH31+uSOUUHA9wv7Nvf0cTqGpD8TgdFSPe3QCBcCTOB/I2vy54LYCDgB13fuh7gLVz6PB2gECDHTH7eUx7YxiuR7ij4I7EvjNY1gXnAW2Fs4HbDNOAQgDeuEs5F1LaYdpOB1THBAODHQfP819Xi93ui8CPxf7IEwB6uJ0ooeAH4C2QCzOgjuqWFv9z53WQJwC2rGUD8bRhcPzQ+dxvxdOx9vXbddR7vtZC4jA+QD8w52fi3E+iH8puCW1NX8s9O+5bVkbpxNd485bNPA58H6x8ccBkcCZQB7wBdAQaOZmHVjGsng/bscLLMZd8Ev7wHu2B06xe8RjWLibdazbFqfhdCRH2nqC+xeFs9xsLqGtv8dZ5o+sXF6JU6jDcHa/bwMi3WFPAL+447cAllBKwcX5DG11pxHp3u/rDvs3MMdtswSclaj/FFt+HnDn7waczuAjdxpd3TZv6/FZOey+9+HAnTgdR7g7/BKcTiUEuAxnWWzi0d4FwC3u/Nam7H7gbJxC0A1nefmI0gtuJE7xi3envQ2nSMe4r3MQaFC8f8Ljs+8xrZk4HXkH97kzgScqsox7TDMPZ6UlFHgcmONRKFLd9o7AWe7XAWeVMf3u7vMS3fY4suLQ2m2PN9ycPXD6is7u8N5AP7c9WuMUxNtLWdaXAYM9hk0G7iinja73eM+34BRuwSnirSq4PJRYcHE+P/uAATh9z/9wlp0j79vtOMt0c3f4a8DHZfQzRx4Lw/l85gLtPV5vPjC8Em3uuXJydD7cvJsBce/H4Sx7R9qgwu/90emXNbD4Al3s8Tk4ayriNnw7j2H9+WOt/xSctarIMjrvX4GR7u0zgLVl5PkCuK2cD8iRghvjZjuywDwKvOXevgz4pdhzXwMeLOE1m+Cs+caVMOxN4CmP+9E4nVhrjw/CiR7DU4F7PO7/F3iuWIdZx2P4p8D9xT8YJS3k/LXgvorbGXs8thKnkA/A6cTEY9jvVL7gtvV47AfgZo/7Hd22CPMYv5nH8GzcLXH3/iQ8OpESMqw+Mhz4J+4aemkfeMouuCfjdOQhHo997C4/oW7ujh7DStrCPa2cz85u3C1GnA/j2R7DRlN6wb0cWFjKsLXAOR73zwI2eLxHB4FQj+Xf4BZrj+XvSIfzEG7xcO+H4BT6k0t57TRgqEd7b/IYVl4/8BYehQ6nAJZYcN3hvwAX4RSZ73A+B2fjbP2me4y3gfIL7r887t8MfFuRZdxjmjM87ncBDrq3+3q2gcdy+XZZy4XHuM8Bzxb7PDX3GD4Pt3CU8NzbgcmlLOv3AB+6t+vjbKQ08ZifsgrudNz+tQL5iy8PpRXcB4AJHvfr4NSEI+/bcuB0j+FN+Gu/4dnPHHkszL3/AfCAe7s9TgGOqkSbl1ZwBWfDcoB7/wbgx2N574/lLOVmOLuSEnDWMlLdk6pycHblJniMm2WMyStjWh/hdDIAV7j3ARCRwSIyR0R2udM+B2fNt1zGmFycLdPh7kPDgQ/d262Avkcyu9MeAZR0QlALYJcxZncJw5ribCkeec19OIWkmcc42z1uHyzhfrTH/d3GmP0e9ze6r1EVrYA7is1jC3d6TYEtxl1SPF6rsjZ73P5TW7i3w4BGHo9Vpi2OEpETgTY4W53gLCPdRSSpCpmPZN1sjCkqlrcZzrIbxp/nzfN2iY+JyB0islxE9rhtHcsfy2rTYuOX1dYtcAprabmLt7Hn8pFt/jiZ5KD7v6w2PprJbYuMI9MTkZEikuax7HTjz589z/kprx+ozPwDzMIpgAPc2zNxVhQHuvcrw/Os+gOUsoxV4vmR7rHEVkDTYp+vsfx5eT9KRPqKyE8ikiUie3AOqRTvy0rMKiIdROQrEdkmInuBx0p47hEfAOeLSDRwKc6GxdYKzmupy14FlofS/Om9d/u3bI/hrYDJHtNdDhTy53Ys6fN3RPH68YUx5oCbuSJtXiK3b5xQbNqe9aPC7/0RVSq4ItIHp2P6FWd36kGcXbH13L9YY4znQm1Kmo6Hz4BTRKQ5cCFuwRWRWjhbPc/gHK+rB3yNs+ZRkemCs9VyuYj0x9kd8ZP7+GZglkfmesY58+1vJUxjM1BfROqVMCwTp/FxM9fB2a24pQLZShLnTuOIlu5rgLMFEeUxrLyzhTcDjxabxyhjzMc4WzLNREQ8xm9Zhbye78Gf2sKdXgF/7vCrahTO+57mfi1prvv4SPf/n9pGRIq3TfFlJRNoUexs+5Y471uWm7u5x7AWJWQ6Ok0RORlny+JSnD0h9YA9/LGsbi02jbLaejPOYZSSlNTGmaWMWxFHM7lt0RzIFJFWOLs3x+Dsvq2Hsxvcc3nxbNPy+oHKzD/8teDOovyCW5H+oCyVff5mnC14z89XjDHmnFLG/whnl3sLY0wszuEVKWXc4l4FVuDsOq2L07mX+FxjzBac470X4hw7fd9zcAXm6S/LXgWXh9L86b0XkSicPtLzNQcXa8dIdz4qkvs7IN5d+b4cjw02ym7zitaPi93574tTj45krsx7D1Sy4IpIXRE5D6fqf2CMWeyuFb8BPCsiDd3xmonIWRWdrjEmC2cN9m13Jpa7gyJw9ulnAQUiMhjn2N8R24EGIhJbxuS/xumg/g184rFF8xXQQUSuEpFw96+PiHQuId9W4BvgFRGJc8cd4A7+CLhGRJLcFYTHgLnGmA0Vnf8SPCwiEW4nfh7OCgk4u3AuEpEo9+s/1xV73nacYwlHvAHc5K7liYjUEZFzRSQG5wNZANwqImEichHOiRnH4mPgHyLSxl27fgynzQuOZaIiEolTyEYDSR5/twAj3K2NRUBX932IxNl15ql428zFKdJ3u+/nKcD5OLu+CnGOPz/ktnUn/ijspYnBac8sIExEHsA5bn/Ep8A/3eWnuZu9NF8BjUXkdhGpJSIxItLXHfYx8C8RSRCReJzddcfyPdHeInKR24a34xw3nIOz28+484OIXIOzRVOiCvQDnwJXi0gXt8N9sJxcv+Mckjge57j9Uty9UjgnzpRkO9Baqv6VxYr0J57mAXtF5B4RqS0ioSLSzd0gKUkMzp6yPBE5HmeLqaJicE782ecujyVtGHh6D+fkue44x3CPKK+NxgN3ikhvt884zi02lVoeipkInCciJ4lIBE5f7Pn644BH3dfBXbaHVnDauP3LROBpnF3o33sMLqvNs3AOFXr2C8WnvdAdbzww3RiT4w6q7HsPVLzgThWRXJyqfh/OQe9rPIbfg3MCyhxxdnfMwPmwVMZHOGcbHl07cXcJ34rzYd2N01hTPIavwOmA1rmb9X/Z9WqMOYTTeZY07TNxdjNn4uzKOXJiV0muwjmusALn5J7b3en8gHMyzyScNbl2/LELuyq24cxrJs7ui5vc+QTnjNB8nA/Nu/yxe+OIh4B33ba41BiTgnPc4SV3mmtwjlFgjMnHOUZ2tTvsMpx2OhZv4axN/4xz8k0eZReWiroAZ+vpPWPMtiN/OMfPQ3GOja7C+SDPwDnWW/xL+G8CXdy2+cKd/yHAYJyts1dwziM40tZjcHYJHzmz/mOcYlSa6TgrZatwdpfm8efdYA+7j6/HWSN/v/gEjnCXzTNwVgC2ufNzqjv4ESAF58zuxThnFh/L94u/xHnvd+Ms4xcZYw4bY5bhnF8wG2d5645zpm9ZSu0HjDHf4Bw/+9Ed58eyJuTudlwALHXfK9wsG00pX03jjxXTbBFZUE7Wkl6z3P6k2PiFOO9REs77uhOnYy6tYN8M/NvtSx/A6dcq6k6c/i8XZ8Xmk3LGn4y7q7bYIaoy28gY8xnOuS4fua/1Bc4Zw1VZHo5McynOGf0f4fSRu3EOXRzxPE6//p3bNnNwVqwq40j9+KzYCn6pbe7udn4U+M19v/uVMu2P+Wv9qOx7D/xx9pXyAe5W1gfGmObljKq8TESeBBobY0bZzlJdROQhnBNtrrSdRVU/EVmL8zXKGbazKIf+lrJSJRCRTiKS6O5WOx5n9/3k8p6nlC8QkWE4u4DL3JOgvMu/f7VDqZoTg7MrqSnOIYT/4ux+VcqnichMnK8vXVXsLHxlme5SVkoppbxAdykrpZRSXqC7lMsQHx9vWrdubTuGUkr5ldTU1J3GmITyxwwuWnDL0Lp1a1JSUmzHUEopvyIiVfnVuoCnu5SVUkopL9CCq5RSSnmBFlyllFLKC7TgKqWUUl6gBVcppZTyAi24SimllBcEVcEVkX+IyFIRWSIiH7uXcVNKKaVqXNB8D1dEmuFc6q+LMeagiHyKcxm9d6wGU0oFnYLD+WxcsYBd6xZQmLsDk38AqRVNWL0mJLTrTfPjEgkNC5ruOWgE2zsaBtQWkcNAFM41Z5VSqsaZoiIWz/qc/IUf02nPb7STg7QracT5kE0saxqcQtyJ19Kh1yleTqpqStAUXGPMFhF5BtiEczHz74wx3xUfT0RGA6MBWrZs6d2QSqmAY4qKWPDtu9RPeZbEoo3kEM2y+qcR0nYgCR36EteoBVF1Yti/by/ZW9ayc/U8QtfOoPvOb4ma8iVLv+1O6JkP0yn5dNuzoo5R0FwtSETigEnAZUAO8Bkw0RjzQWnPSU5ONvrTjkqpqtq0Ko29k26j26E0NoS0JKvH3+hx9rVE1Cr/9JF9e3ezZOoLtFv9NgnsZl69c+g46kVi4+K9kPzYiEiqMSbZdg5fE0wnTQ0C1htjsowxh4HPgRMsZ1JKBSBTVMS8Sc/S8MNBtDy0mrmdx9Ji7EL6DL25QsUWILpuHP1GPEjUHWnMbnIlPXdP58Dz/Vgxf0YNp1c1JZgK7iagn4hEiYgApwPLLWdSSgWYvIP7SXl+OMcvfojVkd3Iv2kOfS+7p8onQdWJqUf/G19m3fkTMSK0++pS5n/xcjWnVt4QNAXXGDMXmAgsABbjzPvrVkMppQLKnl1ZrHv2LPrsmc7sljfS5a4ZxDeunnNBOiafRp1bZ7Mysjt90sYy+627MUVF1TJt5R1Bcwy3KvQYrlKqonZu28y+18+haeEW0o9/kuRzb6iR18k/lMeiV0bRZ8+3zG46kn7XP4+E+Na2kx7DLZlvvUtKKeWHdmdtJff1c2lYuJ1Vg96usWILEFErkuTbPmZug6H0z3yPOe/cW2OvpaqXFlyllDoGe3bvJHvcuTQtzGTdoDfodvLQGn9NCQmhz81vM7/eYPpveo25nzxR46+pjp0WXKWUqqL8Q3lkvHohLQs2sOKUV71SbI8ICQ2l15gPWBh1AsnLniD9p4lee21VNVpwlVKqCkxREWmvXkPX/HTSez9Gj1Mv8XqG0LAwOvztYzaEtaHNzDFsXJ7q9Qyq4rTgKqVUFcz96GGOz/maOc2vI3nITdZy1ImpR/Q1EzkkteCzUezPzbGWRZVNC65SSlXSkl+ncPzq51kQPZDjr3nadhwaNW/HtkEv0qIwg2Xjb9CvC/koLbhKKVUJO7dtpvGMW9gc2oyON75HSGio7UgAdDtpCHNb3UCfPd8x/8uXbMdRJdCCq5RSFVRUWMjWt0cSbfZTNOwt6sTUsx3pT44f+ThLI7rTJe0xtm1abTuOKkYLrlJKVdC89++n+6EFpHcfS5uufW3H+YvQsDDqDX+DEIrI+lB3LfsaLbhKKVUBa9N/p/f6caTGnEqfi263HadUzdp2ZnHXO+l+aCHzJv3PdhzlQQuuUkqVI/9QHnx5M3skhnajxvncTykW12fYHSyplUS3JU+zPWOt7TjK5dtLjVJK+YDUD/5Fu8L1bD7hUerFN7Ydp1whoaHEXTaOUArJmPB/tuMolxZcpZQqw9r030ne9BYpdQfR88wrbcepsGZtO7Ow1bX03jeTxT9/aTuOQguuUkqVqrCggMIpt7FHYmg/6hXbcSqt5+UPkiGNqffTvRzKO2A7TtDTgquUUqVImfwcHQpWsaHXP4lt0Mh2nEqLrF2HXQMepYXJZOGnj9uOE/TCbAcIRHM/+y/Rq78gvCiP8KJDFEg4h0Nrcyg8lkPRLZD6bajXrg+tu/WjVmSU7bhKqRLs2rGFTkv/x9KIRHqfN9p2nCpLPPVi0ua9QZd148nZ+Xe/OAYdqLTg1oTCfEJMIXlhddkfUosQc5iwwoPUP7iRRvvmErn9MCyH/KlhLK3VmdxWZ9Ci/8U0a9vVdnKllGvNR3fS0+QRfeFzPn9WcnnihjxGnQmnM//T++l38xu24wQtMcbYzuA1IlIPGA90AwxwrTFmdmnjJycnm5SUlGrNYIqK2L5lHVuX/cah9XNptOM32hRtAGBZeDf2d72C7meOIjIqulpfVylVcSvmfU+nry9mdpMr6X/jy7bjVIt5L1xJUvbXZI36pcZX7kUk1RiTXKMv4oeCreC+C/xijBkvIhFAlDEmp7Txa6LgliRz/Qo2/vIhzddPpIXJZCf1WHPcNXQfervP/XScUoGuqLCQNY/3I7Ygm+g7FgTMZ3Bn5kaiXuvDiph+9LpzSo2+lhbckvn3fpJKEJG6wADgTQBjTH5ZxdabmrbpRP+R/6H5/UtZMuh9ttVqTb81z5L/3+7M/eQJCg7n246oVNBY8M2bdChYxaaedwZMsQWIb9qKRS1H0mvfLFYtmGU7TlAKmi1cEUkCXgeWAT2AVOA2Y8z+YuONBkYDtGzZsvfGjRu9nNSxYv4MCmb8h26H0lgf0poDgx6n6wnnWMmiVLDIO7if3U/2YH9oXdqOne8zVwKqLrl7dlH4bHc21u5Cj3u+r7HX0S3ckgXNFi7OCWK9gFeNMT2B/cC9xUcyxrxujEk2xiQnJCR4O+NRnfoMous9P7Gw/wtEFu2n63eXM/elaziwb4+1TEoFurTPnqAJWeSd+lDAFVuAmNj6LG8zih4H57Ey5UfbcYJOMBXcDCDDGDPXvT8RpwD7LAkJoedZo4i7O405jS6nT9Zkdv23Lyvmz7AdTamAsztrK13XvsGi2n3pdtIQ23FqTPcL72I3MRya8ZjtKEEnaAquMWYbsFlEOroPnY6ze9nnRUZF0+9v41h+1keEUEi7ry5lzseP6qW3lKpGqz79F1HkUW9IYP9ARHTdOFa0vYbEvPm68u5lQVNwXbcAH4pIOpAE+NUqXtcTziH6tjksqdOXfiufIvXZi3UXs1LVIHPDSnrumExKg/Np1bm37Tg1LvHCO9hNXfJ/8Ksu0O8FVcE1xqS5x2cTjTEXGGN2285UWXXrNaDHHV8xp/Xf6bX3RzKfPZWd2zbZjqWUX8v44iEMIbQZ9m/bUbyiTkw9VrS9msS8VFan/WI7TtAIqoIbKEJCQ+l39WMsHjiOpgUZHB53GhtXLLAdSym/tHnNYnrt/paFjS6kYbM2tuN4Tdcht5NrarN3xjO2owQNLbh+rMdpw9lywUTCOUzchPP0eIxSVbB9ykMcJozjLrrfdhSvqluvAUuaDiMpdxZb1i21HScoaMH1c+17DiD/6u/YK3Vp8dUIls+dbjuSUn5j4/JUeu35gUVNLiG+cUvbcbzuuPPvopBQMqY9ZTtKUNCCGwCatu5IxPXfkB3agFZfX8XS36bZjqSUX8ie9jAHqUXHYcG1dXtEQtPWpNU/ix47p5G9PcN2nICnBTdANGzWhqjR37IjtCFtv7tai65S5Vi/dC699s0ivfnlxCU0sR3HmsaD7yaCAlZN0WO5NU0LbgCJb9ySujdNZ3toI1p9d52efahUGXZ9+wT7TSRdLvqn7ShWteyQxKLoE+m65VP9mmEN04IbYOo3bEbUdVPJlWjiv7iCTavSbEdSyudsXrOYpL0/kd70YmIbNLIdx7rIAbdSl/0s/kavlVuTtOAGoIbN2lBw5WSKECI+Gsb2jLW2IynlU7ZOe5zDhNF+6F9+Tj0odepzBmtC29Fo+Tv6C3Y1SAtugGpxXHdyLppAtNnPvrcvZn9uju1ISvmEbZvXkLTrWxYlnE984xa24/gECQlhV7draF20maW/TbUdJ2BpwQ1g7RJPYN0pL9G6YD2rXr2cwoIC25GUsm7D1CcRoOX5unXrKfHsa9lFXQ7PHmc7SsDSghvgEk+9mJTO99DzwO/MH3+r7ThKWbVrxxZ6bP+CtLgzadKqY/lPCCKRteuwsvnF9Ng/my3rltuOE5C04AaBvsP/ydz4YfTb9iHzJj1nO45S1qz88ilqcZhG5+jWbUnaDb6VQkLYPP1521ECkhbcINH7xnGkRyaTlP4fVi2YZTuOUl63NyebbhmfkBYzgJYdkmzH8UkNm7Uhve5Aumz/Us/7qAFacINEWHgErW74iGyJo+6Ua9mdtdV2JKW8atnU54mRg9Q9427bUXxanQF/py4HWDrjfdtRAo4W3CAS26AR+y54mzizh4zxV+hJVCpoHM4/RJu1H7A0ogfH9TjJdhyf1rH3aSw7awLJQ262HSXgaMENMu2TTmZR4n10P7SA+e/omr4KDou+fZtGZFPQT4tIeSQkhC79BxMSGmo7SsAJuoIrIqEislBEvrKdxZY+F97GvHrn0C/jTRb/PNl2HKVqlCkqot6i19kY0pzuAy+xHUcFsaAruMBtQFCf8y4hIXS/4XU2hrSg8Y//YNeOLbYjKVVjls6exnGFa9ne5TrdalNWBVXBFZHmwLnAeNtZbKtdJ4aCC8cTa3LZ9M51+nNuKmAV/voiu6hL4rk32o6iglxQFVzgOeBuQKsL0K57PxZ0/AdJB2Yz77OnbcdRqtptXLGAHgfnsrLlcCJr17EdRwW5oCm4InIesMMYk1rOeKNFJEVEUrKysryUzp6+w8eyKLIPPZY9zfpl823HUapabf/uf+SZcDqed7vtKEoFT8EFTgSGiMgGYAJwmoh8UHwkY8zrxphkY0xyQkKCtzN6nYSE0Ozqt9kvUZiJ15N/KM92JKWqRfb2DHpkf8uiBoOp37CZ7ThKBU/BNcb80xjT3BjTGhgO/GiMudJyLJ8Q37gFm098grZFG0j94D7bcZSqFqumvUAtOUzjs+6wHUUpIIgKripb0hlXkFL3DJI3vc2aRb/ZjqPUMck/lEf7TZ+QHtmHVh2TbMdRCgjSgmuMmWmMOc92Dl/TftTL7JEYQr68WXctK7+WPuN94snBHD/adhSljgrKgqtKFtugERknObuWF7w/1nYcpaosOu0tMqQJ3QcOsx1FqaO04Ko/SRp0OfNjzyR589usWfSr7ThKVdqaRb/S6fAyMtqP0B+6UD5FC676iw6jXmG3xCJTxlBwON92HKUqZfdPL3HA1KLLOfq7ycq3aMFVfxFbP4GM/g/TrnA9KZ88ZjuOUhW2a8cWEnfPYHH8YOrWa2A7jlJ/ogVXlSjpjKtYGHUCiatfIXPDSttxlKqQld+87HwVaNCttqMo9RdacFWJJCSEJsNfACDrkzH6W8vK5xUczqfN+k9YUiuJVp17246j1F9owVWlatyyPekdxtDj4DwWTn/XdhylyrT4x49pzE4O977BdhSlSqQFV5Up+dJ7WRPajhZzH2ZvTrbtOEqVKiJ1PJnSkMTThtuOolSJtOCqMoWFR2DOf576JoflH9xpO45SJVq/dC5d89PZ1PZyQsPCbMdRqkRacFW52iedTErCRSRnTWbt4jm24yj1Fzt+eIk8E06nwfpVIOW7tOCqCul0xZPslRgOTblDT6BSPmXP7p10z55Oev0zqRff2HYcpUqlBVdVSGz9BFZ3+wddDi8hddobtuModdSK6W8QJYeIG/g321GUKpMWXFVhvS+4ldWhx9Ey9Qn25+bYjqMUpqiIRqs+YlVYB9onnWw7jlJl0oKrKiw0LIzCwU/RkF2kf3S/7ThKsXL+DFoXbSKn8wjbUZQqlxZcVSmdkk9nfuzZ9M78kM2rF9mOo4Lcvt9eJ9fUpttZ19iOolS5tOCqSmtz+dPkE8GuSXoClbInZ+c2uu+ZybKEc4iKjrUdR6lyacFVlRbfuCVLOvyNHnnzSZ/5qe04KkitmP4ateQwDU/Tk6WUfwiagisiLUTkJxFZLiJLReQ225n8Wa+L72GzNKXer4/oJfyU15miIpqtmcDy8C606dLHdhylKiRoCi5QANxhjOkM9AP+LiJdLGfyWxG1ItnZ/z5aFW0mdfJztuOoILP0969oYTLZ1+0q21GUqrCgKbjGmK3GmAXu7VxgOdDMbir/ljToCpZGdKfDshf1d5aVVx2a8yY5RNP9jJG2oyhVYUFTcD2JSGugJzDXchS/JiEhRJzzOHHsZeknD9mOo4LEzm2bScz9hRWNzicyKtp2HKUqLOgKrohEA5OA240xe0sYPlpEUkQkJSsry/sB/Uz7pJOZH3sWvTM/0gvVK69YPf1VwqWQpqffZDuKUpUSVAVXRMJxiu2HxpjPSxrHGPO6MSbZGJOckJDg3YB+quUlj1NICFsn3Ws7igpwRYWFtNrwGUsjetCyQ5LtOEpVStAUXBER4E1guTHmf7bzBJJGzduR1uJKeuf+yIqUH2zHUQFsyc+TaWp2kJc0ynYUpSotaAoucCJwFXCaiKS5f+fYDhUoul/6ADupB9/epz+GoWpM4fw3ySaW7qfrTzkq/xM0BdcY86sxRowxicaYJPfva9u5AkV03TjWdbuNTgXLSfv+fdtxVADanrGW7vvnsKrpUCJqRdqOo1SlBU3BVTWv19AxbAxpToO5T+qPYahqt+67cYRgaDlILzKv/JMWXFVtwsIjyO57Ly2LtrBgysu246gAUnA4n3abJrKkdm+ate1sO45SVaIFV1WrnmeMYGVYJ1ovfoG8A/tsx1EBYsnMiTRkF4W99KpAyn9pwVXVSkJCKDjtQRqyi7SJT9qOowKEpL7FDurT/dRLbUdRqsq04Kpq1/WEc1gU2Ycu695kzy798RB1bDI3rKT7wRTWtriIsPAI23GUqjItuKpGRJ/7CNHmAMs++7ftKMrPbfz+FQzQ5ky9DJ/yb1pwVY1o170fC2JPp2fmx+zYst52HOWn8g/l0X7LFyyu04/GLY6zHUepY6IFV9WYphc+SghFbJh0v+0oyk8t/uFj4slBkq+1HUWpY6YFV9WYpm06saDhRfTKnsamVWm24yg/VGvRO2wlgW4DLrIdRaljpgVX1aj2Fz/EISLY+aVu5arK2bx6Ed0OpbGh9cWEhoXZjqPUMdOCq2pUg0bNSW95Fb32/8zqtF9sx1F+ZMuMVzlsQmmvJ0upAKEFV9W4bhePJYdoDkzXM5ZVxeQd3E/H7V+xOOZE4pu2sh1HqWqhBVfVuJjY+qxoey09Ds5jxbzvbcdRfmDJ9+8TRy7hx19vO4pS1UYLrvKKxIvuZCf1KJzxH9tRlB+os/g9MqQJXU88z3YUpaqNTxZcERkjInG2c6jqExUdy5qOo+mav4glv06xHUf5sA3LU+h8eCkZbS8lJDTUdhylqo1PFlygMTBfRD4VkbNFRGwHUscu6YLb2U4DwmY+phepV6Xa/uOr5JswOp6tJ0upwOKTBdcY8y+gPfAmcDWwWkQeE5F2VoOpYxJZuw4bu42hU8Fy0md+ajuO8kEH9+fSOetr0mNPIS6hie04SlUrnyy4AMYYA2xz/wqAOGCiiDxV1Wm6W8srRWSNiNxbTVFVJfQc8ne2SCPq/PYURYWFtuMoH7P4u7epywGiTrjBdhSlqp1PFlwRuVVEUoGngN+A7saYvwG9gWFVnGYo8DIwGOgCXC4iXaopsqqg8IhaZCbdznGFa0n7/n3bcZSPqbf0fTaEtKDz8WfajqJUtfPJggvEAxcZY84yxnxmjDkMYIwpAqp62uLxwBpjzDpjTD4wARhaPXFVZfQ6dzQbQ1rQYN4zFBYU2I6jfMSaRb/SoWAV29pfjoT4atekVNX55FJtjHnAGLOxlGHLqzjZZsBmj/sZ7mN/IiKjRSRFRFKysvRarjUhNCyM7D530KpoMwu/fsN2HOUjsme9xkETQeezRtuOolSN8MmCW0NKOtPZ/OUBY143xiQbY5ITEhK8ECs4JZ05krWhbWm88DkO5x+yHUdZlrtnF92zp7M4bhCx9fVzpwJTMBXcDKCFx/3mQKalLEEvJDSU3BPuobnZxsIpL9uOoyxbNv1NouQQ9QbcaDuKUjUmmArufKC9iLQRkQhgOKC/wGBRj1MvZWVYJ1oteYm8g/ttx1GWmKIi4ld+yNrQtrRPGmA7jlI1JmgKrjGmABgDTAeWA58aY5baTRXcJCSEwwPH0ohs0r54znYcZcnKBT/RrnA9OzuN0JOlVEALqqXbGPO1MaaDMaadMeZR23kUdD3xfJZGJHLcytc5uD/XdhxlQe4vr7PfRNL1rOtsR1GqRgVVwVW+R0JCCDn9X8STw6LPn7YdR3nZnl1ZdM/5gSXxZxNdV38+XQU2LbjKus59zyI9MpmOa99i397dtuMoL1r+7WtEymHiB+rJUirwacFVPiHyzAeII5fFk56wHUV5iSkqosmaj1kZ1pF2iSfYjqNUjdOCq3xCh14DWRh1Al03vseeXfqDI8Fg2ZxvaVWUwZ6uV9mOopRXaMFVPiP2nIeoywGWff6Y7SjKCw7OHs9eouh+5tW2oyjlFVpwlc9o260vqdGnkLj5I3bt2GI7jqpB2dszSNw7k2UJ51K7ToztOEp5hRZc5VPiz3uQSA6x6nP91lYgWz39NSKkkCan6UXmVfDQgqt8SqtOvVhQ70yStn7KzswSr1+h/FxhQQEt1n/CsvButOrc23YcpbxGC67yOU2HPkgYhayd/G/bUVQNWPLzRJqZ7RxMusZ2FKW8Sguu8jnN2nZlQYNz6bnjC7ZtWm07jqpu88aTRRzdB11pO4lSXqUFV/mklhc8CMCmLx62nERVp4w1S+iRN581LS8holak7ThKeZUWXOWTGrdsz8KEofTM/pot6/QaE4Ei4/uXOGxCaX/2GNtRlPI6LbjKZ7W76EEKCSHzS93KDQQH9+fSZfsU0usOIL5pK9txlPI6LbjKZ8U3bUVak0vplfMdG1cssB1HHaPF346nLvupfeJNtqMoZYUWXOXTOlx0H3nUYudXupXrz0xREQ2Wvcu6kNZ0Pv5M23GUskILrvJp9Rs2I7355fTeN5N1S+bajqOqaMX872lXuJ6sziP1IvMqaOmSr3xel2H3sZco9nz9kO0oqooO/DrO+d3kwdfbjqKUNUFRcEXkaRFZISLpIjJZROrZzqQqLrZ+AktbjaTngd9ZtWCW7TiqknZmbiRx7yyWNTyfqOhY23GUsiYoCi7wPdDNGJMIrAL+aTmPqqTuw+5lNzHkfae/PuVvVn/7MuFSSLMzb7EdRSmrgqLgGmO+M8YUuHfnAM1t5lGVF103jpXtriUxL4Xlc6fbjqMq6HD+IY7b9CnpkX1ocVx323GUsiooCm4x1wLflDZQREaLSIqIpGRl6YXQfUmPi+5iJ/Uo+uE/mKIi23FUBaR//z4J7Kaojx67VSpgCq6IzBCRJSX8DfUY5z6gAPiwtOkYY143xiQbY5ITEhK8EV1VUO06MazpOJqu+YtZ+ttU23FUBUQvfJ0MaULiKZfYjqKUdQFTcI0xg4wx3Ur4+xJAREYB5wEjjDHGblpVVUkX3M52GhA263HdyvVxK+bPoGPBSrZ0HEVIaKjtOEpZFzAFtywicjZwDzDEGHPAdh5VdZG167Ch69/pVLCc9JkTbcdRZdg/60X2EkW3c/Ui80pBkBRc4CUgBvheRNJEZJztQKrqeg0dwxZpRJ3fntCtXB+1bdNqeuT+zLLGF1Inpp7tOEr5hKAouMaY44wxLYwxSe6f/pirHwuPqEVmj1s5rnAtad+/bzuOKsGGr58FoPU5/7CcRCnfERQFVwWenueOZlNIM+LmPkNhQUH5T1Besz83hy7bJrMoZgCNW7a3HUcpn6EFV/mlsPAIsnr/H62LNrHw27dsx1Eelkx7lbocoM4pt9qOopRP0YKr/FbPs69hfUhrGqU+S8HhfNtxFFBUWEizle+yMqwjnZJPtx1HKZ+iBVf5rZDQUHL63UULk8mCKa/YjqOA9B8/obnZyr6eo21HUcrnaMFVfi1p0BWsDOtI68XPc3B/ru04QS98/qtsI54eZ460HUUpn6MFV/k1CQmh8PSHacgu0iY+YTtOUFu1YBZd89PZcNyVhIVH2I6jlM/Rgqv8Xpf+g0mr3Y+u698iZ+c223GC1r4fnmEvUXQ9/zbbUZTySVpwVUCoN+RR6piDrPjsQdtRgtLm1YtI2vcLS5teQkxsfdtxlPJJWnBVQGjdOZnUuMH02jaRzA0rbccJOlu/forDhNF+yF22oyjls7TgqoDR6uJHKELI/Hys7ShBZWfmRpJ2fUta/LnEN25hO45SPksLrgoYjZq3Y2HTy0neO4M1i36zHSdorJ76FKEU0vzcu21HUcqnacFVAaXLpQ+SQzQHvrnfdpSgsDcnm26Zk0iLGUiztl1tx1HKp2nBVQElNi6eFe1Hk5iXyuKfv7QdJ+AtnfIcMXKQ2DP02K1S5dGCqwJOz2F3sZUEas96mKLCQttxAtbB/bm0X/cei2v15LgeJ9mOo5TP04KrAk6tyCi29LqD4wrXkjpVL31cUxZ9+Tzx5BB6yj22oyjlF7TgqoDU69zRrArrQKu0Z9ifm2M7TsDJO7ifdqvGszSiO136D7YdRym/oAVXBaSQ0FCKznqchuwi/ZOHbccJOIumvEgCuzEDdetWqYoKqoIrIneKiBGReNtZVM3r1GcQKTGn03Pz+2zdqD+GUV0O5R2g9fLXWR7ela79z7UdRym/ETQFV0RaAGcAm2xnUd7T/NKnMAiZE++1HSVgpE15mUZkU3DyXUhI0HQhSh2zYPq0PAvcDRjbQZT3NG5xHGktR9I790dWzP3Odhy/l38oj1bLXmNFWGe6nTTUdhyl/EpQFFwRGQJsMcYsqsC4o0UkRURSsrKyvJBO1bQelz3ADuoT+v1Y/ZrQMVo45WUak0X+iXfq1q1SlRQwnxgRmSEiS0r4GwrcBzxQkekYY143xiQbY5ITEhJqNrTyiqjoWDb1upv2BatJnfqq7Th+K+/APtosfYmVYZ3oPvAi23GU8jsBU3CNMYOMMd2K/wHrgDbAIhHZADQHFohIY5t5lXf1Onc0K8M60SbtafbmZNuO45fSPn+Ghuyi4NQHdOtWqSoI+E+NMWaxMaahMaa1MaY1kAH0MsbolcqDSEhoKCHnPUOc2cPyD/VH9isrd88uOq4ZT3pkb7qeqGcmK1UVAV9wlTqifdLJpCRcSPKOSXo1oUpaMvFR4sil9tn6nWalqiroCq67pbvTdg5lR6cRT5MjdSmc+g89gaqCdu3YQuKmD1gQPYD2SSfbjqOU3wq6gquCW2xcPOt63kvHgpWkTH7edhy/sGriw0RyiAbn/8d2FKX8mhZcFXSSz7+JZRHd6bDkv+zO2mo7jk/LWLOEXtsnklr/HFp1TLIdRym/pgVXBR0JCaH2Bc9Sxxxk9Ud32I7j07I+v5sCwmh7yeO2oyjl97TgqqDUpksfUpsM5/jd01j62zTbcXzSkt+m0vPAb6S3uZ74pq1sx1HK72nBVUGrx1VPskUaETvj/zi4P9d2HJ9SWFBA7R8fYBsJJF061nYcpQKCFlwVtGrXiWH3oP/S3Gxj0ft6mTlPC6a8TLvCdWT0uYfIqGjbcZQKCFpwVVDrduL5zK0/hD5bP2LVglm24/iEfXt30yb9f6wI60zvwdfZjqNUwNCCq4Je55HPkS1xhH91C/mH8mzHsW7JB/dQ3+wh5Jwn9CcclapGYbYD+JvDhw+TkZFBXp52zFURGRlJ8+bNCQ8Ptx3lqLr1GrDu5MdJ+uVGZn9wH/2v+6/tSNasWfQbfbZ/yvz4ofTtdYrtOEoFFC24lZSRkUFMTAytW7dGRGzH8SvGGLKzs8nIyKBNmza24/xJ0unDmZ8+iT6b3mJlyrl0TD7NdiSvKywooGjq7eRIXTqNeMZ2HKUCju4vqqS8vDwaNGigxbYKRIQGDRr47N6BjteMY6c0oM60m9mfm2M7jtelfP4sHQpWsb7XWGLr66UplapuWnCrQItt1fly29Wt14DsM1+gadE2lrx9i+04XrVz2yY6L/sfS2ol0fu80bbjKBWQtOAq5aHrCecwt+kI+u6aQtqMj23H8QpTVMTm926kljlM3WHP64lSStUQ/WQFgS+++IJly5Z5/XUfeughnnnG/44F9hr1NGtD29Dy17vJytxgO06NS/3qdXoe+J2F7cfQskOS7ThKBSwtuEGgOgtuQUFBtUzHl9WKjCLskreINIfIensEBYfzbUeqMTszN9J+wb9ZEd6FPsP/ZTuOUgFNz1I+Bg9PXcqyzL3VOs0uTevy4Pldyx3vggsuYPPmzeTl5XHbbbcxevRooqOj2bdvHwATJ07kq6++YvTo0UyZMoVZs2bxyCOPMGnSJHJzc7nppps4cOAA7dq146233iIuLo758+dz3XXXUadOHU466SS++eYblixZwjvvvMO0adPIy8tj//79TJkyhaFDh7J7924OHz7MI488wtChQwF49NFHee+992jRogUJCQn07t27WtvHW1p16kVK8n9ITr2b2W/9g/43vmw7UrUzRUVkfHAjnUw+dS4dR2iYdgdK1STdwvVTb731FqmpqaSkpPDCCy+QnZ1d4ngnnHACQ4YM4emnnyYtLY127doxcuRInnzySdLT0+nevTsPP/wwANdccw3jxo1j9uzZhIaG/mk6s2fP5t133+XHH38kMjKSyZMns2DBAn766SfuuOMOjDGkpqYyYcIEFi5cyOeff878+fNrvB1qUvL5NzK3wQX03/oBad9/ZDtOtZv32VMkHZhNWsfbadG+h+04SgW8oFmlFZFbgDFAATDNGHP3sU6zIluiNeWFF15g8uTJAGzevJnVq1dX6Hl79uwhJyeHgQMHAjBq1CguueQScnJyyM3N5YQTTgDgiiuu4Kuvvjr6vDPOOIP69esDzvdpx44dy88//0xISAhbtmxh+/bt/PLLL1x44YVERUUBMGTIkGqbX1t6XP8Ka55ZTNvf7mRLu540a9vZdqRqsTb9d3oue5pFUX3pO1wvTqCUNwTFFq6InAoMBRKNMV0B/zuTx8PMmTOZMWMGs2fPZtGiRfTs2ZO8vLw/feWmst91NcaUObxOnTpHb3/44YdkZWWRmppKWloajRo1Ovp6vvy1n6qIrF2HqCs/wgCHP7iEvTkl70nwJ/tzcwj/4npypC4trnlbz0pWykuC5ZP2N+AJY8whAGPMDst5jsmePXuIi4sjKiqKFStWMGfOHAAaNWrE8uXLKSoqOrr1CxATE0NurnP5udjYWOLi4vjll18AeP/99xk4cCBxcXHExMQcndaECRPKfP2GDRsSHh7OTz/9xMaNGwEYMGAAkydP5uDBg+Tm5jJ16tQamX9va9qmE5vPeI1mhZlsGHeZX59EZYqKWP7G9TQvzGTHGS9Qv2Ez25GUChrBUnA7ACeLyFwRmSUifUobUURGi0iKiKRkZWV5MWLFnX322RQUFJCYmMj9999Pv379AHjiiSc477zzOO2002jSpMnR8YcPH87TTz9Nz549Wbt2Le+++y533XUXiYmJpKWl8cADDwDw5ptvMnr0aPr3748xhtjY2BJff8SIEaSkpJCcnMyHH35Ip06dAOjVqxeXXXYZSUlJDBs2jJNPPrmGW8J7up14Pgu7/4vEvPmkvPF323GqbO5H/yZ57/fMbX0j3U4833YcpYKKlLcr0V+IyAygcQmD7gMeBX4EbgP6AJ8AbU05M5+cnGxSUlL+9Njy5cvp3DkwjuMVt2/fPqKjnWufPvHEE2zdupXnn3++2l/Hn9twziuj6bfjE+Z0vJt+l99nO06lpM+cRNefrmNR9Mkk/d8XhBQ7MU6p6iIiqcaYZNs5fE3AnDRljBlU2jAR+RvwuVtg54lIERAP+OYmrCXTpk3j8ccfp6CggFatWvHOO+/YjuRz+ox+hYX/y6DfyqdImVKf5CF/sx2pQjavXkTrmbewMbQVHW96X4utUhYETMEtxxfAacBMEekARAA7rSbyQZdddhmXXXaZ7Rg+LTQsjM63fMbSZweTlDqWtDpxJJ0+3HasMmVlbiDsw2EUEErkyE+pE1PPdiSlglKwHMN9C2grIkuACcCo8nYnK1WayNp1aDXmS9aHt6PTz2NInznJdqRS7dmVxb7xQ6hrcsm+4COatu5oO5JSQSsoCq4xJt8Yc6Uxppsxppcx5kfbmZR/i64bR/yNU9kS1oJOP41m4Xcf2I70F7l7drH11SE0K9zC+kGv0z4pcE5iU8ofBUXBVaomxCU0If7v37EhvC3dfruV1GnjbUc6as+uLLa+eBbt8ley9IT/0e3kobYjKRX0tOAqdQxi6yfQ5JbprI7oTO/5dzDnvfsxRUVWM2VvzyDr5bNofXgdS096iZ5njbKaRynl0IIbxN555x3GjBlT7jiZmZlH719//fVWLvXny2Ji69P2H9NJjTmVfuteYP4LI8g/VLlf+qou65fN59C4U2lesInlp4wj6YwrrORQSv2VFlxVpuIFd/z48XTp0sViIt8UGRVNz9snMbv5tRyf8zUbnj6ZLeuWezXDoh8nkPDJ+YSbw2waOokep17i1ddXSpUtWL4WVDO+uRe2La7eaTbuDoOfKHe00i7Pd9ttt/HVV19Ru3ZtvvzySxo1asTUqVN55JFHyM/Pp0GDBnz44Yc0atTo6LRyc3NJTExk1apVhIeHs3fvXhITE3n66adJSUlhxIgR1K5dm9mzZzN48GCeeeYZkpOT+fbbbxk7diyFhYXEx8fzww8/VG9b+JmQ0FD6X/8sC6b34rjZ9yDvnkpK74fofd7oGv294kN5B1j41u302/EJa8LaEXPNZ3Ro3q7GXk8pVTW6heunSro83/79++nXrx+LFi1iwIABvPHGGwCcdNJJzJkzh4ULFzJ8+HCeeuqpP00rJiaGU045hWnTpgHO7ygPGzaMSy655OjPN6alpVG7du2jz8nKyuKGG25g0qRJLFq0iM8++8x7M+/jep11Ffuv+YktEW1IXnAPi58axJZ1S2vktZbPnU7mU/3ot+MT5sYPo/mdv9BIi61SPkm3cI9FBbZEa0pJl+eLiIjgvPPOA6B37958//33AGRkZHDZZZexdetW8vPzadOmzV+md/311/PUU09xwQUX8Pbbbx8t1qWZM2cOAwYMODqtI5fuU44mrTrS8J5fmDvxabouf57wdwcyp/Ew2l90Pw0aNT/m6W9Zt5zMLx+kz57pbCOetJNfo6+P/wCHUsFOt3D9UGmX5wsPDz96ebzQ0FAKCgoAuOWWWxgzZgyLFy/mtddeK/HSfSeeeCIbNmxg1qxZFBYW0q1btzIzGGMC7lJ81S00LIy+w//JgdGzSa93On22fULtV3ox98WRrE3/vdLTM0VFrJj3PfOfvZRG755AYs6PzG5yJTF3pPr8r10ppXQL1y+Vdnm+ssZv1sy5DNu7775b6ngjR47k8ssv5/777z/6mOel/Tz179+fv//976xfv542bdqwa9cu3cotRcNmbWj4j0/YtCqN7V8/QY+dXxP5+Zds/KI5WxsOJLL9AJp07EPDpm3+dKy3qLCQHZnr2bYqhbxVP9Fsx890MpkcMLVIaXQJ7S4YS/+mre3NmFKqUrTg+qGzzz6bcePGkZiYSMeOHY9enq80Dz30EJdccgnNmjWjX79+rF+/vsTxRowYwb/+9S8uv/zyo49dffXV3HTTTUdPmjoiISGB119/nYsuuoiioiIaNmx4dBe2KlnLDkm07DCBPbuyWPTdeKLWT6fX1glEbPsQfoFDJpw9EkO+1CLCHCLG7KOx5NMYZ9iqyG5s63gTnQeNpF/dONuzo5SqpIC5PF9NCLbL802cOJEvv/yS999/v0ZfJ5DbsLL25+awedk89qxPxeRsJiRvNyGFhygKrUVRrVgk/jiim3WhXc+BRNauYzuuUhWil+crmW7hKsA5zvvNN9/w9ddf244SVOrE1KNT3zOh75m2oyilapgWXAXAiy++aDuCUkoFND1LuQp0N3zVadsppYKVFtxKioyMJDs7WwtHFRhjyM7OJjIy0nYUpZTyOt2lXEnNmzcnIyODrKws21H8UmRkJM2bH/sPPyillL/RgltJ4eHhJf5Sk1JKKVUW3aWslFJKeYEWXKWUUsoLtOAqpZRSXqC/NFUGEckCNlbx6fHAzmqM4w90noODznNwOJZ5bmWMSajOMIFAC24NEZGUYPtpM53n4KDzHByCcZ5rmu5SVkoppbxAC65SSinlBVpwa87rtgNYoPMcHHSeg0MwznON0mO4SimllBfoFq5SSinlBVpwlVJKKS/QgnuMRORsEVkpImtE5N4ShouIvOAOTxeRXjZyVqcKzPMId17TReR3EelhI2d1Km+ePcbrIyKFInKxN/NVt4rMr4icIiJpIrJURGZ5O2N1q8ByHSsiU0VkkTvP19jIWZ1E5C0R2SEiS0oZHnD9l1XGGP2r4h8QCqwF2gIRwCKgS7FxzgG+AQToB8y1ndsL83wCEOfeHhwM8+wx3o/A18DFtnPX8HtcD1gGtHTvN7Sd2wvzPBZ40r2dAOwCImxnP8b5HgD0ApaUMjyg+i/bf7qFe2yOB9YYY9YZY/KBCcDQYuMMBd4zjjlAPRFp4u2g1ajceTbG/G6M2e3enQP4+/X4KvI+A9wCTAJ2eDNcDajI/F4BfG6M2QRgjAmGeTZAjIgIEI1TcAu8G7N6GWN+xpmP0gRa/2WVFtxj0wzY7HE/w32ssuP4k8rOz3U4a8j+rNx5FpFmwIXAOC/mqikVeY87AHEiMlNEUkVkpNfS1YyKzPNLQGcgE1gM3GaMKfJOPGsCrf+ySq+He2ykhMeKf8+qIuP4kwrPj4icilNwT6rRRDWvIvP8HHCPMabQ2QDyaxWZ3zCgN3A6UBuYLSJzjDGrajpcDanIPJ8FpAGnAe2A70XkF2PM3hrOZlOg9V9WacE9NhlAC4/7zXHWfis7jj+p0PyISCIwHhhsjMn2UraaUpF5TgYmuMU2HjhHRAqMMV94JWH1quhyvdMYsx/YLyI/Az0Afy24FZnna4AnjHNwc42IrAc6AfO8E9GKQOu/rNJdysdmPtBeRNqISAQwHJhSbJwpwEj3bL9+wB5jzFZvB61G5c6ziLQEPgeu8uMtHk/lzrMxpo0xprUxpjUwEbjZT4stVGy5/hI4WUTCRCQK6Ass93LO6lSRed6Es0WPiDQCOgLrvJrS+wKt/7JKt3CPgTGmQETGANNxznJ8yxizVERucoePwzlj9RxgDXAAZy3Zb1Vwnh8AGgCvuFt8BcaPrzpSwXkOGBWZX2PMchH5FkgHioDxxpgSv1riDyr4Hv8HeEdEFuPsar3HGOPXl+wTkY+BU4B4EckAHgTCITD7L9v0px2VUkopL9BdykoppZQXaMFVSimlvEALrlJKKeUFWnCVUkopL9CCq5RSSnmBFlyllFLKC7TgKqWUUl6gBVcpP+JebzddRCJFpI57XdZutnMppcqnP3yhlJ8RkUeASJyLBmQYYx63HEkpVQFacJXyM+5v/c4H8oATjDGFliMppSpAdykr5X/q41wAPQZnS1cp5Qd0C1cpPyMiU4AJQBugiTFmjOVISqkK0KsFKeVHRGQkztWXPhKRUOB3ETnNGPOj7WxKqbLpFq5SSinlBXoMVymllPICLbhKKaWUF2jBVUoppbxAC65SSinlBVpwlVJKKS/QgquUUkp5gRZcpZRSygv+H5eG7m/qSqURAAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The max absolute difference is: 1.77636e-15\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "\n", @@ -3902,9 +3521,7 @@ { "cell_type": "markdown", "id": "5ab6d83d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using autograd\n", "\n", @@ -3917,13 +3534,19 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 11, "id": "751d1395", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The gradient of f1 evaluated at a = 1 using autograd is: 3\n", + "The gradient of f1 evaluated at a = 1 by finding the analytic expression is: 3\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -3947,9 +3570,7 @@ { "cell_type": "markdown", "id": "838d2638", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Autograd with more complicated functions\n", "\n", @@ -3960,13 +3581,24 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 12, "id": "2241609f", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Evaluating at x1 = 1, x2 = 3\n", + "------------------------------\n", + "The derivative of f2 w.r.t x1: 12\n", + "The analytical derivative of f2 w.r.t x1: 12\n", + "\n", + "The derivative of f2 w.r.t x2: -4\n", + "The analytical derivative of f2 w.r.t x2: -4\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4006,9 +3638,7 @@ { "cell_type": "markdown", "id": "b8a70461", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable." ] @@ -4016,22 +3646,26 @@ { "cell_type": "markdown", "id": "a5bc4872", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More complicated functions using the elements of their arguments directly" ] }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 13, "id": "33f0541e", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The computed gradient of f3 is: [ 2. 3. 5. 7. 88.]\n", + "The analytical gradient of f3 is: [ 2. 3. 5. 7. 88.]\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4055,9 +3689,7 @@ { "cell_type": "markdown", "id": "50e2ba5d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that in this case, when sending an array as input argument, the\n", "output from Autograd is another array. This is the true gradient of\n", @@ -4070,22 +3702,26 @@ { "cell_type": "markdown", "id": "fe1a3058", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Functions using mathematical functions from Numpy" ] }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 14, "id": "fda91b69", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The computed derivative of f4 at x = 2.7 is: 13.8759\n", + "The analytical gradient of f4 at x = 2.7 is: 13.8759\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4109,22 +3745,25 @@ { "cell_type": "markdown", "id": "1a48d07c", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## More autograd" ] }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 15, "id": "6c8baf8d", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The computed derivative of f5 at x = 2.7 is: 5.4\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4145,22 +3784,26 @@ { "cell_type": "markdown", "id": "3ac7c6c0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And with loops" ] }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 16, "id": "ac60cd6c", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The computed derivative of f6_for at x = 0.5 is: 3.95703\n", + "The computed derivative of f6_while at x = 0.5 is: 3.95703\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4190,13 +3833,18 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 17, "id": "59735281", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The analytical derivative of f6 at x = 0.5 is: 3.95703\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4212,22 +3860,26 @@ { "cell_type": "markdown", "id": "b247b6ad", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using recursion" ] }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 18, "id": "da05505a", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The computed derivative of f7 at n = 2 is: 1\n", + "The analytical derivative of f7 at n = 2 is: 1\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4261,9 +3913,7 @@ { "cell_type": "markdown", "id": "feb7935f", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input." ] @@ -4271,9 +3921,7 @@ { "cell_type": "markdown", "id": "714f2b8d", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Unsupported functions\n", "Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.\n", @@ -4283,13 +3931,28 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 19, "id": "0ac7f4e6", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "ename": "TypeError", + "evalue": "'ArrayBox' object does not support item assignment", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mTypeError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_60598/1122558214.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m8.4\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"The derivative of f8 is:\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf8_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36mnary_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0margnum\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0munary_operator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0munary_f\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mnary_op_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mnary_op_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 21\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_operator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/differential_operators.py\u001b[0m in \u001b[0;36mgrad\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 23\u001b[0m \u001b[0marguments\u001b[0m \u001b[0;32mas\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbut\u001b[0m \u001b[0mreturns\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mgradient\u001b[0m \u001b[0minstead\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mThe\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 24\u001b[0m should be scalar-valued. The gradient has the same type as the argument.\"\"\"\n\u001b[0;32m---> 25\u001b[0;31m \u001b[0mvjp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mans\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 26\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mans\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 27\u001b[0m raise TypeError(\"Grad only applies to real scalar-output functions. \"\n", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/core.py\u001b[0m in \u001b[0;36mmake_vjp\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmake_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mVJPNode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_root\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_node\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/tracer.py\u001b[0m in \u001b[0;36mtrace\u001b[0;34m(start_node, fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtrace_stack\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_box\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_node\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_box\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misbox\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mend_box\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mstart_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_node\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36munary_f\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0msubargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msubvals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msubargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_60598/1122558214.py\u001b[0m in \u001b[0;36mf8\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mautograd\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mf8\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# Assume x is an array\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mx\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 5\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mTypeError\u001b[0m: 'ArrayBox' object does not support item assignment" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4307,9 +3970,7 @@ { "cell_type": "markdown", "id": "2db47b66", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible." ] @@ -4317,22 +3978,35 @@ { "cell_type": "markdown", "id": "0c29b75a", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## The syntax a.dot(b) when finding the dot product" ] }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 20, "id": "6cfc0ef8", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "ename": "AttributeError", + "evalue": "'ArrayBox' object has no attribute 'dot'", + "output_type": "error", + "traceback": [ + "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", + "\u001b[0;31mAttributeError\u001b[0m Traceback (most recent call last)", + "\u001b[0;32m/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_60598/546166676.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1.0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m0.0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"The derivative of f9 is:\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mf9_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36mnary_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 18\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 19\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0margnum\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 20\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0munary_operator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0munary_f\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0mnary_op_args\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mnary_op_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 21\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 22\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mnary_operator\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/differential_operators.py\u001b[0m in \u001b[0;36mgrad\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 23\u001b[0m \u001b[0marguments\u001b[0m \u001b[0;32mas\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbut\u001b[0m \u001b[0mreturns\u001b[0m \u001b[0mthe\u001b[0m \u001b[0mgradient\u001b[0m \u001b[0minstead\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mThe\u001b[0m \u001b[0mfunction\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 24\u001b[0m should be scalar-valued. The gradient has the same type as the argument.\"\"\"\n\u001b[0;32m---> 25\u001b[0;31m \u001b[0mvjp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mans\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 26\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mans\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 27\u001b[0m raise TypeError(\"Grad only applies to real scalar-output functions. \"\n", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/core.py\u001b[0m in \u001b[0;36mmake_vjp\u001b[0;34m(fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mmake_vjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mVJPNode\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_root\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_node\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mend_node\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mvjp\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mvspace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzeros\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/tracer.py\u001b[0m in \u001b[0;36mtrace\u001b[0;34m(start_node, fun, x)\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;32mwith\u001b[0m \u001b[0mtrace_stack\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnew_trace\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 9\u001b[0m \u001b[0mstart_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnew_box\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstart_node\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m \u001b[0mend_box\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstart_box\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 11\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misbox\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mend_box\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mstart_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_trace\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 12\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_value\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mend_box\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_node\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m~/anaconda3/lib/python3.8/site-packages/autograd/wrap_util.py\u001b[0m in \u001b[0;36munary_f\u001b[0;34m(x)\u001b[0m\n\u001b[1;32m 13\u001b[0m \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 14\u001b[0m \u001b[0msubargs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msubvals\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mx\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0mfun\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0msubargs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 16\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 17\u001b[0m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0margnum\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;32m/var/folders/jy/g42mrgv128v34gnnhxwk9nrc0000gp/T/ipykernel_60598/546166676.py\u001b[0m in \u001b[0;36mf9\u001b[0;34m(a)\u001b[0m\n\u001b[1;32m 3\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mf9\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;31m# Assume a is an array with 2 elements\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1.0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;36m2.0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0ma\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 7\u001b[0m \u001b[0mf9_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgrad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf9\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", + "\u001b[0;31mAttributeError\u001b[0m: 'ArrayBox' object has no attribute 'dot'" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4350,9 +4024,7 @@ { "cell_type": "markdown", "id": "ee6bf2f1", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "Here we are told that the 'dot' function does not belong to Autograd's\n", "version of a Numpy array. To overcome this, an alternative syntax\n", @@ -4361,13 +4033,18 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 21, "id": "37c14206", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The gradient of f9 is: [1. 2.]\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4388,9 +4065,7 @@ { "cell_type": "markdown", "id": "645d9ee0", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Recommended to avoid\n", "The documentation recommends to avoid inplace operations such as" @@ -4400,10 +4075,7 @@ "cell_type": "code", "execution_count": 29, "id": "27ffa613", - "metadata": { - "collapsed": false, - "editable": true - }, + "metadata": {}, "outputs": [], "source": [ "a += b\n", @@ -4415,9 +4087,7 @@ { "cell_type": "markdown", "id": "32628932", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Using Autograd with OLS\n", "\n", @@ -4428,13 +4098,36 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 22, "id": "47a1833d", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[3.99799814]\n", + " [3.06594678]]\n", + "Eigenvalues of Hessian Matrix:[0.30328909 4.16178958]\n", + "theta from own gd\n", + "[[3.99799814]\n", + " [3.06594678]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAArNklEQVR4nO3deXxU9b3/8dcnQFiCooAsCgluRar1utBa666xrrW21VsttVoXrnSxt7uVa6219NbWXqu97VWsdUuqV39Ve+2q0QRBRA0CsqpgCbLJIpvsJJ/fHzMJQ5jlTObMnJnk/Xw85pHMzFk+5+Tk+znf7/ec7zF3R0REJJOyqAMQEZHSoIQhIiKBKGGIiEggShgiIhKIEoaIiASihCEiIoEoYUinZWY/MrOaqOMIk5mdbmZLo45DuiYlDCkoM1tsZlvN7AMzW2lmD5pZ36jjEpHMlDAkCp9y977AMcCxwA+iDadrMrPuUccgpUUJQyLj7iuBfxBLHACY2Y1mtsjMNpnZPDP7TMJ3V5nZFDO7w8zWmdk/zey8hO8PNrNJ8XmfAwYmrs/MLjKzuWa23swazGxUwneLzey7ZvaGmW02s/vNbLCZ/S2+vDoz2z/ZdrQ2E5nZt81slZmtMLMvJ3zfYGbXtt+OhPduZl8xs7fj67rNzA41s5fNbKOZPW5m5e3WeZOZrYnHPSbh857x/bPEzN4zs3vMrHe7OL9vZiuBB8xsoJn9Ob5P3jezyWamckGS0oEhkTGzYcB5wMKEjxcBpwD9gFuBGjMbmvD9CcCbxJLBz4H7zczi3/0BmB7/7jbgyoR1fQh4FPh34ADgr8Az7QrizwFnAx8CPgX8Dbgpvrwy4IY0mzMkHvNBwDXAb1IlmBTOBY4HPg58D5gIjAGGA0cBl7db18D4uq4EJprZyPh3t8fjPwY4LD7ND9vN2x+oAsYC3waWEtsng+Pbq/GCJCklDInC02a2CXgXWAXc0vqFuz/h7svdvcXd/xd4G/hYwrxN7n6fuzcDDwFDgcFmVgl8FLjZ3be7+4vAMwnzfR74i7s/5+47gTuA3sAnEqb5tbu/5+7LgMnAK+4+w923A08Raz5LZSfwY3ff6e5/BT4ARqaZvr3b3X2ju88F5gDPuvs77r6BWOJqv+7W7ZwE/AX413jivA74pru/7+6bgJ8ClyXM1wLcEp93azzuoUBVPPbJrgHmJAUlDInCxe6+D3A6cAQJTUdm9iUzmxlvIllP7Ow6sWlpZesv7r4l/mtf4EBgnbtvTpi2KeH3AxPfu3sLsYR1UMI07yX8vjXJ+3Sd82vdfVfC+y0Zpm8vm3Un284DidUS+gDTE/bf3+Oft1rt7tsS3v+CWA3vWTN7x8xuzCJm6WKUMCQy8bPjB4md7WNmVcB9wNeAAe6+H7GzbUuxiEQrgP3NrCLhs8qE35cTa4Yhvi4j1tyzrONbENhmYgV5qyE5Li/Zdi4H1hBLLke6+37xV7/4BQat9qg9uPsmd/+2ux9CrBnuW2Z2Vo7xSSelhCFR+xVwtpkdA1QQK9BWA8Q7jo8KshB3bwIagVvNrNzMTiZWALZ6HLjAzM4ysx7E2u63A1ND2o50ZgKfNbM+ZnYYsT6OXLVu5ynAhcAT8VrTfcCdZjYIwMwOMrNzUi3EzC40s8PiCXQj0Bx/iexFCUMi5e6rgYeJtcnPA34JvEysSeYjwEtZLO4LxDrF3yfWL/JwwnreBL4I/JrYmfiniF3euyOEzcjkTmAHsW16CKjNcXkrgXXEahW1wPXuviD+3feJNTFNM7ONQB3p+1IOj0/zAbH9/lt3b8gxPumkTP1bIiIShGoYIiISiBKGiIgEooQhIiKBKGGIiEggJTX42MCBA33EiBFRhyEiUlKmT5++xt0PyDxleiWVMEaMGEFjY2PUYYiIlBQza8o8VWZqkhIRkUCUMEREJBAlDBERCUQJQ0REAlHCEBGRQJQwREQkECUMEREJRAlDREQCUcIQEZFAlDBERCSQvCcMM/u9ma0yszlJvvuOmbmZDcx3HCIikptC1DAeBM5t/6GZDQfOBpYUIAYREclR3hOGu79I7BnL7d0JfA/QM2JFREpAJH0YZnYRsMzdZwWYdqyZNZpZ4+rVqwsQnYiIJFPwhGFmfYDxwA+DTO/uE919tLuPPuCAnIdzFxGRDoqihnEocDAwy8wWA8OA181sSASxiIhIQAV/gJK7zwYGtb6PJ43R7r6m0LGIiEhwhbis9lHgZWCkmS01s2vyvU4REQlf3msY7n55hu9H5DsGERHJne70FhGRQJQwREQkECUMEREJRAlDREQCUcIQEZFAlDBERCQQJQwREQlECUNERAJRwhARkUCUMEREJBAlDBERCUQJQ0REAlHCEBGRQJQwREQkECUMEREJRAlDREQCUcIQEZFAlDBERCQQJQwREQlECUNERALJe8Iws9+b2Sozm5Pw2S/MbIGZvWFmT5nZfvmOQ0REclOIGsaDwLntPnsOOMrdjwbeAn5QgDhERCQHeU8Y7v4i8H67z551913xt9OAYfmOQ0REclMMfRhXA39L9aWZjTWzRjNrXL16dQHDEhGRRJEmDDMbD+wCalNN4+4T3X20u48+4IADCheciIjsoXtUKzazK4ELgbPc3aOKQ0REgokkYZjZucD3gdPcfUsUMYiISHYKcVnto8DLwEgzW2pm1wD/DewDPGdmM83snnzHISIiucl7DcPdL0/y8f35Xq+IiISrGK6SEhGRjqqthREjoKws9rM25TVEOYus01tERHJUWwtjx8KWeFdwU1PsPcCYMaGvTjUMEZFSNX787mTRasuW2Od5oIQhIlKqlizJ7vMcKWGIiJSqysrsPs+REoaISKICdiLnbMIE6NNnz8/69Il9ngdKGCIirVo7kZuawH13J3KxJo0xY2DiRKiqArPYz4kT89LhDUoYIiK7FbgTOWe1tbHYliyJNUNNmJC3ZAFKGCIiuwXpRC6WJqsIakNKGCIirTJ1IhdTk1UEtSElDBGRVpk6kYupyarAl9SCEoaIyG6ZOpEjKKRTKvAltaCEISKypzFjYPFiaGmJ/UzsRI6gkE6pwJfUghKGiEhwERTSKRX4klrQ4IMiItnp3Xt3P8aAAXDXXXktpNMaM6ag61YNQ6SrKJbLQUtV6xVSa9fu/mzr1ujiiYAShkhXUEyXg5aqKK6QCprkC3QyYO6elwXnw+jRo72xsTHqMERKz4gRsSTRXlVVrGNXMisriyXb9sxiHeRha/+sC4j1l7TvpwgwnZlNd/fRuYakGoZIV1BMl4OWatNYoa+QClqjKWDNRwlDpCsolstBS7lpLKwrpIImzKBJvoAnA3lPGGb2ezNbZWZzEj7rb2bPmdnb8Z/75zsOkS6tWC4HLaY7pbMVxmWs2STMoEk+xXTb9h/Cj89s4NR+s4LHl4m75/UFnAocB8xJ+OznwI3x328Ebg+yrOOPP95FpINqatyrqtzNYj9ragofg5l7rKjc82VW+FiiUFWVfPurqvaetqbGvU+fPafr02fvv1tNjbe0m24zvf1yatxo9uN6z3Og0UMozwvS6W1mI4A/u/tR8fdvAqe7+wozGwo0uPvITMtRp7dIievqne/ZdpynGb58WeMKnp+4iLrnoe87b/A9fk4lS1jBgfz9wC/T77LzOeP6kQw4vH9ond5RJYz17r5fwvfr3D1ps5SZjQXGAlRWVh7flOxgE5HSEPTKn84qh4S5YckGJt27gLo/b6NuwTDm7zgUgIG2hrOGvUX1Gc2cdc0IDj51+F7zhpUwiv5Ob3efCEyEWA0j4nBEJBetSaGAD/0pKhMmJE+YSfqSdnywg2kPzKfuiXXUzRjAqx+MopkT6M0WTh0wj6s/3kD1FUM5+nOHU9b9EwUJP6qE8Z6ZDU1okloVURwiUmgFHs6iqKRJmC27Wpjz9ELqHllO3csVTFr9YbbwL5TRzEcr5nPjSVOovmQ/Trx6FD33zbmy0CFRJYz/A64Efhb/+aeI4hARKZx2fRKrr/w2z9RVUjf+JZ5f8iFW+YeADzGy/B2+/JFGqi/oxenXH8F+VUdFHTlQgD4MM3sUOB0YCLwH3AI8DTwOVAJLgEvd/f1My1Knt4iUrNpa/NrrsG27x5/aTB+uYyIvlJ1NdeXbVJ/lnHXdIQw/4cBQV10yfRjufnmKr87K97pFRKK0bf02XrpvHnVPbuSGV77LUN9zsMIKtvDgAd+jx8qlWNmgiKIMrug7vUVESkXzjmZmPv4WdbXvUffqPkx5/8Ns4zi6s5MJrEw6T/maFVBm4QbSrulrIPQPY7EaGkRE8ifZMBilOpZUEt7iLHqhiXvHvMilw15mUK8NjL5iFDf+/XRWbt6X6499lT/f8hrvL9tGWVWBhmdJcjd5JVSFsWiNVisi+ZHsnovy8lghtnPn7s9K7D6M1fPX8MI9b1H3j13ULTqYxbti9z0cVLaCsw9ZSPXZZZw59jCGHjN4zxkLdQ9Kkns9RgON7jlXY5QwRCQ/Ut2klkwR3+m9Zc0WJt87j7qnP6Bu7hBmbj0CgH5s4Iyh86k+eTvVXx7Oh845GMvUtJTmzu3QJLmbXAlDRIpbqmEwksnXMyU6YNe2XUyvXUDdY2uoa+zH1PUfZgc96cEOTtpvLtWjN1B92UCOH3ME3XsVUTdwazJKkqTDShhFtLUi0qlUVgavYRR6mPUE3uK8+fd/8vyD71I3pSf1K0axgdh9D8f0XsANo1+m+uK+nHzdKCoGHRtZnGkla+5K4BBKNlant4gEk21ndbIh1cvLoUePPT/LdZj1DnSir5j5HjXjXuKqw6YwvMdKRl1wCF974jRmrBrGpSNn8dgNU1k1bw0zthzBL147nXPGj6ZiUEXHY8y3ZMPGt6qqYgmEMwhfGEPeFuql4c1FIhJ0qO1k87UfUj3MYdYDxrVx2UZ/5uZX/BvHNPiRPd9qm7S/rfVLDprq93xhki98frG3NLd0PJYoZRg2npCGN488CWTzUsIQ6YBsCuhU02bzHIdCShFXy/BKn/ybWX7LafV+0j6zvDs7HNx7scXP7t/ot59X79Nr5nnzzuZw4oj6WSMZ/j5KGCKSWTY1g3TTZnrwUVQFZoq4mrFYeDT7Ryvm+A9OrPfn73jdt67bGn4MHa19FTAGJQwRySybmkG6adN9F1GBuWTaMt9UMThpXGt6DPY/fvdlX7vw/bzG4O7FU/tKk7TDShi6rFakM8vmCW/ppn3kkdQ3naW4lDPseyvWN22g4Z4F1P1lG3VvDufNHYdwObX8juvoQ8IYTYW+ETDbp+hFIKzBB3WVlEhnlupy1WSfp5rWPZYUrrwylgTMYj9bC+UlS5LPl+rzgLZv3E7Dr2byHyc38PG+cxgwoi+f+dkJPDB7NIf0W8svL2rgxic+Sq+HJyaPq1D6pximKcJLhfMmjGpKoV5qkhLJUq59GEHmC6lJpnlns7/+h/n+iwvq/ZwBr3lvNju4d2Onn9j3Db/5lHqfdPdM375pe4d2xV7bGkafS02Ne3n53tveo0fhO77TQH0YIhJIR66SSpU0kiWBHPow3vuPu3xjxWBvxnwJw/xyahzcR5Uv9K8f3eB/ummar29a38ENT7ONYfW5pNpXAwaEG3OOwkoY6sMQkb1l2y4fcIyktW+/zwv/8yZ1f9tJ7zdnMMFvooLd/SK7uvdk4w9/Sf+bvxrm1uwp1RhXHelzKYH+CwivD0MJQ0T2lkuhmpA8Wg4axtxPjKVm8cnUzR7MjK0jccrYh40sLBvJoJYkz4jI90CEYRbyYSafPFKnt4gkF8bzJpIN6xFgCI+Whx6h+epr257FULb0XQ55/D9Z/moTfcu3c+uZLzL13tms3dybQf5e8oXk2FmeUTYXAmTSwf1UssJo1yrUS30YIhmE2T4foO+jpbnF33r2n/7byyb5Zw982ZsYlrRNv3nY8L2XH9X9C2HfNxL1Xd4BoD4MEdlLAZpI3puzmhfufZu6fzRT987BLGkeBsDwbstoah6OEbC5p1APFEqmEM+lKCKdog/DzL4JXAs4MBv4srtvSzW9EoZIBnnohP1g5QdMnjifuv/bTN3cobyxbSQA+9l6zhy6gOpTd1B9dSWHnVWFHXJwdgmrixXcUQkrYUTWvAQcBPwT6B1//zhwVbp51CQlJS3xktVu3XY3v4TZhBFCM8/OrTt96r1v+I/PrPdT+83wHmx3cC9nm5+5/3T/6Sfr/dUH5/qu7bv2nrkYxlWSvRBSk1TUD1DqDvQ2s51AH2B5xPGI5Ef75pfm5tjPpqbY5xDOmfWECcmbedJ0wnqLM//Pi6h7aBnPT+3FgJWz+SE/YTxLuMaGMOmQqxl43Wc46dpR9Bl4XPr1t26Dag2dUtRNUt8AJgBbgWfdfa+jyszGAmMBKisrj28K+gQvkWKS6fnWYV6GGaCZZ1njCp6fuIi656Fu8WGsaBnC5dTy33yN/VnPHs/yLPTYTBK6ztAktT/wAnAA0AN4GvhiunnUJCUpFdOVKsliSTU8ePthwvNkfdN6/9NN0/zrRzf4qPKFbasdaKv988Nf8udPvtmbe/ZOHV9Hr1wqpr9LF0apDw0CXArcn/D+S8Bv082jhCFJFardPEjhlyqWAQPSJ4yQLyXdvmm7T7p7pt98Sr2f2PcN78ZOB/febPZzBrzmv7ig3mc8tmD3A4TSDQcSNKG13z/jxqk/o0h0hoRxAjCXWN+FAQ8BX083jxKGJFWI6/mTJYLy8lgiSEwg6cYWSjWwXwiFaPPOZp/1xJv+y4vq/bwDXvU+fODgXsYuP6Fito8/qd7r75zh2zZsS76ATDWgTPsy2f5Jtcxs/y6qpeSs5BNGbBu4FVgAzAEeAXqmm14JQ5LK9DS4MGQ6A28t+NOdoYd8lVTT1KV+/1Uv+uVVU3yQrWpb1cjyRf7VjzT4UzdO83WL1wdbWLoaUJCEFmT/dOTvoquuQtEpEka2LyWMElHoM8JC1DAynYG3vloTQR5ief+ddf7H777s446c5If3eKdt0YPL3vMxI6b4A9dM9iXTlmW/4FRDdLfWjIL8/YLun2z3RbE8za7EFSxhAHXAv4SxslxfShglIIozwkKsM5sz6JBi2bpuqz9/x+v+gxPr/aMVc9xodnCvYJNfMOgVv/PiBp/95Fve0tySn23LZojuVMton0iy3ReFqD12AYVMGMfFr2Z6ABgaxko7+lLCKAFRjg+Uz1pNpocLJW5nB2Np3tns02vm+e3n1fvZ/Ru9F1tilRZ2+kn7zPJbTqv3yb+ZFc4DhBKFUSinStrjxuX2d1ENIxQFb5ICPgfMAm4hfnd2oV9KGEWupiZ1QdoZzggTE8GAAbGnqrXfzoqKwIViS3OLL3x+sd/zhUl+yUFTvb+tbVvMkT3f8m8c0+DP3PyKb3h3Q363K6xCOR9JO1WiDtpUJu5e4IQRv4rpKOB6YA2wFLgijACyeSlhxBXjVSOZzsA74xnhuHHJtzXN4zlXzVvtj93wkl87cpKP6L6kbZaDypb7VYe96DXjpvjyGSsLux3F3rFcU5O8U76YYixyhWySmkJsyI7ngNuAC4HDgF8DE8MIIuhLCcOL9587XRt/McSXDwEeZbp59Wb/+09e8++Mrvdjes9v+3pf1vvFQ1/2/760wRf8dVHu/RC5KsaTkERqmspJWAkj49AgZnYUMNeTTGhm8919VNoFhEij1VK8T/hKNUoqQE1N5xxWIs02OzCh90+4bet32EFPerCDk/abS/XoDVRfNpDjxxxB914hDuXW2Ud9LZFHoRarsIYGyXjEuvucNF9fkGsAkqVUTyPL91PKMqmsTJ3IOlPBlSjVNhNrw/321p9wyMFbGXDNxZx83SgqBh2bnzjaD2wY9oCGxSDVvu7IU/Kkw3J6RKu7vxNWIBJQmI+XDFOUj6qsrYWBA2Nnm2ax3zvyWNKAVr6xitqvvMS9W7/EdnqknK432/hCSw3njB9NxaCKvMXD+PF7jk4Lsffjx+dvnYXW1R6FWqzCaNcq1Et9GF68fRitsRW6HbymJvnVSuXloa1/47KN/udbXvV/P7bBj+r5Vtsq+tta/9V+t/iW8n7eEuXVYV3lXoVi72cpYuhO7y5M/zi7Beh4ztbO3z/kW/sP9RbMV9hQ/yIPObj3ZKtX92/0n51b742PzNvzAUKZOmXz+TdTh7BkoIQh4p5+SIqAZ9gtzS0++8m3/M6LG/zn+9zqm9lzmO/tZT19yccv8eZhw1MX+OlqfvmuFRZjrVMnNUVFCUPEvcM1jHdfXe4PXDPZx4yY4kPKVrbNspQDUyefTAVyqkIyVYzduoVXoOZaQIdZwBdjAuvilDAks65wlhewD2Pd4vX+1I3T/KsfafCR5YvaJjvAVvnlVVP8/qte9MVT3s3PIHpBlhllgZqpgM/2OFITWdFRwpD0utJZXvs7gQcM8B33Pej1d87w8SfV+wkVs72MXbFdwAd+3gGv+i8vqvdZT7y5+wFCrfIxTHfQZUZVoKYr4DtyHHWVTvgSooQRRFc4w06li53lNe9s9hmPLfBfXFDv5wx4zXuzOdbqw04/se8bfvMp9T7p7pmZB+7Lx4OAgg5cGFWBmq6A78hx1MWOvVKghJFJVzrDTiYfZ3lFloD/Ofldv+9LL/rnh7/kA2112yaOKl/oXz+6wf900zRf37Q++wXn41GjicvM4zMzOiRdAd+R46ir/+8VISWMTLr6WU7Y218EhcCat9b6E9+a6v82apIf2n1xWxhDy1b4FYdM9oeum+xLX1uen5WHlSyLcSC9dH/bjh5HRXZy0dUpYWTSWdpRO/qPF3YBH0EC3rJ2iz93+3T//gn1fnyfuW0PENqHDf6pwdP8rs82+Nw/vR0buC/dfiqWwquYh+pOtY+K4ERBcqeEkUlnODPK9Z81zG0pQALetX2Xv/rgXP/pJ+v9zP2ne0+2Orh3Z4efsu9Mv/WMen/pnjd8x+Yde84Y5T0Q2SjVWm8x/U9IhyhhZNKRgqKYChf34ipg8hBLS3OLL//+nW1Da7SAr2KAX06NH91rgX/r+Hr/y49e9U0rNnU8tnw0zWUqPFNN01lqvVJyOkXCAPYD/h+wAJgPnJhu+rxfJVVMBbR7cRUwISXTlbNX+R++9pJfffiL/nW727dRvtf2NXdP/QCipNLtpzD3YZB9kI/+AJEcdZaE8RBwbfz3cmC/dNPn/T6MYiqg3QtbwORy5pzGphWb/K+3vurfOr7ej+61oG0T9rN1/l63Icm3L9ttDKOGEWTbgiwr7HsaREJQ8gkD2Bf4J8Qe4hTklfeEUWxngIUqYEJcz86tO33qvW/4j8+s91P7zfAebHdwL2ebn7n/dP/pJ+v91QfnxgbuC2EcqIzx51orSBTkhCLTNOoPkAh0hoRxDPAq8CAwA/gdUJFkurFAI9BYWVkZ8m5spxjPAAtRwOSQKFuaW3zeMwv97s81+EVDpvm+rI+VjzT7cb3n+fc+Vu/P/mejb169Ofh6E9cddPtzuUoq6PbnWsMQiUhnSBijgV3ACfH3dwG3pZunIEODlMoZYIRXQC2bvsIf/rfJ/qVDJvuBZcvbJj+k+2Ife8Qkf/ybU331gjXBtqF87z4M79GjsFc4Bd3+MGsrIgXUGRLGEGBxwvtTgL+km0djScUV+B6LDe9u8D/dNM2/fnSDf7jn221fD7A1/q/DX/KJV0zylePv6vj9Iu3GgWqbt1Bn69msJ099PSL5VPIJI7YNTAZGxn//EfCLdNMrYcQV4C7uXT16+eMjx/uJfd/wbux0cO/NZv/kgNf85+fX++t/mL974L6wElj7gjZo/0auBbRqBdLJdZaEcUy8f+IN4Glg/3TTK2HEhXw1V0tzizfdcIdv6D3YmzFvYrhfTo2Xscs/VjHbb/pEvb/wy9d967qtyRcQRgLr6KB/+UpWShbSiXSKhJHtqyQTRkcKorA6adNomrrU77/qRb+8aooPslVtixhZvsi/+pEGf+rGab5u8fpgCwsjgaWrUbR/jRsX6r4Q6eyUMHJRqLPJfN1t3oHlvv/OOv/jd1/2cUdO8sN7vNM22+Cy93zMiCn+wDWTfcm0ZR3bzjAK7Y4+uKjY7p0RKUJKGB1VyPbqjhSkId1otnXdVn/+jtf9ByfW+0cr5rQ9QKiCTX7BoFf8zosbfPaTb8UG7stVGPs0mxpGYjJQDUMkIyWMjipkAdORs98OnjE372z26TXz/Pbz6v3s/o3eiy0OsQcInbTPLL/ltHqf/JtZmR8glEm6UU3D7nguZB+GJKe+nU5BCaOjCtmEkc8ahrsvqm/ye8dM8kuHTfUBtqZt0iN7vuXfOKbBn7n5Fd/w7obwtidV4TxuXDiFSvvCKeiDi1So5YeScaehhNFRhR6fKdVZc7pr+FP8k66at9ofu+Elv3bkJB/RfUnb1weVLfcrD53sj1w/xZfPWBn+drRKte/aJ+EwCxUlg+ioua/TUMLoqEKcNSUWcgMG7L4xLWjBGp+/xcy39BvitQeP92N6z2+bbV/W+6eHTPNfX9Lg8/+yKJx+iCA62jGdLSWJ4qALCjoNJYxc5LNASpWQkj2Ws13BunPrTp/2u9n+k+p6P32/172cbQ7uPdjup/Wb4bedVe8v3zfbd27dGV682ehox3Q21AxSPFTD6DTCShgWW1ZpGD16tDc2NkYdRnojRkBTU+DJ3YzfXlJP3ZSe1K8YxQb6AXBM7wVUH7mS6ov7cvJ1o6gYVJGngLNQWwtjx8KWLbs/M4sVI+1VVcHixdmvI9X+6+jypOOS/b379IGJE2HMmOjikqyZ2XR3H53rcrqHEYwkWLIkq8mbvJKvPXEaVd2WcunIWVSf050zxh7OoCOPAI7IT4wd1VpIjB8f287KSjj/fHjoob0LlQkTOraOVPsvy/0qIUj2954wQcmiC1MNI2wpzpA/KNuXbi076M22ts+2WU+mfPy7HPyTaznk9EqszAoYaIhqa8MrVFTDEAldWDWMsjCCiURtbaxwKSuL/aytjToiAHbd8mOay3vt8dlm+vDVlru4s2I8G3oNxjG8spJej9xP9dTbOPTMqtJNFhBLDosXQ0tL7GcuZ6ATJsRqKIlyqbGISGhKs0mqfdtqU1PsPRS8uuwtzrxnFlH30DLqXupNw6rP8Cm68VNuopJ32VA+iGWf/Tr3/Obz9O7fG/iPgsZXctQMIlK0SrNJKuJmi7W3/Yaet99GxeZVLOUgvs/PeJQxHNZjMdWHL6H6/B6ccf0R9D90/7zHIiKSSVhNUqWZMMrKkl+ZYxZrFgnZ+qYNNNyzgLq/bKPnvNeZ0Px9erGz7XsHNl18Bfs+9XDo6+50wuzvEJFAuvZVUpWVyWsYlZWhLH77xu28/Pv51P2/9dTNHMhrm0fRwgn0YTPLuWiPZAFgwL5PPwK156jwS6eImhJFJHul2emdrGPULHaJZwe07Gph5v++yR0XNnDuwEb279fMGd88hp+9dDJl5ow/ZTKT7p7Fuk096MfG1AsaPz77lRdp531ejB+/5+W3EHvfkf0mIgVXmk1SAF/5Ctxzz55NU1ncVLR4ylLq7vsndfXdeH7ph1jjAwEYVb6I6iOWUn1hL077tyPoV9lvzxktzdVM2TaJdbUbowrclCgiMV27DwOy7vhe+/b71N/zJnV/20nd21Us2lUFwNCylVSPWEj1WXDW2EM5aPTQ9EEMHAhr1yb/LttO9652z0FX216RItE1+zCmT48VOhMmZLwjeNc1Yyl74H7MW2imjMcYy9f4H/ZhI6cPns8NJy2m+sqDGHXhoVjZkOAx3HUXXHUV7Nq15+c9emR/r0BXu6t5woTkNSrdYyFSGsIYkKpQr+MDDOa3vtdgf6rHJd7S7vMW8OWn/qvv2Lyj4yN4taqp2XP9AwZ0bHC8rji4m0aiFSk4Osvgg2bWDWgElrn7hemmHW3mrQOD+IAB+AdbKNu+te37zfThOibyMFfSnea9F9Ct2941g1QKcflnV+vDEJFIdKahQb4BzM92Jl/7Pl/cfh+LqaIFY033wcys/g7/Nevs5MkCoDnF5+21FuRNTbFz/tbLP8O+gmnMmFhyqKqKdfxWVSlZiEjRirSGYWbDgIeACcC3sqlhrOo2hD9e+r9UX13JYWe1G4upe/fkySFoDUOdsyLSiXSWGsavgO8BKa+pNLOxZtZoZm3D1HqfPgx66A7GPXoqh589Yu+B+1pvBmsv1eft5bszuivdeyEinUZkCcPMLgRWufv0dNO5+0R3H92WHauqsEzNNr/9LYwbF6tRQOznuHGxz4NIdcd4GHeSF6q5S0QkZJE1SZnZfwJXALuAXsC+wJPu/sVU8xTseRj57IxWc5eIFFjJN0m5+w/cfZi7jwAuA15IlywKKp+d0V3t3gsR6TRK68a9QhozJj9XK+V54EQRkXyJutMbAHdvyHSFVKehJ8qJSIkqioSRs1K66kj3XohIiSr9JqlSfMZCvpq7RETyqPRrGHrGgohIQZR+wtBVRyIiBVH6CSOfN9mJiEib0k8Y6a46KqXOcBGRIlf6CSPVVUegIThEREIU+fMwspHV0CAagkNEBOgEQ4PknTrDRURC1XkThjrDRURC1XkThobgEBEJVedNGBqCQ0QkVKU/NEg6GoJDRCQ0nbeGISIioVLCEBGRQJQwREQkECUMEREJRAlDREQCUcIQEZFAlDBERCSQyBKGmQ03s3ozm29mc83sG1HFIiIimUV5494u4Nvu/rqZ7QNMN7Pn3H1ehDGJiEgKkdUw3H2Fu78e/30TMB84KKp40tKDmEREimNoEDMbARwLvJLku7HAWIDKKEaara2NPXhpy5bY+9YHMYGGHRGRLiXyByiZWV9gEjDB3Z9MN21WD1AKix7EJCIlrlM8QMnMegB/BGozJYvI6EFMIiJAtFdJGXA/MN/d/yuqODLSg5hERIBoaxgnAVcAZ5rZzPjr/AjjSU4PYhIRASLs9Hb3KYBFtf7AWju2x4+PNUNVVsaShTq8RaSLKYqrpIqeHsQkIqKhQUREJBglDBERCUQJQ0REAlHCEBGRQJQwREQkECUMEREJRAlDREQCUcIQEZFAlDBERCQQJQwREQlECUNERAJRwhARkUCUMEREJBAlDBERCUQJQ0REAlHCEBGRQJQwREQkECUMEREJRAlDREQCiTRhmNm5ZvammS00sxujjEVERNKLLGGYWTfgN8B5wIeBy83sw1HFIyIi6UVZw/gYsNDd33H3HcBjwKcjjEdERNLoHuG6DwLeTXi/FDih/URmNhYYG3+73czmFCC2XA0E1kQdRACKMzylECMozrCVSpwjw1hIlAnDknzme33gPhGYCGBmje4+Ot+B5UpxhqsU4iyFGEFxhq2U4gxjOVE2SS0Fhie8HwYsjygWERHJIMqE8RpwuJkdbGblwGXA/0UYj4iIpBFZk5S77zKzrwH/ALoBv3f3uRlmm5j/yEKhOMNVCnGWQoygOMPWpeI09726DURERPaiO71FRCQQJQwREQmkKBJGpiFCLObu+PdvmNlxQectcJxj4vG9YWZTzexfEr5bbGazzWxmWJe45RDn6Wa2IR7LTDP7YdB5CxzndxNinGNmzWbWP/5dQfanmf3ezFaluv+niI7NTHEWy7GZKc5iOTYzxVkMx+ZwM6s3s/lmNtfMvpFkmnCPT3eP9EWsw3sRcAhQDswCPtxumvOBvxG7d+PjwCtB5y1wnJ8A9o//fl5rnPH3i4GBRbI/Twf+3JF5Cxlnu+k/BbwQwf48FTgOmJPi+8iPzYBxRn5sBowz8mMzSJxFcmwOBY6L/74P8Fa+y85iqGEEGSLk08DDHjMN2M/Mhgact2BxuvtUd18XfzuN2L0lhZbLPimq/dnO5cCjeYolJXd/EXg/zSTFcGxmjLNIjs0g+zOVotqf7UR1bK5w99fjv28C5hMbQSNRqMdnMSSMZEOEtN/oVNMEmTcs2a7rGmKZvZUDz5rZdIsNd5IvQeM80cxmmdnfzOzILOcNQ+B1mVkf4FzgjwkfF2p/ZlIMx2a2ojo2g4r62AysWI5NMxsBHAu80u6rUI/PKIcGaRVkiJBU0wQaXiQkgddlZmcQ+6c8OeHjk9x9uZkNAp4zswXxs5go4nwdqHL3D8zsfOBp4PCA84Ylm3V9CnjJ3RPP+Aq1PzMphmMzsIiPzSCK4djMRuTHppn1JZaw/t3dN7b/OsksHT4+i6GGEWSIkFTTFHJ4kUDrMrOjgd8Bn3b3ta2fu/vy+M9VwFPEqoSRxOnuG939g/jvfwV6mNnAIPMWMs4El9Guyl/A/ZlJMRybgRTBsZlRkRyb2Yj02DSzHsSSRa27P5lkknCPz3x3zATouOkOvAMczO7OlyPbTXMBe3bcvBp03gLHWQksBD7R7vMKYJ+E36cC50YY5xB237T5MWBJfN8W1f6MT9ePWFtyRRT7M76OEaTupI382AwYZ+THZsA4Iz82g8RZDMdmfL88DPwqzTShHp+RN0l5iiFCzOz6+Pf3AH8l1tu/ENgCfDndvBHG+UNgAPBbMwPY5bGRLAcDT8U/6w78wd3/HmGclwDjzGwXsBW4zGNHUbHtT4DPAM+6++aE2Qu2P83sUWJX7gw0s6XALUCPhBgjPzYDxhn5sRkwzsiPzYBxQsTHJnAScAUw28xmxj+7idjJQV6OTw0NIiIigRRDH4aIiJQAJQwREQlECUNERAJRwhARkUCUMEREJBAlDBERCUQJQ0REAlHCEMlB/HkEZ8d//4mZ3R11TCL5Evmd3iIl7hbgx/GB5o4FLoo4HpG80Z3eIjkys0lAX+B0jz2XQKRTUpOUSA7M7CPEnny2XclCOjslDJEOij+5rJbYk8o2m9k5EYckkldKGCIdEH/S2pPAt919PnAb8KNIgxLJM/VhiIhIIKphiIhIIEoYIiISiBKGiIgEooQhIiKBKGGIiEggShgiIhKIEoaIiATy/wEEOzDtKL7twAAAAABJRU5ErkJggg==\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + } + ], "source": [ "# Using Autograd to calculate gradients for OLS\n", "from random import random, seed\n", @@ -4490,9 +4183,7 @@ { "cell_type": "markdown", "id": "d4dde158", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## Including Stochastic Gradient Descent with Autograd\n", "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**." @@ -4500,13 +4191,45 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 23, "id": "b86c1477", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Own inversion\n", + "[[4.1518464 ]\n", + " [2.88932963]]\n", + "Eigenvalues of Hessian Matrix:[0.27717669 4.61390898]\n", + "theta from own gd\n", + "[[4.1518464 ]\n", + " [2.88932963]]\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEWCAYAAAB1xKBvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAAAmT0lEQVR4nO3debgcdZ3v8fc3J8tJCBBIQhIIJ1GBgGwBIiCMTkZc2HG5ozAHLnjhZoiDgqK44AgumRmFGbhuo2ERyIkojsIogopjDmELcIIsgSTsCSSQPUBWsnzvH9UH+nR6qe6urqX783qefs7p7uruX1VX16d+S1WZuyMiIlJJv6QLICIi2aDAEBGRUBQYIiISigJDRERCUWCIiEgoCgwREQlFgSFNy8wuN7OupMsRJTObbGYvJ10OaU0KDImVmb1oZhvNbJ2ZvWpmN5jZ0KTLJSKVKTAkCae4+1BgInAY8NVki9OazKx/0mWQbFFgSGLc/VXgjwTBAYCZfcXMnjOzN8zsKTP7WN5z55jZvWZ2pZmtMbMXzOyEvOffYWZ35157FzAi//PM7FQze9LM1ppZt5kdkPfci2b2JTN73MzWm9l1ZjbKzO7Mvd+fzWy3YvPR20xkZheb2XIze8XMPp33fLeZnVc4H3n33cw+Y2bP5D7r22b2LjN7wMxeN7NbzGxgwWd+zcxW5srdmff4oNzyWWxmy8zsJ2Y2uKCcXzazV4GfmdkIM7s9t0xWm9k9ZqbtghSlFUMSY2ZjgROAZ/Mefg54H7Ar8E2gy8zG5D1/FLCQIAy+B1xnZpZ77ufA3Nxz3wbOzvus/YCbgYuAkcAdwO8KNsSfAD4E7AecAtwJfC33fv2Az5WZndG5Mu8FnAv8qFTAlHA8cARwNHAJMB3oBPYGDgLOKPisEbnPOhuYbmYTcs99N1f+icA+uWm+UfDa3YFxwBTgYuBlgmUyKje/Ol+QFKXAkCTcZmZvAC8By4HLep9w91+5+1J33+7uvwSeAY7Me+0id7/G3bcBNwJjgFFm1gG8B/hnd9/s7rOB3+W97lPA7939LnffAlwJDAaOyZvmB+6+zN2XAPcAD7r7X919M3ArQfNZKVuAb7n7Fne/A1gHTCgzfaHvuvvr7v4kMA/4k7s/7+6vEQRX4Wf3zufdwO+BT+aC8/8Cn3f31e7+BvAvwOl5r9sOXJZ77cZcuccA43Jlv8d1gjkpQYEhSfiou+8MTAb2J6/pyMz+t5k9mmsiWUuwd53ftPRq7z/uviH371BgT2CNu6/Pm3ZR3v975t939+0EgbVX3jTL8v7fWOR+uc75Ve6+Ne/+hgrTF6rms4vN554EtYQhwNy85feH3OO9Vrj7prz7VxDU8P5kZs+b2VeqKLO0GAWGJCa3d3wDwd4+ZjYOuAa4ABju7sMI9ratxFvkewXYzcx2ynusI+//pQTNMOQ+ywiae5bUPgehrSfYkPcaXef7FZvPpcBKgnA50N2H5W675gYY9OpTe3D3N9z9Ynd/J0Ez3BfM7Lg6yydNSoEhSbsa+JCZTQR2ItigrQDIdRwfFOZN3H0R0AN808wGmtnfEGwAe90CnGRmx5nZAIK2+83A/RHNRzmPAh83syFmtg9BH0e9eufzfcDJwK9ytaZrgKvMbA8AM9vLzD5S6k3M7GQz2ycXoK8D23I3kR0oMCRR7r4CuImgTf4p4N+BBwiaZA4G7qvi7f6BoFN8NUG/yE15n7MQOBP4AcGe+CkEw3vfjGA2KrkKeJNgnm4EZtb5fq8CawhqFTOB8919Qe65LxM0Mc0xs9eBP1O+L2Xf3DTrCJb7j929u87ySZMy9W+JiEgYqmGIiEgoCgwREQlFgSEiIqEoMEREJJRMnXxsxIgRPn78+KSLISKSKXPnzl3p7iMrT1lepgJj/Pjx9PT0JF0MEZFMMbNFlaeqTE1SIiISigJDRERCUWCIiEgoCgwREQlFgSEiIqEoMEREJBQFhoiIhKLAEBGRUBQYIiISigJDRERCaXhgmNn1ZrbczOYVee6LZuZmNqLR5RARkfrEUcO4ATi+8EEz2xv4ELA4hjKIiEidGh4Y7j6b4BrLha4CLgF0jVgRkQxIpA/DzE4Flrj7YyGmnWJmPWbWs2LFihhKJyIixcQeGGY2BLgU+EaY6d19urtPcvdJI0fWfTp3ERGpURI1jHcB7wAeM7MXgbHAI2Y2OoGyiIhISLFfQMndnwD26L2fC41J7r4y7rKIiEh4cQyrvRl4AJhgZi+b2bmN/kwREYlew2sY7n5GhefHN7oMIiJSPx3pLSIioSgwREQkFAWGiIiEosAQEZFQFBgiIhKKAkNEREJRYIiISCgKDBERCUWBISIioSgwREQkFAWGiIiEosAQEZFQFBgiIhKKAkNEREJRYIiISCgKDBERCUWBISIioSgwREQkFAWGiIiEosAQEZFQGh4YZna9mS03s3l5j11hZgvM7HEzu9XMhjW6HCIiUp84ahg3AMcXPHYXcJC7HwI8DXw1hnKIiEgdGh4Y7j4bWF3w2J/cfWvu7hxgbKPLISIi9UlDH8b/Ae4s9aSZTTGzHjPrWbFiRYzFEhGRfIkGhpldCmwFZpaaxt2nu/skd580cuTI+AonIiJ99E/qg83sbOBk4Dh396TKISIi4SQSGGZ2PPBl4G/dfUMSZRARkerEMaz2ZuABYIKZvWxm5wI/BHYG7jKzR83sJ40uh4iI1CeOUVJnuPsYdx/g7mPd/Tp338fd93b3ibnb+Y0uh4hIJs2cCePHQ79+wd+ZJbt8Gy6xPgwREalg5kyYMgU25FruFy0K7gN0dsZenDQMqxURkWIuvfTtsOi1YUPweAIUGCIiabV4cXWPN5gCQ0QkrTo6qnu8wRQYIiJpNW0aDBnS97EhQ4LHE6DAEBGpVaNHMHV2wvTpMG4cmAV/p09PpMMbNEpKRKQ2cY1g6uxMLCAKqYYhIlKLakcwpeh4ilqphiEiUotqRjCl7HiKWqmGISJSi2pGMEVxPEUKaigKDBGRWlQzgqne4yl6ayiLFoH72zWUmENDgSEiUotqRjDVezxFSo74VmCIiNSqsxNefBG2bw/+luqPqPd4ipQc8a3AEBFptHqPp0jJEd8KDBGROIStjRSTkiO+FRgiImmXkiO+FRgi0tpSMFw1lHpqKBHRgXsi0rqa5IC6uKiGISKtKyXDVbNCgSEirSslw1WzouGBYWbXm9lyM5uX99juZnaXmT2T+7tbo8shIi2qXB9FksNVG9l3UvDeI2D3KN42jhrGDcDxBY99Bfgfd98X+J/cfRGRaFU6pUZSw1XLlaveICny3h0wLpJyu3vDb8B4YF7e/YXAmNz/Y4CFYd7niCOOcBGR0MaNcw82m31v48a9PU1XV3DfLPjb1ZVcuYYODcqR/9iQIX3LVKm8Rd77CHCPYFtuHmy0G8rMxgO3u/tBuftr3X1Y3vNr3L1os5SZTQGmAHR0dByxaNGihpdXRJpEv37BJrOQWTA8NYyZM4NO8MWLg6aqE0+EO+54+/60adWPqCpVrlLGjQuG0haO6oKgRpR/TEaR954E9LhbdYXcUeoDI9+kSZO8p6enYeUUkSYzfnzQNFOodwNcSbENdKHCDXY95SqlN+BKvM6B9UNH8fOdpnDSsuvZiyV9no8qMJIaJbXMzMYA5P4uT6gcItLM6u2jKDbstlAtw3CLlauc3k74EqO3DBi6bhlnLruSRwcfxZa2QX2edwhZnSovqcD4LXB27v+zgf9OqBwi0szqPaVG2OG11Q7DLVau4cOLT2vGqk9/kRnn38vKtj3Kvu0QNnLSHnMZcON1fd57MUTSlt/wJikzuxmYDIwAlgGXAbcBtwAdwGLg7919daX3UpOUiMQqbNNR2Caucoo0fzkwgzM5mxkAnMd0fsCFtLOp9PsU6Z8xs7nuPqm+AsZQw3D3M9x9jLsPcPex7n6du69y9+Pcfd/c34phISJSVCOPZwjTdBTBMNxV3/wh686/GN+wga20sR14kXFM4afcOuafuPpjd/PoLxfy0y3n0d51bRBQpTTyGJIohlrFddOwWpEWUM0w166uYNhpuWGoUZdn6tS6h+EunrPEZ5x/r5+732y/yK72dfSdhzfbBvmLF1zh27ZsK1+ukPMO9HgUh0hE8SZx3RQYIk2u2gAIc5xFqc+J8diLlx5a+lZAvKv/i28Vc5it8WX9Rtc2D1XMR1SBEcuw2qioD0OkhMJjBWo5NiANqh0GW8txFmGOZajTyw+/Qvf1z9P9l+10P9/Bc1uDJqRhtpa/HbWQyUdvYvLpozn4Y/vQ1j6g/mNFKoiqD0OnNxfJumY6RXepDuZSo5A6Ooq/plw7fqkz1J6dG7hZwzIrHhBj3gqIC45+8e2AGHhU/fOQlCiqKXHd1CQlUkStzTJp09W142kxKs1LLX0YpT6jiv6Plx5a6l1T7/XzJtzt+wx4oU8T02mj5/hVH+32v/5igW/dvDXcfDe4Hwb1YYiIu5feAJolXbLqlAo+s8od39X0R5T6nDLhFGlARDEPVYoqMNSHIenTLO3xcan39BdpUe78SlFupyqd7sOMJQ8tyTUxbaP7+Q6e3TIeCPog3j9qIZOP2sjkT43mkE/sS9vAtujK1iBR9WEkXmuo5qYaRguIY5hks2mWZRZn01pXl3tbW9HPe5m9+tQgTh09x//jtFn+yM/n116DiFqVNRLUJCVNKan2+CROcR2lrJffvXzwRTl/XV2+ZfRevh18e8F6tp7BfuUul6cvIPLVsIOgwJDmlER7fLPsoTeDYsEQwffz8sNLfeZn7vVrxnzd1zO4z3v1BsfmEXv6thtuatisRabUTlVbW8lAjSow1Ich6ZJEe3yz9AE0qxq+nyU9r3D3z3J9EM/tzTNb3gHAIjro4KWq3it1wlxLo+C4kqj6MBQYki4xHFS1gygusiONE+L7KRUQu/Ia7x+1IOik/uQoDjvrQCzr33UNJ0TMzMkHRapS7+moC4U5MV2pA6TSeOBUvkaedC9NSnwP63fagykHzGa/gS8w9j1j6Pzxsdyy4FD23305tx/+dTYN35M1thu/bf8UX/jkEg7vPADL6nedL+y1NKo95XoYUbRrxXVTH4ZUJWzbdxb7MLJY5lp1dfm29r79DusY4mfQ5buy1k8ZNcf//dRZPrfrqaCTulLnebXLLY0DCvLLVGK0V/5AEdTpLVJBNSOu0rhRcC9drmY5uruEJXNf8Z9fcJ9P2f9u32/A834GXf4C43wb5sv6jfbbD7/07YAoVGnZpO1suPUKUcaoAkN9GNK8st43Ua4/56yzsj1vBZY+8mrQB/HnrXQ/tzdPl+iDOPTv96t8oFyU33tWBkRUONhVnd4ilWTlx15KufJDpuetVEDswmu8f4+FTD5yA5M/NYqJnwwREIWi/N6zvtORo05vkUKFncAnnrhj52AEV0eLTalOy8WLi3d8pnjelj7yKjd/9n7+8YDZTBj4AnsdMZp/+OEx/GLBoey32wquPLmbnhnzWb15KL9bdiQX/24yR5x5AG2/+kX1HftRLptm6CSPUhTtWnHd1IchJZVqx43g6miJibItPk5dXb5lzFjfjvnK/qP8C/2ueqvou7DWT97jQb/y5FneM6NEH0Te+9Tcf1BtP0WpabPQhxEC6vQWydOMncAZ2lgt/eurfvNn7/Nr9/y6byg4knoj7X77xEsrB0ShOL7TMMs4rcFchaYIDODzwJPAPOBmoL3c9AoMKalZTvFdqNaNVYM3cr0B8Y8H3O0TBj731uJexN7RbeTj+E6bcUejiMwHBrAX8AIwOHf/FuCccq9RYEhJWf/hR3xyvahrJqUCYhfW+kl7POhXnDTLH77xSd8e5UY+ju+0WXc0CjRLYLwE7E5wqdjbgQ+Xe40CQ0rKUPPNDqIuewQb2lIBsTOv9QmIrT+7qW/QDR8e3UY+ju806zsaIWU+MIJ54EJgHbACmFlimilAD9DT0dER6UKUJpPVtuaoN1o17DWHDYgtG7e8/aJiG/SBA90HDIhuI9/o7zTLOxpVyHxgALsBfwFGAgOA24Azy71GNQxpSlE3i4QIoFceW+b3f+QyX9l/lG/D/AXG+Rl0lQ+IsJ8zfHi2gjurOxpViCow+jdmsG4oHwRecPcVAGb2G+AYoCvBMonEr6Oj+IFmtY71nzZthyPEt7cP5sH9z+GmA2fT/exeHPbmHK7hCnYimGY8i5gx8Dz46XTazjkr3OeUOk5k1SoYOrS2siehs1OXAA4pyQP3FgNHm9kQMzPgOGB+guWRtGmVs7FGfRBeZydrv34l63cehWMsYU/O3HQNx/zxcmY+NZF37rqa/2y/+K2w6NX25ibaLv/n8J9TKtDMggB0D/5OmdK8312riaKaUusN+CawgGBY7QxgULnp1STVQlqkbfkt+c0iw4cHtyqaSF55bJn/4nP3+fnvvtv3H/hsnz6IE0c+5N87cZY/dENeE1MUzWDFvqNS79tknchZQ9b7MGq5KTBaSIuMXtlByKCsOiAKRbV8C9v/i71nrf0xLdC3EJfYAgP4M3BoFB9W702B0UKi7giucw8+NiU2ulvHjPVfXnSfTz3wbj+gloAo1KgaXJRB1Eo1zAaLMzAOJxjN9DNgTBQfWutNgdFCoqxhTJ1aOoDStiEqUc5tWPUBMXXq2xfXaWsL7udrxB58VBv6Vq1hNkjsTVLAJ4DHgMvIHZ0d902B0UKi2vB0dZUPi5RsiF59Yrn/8qL7fOWAUUXLt7Z9lD94/bzwNYipU4vPZ2FoNEIUQdQiR2DHJdbAAAw4CDgfWAm8DJwVRQGquSkwWkwUG55y7eoJboh6A6KwienTXOsbaa8/KEtdtrOtrTEzFDXVMCIVVWBUHFZrZvcCS4CrCE7ncQ4wGTjSzKbXM0JLpKzOzuCCN9u3B39rGStf6liBQg2+vsGyeSt44ITLWTVwNNutHxsPfg+3Xf0CM548jPG7rua7J3Tz4PVPMn3j2bR3XRtc7Mcs+Dt9evXzvm1bdY/HrdKQ6Yxd76NlVEoUgpqFlXhufhSpFfamGkYdWnXESZgaRgP6MJbNW+63fP5+/8xB3f7uQc/4GXT5Ovo2sW0d2B6ci6kR0lzDCNvc2KrrbAOQhmG1wDujKETYW6oCI0src5wjTqJeLvW+X7F5L9yARrAcCgOi9+2H8rqfMPIhX9tevG+iYU0sSfZhVKLmptilIjDivqUmMLI25C+uH2jUyyWqK66VOoNqHX0X5QLi+BEP+b8dP8vnXPuEv7n+zeAFSXTiVhollRR1aMdOgZGkrO0hxfUDrWa5hKk51LqcK9UqavjOSgXETrxRPCCimpdmpGUROwVGktK6h1RqIxzXDzTscglbc6h1OVczMqpEbaXugCiUhlppWppR07AsWowCI0lp3EMq9yOM6wcadrlEPV2hMMdd9N5yls1b7r/6wv3+Twd3+4GDnq4/IIpJcoMd1zoQdh7TEl4tQoGRpDTuIVXauMbxA4265hDm/YrNV8gaxjbr59sxX8KefgZdbwXER4Y/7P/6kVn+wDU1BkQaN4aN2MkpnM+pU9P3uxB3V2AkL20bhbQ0k0XdN1Hu/UoFynHH7bA8thd8VuH9N/sN8qfP/bf6ahDlytRs64fOVJspUQWGBe+VDZMmTfKenp6ki5FO48cXvwjPuHHBQW9pMnPmDhf4YciQ6g9QKzXPZsGmKmc7xl18gAk8SweLcYw2tu/4uiiWVVq/h6jLVer9ijELDr6UxJjZXHefVO/7JHkBJYlSlo6M7ewMwqHeo5lLHcVdsBPUD+fo9nm8es3tbFu/mTYrsZMU9qjwWsoUxXvXI+r1o5r5afBR9BIfBUaziGojHJc6T/uxYv5K1u+8R+jpd928nKPPO4gBQwaU3oBFsWFr5HvXI+r1o9zV9vKldadFahNFu1Zct1T1YWRd2vpgKlj+1Ar/ry/e7xcc3O0H5UYxBafbGNy3byJMO3oj+xnS2ocRtVLzOXVqptarVoE6vaVmGdioFQsIcB/COv/w8If9Xz48y+//6eO+5fobaxup08jAzFgY16xV5rMJKDCkdlEOsYxooxE2IEKNYqo0skobOWkxCgypXVRDLMsNraywMV6xYGV0AVFPeVNWs4qVwrNlNEVgAMOA/wIWAPOB95abPtWBkaUfX1Q1jEoHyOVtjFcsWOm//tID/tlDuv2S/lf6C4zzbZgvYm//152+9VZAbH5jc9RzW7m8rXicgMKzpTRLYNwInJf7fyAwrNz0qQ2MrP34oipviFNwrBowyg9uX/jWQ+dwXTRXlKtFWg5uTAOFZ0uJKjASG1ZrZrsA7weuA3D3N919bVLlqcull/Y9CA2C+5demkx5KolqiGWIoaLDtixn9JA3mPahbu77yRNcv/c3aWdT34niWlZpHfKahLQeLyK1qXQFw6hEkTq13ICJwEPADcBfgWuBnYpMNwXoAXo6OjoiTd3INOuea6lmtq4u37rnWN/OjqfYKLxt37vgO0tyWWWtJthIqmE0jxDrNVlvkgImAVuBo3L3/x/w7XKvSW2TVKN+fCk7u+mW/u1+9+6n+YaCJqVtueDYITyKbYyT3lBlqa+pkRSezSPEb6oZAmM08GLe/fcBvy/3mtQGRiN+fAn/oLfuObboSriFEteKzj9TbLmNsQ74Sg+FZ3MIUWvPfGAE88A9wITc/5cDV5SbPrWB4R79jy/mPfGVT6/y31zygH/u0G4/pH2Bb6P4SliyCaqaJiWdFlskOjHWMBI9W62ZTSTouxgIPA982t3XlJq+pc5W269f8LUXiujMn6ueWc3sa5+m+4+b6V44msc3TQBgMBs4dvf5/GrDSQzbtGzHF7a1wbZtOz5ez9lY03qGV5EsCHH256Y4W627P+ruk9z9EHf/aLmwaDkRj+hZ9cxqbv3yHC6ceDeHDl7IiP125+PfO5prHnsPewxZx3c+2M29P36ctW/0567vL2DYTlt3fJMhQ4IVM+qz4mrETnhxjYaR7IjzxKNRVFPiuoVukmqGttk6+zAKm5h632Iw6/2Du/f4dz44y+/98WM7HihX7HPBffjwPqOkstz8llnqqJYa0Qx9GNXeQgVGM/2oqtgwr3p2td/6lTl+4cRuP7R9gRvbwgVEoSQ23s30nTWSglVqFFVgNN8V9yq1h8+cGRwktnhx0LwzbVp6rxlRxurn1gR9EH/YRPeC0Ty+aV+cfgxmA8fstoDJh7/O5I/vznvOnMCgXQaFf+MG952U1CTfS0Ml9d1I5kXVh9F8gVHuRzVjRjSXBk1AwwKikDqg00vfjdRIgVFKuR8VZOYHVyog2tnIsbvNjy4gChUbcTFwIOy8M6xerb3/JEV1LXRpOVEFRuL9EtXc6u7DSPEpPEr1QbSzwY/bba5/+7hZfs+PHvNNr22q7QOq6ajOn3b4cPcBA9S/kBbNMKBDYof6MMoo1R6eoir96ufWcM91vTWIUTy2cb/G1SDq2TNN0TITkdqohlGLRo3GCbHXt+rZ1X7bV+f4RYd1+8TB8/vUID6w21z/1gdm+ewfPFp7DaKcekbXJF0ri3OPWnvv0qTQsNoaRb1RKBFCb1w1PbmAKFTPRj/JoZxxDrfV0F5pYgqMJBQLmxIb1BfoSC4gCtWz0U9yQxpnWOkYB2liCoy4lTjdd6mT8W3HkguIEGWvaqOfVFNNnM1hSTe9iTRQVIGR6LmksmTbJV/Z4ap6/bduYhttRae3cR2874JDy3dYx3VeoHrPNdPZGXRwb98e/I1rCGcjrpBXapnranwilUWROnHd4qxhrH5+jf/31+b45w+f5YcNfqr86b5r2XtvpjbzRtVAol5G5d6vmb4PkQKoSSpahQHR20k9iI3+d8Me8TWDRhUNjNAXDirULG3mjd7QRhlGlZa5RklJk4oqMJrzOIwQ1rywlnuuXUj3HzbSPX8Uj26cgNOPQWzimGHzmXzYa0z+2G4cedYE2oe1R3+UbbOcFyhLx2k0yzIXqVJUx2H0j6IwWbB20WtBQNy5ge6nRvHXjRNwjnorIC4/enZeQBy24xv0hkJUJ8jr6Ci+oc1am3mWrmXRLMtcJCFNGxjFA+LI8AFRTGdndB2+06YVr7HUcyGiJGRpI9wsy1wkIU0TGOUC4r3D5nPZUbOZ/NFhHHX2/uEDopGirrEkJUsb4WZZ5iIJyWwfRvGA6PdWQEye+FpeQLQnXPKcZr3mQ7POl0iTaMnTm++75wF+6p7/mZ2AyKdTU4tIQpomMMysDegBlrj7yeWnneSDuDcbAVEoS6OJRKSpNNMoqQuB+cAulSbcb+wGHnuC4n0QaW8WydJoIhGRIhI9NYiZjQVOAq4NM/3Oo4YUr030NvcsWhSMs1+0KLjfqFNt1EKnnhCRjEv6XFJXA5cAJY+aMrMpZtZjZj0rVqwoPtGll+5wnic2bAgeT4tp04I+i3xpHU0kIlJEYoFhZicDy919brnp3H26u09y90kjR44sPlEWmnvqPQGg7CiukzeKCJBsH8axwKlmdiLQDuxiZl3ufmbV75SVg8eiPPCv1RWOOutthgQtY5EGSayG4e5fdfex7j4eOB34S01hAdU392jPNPuy0Awp0mSS7sOIRjXNPVnoIJfKstAMKdJkUhEY7t5d6RiMisJe5KdV9kybvRalUWcisUtFYMSqFfZMW6EWpVFnIrFrvcBohT3TVqhFadSZSOxaLzCK7ZkOHAjr1jVP800r1KIguWuNi7So1gmM3jb9s86CwYNh+PBgz3T48KDZZtWq5mm+aYValIjErjUCo7BNf9Uq2LgRZsyAoUNhy5a+02e9+Ubt+yLSAK0RGOXa9Jux+Ubt+yLSAGk4W23jlQuFrBwlXi0dVS4iEWuNGka5Nn0134iIhJKtwHjiidpGMpULBTXfiIiEkvgV96oxycx7eu9Ue3nTtF9gSUSkQZrmEq3V6BMYoMubioiEEFVgZKtJqlCWRzKJiGRMtgMj6yOZREQyJLuBoZFMIiKxylZgDByokUwiIgnJ1oF7Bx8MPT2VpxMRkchlq4YhIiKJUWCIiEgoCgwREQlFgSEiIqEkFhhmtreZzTKz+Wb2pJldmFRZRESksiRHSW0FLnb3R8xsZ2Cumd3l7k8lWCYRESkhsRqGu7/i7o/k/n8DmA/slVR5RESkvFT0YZjZeOAw4MEiz00xsx4z61mxYkXsZRMRkUDigWFmQ4FfAxe5++uFz7v7dHef5O6TRo4cGX8BRUQESDgwzGwAQVjMdPffJFkWEREpL8lRUgZcB8x39/9IqhwiIhJOkjWMY4GzgA+Y2aO524kJlkdERMpIbFitu98LWFKfLyIi1Um801tERLJBgSEiIqEoMEREJBQFhoiIhKLAEBGRUBQYIiISigJDRERCUWCIiEgoCgwREQlFgSEiIqEoMEREJBQFhoiIhKLAEBGRUBQYIiISigJDRERCUWCIiEgoCgwREQlFgSEiIqEoMEREJBQFhoiIhJJoYJjZ8Wa20MyeNbOvJFkWEREpL7HAMLM24EfACcC7gTPM7N1JlUdERMpLsoZxJPCsuz/v7m8CvwBOS7A8IiJSRv8EP3sv4KW8+y8DRxVOZGZTgCm5u5vNbF4MZavXCGBl0oUIQeWMThbKCCpn1LJSzglRvEmSgWFFHvMdHnCfDkwHMLMed5/U6ILVS+WMVhbKmYUygsoZtSyVM4r3SbJJ6mVg77z7Y4GlCZVFREQqSDIwHgb2NbN3mNlA4HTgtwmWR0REykisScrdt5rZBcAfgTbgend/ssLLpje+ZJFQOaOVhXJmoYygckatpcpp7jt0G4iIiOxAR3qLiEgoCgwREQklFYFR6RQhFvh+7vnHzezwsK+NuZydufI9bmb3m9mhec+9aGZPmNmjUQ1xq6Ock83stVxZHjWzb4R9bczl/FJeGeeZ2TYz2z33XCzL08yuN7PlpY7/SdG6WamcaVk3K5UzLetmpXKmYd3c28xmmdl8M3vSzC4sMk2066e7J3oj6PB+DngnMBB4DHh3wTQnAncSHLtxNPBg2NfGXM5jgN1y/5/QW87c/ReBESlZnpOB22t5bZzlLJj+FOAvCSzP9wOHA/NKPJ/4uhmynImvmyHLmfi6GaacKVk3xwCH5/7fGXi60dvONNQwwpwi5DTgJg/MAYaZ2ZiQr42tnO5+v7uvyd2dQ3BsSdzqWSapWp4FzgBublBZSnL32cDqMpOkYd2sWM6UrJthlmcpqVqeBZJaN19x90dy/78BzCc4g0a+SNfPNARGsVOEFM50qWnCvDYq1X7WuQTJ3suBP5nZXAtOd9IoYcv5XjN7zMzuNLMDq3xtFEJ/lpkNAY4Hfp33cFzLs5I0rJvVSmrdDCvpdTO0tKybZjYeOAx4sOCpSNfPJE8N0ivMKUJKTRPq9CIRCf1ZZvZ3BD/Kv8l7+Fh3X2pmewB3mdmC3F5MEuV8BBjn7uvM7ETgNmDfkK+NSjWfdQpwn7vn7/HFtTwrScO6GVrC62YYaVg3q5H4umlmQwkC6yJ3f73w6SIvqXn9TEMNI8wpQkpNE+fpRUJ9lpkdAlwLnObuq3ofd/elub/LgVsJqoSJlNPdX3f3dbn/7wAGmNmIMK+Ns5x5Tqegyh/j8qwkDetmKClYNytKybpZjUTXTTMbQBAWM939N0UmiXb9bHTHTIiOm/7A88A7eLvz5cCCaU6ib8fNQ2FfG3M5O4BngWMKHt8J2Dnv//uB4xMs52jePmjzSGBxbtmmannmptuVoC15pySWZ+4zxlO6kzbxdTNkORNfN0OWM/F1M0w507Bu5pbLTcDVZaaJdP1MvEnKS5wixMzOzz3/E+AOgt7+Z4ENwKfLvTbBcn4DGA782MwAtnpwJstRwK25x/oDP3f3PyRYzv8FTDWzrcBG4HQP1qK0LU+AjwF/cvf1eS+PbXma2c0EI3dGmNnLwGXAgLwyJr5uhixn4utmyHImvm6GLCckvG4CxwJnAU+Y2aO5x75GsHPQkPVTpwYREZFQ0tCHISIiGaDAEBGRUBQYIiISigJDRERCUWCIiEgoCgwREQlFgSEiIqEoMETqkLsewYdy/3/HzL6fdJlEGiXxI71FMu4y4Fu5E80dBpyacHlEGkZHeovUyczuBoYCkz24LoFIU1KTlEgdzOxggiufbVZYSLNTYIjUKHflspkEVypbb2YfSbhIIg2lwBCpQe5Ka78BLnb3+cC3gcsTLZRIg6kPQ0REQlENQ0REQlFgiIhIKAoMEREJRYEhIiKhKDBERCQUBYaIiISiwBARkVD+Py/jcaIE7+D+AAAAAElFTkSuQmCC\n", + "text/plain": [ + "
" + ] + }, + "metadata": { + "needs_background": "light" + }, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "theta from own sdg\n", + "[[4.04721612]\n", + " [2.95074375]]\n" + ] + } + ], "source": [ "# Using Autograd to calculate gradients using SGD\n", "# OLS example\n", @@ -4586,22 +4309,26 @@ { "cell_type": "markdown", "id": "27576d79", - "metadata": { - "editable": true - }, + "metadata": {}, "source": [ "## And Logistic Regression" ] }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 24, "id": "c6d0db97", - "metadata": { - "collapsed": false, - "editable": true - }, - "outputs": [], + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Initial loss: 2.772588722239781\n", + "Trained loss: 1.067270675787016\n" + ] + } + ], "source": [ "import autograd.numpy as np\n", "from autograd import grad\n", @@ -4637,9 +4364,35 @@ "\n", "print(\"Trained loss:\", training_loss(weights))" ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "0c4cb324", + "metadata": {}, + "outputs": [], + "source": [] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.8.12" + } + }, "nbformat": 4, "nbformat_minor": 5 }