diff --git a/doc/pub/week39/html/week39-reveal.html b/doc/pub/week39/html/week39-reveal.html
index c384bea00..bdbae8e52 100644
--- a/doc/pub/week39/html/week39-reveal.html
+++ b/doc/pub/week39/html/week39-reveal.html
@@ -228,7 +228,7 @@ MathJax.Hub.Config({
Stochastic Gradient descent with examples and automatic differentiation
-
Video of lecture
+
Video of lecture
Whiteboard notes TBA at https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf
diff --git a/doc/pub/week39/html/week39-solarized.html b/doc/pub/week39/html/week39-solarized.html
index 7f05ab1ce..99854828b 100644
--- a/doc/pub/week39/html/week39-solarized.html
+++ b/doc/pub/week39/html/week39-solarized.html
@@ -364,7 +364,7 @@ MathJax.Hub.Config({
Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent
Stochastic Gradient descent with examples and automatic differentiation
- Video of lecture
+ Video of lecture
Whiteboard notes TBA at https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf
Readings and Videos:
diff --git a/doc/pub/week39/html/week39.html b/doc/pub/week39/html/week39.html
index 3c97fecef..1893f246a 100644
--- a/doc/pub/week39/html/week39.html
+++ b/doc/pub/week39/html/week39.html
@@ -441,7 +441,7 @@ MathJax.Hub.Config({
Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent
Stochastic Gradient descent with examples and automatic differentiation
- Video of lecture
+ Video of lecture
Whiteboard notes TBA at https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf
Readings and Videos:
diff --git a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz
index 1f109e46c..40bcc18f5 100644
Binary files a/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz and b/doc/pub/week39/ipynb/ipynb-week39-src.tar.gz differ
diff --git a/doc/pub/week39/ipynb/week39.ipynb b/doc/pub/week39/ipynb/week39.ipynb
index 829ff588c..9bb0ab728 100644
--- a/doc/pub/week39/ipynb/week39.ipynb
+++ b/doc/pub/week39/ipynb/week39.ipynb
@@ -2,8 +2,10 @@
"cells": [
{
"cell_type": "markdown",
- "id": "9d3794a3",
- "metadata": {},
+ "id": "583ae0c0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
@@ -12,8 +14,10 @@
},
{
"cell_type": "markdown",
- "id": "1dd96d80",
- "metadata": {},
+ "id": "bf393969",
+ "metadata": {
+ "editable": true
+ },
"source": [
"# Week 39: Optimization and Gradient Methods\n",
"**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n",
@@ -23,8 +27,10 @@
},
{
"cell_type": "markdown",
- "id": "c7488059",
- "metadata": {},
+ "id": "71ed0c72",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Plan for week 39\n",
"\n",
@@ -49,7 +55,7 @@
"\n",
" * Stochastic Gradient descent with examples and automatic differentiation\n",
"\n",
- " * [Video of lecture](https://youtu.be/)\n",
+ " * [Video of lecture](https://youtu.be/bFRVuIJroHs)\n",
"\n",
" * Whiteboard notes TBA at \n",
"\n",
@@ -68,8 +74,10 @@
},
{
"cell_type": "markdown",
- "id": "63e36f88",
- "metadata": {},
+ "id": "e51abfa3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Optimization, the central part of any Machine Learning algortithm\n",
"\n",
@@ -87,8 +95,10 @@
},
{
"cell_type": "markdown",
- "id": "f23813a0",
- "metadata": {},
+ "id": "69471b3f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Revisiting our Logistic Regression case\n",
"\n",
@@ -102,8 +112,10 @@
},
{
"cell_type": "markdown",
- "id": "62e71897",
- "metadata": {},
+ "id": "c7981416",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{align*}\n",
@@ -115,16 +127,20 @@
},
{
"cell_type": "markdown",
- "id": "973f7f7b",
- "metadata": {},
+ "id": "4095b05f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\boldsymbol{\\beta}$ are the weights we wish to extract from data, in our case $\\beta_0$ and $\\beta_1$."
]
},
{
"cell_type": "markdown",
- "id": "2d5b7dc0",
- "metadata": {},
+ "id": "5e9fe500",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The equations to solve\n",
"\n",
@@ -137,8 +153,10 @@
},
{
"cell_type": "markdown",
- "id": "05c38ec5",
- "metadata": {},
+ "id": "d418778c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}} = -\\boldsymbol{X}^T\\left(\\boldsymbol{y}-\\boldsymbol{p}\\right).\n",
@@ -147,8 +165,10 @@
},
{
"cell_type": "markdown",
- "id": "cdac7312",
- "metadata": {},
+ "id": "4673766e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"If we in addition define a diagonal matrix $\\boldsymbol{W}$ with elements \n",
"$p(y_i\\vert x_i,\\boldsymbol{\\beta})(1-p(y_i\\vert x_i,\\boldsymbol{\\beta})$, we can obtain a compact expression of the second derivative as"
@@ -156,8 +176,10 @@
},
{
"cell_type": "markdown",
- "id": "0b49a2d8",
- "metadata": {},
+ "id": "34a20bca",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T} = \\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X}.\n",
@@ -166,16 +188,20 @@
},
{
"cell_type": "markdown",
- "id": "514a84be",
- "metadata": {},
+ "id": "e4347ab8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This defines what is called the Hessian matrix."
]
},
{
"cell_type": "markdown",
- "id": "cd3c8f0e",
- "metadata": {},
+ "id": "a7e5f736",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Solving using Newton-Raphson's method\n",
"\n",
@@ -186,8 +212,10 @@
},
{
"cell_type": "markdown",
- "id": "8b8f77b0",
- "metadata": {},
+ "id": "61da07b0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\frac{\\partial^2 \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}\\partial \\boldsymbol{\\beta}^T}\\right)^{-1}_{\\boldsymbol{\\beta}^{\\mathrm{old}}}\\times \\left(\\frac{\\partial \\mathcal{C}(\\boldsymbol{\\beta})}{\\partial \\boldsymbol{\\beta}}\\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}},\n",
@@ -196,16 +224,20 @@
},
{
"cell_type": "markdown",
- "id": "108d8194",
- "metadata": {},
+ "id": "74eb4429",
+ "metadata": {
+ "editable": true
+ },
"source": [
"or in matrix form as"
]
},
{
"cell_type": "markdown",
- "id": "95e1653c",
- "metadata": {},
+ "id": "88d606a1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\beta}^{\\mathrm{new}} = \\boldsymbol{\\beta}^{\\mathrm{old}}-\\left(\\boldsymbol{X}^T\\boldsymbol{W}\\boldsymbol{X} \\right)^{-1}\\times \\left(-\\boldsymbol{X}^T(\\boldsymbol{y}-\\boldsymbol{p}) \\right)_{\\boldsymbol{\\beta}^{\\mathrm{old}}}.\n",
@@ -214,8 +246,10 @@
},
{
"cell_type": "markdown",
- "id": "ba92e6a6",
- "metadata": {},
+ "id": "91781994",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The right-hand side is computed with the old values of $\\beta$. \n",
"\n",
@@ -224,8 +258,10 @@
},
{
"cell_type": "markdown",
- "id": "76c0a7be",
- "metadata": {},
+ "id": "36d56722",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Brief reminder on Newton-Raphson's method\n",
"\n",
@@ -242,8 +278,10 @@
},
{
"cell_type": "markdown",
- "id": "60fdb840",
- "metadata": {},
+ "id": "2b4ea29c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The equations\n",
"\n",
@@ -256,8 +294,10 @@
},
{
"cell_type": "markdown",
- "id": "7d1e7ba3",
- "metadata": {},
+ "id": "68947156",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"
\n",
@@ -270,8 +310,10 @@
},
{
"cell_type": "markdown",
- "id": "9f8797e3",
- "metadata": {},
+ "id": "079d2d11",
+ "metadata": {
+ "editable": true
+ },
"source": [
"For small enough values of the function and for well-behaved\n",
"functions, the terms beyond linear are unimportant, hence we obtain"
@@ -279,8 +321,10 @@
},
{
"cell_type": "markdown",
- "id": "55ec6b4d",
- "metadata": {},
+ "id": "0a8251fc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(x)+(s-x)f'(x)\\approx 0,\n",
@@ -289,16 +333,20 @@
},
{
"cell_type": "markdown",
- "id": "b170829a",
- "metadata": {},
+ "id": "b15408dc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"yielding"
]
},
{
"cell_type": "markdown",
- "id": "ede51475",
- "metadata": {},
+ "id": "a9fc1544",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"s\\approx x-\\frac{f(x)}{f'(x)}.\n",
@@ -307,16 +355,20 @@
},
{
"cell_type": "markdown",
- "id": "faaa21eb",
- "metadata": {},
+ "id": "f8d344f4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Having in mind an iterative procedure, it is natural to start iterating with"
]
},
{
"cell_type": "markdown",
- "id": "0df729a1",
- "metadata": {},
+ "id": "6da8a298",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"x_{n+1}=x_n-\\frac{f(x_n)}{f'(x_n)}.\n",
@@ -325,8 +377,10 @@
},
{
"cell_type": "markdown",
- "id": "61c17cb2",
- "metadata": {},
+ "id": "11188fc1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Simple geometric interpretation\n",
"\n",
@@ -345,8 +399,10 @@
},
{
"cell_type": "markdown",
- "id": "3bfa64c0",
- "metadata": {},
+ "id": "7e36eb5a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Extending to more than one variable\n",
"\n",
@@ -356,8 +412,10 @@
},
{
"cell_type": "markdown",
- "id": "ee3079ee",
- "metadata": {},
+ "id": "965fac17",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{array}{cc} f_1(x_1,x_2) &=0\\\\\n",
@@ -367,16 +425,20 @@
},
{
"cell_type": "markdown",
- "id": "c8d7c6e8",
- "metadata": {},
+ "id": "e8ce62ac",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which we Taylor expand to obtain"
]
},
{
"cell_type": "markdown",
- "id": "11f8dddf",
- "metadata": {},
+ "id": "59889f63",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\begin{array}{cc} 0=f_1(x_1+h_1,x_2+h_2)=&f_1(x_1,x_2)+h_1\n",
@@ -391,16 +453,20 @@
},
{
"cell_type": "markdown",
- "id": "67c6a15d",
- "metadata": {},
+ "id": "7ed9fe19",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Defining the Jacobian matrix ${\\bf \\boldsymbol{J}}$ we have"
]
},
{
"cell_type": "markdown",
- "id": "5bdd724a",
- "metadata": {},
+ "id": "d472c301",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"{\\bf \\boldsymbol{J}}=\\left( \\begin{array}{cc}\n",
@@ -412,16 +478,20 @@
},
{
"cell_type": "markdown",
- "id": "5b0ceecc",
- "metadata": {},
+ "id": "936556af",
+ "metadata": {
+ "editable": true
+ },
"source": [
"we can rephrase Newton's method as"
]
},
{
"cell_type": "markdown",
- "id": "f4d450ce",
- "metadata": {},
+ "id": "3f2f7069",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\left(\\begin{array}{c} x_1^{n+1} \\\\ x_2^{n+1} \\end{array} \\right)=\n",
@@ -432,16 +502,20 @@
},
{
"cell_type": "markdown",
- "id": "c9336abc",
- "metadata": {},
+ "id": "2664c763",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have defined"
]
},
{
"cell_type": "markdown",
- "id": "5bf43834",
- "metadata": {},
+ "id": "9ce01b74",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\left(\\begin{array}{c} h_1^{n} \\\\ h_2^{n} \\end{array} \\right)=\n",
@@ -452,8 +526,10 @@
},
{
"cell_type": "markdown",
- "id": "df80d402",
- "metadata": {},
+ "id": "813e3282",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We need thus to compute the inverse of the Jacobian matrix and it\n",
"is to understand that difficulties may\n",
@@ -465,8 +541,10 @@
},
{
"cell_type": "markdown",
- "id": "567d5244",
- "metadata": {},
+ "id": "16f16e4f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Steepest descent\n",
"\n",
@@ -480,8 +558,10 @@
},
{
"cell_type": "markdown",
- "id": "40b81b1a",
- "metadata": {},
+ "id": "30f1159d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k),\n",
@@ -490,8 +570,10 @@
},
{
"cell_type": "markdown",
- "id": "1702738e",
- "metadata": {},
+ "id": "e6ae1d1f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with $\\gamma_k > 0$.\n",
"\n",
@@ -502,8 +584,10 @@
},
{
"cell_type": "markdown",
- "id": "62b2b527",
- "metadata": {},
+ "id": "f34c4b79",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More on Steepest descent\n",
"\n",
@@ -515,8 +599,10 @@
},
{
"cell_type": "markdown",
- "id": "c7e8962e",
- "metadata": {},
+ "id": "ea2806d0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{x}_{k+1} = \\mathbf{x}_k - \\gamma_k \\nabla F(\\mathbf{x}_k), \\ \\ k \\geq 0.\n",
@@ -525,8 +611,10 @@
},
{
"cell_type": "markdown",
- "id": "d70bba43",
- "metadata": {},
+ "id": "1647c727",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The parameter $\\gamma_k$ is often referred to as the step length or\n",
"the learning rate within the context of Machine Learning."
@@ -534,8 +622,10 @@
},
{
"cell_type": "markdown",
- "id": "75723df7",
- "metadata": {},
+ "id": "665a4235",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The ideal\n",
"\n",
@@ -560,8 +650,10 @@
},
{
"cell_type": "markdown",
- "id": "654b214d",
- "metadata": {},
+ "id": "72016284",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The sensitiveness of the gradient descent\n",
"\n",
@@ -580,8 +672,10 @@
},
{
"cell_type": "markdown",
- "id": "9495f11b",
- "metadata": {},
+ "id": "c3934c34",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Convex functions\n",
"\n",
@@ -600,8 +694,10 @@
},
{
"cell_type": "markdown",
- "id": "6944313a",
- "metadata": {},
+ "id": "8019895f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Convex function\n",
"\n",
@@ -619,8 +715,10 @@
},
{
"cell_type": "markdown",
- "id": "7e9a67b4",
- "metadata": {},
+ "id": "5c64bff2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conditions on convex functions\n",
"\n",
@@ -656,8 +754,10 @@
},
{
"cell_type": "markdown",
- "id": "798e48b8",
- "metadata": {},
+ "id": "27a4f978",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More on convex functions\n",
"\n",
@@ -682,8 +782,10 @@
},
{
"cell_type": "markdown",
- "id": "202f23a9",
- "metadata": {},
+ "id": "e9b3d8d2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Some simple problems\n",
"\n",
@@ -710,8 +812,10 @@
},
{
"cell_type": "markdown",
- "id": "ef5e013d",
- "metadata": {},
+ "id": "d09eceeb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Standard steepest descent\n",
"\n",
@@ -728,8 +832,10 @@
},
{
"cell_type": "markdown",
- "id": "c877d472",
- "metadata": {},
+ "id": "9c7c69f6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{A}\\boldsymbol{x} = \\boldsymbol{b}.\n",
@@ -738,16 +844,20 @@
},
{
"cell_type": "markdown",
- "id": "84db7aa1",
- "metadata": {},
+ "id": "059f8341",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In the iterative process we end up with a problem like"
]
},
{
"cell_type": "markdown",
- "id": "5f2a0547",
- "metadata": {},
+ "id": "b34e3cbf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{r}= \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x},\n",
@@ -756,8 +866,10 @@
},
{
"cell_type": "markdown",
- "id": "63834dfc",
- "metadata": {},
+ "id": "757d3a71",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\boldsymbol{r}$ is the so-called residual or error in the iterative process.\n",
"\n",
@@ -766,8 +878,10 @@
},
{
"cell_type": "markdown",
- "id": "382780f2",
- "metadata": {},
+ "id": "1062652b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Gradient method\n",
"\n",
@@ -776,8 +890,10 @@
},
{
"cell_type": "markdown",
- "id": "84a25410",
- "metadata": {},
+ "id": "880644fb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"P(\\boldsymbol{x})=\\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T\\boldsymbol{b},\n",
@@ -786,8 +902,10 @@
},
{
"cell_type": "markdown",
- "id": "5ff06e1f",
- "metadata": {},
+ "id": "73ab5acb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with the constraint that the matrix $\\boldsymbol{A}$ is positive definite and\n",
"symmetric. This defines also the Hessian and we want it to be positive definite."
@@ -795,8 +913,10 @@
},
{
"cell_type": "markdown",
- "id": "34adc355",
- "metadata": {},
+ "id": "3125f913",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Steepest descent method\n",
"\n",
@@ -806,8 +926,10 @@
},
{
"cell_type": "markdown",
- "id": "78b1ca61",
- "metadata": {},
+ "id": "e6ea9762",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{x}_0=0,\n",
@@ -816,16 +938,20 @@
},
{
"cell_type": "markdown",
- "id": "4e13486b",
- "metadata": {},
+ "id": "5988146c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"or consider the system"
]
},
{
"cell_type": "markdown",
- "id": "4f0a3182",
- "metadata": {},
+ "id": "6648cbdf",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n",
@@ -834,16 +960,20 @@
},
{
"cell_type": "markdown",
- "id": "265b0272",
- "metadata": {},
+ "id": "f3e28a65",
+ "metadata": {
+ "editable": true
+ },
"source": [
"instead."
]
},
{
"cell_type": "markdown",
- "id": "5b67dd4b",
- "metadata": {},
+ "id": "22d951fb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Steepest descent method\n",
"One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form"
@@ -851,8 +981,10 @@
},
{
"cell_type": "markdown",
- "id": "1891f2b1",
- "metadata": {},
+ "id": "4913dea8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n",
@@ -861,8 +993,10 @@
},
{
"cell_type": "markdown",
- "id": "b5346b61",
- "metadata": {},
+ "id": "760cd864",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This suggests taking the first basis vector $\\boldsymbol{r}_1$ (see below for definition) \n",
"to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n",
@@ -871,8 +1005,10 @@
},
{
"cell_type": "markdown",
- "id": "3721dae9",
- "metadata": {},
+ "id": "949c6670",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n",
@@ -881,8 +1017,10 @@
},
{
"cell_type": "markdown",
- "id": "604ff6d8",
- "metadata": {},
+ "id": "b78c24e3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and \n",
"$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$."
@@ -890,8 +1028,10 @@
},
{
"cell_type": "markdown",
- "id": "a7a9f227",
- "metadata": {},
+ "id": "c0a0df7d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Final expressions\n",
"We can compute the residual iteratively as"
@@ -899,8 +1039,10 @@
},
{
"cell_type": "markdown",
- "id": "b849223d",
- "metadata": {},
+ "id": "74efdd7b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n",
@@ -909,16 +1051,20 @@
},
{
"cell_type": "markdown",
- "id": "deb44c9e",
- "metadata": {},
+ "id": "0f65ae12",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which equals"
]
},
{
"cell_type": "markdown",
- "id": "2b27982e",
- "metadata": {},
+ "id": "620ee7b9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_k),\n",
@@ -927,16 +1073,20 @@
},
{
"cell_type": "markdown",
- "id": "9214b70a",
- "metadata": {},
+ "id": "19fcd80e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"or"
]
},
{
"cell_type": "markdown",
- "id": "662bf19c",
- "metadata": {},
+ "id": "74967d0c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{r}_k,\n",
@@ -945,16 +1095,20 @@
},
{
"cell_type": "markdown",
- "id": "b01ada03",
- "metadata": {},
+ "id": "1de35d60",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which gives"
]
},
{
"cell_type": "markdown",
- "id": "b0d2d200",
- "metadata": {},
+ "id": "030c705b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\alpha_k = \\frac{\\boldsymbol{r}_k^T\\boldsymbol{r}_k}{\\boldsymbol{r}_k^T\\boldsymbol{A}\\boldsymbol{r}_k}\n",
@@ -963,16 +1117,20 @@
},
{
"cell_type": "markdown",
- "id": "45f88381",
- "metadata": {},
+ "id": "fc5e75d4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"leading to the iterative scheme"
]
},
{
"cell_type": "markdown",
- "id": "3af9ed0f",
- "metadata": {},
+ "id": "d05b2059",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{x}_{k+1}=\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{r}_{k},\n",
@@ -981,8 +1139,10 @@
},
{
"cell_type": "markdown",
- "id": "7a1af4a0",
- "metadata": {},
+ "id": "e23f881c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Steepest descent example"
]
@@ -990,8 +1150,11 @@
{
"cell_type": "code",
"execution_count": 1,
- "id": "670fa483",
- "metadata": {},
+ "id": "251aa6de",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"%matplotlib inline\n",
@@ -1020,8 +1183,10 @@
},
{
"cell_type": "markdown",
- "id": "40d73965",
- "metadata": {},
+ "id": "1f0f1ed5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"And then as countor plot"
]
@@ -1029,8 +1194,11 @@
{
"cell_type": "code",
"execution_count": 2,
- "id": "5dfb2ddc",
- "metadata": {},
+ "id": "0409acdb",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"pt.axis(\"equal\")\n",
@@ -1040,8 +1208,10 @@
},
{
"cell_type": "markdown",
- "id": "77d58e5a",
- "metadata": {},
+ "id": "616c1455",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Find guesses"
]
@@ -1049,8 +1219,11 @@
{
"cell_type": "code",
"execution_count": 3,
- "id": "44d8a3c9",
- "metadata": {},
+ "id": "ad1f5a1e",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"x = guesses[-1]\n",
@@ -1059,8 +1232,10 @@
},
{
"cell_type": "markdown",
- "id": "614db792",
- "metadata": {},
+ "id": "ba9c7a98",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Run it!"
]
@@ -1068,8 +1243,11 @@
{
"cell_type": "code",
"execution_count": 4,
- "id": "e13065ea",
- "metadata": {},
+ "id": "e312de69",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"def f1d(alpha):\n",
@@ -1083,8 +1261,10 @@
},
{
"cell_type": "markdown",
- "id": "e17f3f24",
- "metadata": {},
+ "id": "d711e345",
+ "metadata": {
+ "editable": true
+ },
"source": [
"What happened?"
]
@@ -1092,8 +1272,11 @@
{
"cell_type": "code",
"execution_count": 5,
- "id": "3a192039",
- "metadata": {},
+ "id": "2218e8cf",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"pt.axis(\"equal\")\n",
@@ -1104,16 +1287,20 @@
},
{
"cell_type": "markdown",
- "id": "25133be0",
- "metadata": {},
+ "id": "9a597d61",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Note that we did only one iteration here. We can easily add more using our previous guesses."
]
},
{
"cell_type": "markdown",
- "id": "8366edee",
- "metadata": {},
+ "id": "ce6d67b8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conjugate gradient method\n",
"In the CG method we define so-called conjugate directions and two vectors \n",
@@ -1124,8 +1311,10 @@
},
{
"cell_type": "markdown",
- "id": "6713c482",
- "metadata": {},
+ "id": "e13f279a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{s}^T\\boldsymbol{A}\\boldsymbol{t}= 0.\n",
@@ -1134,8 +1323,10 @@
},
{
"cell_type": "markdown",
- "id": "68914ee4",
- "metadata": {},
+ "id": "6e52272c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The philosophy of the CG method is to perform searches in various conjugate directions\n",
"of our vectors $\\boldsymbol{x}_i$ obeying the above criterion, namely"
@@ -1143,8 +1334,10 @@
},
{
"cell_type": "markdown",
- "id": "04879349",
- "metadata": {},
+ "id": "f94e9812",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{x}_i^T\\boldsymbol{A}\\boldsymbol{x}_j= 0.\n",
@@ -1153,8 +1346,10 @@
},
{
"cell_type": "markdown",
- "id": "274806c6",
- "metadata": {},
+ "id": "b3b9eda9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Two vectors are conjugate if they are orthogonal with respect to \n",
"this inner product. Being conjugate is a symmetric relation: if $\\boldsymbol{s}$ is conjugate to $\\boldsymbol{t}$, then $\\boldsymbol{t}$ is conjugate to $\\boldsymbol{s}$."
@@ -1162,8 +1357,10 @@
},
{
"cell_type": "markdown",
- "id": "4a5bcfa2",
- "metadata": {},
+ "id": "a9b1af0e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conjugate gradient method\n",
"An example is given by the eigenvectors of the matrix"
@@ -1171,8 +1368,10 @@
},
{
"cell_type": "markdown",
- "id": "9f016300",
- "metadata": {},
+ "id": "eaa428b7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{v}_i^T\\boldsymbol{A}\\boldsymbol{v}_j= \\lambda\\boldsymbol{v}_i^T\\boldsymbol{v}_j,\n",
@@ -1181,16 +1380,20 @@
},
{
"cell_type": "markdown",
- "id": "abeeac99",
- "metadata": {},
+ "id": "42fbcdc6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which is zero unless $i=j$."
]
},
{
"cell_type": "markdown",
- "id": "b58ba91e",
- "metadata": {},
+ "id": "f5f71e71",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conjugate gradient method\n",
"Assume now that we have a symmetric positive-definite matrix $\\boldsymbol{A}$ of size\n",
@@ -1199,8 +1402,10 @@
},
{
"cell_type": "markdown",
- "id": "177ab271",
- "metadata": {},
+ "id": "acc6967e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{x}_{i+1}=\\boldsymbol{x}_{i}+\\alpha_i\\boldsymbol{p}_{i}.\n",
@@ -1209,8 +1414,10 @@
},
{
"cell_type": "markdown",
- "id": "80127879",
- "metadata": {},
+ "id": "8b7294d4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We assume that $\\boldsymbol{p}_{i}$ is a sequence of $n$ mutually conjugate directions. \n",
"Then the $\\boldsymbol{p}_{i}$ form a basis of $R^n$ and we can expand the solution \n",
@@ -1219,8 +1426,10 @@
},
{
"cell_type": "markdown",
- "id": "2d71db3f",
- "metadata": {},
+ "id": "fc226dfb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i \\boldsymbol{p}_i.\n",
@@ -1229,8 +1438,10 @@
},
{
"cell_type": "markdown",
- "id": "3b848f20",
- "metadata": {},
+ "id": "8b14996b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conjugate gradient method\n",
"The coefficients are given by"
@@ -1238,8 +1449,10 @@
},
{
"cell_type": "markdown",
- "id": "4bb8943d",
- "metadata": {},
+ "id": "7dd48d5b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{A}\\mathbf{x} = \\sum^{n}_{i=1} \\alpha_i \\mathbf{A} \\mathbf{p}_i = \\mathbf{b}.\n",
@@ -1248,16 +1461,20 @@
},
{
"cell_type": "markdown",
- "id": "10ea5cc8",
- "metadata": {},
+ "id": "af228cde",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Multiplying with $\\boldsymbol{p}_k^T$ from the left gives"
]
},
{
"cell_type": "markdown",
- "id": "b64e16e4",
- "metadata": {},
+ "id": "2b080da7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{x} = \\sum^{n}_{i=1} \\alpha_i\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{p}_i= \\boldsymbol{p}_k^T \\boldsymbol{b},\n",
@@ -1266,16 +1483,20 @@
},
{
"cell_type": "markdown",
- "id": "55cf46ae",
- "metadata": {},
+ "id": "ef1b93ef",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and we can define the coefficients $\\alpha_k$ as"
]
},
{
"cell_type": "markdown",
- "id": "572d161f",
- "metadata": {},
+ "id": "c2e496e6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\alpha_k = \\frac{\\boldsymbol{p}_k^T \\boldsymbol{b}}{\\boldsymbol{p}_k^T \\boldsymbol{A} \\boldsymbol{p}_k}\n",
@@ -1284,8 +1505,10 @@
},
{
"cell_type": "markdown",
- "id": "b3efc955",
- "metadata": {},
+ "id": "da295a05",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conjugate gradient method and iterations\n",
"\n",
@@ -1302,8 +1525,10 @@
},
{
"cell_type": "markdown",
- "id": "bf2e5701",
- "metadata": {},
+ "id": "a7181f43",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{x}_0=0,\n",
@@ -1312,16 +1537,20 @@
},
{
"cell_type": "markdown",
- "id": "0a82ccd3",
- "metadata": {},
+ "id": "95d4d253",
+ "metadata": {
+ "editable": true
+ },
"source": [
"or consider the system"
]
},
{
"cell_type": "markdown",
- "id": "d8968174",
- "metadata": {},
+ "id": "3d34aa7a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{A}\\boldsymbol{z} = \\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_0,\n",
@@ -1330,16 +1559,20 @@
},
{
"cell_type": "markdown",
- "id": "a8668d46",
- "metadata": {},
+ "id": "595b8ba3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"instead."
]
},
{
"cell_type": "markdown",
- "id": "8717df42",
- "metadata": {},
+ "id": "e45646ca",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conjugate gradient method\n",
"One can show that the solution $\\boldsymbol{x}$ is also the unique minimizer of the quadratic form"
@@ -1347,8 +1580,10 @@
},
{
"cell_type": "markdown",
- "id": "a316cb92",
- "metadata": {},
+ "id": "85073560",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(\\boldsymbol{x}) = \\frac{1}{2}\\boldsymbol{x}^T\\boldsymbol{A}\\boldsymbol{x} - \\boldsymbol{x}^T \\boldsymbol{x} , \\quad \\boldsymbol{x}\\in\\mathbf{R}^n.\n",
@@ -1357,8 +1592,10 @@
},
{
"cell_type": "markdown",
- "id": "e5c19fab",
- "metadata": {},
+ "id": "4b0eca64",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This suggests taking the first basis vector $\\boldsymbol{p}_1$ \n",
"to be the gradient of $f$ at $\\boldsymbol{x}=\\boldsymbol{x}_0$, \n",
@@ -1367,8 +1604,10 @@
},
{
"cell_type": "markdown",
- "id": "d961fd12",
- "metadata": {},
+ "id": "2ee2810d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{A}\\boldsymbol{x}_0-\\boldsymbol{b},\n",
@@ -1377,8 +1616,10 @@
},
{
"cell_type": "markdown",
- "id": "d7bb8bd2",
- "metadata": {},
+ "id": "375b3f9c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and \n",
"$\\boldsymbol{x}_0=0$ it is equal $-\\boldsymbol{b}$.\n",
@@ -1388,8 +1629,10 @@
},
{
"cell_type": "markdown",
- "id": "47a808b4",
- "metadata": {},
+ "id": "809c4f47",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conjugate gradient method\n",
"Let $\\boldsymbol{r}_k$ be the residual at the $k$-th step:"
@@ -1397,8 +1640,10 @@
},
{
"cell_type": "markdown",
- "id": "1c4dc759",
- "metadata": {},
+ "id": "b624314d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{r}_k=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k.\n",
@@ -1407,8 +1652,10 @@
},
{
"cell_type": "markdown",
- "id": "3592a56d",
- "metadata": {},
+ "id": "1bc4699f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Note that $\\boldsymbol{r}_k$ is the negative gradient of $f$ at \n",
"$\\boldsymbol{x}=\\boldsymbol{x}_k$, \n",
@@ -1421,8 +1668,10 @@
},
{
"cell_type": "markdown",
- "id": "928d9861",
- "metadata": {},
+ "id": "b69344a5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{p}_{k+1}=\\boldsymbol{r}_k-\\frac{\\boldsymbol{p}_k^T \\boldsymbol{A}\\boldsymbol{r}_k}{\\boldsymbol{p}_k^T\\boldsymbol{A}\\boldsymbol{p}_k} \\boldsymbol{p}_k.\n",
@@ -1431,8 +1680,10 @@
},
{
"cell_type": "markdown",
- "id": "73001952",
- "metadata": {},
+ "id": "9b484618",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Conjugate gradient method\n",
"We can also compute the residual iteratively as"
@@ -1440,8 +1691,10 @@
},
{
"cell_type": "markdown",
- "id": "45ac58cc",
- "metadata": {},
+ "id": "ed2d98aa",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{r}_{k+1}=\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_{k+1},\n",
@@ -1450,16 +1703,20 @@
},
{
"cell_type": "markdown",
- "id": "b84fa3e8",
- "metadata": {},
+ "id": "c1d569d8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which equals"
]
},
{
"cell_type": "markdown",
- "id": "cf83831a",
- "metadata": {},
+ "id": "5fee2ff9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{b}-\\boldsymbol{A}(\\boldsymbol{x}_k+\\alpha_k\\boldsymbol{p}_k),\n",
@@ -1468,16 +1725,20 @@
},
{
"cell_type": "markdown",
- "id": "fc1c59d4",
- "metadata": {},
+ "id": "74575df1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"or"
]
},
{
"cell_type": "markdown",
- "id": "6ec0dd91",
- "metadata": {},
+ "id": "22932ed5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"(\\boldsymbol{b}-\\boldsymbol{A}\\boldsymbol{x}_k)-\\alpha_k\\boldsymbol{A}\\boldsymbol{p}_k,\n",
@@ -1486,16 +1747,20 @@
},
{
"cell_type": "markdown",
- "id": "2d9d9e02",
- "metadata": {},
+ "id": "750554fc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which gives"
]
},
{
"cell_type": "markdown",
- "id": "61db277e",
- "metadata": {},
+ "id": "f6dd46b8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{r}_{k+1}=\\boldsymbol{r}_k-\\boldsymbol{A}\\boldsymbol{p}_{k},\n",
@@ -1504,8 +1769,10 @@
},
{
"cell_type": "markdown",
- "id": "81e9eda7",
- "metadata": {},
+ "id": "5bd6edfe",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Revisiting our first homework\n",
"\n",
@@ -1526,8 +1793,11 @@
{
"cell_type": "code",
"execution_count": 6,
- "id": "b7e14150",
- "metadata": {},
+ "id": "ece72270",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"x = 2*np.random.rand(m,1)\n",
@@ -1536,8 +1806,10 @@
},
{
"cell_type": "markdown",
- "id": "314a5ce2",
- "metadata": {},
+ "id": "cb44d91f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"with $x_i \\in [0,1] $ is chosen randomly using a uniform distribution. Additionally we have a stochastic noise chosen according to a normal distribution $\\cal {N}(0,1)$. \n",
"The linear regression model is given by"
@@ -1545,8 +1817,10 @@
},
{
"cell_type": "markdown",
- "id": "43420c44",
- "metadata": {},
+ "id": "5edb9af4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"h_\\beta(x) = \\boldsymbol{y} = \\beta_0 + \\beta_1 x,\n",
@@ -1555,16 +1829,20 @@
},
{
"cell_type": "markdown",
- "id": "3e44b3ac",
- "metadata": {},
+ "id": "d654c751",
+ "metadata": {
+ "editable": true
+ },
"source": [
"such that"
]
},
{
"cell_type": "markdown",
- "id": "480a87a7",
- "metadata": {},
+ "id": "c730e2e0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{y}_i = \\beta_0 + \\beta_1 x_i.\n",
@@ -1573,8 +1851,10 @@
},
{
"cell_type": "markdown",
- "id": "15f589dd",
- "metadata": {},
+ "id": "426303d7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Gradient descent example\n",
"\n",
@@ -1585,8 +1865,10 @@
},
{
"cell_type": "markdown",
- "id": "b9ecaad6",
- "metadata": {},
+ "id": "51186928",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"X \\equiv \\begin{bmatrix}\n",
@@ -1599,16 +1881,20 @@
},
{
"cell_type": "markdown",
- "id": "258933de",
- "metadata": {},
+ "id": "3294449e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"The cost/loss/risk function is given by ("
]
},
{
"cell_type": "markdown",
- "id": "9d1e4418",
- "metadata": {},
+ "id": "4bdef3c4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\beta) = \\frac{1}{n}||X\\beta-\\mathbf{y}||_{2}^{2} = \\frac{1}{n}\\sum_{i=1}^{100}\\left[ (\\beta_0 + \\beta_1 x_i)^2 - 2 y_i (\\beta_0 + \\beta_1 x_i) + y_i^2\\right]\n",
@@ -1617,16 +1903,20 @@
},
{
"cell_type": "markdown",
- "id": "70aa5a66",
- "metadata": {},
+ "id": "994838e3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"and we want to find $\\beta$ such that $C(\\beta)$ is minimized."
]
},
{
"cell_type": "markdown",
- "id": "dd51cd8d",
- "metadata": {},
+ "id": "8fa5a88f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The derivative of the cost/loss function\n",
"\n",
@@ -1635,8 +1925,10 @@
},
{
"cell_type": "markdown",
- "id": "7666587b",
- "metadata": {},
+ "id": "6a889bc0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\nabla_{\\beta} C(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
@@ -1647,16 +1939,20 @@
},
{
"cell_type": "markdown",
- "id": "8606bcbc",
- "metadata": {},
+ "id": "e2e7078b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $X$ is the design matrix defined above."
]
},
{
"cell_type": "markdown",
- "id": "fc6fb851",
- "metadata": {},
+ "id": "3a9e044a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The Hessian matrix\n",
"The Hessian matrix of $C(\\beta)$ is given by"
@@ -1664,8 +1960,10 @@
},
{
"cell_type": "markdown",
- "id": "a0384f91",
- "metadata": {},
+ "id": "108bb82a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{H} \\equiv \\begin{bmatrix}\n",
@@ -1677,16 +1975,20 @@
},
{
"cell_type": "markdown",
- "id": "2029121a",
- "metadata": {},
+ "id": "a90ec725",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This result implies that $C(\\beta)$ is a convex function since the matrix $X^T X$ always is positive semi-definite."
]
},
{
"cell_type": "markdown",
- "id": "87717f56",
- "metadata": {},
+ "id": "8978f17b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Simple program\n",
"\n",
@@ -1695,8 +1997,10 @@
},
{
"cell_type": "markdown",
- "id": "b437c997",
- "metadata": {},
+ "id": "aab2903e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_{k+1} = \\beta_k - \\gamma \\nabla_\\beta C(\\beta_k), \\ k=0,1,\\cdots\n",
@@ -1705,8 +2009,10 @@
},
{
"cell_type": "markdown",
- "id": "fa0fd72c",
- "metadata": {},
+ "id": "ae8f0c83",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We can use the expression we computed for the gradient and let use a\n",
"$\\beta_0$ be chosen randomly and let $\\gamma = 0.001$. Stop iterating\n",
@@ -1718,8 +2024,10 @@
},
{
"cell_type": "markdown",
- "id": "75b9607c",
- "metadata": {},
+ "id": "65f9ad42",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Gradient Descent Example\n",
"\n",
@@ -1728,32 +2036,13 @@
},
{
"cell_type": "code",
- "execution_count": 10,
- "id": "5a39c590",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Eigenvalues of Hessian Matrix:[0.23690242 4.59790424]\n",
- "[[4.]\n",
- " [3.]]\n",
- "[[3.99988355]\n",
- " [3.00009593]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABE2ElEQVR4nO3deZzN9eLH8feZfWxTMxjGDCZkLKFUlJ2QmGtJ0iLcFl1KqMR1u1ExdSuXm4QW6hcqjCWVTDFSqktoT3Ttu2jG1uDM5/fHcU5zZs7syzlnvq/n4zGPOt/z+X6/n+85823efb6fxWaMMQIAALCoAG9XAAAAwJsIQwAAwNIIQwAAwNIIQwAAwNIIQwAAwNIIQwAAwNIIQwAAwNIIQwAAwNIIQwAAwNIIQ4AH3377re6++27Vq1dP4eHhCg8PV4MGDTRs2DBt2rSpzOoxceJE2Ww2t21169bVkCFDSvW8GzZs0MSJE/X7778X6zhDhgxR3bp1S6ROvuLAgQOaOHGitm7d6u2q+IzU1FTZbDalpqZ6uypAkQR5uwKAr5k9e7YeeOABNWzYUA899JCaNGkim82mn376SQsXLtQ111yjHTt2qF69el6p39KlS1WlSpVSPceGDRs0adIkDRkyRJdcckmpnsvfHDhwQJMmTVLdunXVokULb1cHQAkgDAFZfP755xo+fLh69uypxYsXKyQkxPVe586dNWLECC1atEjh4eF5HufMmTOqUKFCqdTxyiuvLJXjAoBV8ZgMyGLKlCkKDAzU7Nmz3YJQVrfccotiYmJcr4cMGaJKlSrpu+++U7du3VS5cmV16dJFkpSSkqLevXsrNjZWYWFhql+/voYNG6Zjx47lOO7777+vFi1aKDQ0VPHx8Xr++ec9nt/TY7L09HQ98sgjio+PV0hIiGrVqqVRo0bp9OnTbuVsNpseeOAB/d///Z8aNWqkChUqqHnz5lq5cqWrzMSJE/Xoo49KkuLj42Wz2Qr0CGTevHlq2LChQkND1ahRI7355psey507d05PP/20EhISFBoaqmrVqmno0KE6evSoW7k1a9aoY8eOioqKUnh4uGrXrq2bb75ZZ86ccZXJyMjQk08+qUaNGiksLExRUVHq1KmTNmzY4CpjjNHMmTPVokULhYeH69JLL1X//v31v//9z+18HTt2VNOmTbVx40a1a9dOFSpU0GWXXaZnnnlGmZmZkhyPg6655hpJ0tChQ12fzcSJE/P8bA4dOqRhw4YpNjZWISEhio+P16RJk3ThwgVXHW+66SZFRUVpz549rv3OnDmjJk2aqFGjRq7vcseOHRo6dKgaNGigChUqqFatWkpMTNR3333ndk7no6sFCxboscceU82aNVWpUiUlJibq8OHDOnnypO677z5VrVpVVatW1dChQ3Xq1Cm3Yzh/X2bPnq3LL79coaGhaty4sd5+++08r9dp06ZN+stf/qLIyEiFhYXpyiuv1LvvvlugfYEyZQAYY4y5cOGCCQ8PN9ddd12h9hs8eLAJDg42devWNUlJSeaTTz4xH330kTHGmJdfftkkJSWZFStWmHXr1pk33njDNG/e3DRs2NCcO3fOdYyPP/7YBAYGmrZt25rk5GSzaNEic80115jatWub7LdpnTp1zODBg12vT58+bVq0aGGqVq1qpk6daj7++GMzffp0ExERYTp37mwyMzNdZSWZunXrmmuvvda8++675oMPPjAdO3Y0QUFB5tdffzXGGLN3717z4IMPGkkmOTnZfPHFF+aLL74waWlpuX4Gc+fONZJM7969zXvvvWfeeustU79+fRMXF2fq1KnjKme3282NN95oKlasaCZNmmRSUlLMq6++amrVqmUaN25szpw5Y4wxZufOnSYsLMx07drVLFu2zKSmppr58+ebQYMGmRMnThhjjDl//rzp1KmTCQoKMo888oj54IMPzIoVK8zf//53s3DhQtc57733XhMcHGwefvhhs2rVKrNgwQKTkJBgoqOjzaFDh1zlOnToYKKiokyDBg3MrFmzTEpKihk+fLiRZN544w1jjDFpaWmua/3HP/7h+mz27t2b62dz8OBB1+cwe/Zs8/HHH5unnnrKhIaGmiFDhrjKHTt2zMTGxppWrVq5fjcGDx5swsPDzbfffusqt27dOvPwww+bxYsXm3Xr1pmlS5eaPn36mPDwcPPzzz+7yq1du9ZIMnXq1DFDhgwxq1atMrNmzTKVKlUynTp1Ml27djWPPPKIWb16tXn22WdNYGCgefDBB93qLsnExcWZxo0bm4ULF5oVK1aYG2+80UgyixYtynGutWvXuratWbPGhISEmHbt2pl33nnHrFq1ygwZMsRIMnPnzs318wK8gTAEXHTo0CEjyQwcODDHexcuXDDnz593/WQNGIMHDzaSzOuvv57n8TMzM8358+fN7t27jSSzfPly13utWrUyMTEx5uzZs65t6enpJjIyMt8wlJSUZAICAszGjRvdyi1evNhIMh988IFrmyQTHR1t0tPT3a47ICDAJCUlubY999xzRpLZuXNnntdkjCPgxMTEmKuuusrtc9m1a5cJDg52C0MLFy40ksySJUvcjrFx40YjycycOdOt7lu3bs31vG+++aaRZF555ZVcy3zxxRdGknnhhRfctu/du9eEh4ebsWPHurZ16NDBSDJfffWVW9nGjRub7t2756hrQf+gDxs2zFSqVMns3r3bbfvzzz9vJJkffvjBte2zzz4zQUFBZtSoUeb11183ksyrr76a5/EvXLhgzp07Zxo0aGBGjx7t2u4MKImJiW7lR40aZSSZkSNHum3v06ePiYyMdNsmyYSHh7uFxgsXLpiEhARTv379HOfKGoYSEhLMlVdeac6fP+92zF69epmaNWsau92e53UBZYnHZEABtGzZUsHBwa6fF154IUeZm2++Oce2I0eO6P7771dcXJyCgoIUHBysOnXqSJJ++uknSdLp06e1ceNG9evXT2FhYa59K1eurMTExHzrtnLlSjVt2lQtWrTQhQsXXD/du3f3+HirU6dOqly5sut1dHS0qlevrt27dxfos8hu27ZtOnDggG6//Xa3kW916tTR9ddfn6Oul1xyiRITE93q2qJFC9WoUcNV1xYtWigkJET33Xef3njjjRyPtCTpww8/VFhYmP7617/mWreVK1fKZrPpzjvvdDtfjRo11Lx58xyfTY0aNXTttde6bWvWrFmRPxtnHTp16qSYmBi3OvTo0UOStG7dOlfZNm3aaPLkyZo2bZr+9re/6c4779Tdd9/tdrwLFy5oypQpaty4sUJCQhQUFKSQkBBt377d9TuVVa9evdxeN2rUSJLUs2fPHNuPHz+e41FZly5dFB0d7XodGBioW2+9VTt27NC+ffs8XvOOHTv0888/64477nDV2flz00036eDBg9q2bVuenxtQlghDwEVVq1ZVeHi4xz98CxYs0MaNG7VixQqP+1aoUCHHCK/MzEx169ZNycnJGjt2rD755BP997//1ZdffilJOnv2rCTpxIkTyszMVI0aNXIc19O27A4fPqxvv/3WLawFBwercuXKMsbk6J8UFRWV4xihoaGu+hTWb7/9lmtds287fPiwfv/9d4WEhOSo76FDh1x1rVevnj7++GNVr15dI0aMUL169VSvXj1Nnz7ddayjR48qJiZGAQG5/2fs8OHDMsYoOjo6x/m+/PLLUv9snHV47733cpy/SZMmkpSjDnfccYdCQkKUkZHh6ruV1ZgxY/T444+rT58+eu+99/TVV19p48aNat68ucd6RkZGur129oXLbfsff/zhtj2v79X53Xu6Zkl65JFHclz38OHDPV434E2MJgMuCgwMVOfOnbV69WodPHhQNWvWdL3XuHFjSdKuXbs87pt9LiBJ+v777/XNN99o3rx5Gjx4sGv7jh073MpdeumlstlsOnToUI5jeNqWnTPEvf7667m+X5qcAaIg9a9ataqioqK0atUqj8fK2mLVrl07tWvXTna7XZs2bdKLL76oUaNGKTo6WgMHDlS1atX02WefKTMzM9dAVLVqVdlsNq1fv16hoaE53ve0raRVrVpVzZo10+TJkz2+n7Uzvt1u1x133KFLL71UoaGhuvvuu/X555+7deZ/6623dNddd2nKlCluxzl27FipTIOQ1/fqKTxKf/7OjR8/Xv369fNYpmHDhiVUQ6D4CENAFuPHj9eHH36o+++/X4sXL1ZwcHCRj+UMSNn/4M6ePdvtdcWKFXXttdcqOTlZzz33nOtR2cmTJ/Xee+/le55evXppypQpioqKUnx8fJHrm5WzzgVpEWnYsKFq1qyphQsXasyYMa7r3r17tzZs2OD2x75Xr156++23Zbfb1apVqwLVJTAwUK1atVJCQoLmz5+vzZs3a+DAgerRo4cWLlyoefPm5fqorFevXnrmmWe0f/9+DRgwoEDny09hPhtnHT744APVq1dPl156aZ5ln3jiCa1fv16rV69WxYoV1b59ez366KNuLWI2my3H79T777+v/fv3q379+oW8mvx98sknOnz4sOtRmd1u1zvvvKN69eopNjbW4z4NGzZUgwYN9M033+QIbYAvIgwBWbRp00YvvfSSHnzwQV111VW677771KRJEwUEBOjgwYNasmSJJBVo0sOEhATVq1dP48aNkzFGkZGReu+995SSkpKj7FNPPaUbb7xRXbt21cMPPyy73a5nn31WFStW1PHjx/M8z6hRo7RkyRK1b99eo0ePVrNmzZSZmak9e/Zo9erVevjhhwscPJyuuOIKSdL06dM1ePBgBQcHq2HDhm4tN04BAQF66qmndM8996hv376699579fvvv2vixIk5HrEMHDhQ8+fP10033aSHHnpI1157rYKDg7Vv3z6tXbtWvXv3Vt++fTVr1iytWbNGPXv2VO3atfXHH3+4Wr5uuOEGSdJtt92muXPn6v7779e2bdvUqVMnZWZm6quvvlKjRo00cOBAtWnTRvfdd5+GDh2qTZs2qX379qpYsaIOHjyozz77TFdccYX+9re/Feqzcc5KPn/+fDVq1EiVKlVSTEyMW+jL6sknn1RKSoquv/56jRw5Ug0bNtQff/yhXbt26YMPPtCsWbMUGxurlJQUJSUl6fHHH3dNzZCUlKRHHnlEHTt2VN++fSU5wtW8efOUkJCgZs2a6euvv9Zzzz2XazAprqpVq6pz5856/PHHVbFiRc2cOVM///xzvsPrZ8+erR49eqh79+4aMmSIatWqpePHj+unn37S5s2btWjRolKpL1AkXu7ADfikrVu3mqFDh5r4+HgTGhpqwsLCTP369c1dd91lPvnkE7eygwcPNhUrVvR4nB9//NF07drVVK5c2Vx66aXmlltuMXv27DGSzBNPPOFWdsWKFaZZs2YmJCTE1K5d2zzzzDPmiSeeyHc0mTHGnDp1yvzjH/8wDRs2NCEhISYiIsJcccUVZvTo0W4jgSSZESNG5Kinp2OOHz/exMTEmICAgBwjhTx59dVXTYMGDUxISIi5/PLLzeuvv24GDx7sNprMGMeQ+Oeff940b97chIWFmUqVKpmEhAQzbNgws337dmOMYxRY3759TZ06dUxoaKiJiooyHTp0MCtWrHA71tmzZ80///lP13mjoqJM586dzYYNG9zKvf7666ZVq1amYsWKJjw83NSrV8/cddddZtOmTa4yHTp0ME2aNMlxXZ6uYeHChSYhIcEEBwd7/C6zO3r0qBk5cqSJj483wcHBJjIy0rRs2dJMmDDBnDp1yhw4cMBUr17ddO7c2W2UVWZmpklMTDSXXHKJa2TfiRMnzN13322qV69uKlSoYNq2bWvWr19vOnToYDp06ODa1znCK+sQeGP+nAYh++hD5+/a0aNHXducvy8zZ8409erVM8HBwSYhIcHMnz/fbV9Po8mMMeabb74xAwYMMNWrVzfBwcGmRo0apnPnzmbWrFl5fl5AWbMZY4y3ghgAwHfZbDaNGDFCM2bM8HZVgFLFaDIAAGBphCEAAGBpdKAGAHhELwpYBS1DAADA0ghDAADA0ghDAADA0izRZygzM1MHDhxQ5cqVPS6bAAAAfI8xRidPnsx3HcLiskQYOnDggOLi4rxdDQAAUAR79+4ttVnWJYuEIecSAnv37i3QMgoAAMD70tPTFRcX53EpoJJkiTDkfDRWpUoVwhAAAH6mtLu40IEaAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYGmEIAABYmtfD0KeffqrExETFxMTIZrNp2bJluZYdNmyYbDabpk2bVmb1AwAA5ZvXw9Dp06fVvHlzzZgxI89yy5Yt01dffaWYmJgyqhkAALCCIG9XoEePHurRo0eeZfbv368HHnhAH330kXr27FlGNQMAAFbg9Zah/GRmZmrQoEF69NFH1aRJE29XBwAAlDNebxnKz7PPPqugoCCNHDmywPtkZGQoIyPD9To9Pb00qgYAAMoBn24Z+vrrrzV9+nTNmzdPNputwPslJSUpIiLC9RMXF1eKtQQAAP7Mp8PQ+vXrdeTIEdWuXVtBQUEKCgrS7t279fDDD6tu3bq57jd+/HilpaW5fvbu3Vt2lQYAAH7Fpx+TDRo0SDfccIPbtu7du2vQoEEaOnRorvuFhoYqNDS0tKsHAADKAa+HoVOnTmnHjh2u1zt37tTWrVsVGRmp2rVrKyoqyq18cHCwatSooYYNG5Z1VQEAQDnk9TC0adMmderUyfV6zJgxkqTBgwdr3rx5XqoVAACwCq+HoY4dO8oYU+Dyu3btKr3KAAAAy/HpDtQAAACljTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAsjTAEAAAszeth6NNPP1ViYqJiYmJks9m0bNky13vnz5/XY489piuuuEIVK1ZUTEyM7rrrLh04cMB7FQYAAOWK18PQ6dOn1bx5c82YMSPHe2fOnNHmzZv1+OOPa/PmzUpOTtYvv/yiv/zlL16oKQAAKI9sxhjj7Uo42Ww2LV26VH369Mm1zMaNG3Xttddq9+7dql27doGOm56eroiICKWlpalKlSolVFsAAFCayurvd1CpHbmUpKWlyWaz6ZJLLsm1TEZGhjIyMlyv09PTy6BmAADAH3n9MVlh/PHHHxo3bpxuv/32PBNiUlKSIiIiXD9xcXFlWEsAAOBP/CYMnT9/XgMHDlRmZqZmzpyZZ9nx48crLS3N9bN3794yqiUAAPA3fvGY7Pz58xowYIB27typNWvW5PvcMDQ0VKGhoWVUOwAA4M98Pgw5g9D27du1du1aRUVFebtKAACgHPF6GDp16pR27Njher1z505t3bpVkZGRiomJUf/+/bV582atXLlSdrtdhw4dkiRFRkYqJCTEW9UGAADlhNeH1qempqpTp045tg8ePFgTJ05UfHy8x/3Wrl2rjh07FugcDK0HAMD/WGZofceOHZVXHvOhaZAAAEA55DejyQAAAEoDYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFgaYQgAAFia18PQp59+qsTERMXExMhms2nZsmVu7xtjNHHiRMXExCg8PFwdO3bUDz/84J3KAgCAcsfrYej06dNq3ry5ZsyY4fH9f/3rX5o6dapmzJihjRs3qkaNGuratatOnjxZxjUFAADlUZC3K9CjRw/16NHD43vGGE2bNk0TJkxQv379JElvvPGGoqOjtWDBAg0bNqwsqwoAAEqK3S6tXy8dPCjVrCm1aycFBnqlKl5vGcrLzp07dejQIXXr1s21LTQ0VB06dNCGDRty3S8jI0Pp6eluPwAAwEckJ0t160qdOkm33+74Z926ju2STh06pWXjv9IDV+X+t74keb1lKC+HDh2SJEVHR7ttj46O1u7du3PdLykpSZMmTSrVugEAgDw4W37275eOHpWqVZNq1ZKOHZMGDJCMcStu9u2Xbr5ZT1d6Vk+fekjn1EpS2TRm+HQYcrLZbG6vjTE5tmU1fvx4jRkzxvU6PT1dcXFxpVY/AACQRXKy9NBD0r59Od8LDMwRhCTJJqNM2TT41AxN1MOKD9qjbo2+1+zvSr+6Ph2GatSoIcnRQlSzZk3X9iNHjuRoLcoqNDRUoaGhpV4/AAAsL3vfn1xaftzK5yJARrW1V/+b8o5qP3abTp66RLMjSqneWfh0GIqPj1eNGjWUkpKiK6+8UpJ07tw5rVu3Ts8++6yXawcAgMVkDT7Vqzv+/cUXpePH/yyTS8tPYdSpa5MCcn8CVNK8HoZOnTqlHTt2uF7v3LlTW7duVWRkpGrXrq1Ro0ZpypQpatCggRo0aKApU6aoQoUKuv32271YawAALCavR19Z5dHyU2BZngaVBa+HoU2bNqlTp06u186+PoMHD9a8efM0duxYnT17VsOHD9eJEyfUqlUrrV69WpUrV/ZWlQEAsJbkZKl//2K3+OTLZpNiYx3D7MuQzZjSvjLvS09PV0REhNLS0lSlShVvVwcAAN+S15w/drtj2Ht+LULF5RwYtXixdHFuwbL6++31liEAAOAFzgC0fLk0f75j+LtTbKw0fbojlKxfXzpBKDDQ/ZFabKw0bZorCJUlwhAAAFaTX/+f/ful/v2VNvU1fbNit9qX9PltNmnhQsfcQz4wAzVhCACA8srTxIe//io98UTe+xnHnD9po5/QRM3VmqKcO3vLj1NcnNdagHJDGAIAoDwq6OivXDjn/KkfvFdptmqqcu6YbCpEN2Nny0/2Gai92AKUG8IQAADlTQmO/przRqgUOstxPNnyP2ZUlDRnjk+1/OTHpxdqBQAA+bDbpdRUR0tMaqp07pyjRaikBovXrOkINosXO1p2chMVJU2aJB0+7FdBSGJoPQAAvi+3oe+eHoVVq+Y+MqyonHP+7NzpPsw+6wzUknTkSKl1gGZoPQAAVme3S5MnO4a5Z13yIjZWuu026fnnc67+fvSoir2QhXPOn2nT3ANOYKDUsWNxj+5zCEMAAPii5GTpvvuk337L+d6+fdJzz8lIOYJPiazo5cU5f7yBMAQAgK8pYAfoEl3KtFo16Y47pN69fXLEV2kiDAEA4Evs9pLtAJ0bm02aOFFq0MDrkx56G2EIAABfUpLLX1StKh07lnO7D0586E2EIQAAStq5c9LMmY7ZnuvVk4YPl0JC8tzlxK40fTT1B/22MFUjint+50iwHTukDRv8YuJDb2JoPQAAJWnsWGnqVPelKAIDpTFjpH/9y63otlU7tfKl3Vr5WYTW/36F7ApSB6UqVZ0Kfj5btokQPaz+7q8YWg8AgL8ZO1Z67rmc2+126bnnZD9v17rYO7VyQZpWfldH28/HS4p3FWscukOtm9uV8Uu0QtKOyJZfe8WjjzomW8z6WM1iI8FKAi1DAAAURm4TIJ47J1Wo4HlxUklGkl2BCtcZXZDjkVmwzqlj5Hfq1emUeo6IV71OtR2FnaPJJM8dqbMueZFbfcoBWoYAAPA1nmZ8jo11TIq4Z0+uQUhyDIMPkl2P6V/aX7+jevUJUtdRTVSlVsuchZ3LX2Q/V1SUNHKkNGHCn4GnnE6EWJZoGQIAwJOzZ6WHH5Y2bpQuvVRq3Vp66qmc5WyOtdyP1W+tatu/yPewmcOHK+CllwpWh3Lc6lMQtAwBAFCWsgaPl16SPv/c/f2UFM/7GSMjmwK3/1ig0wQ0aFDwOtHqUyYIQwAAeHr8VQgBMopUmjJlk00m95mhAwMdw+zhUwK8XQEAALzK2Vm5BCY6DOjVM+8lMsaMyXe+IZQ9WoYAAOVXfn1uSnrpi4cflho1KvA8Q/ANhCEAQPljtzs6O0+dKp08+ed258gv5xw8JbX0hXPG53btHH18nn660DNQw3sIQwCA8iU5WRo8WDp1Kud7+/ZJ/fvr7Cv/p1W/1NORN9dqWHHP55zxedq0P1udQkKkUaOKe2SUEcIQAMB/ZX8MduyYdMstee6SaYyO3TNO/bVL7fSHhunJ4tWBGZ/9HmEIAOAfPAWf0aPdH3MVYA6eAElx2qdbAxYp+orq+mNXdYWmH81/6YusJk2SGjSw5Nw/5RFhCADg286elfr2lVJTpYyMvMvmMQN0dvPfzJTtjs5S8suO0WTZFzz1JDxceustWoHKGYbWAwB8V58+jvW+Pvoo/yBUSLZaMY5/cS59UauWe4HYWKl7d+nqq6WuXaVVqxydsQlC5Q7LcQAAfFOfPtLy5aVz7GrVHI/bsg+zt/DSF76I5TgAANZ19mzpBSHJsdxG9qDD0heWxWMyAIDPMQ8/UnoHf/TRfEecwVpoGQIA+IRzp85p3Uvfa+XCk7r5m+/UvjgHCwiQMjPdt1WpIr32mqOzNJAFYQgAUPIK2P/myA9H9cG/t2nlqkCt3t9EJ3WVJKmBmqm91hf+vM4JEN9+W4qMdIxAkxyPvzp2pA8QPCIMAQCKJ3vwOXrUsQ5X1vl/Li6DYfr01XfJ2/Xe7ANa+WWUvjrVREZtXcWiA46oV/1tiu3VX2bqS3kveupJ9gkQu3Qp7tXBAghDAICCs9ulNWuk//s/xzDziAjp44+l/fvz3M3s2yfdfLOGBczRK5n3Srrc9d6V4T8r8ZpD6vXX6mp5R4ICgqo73vi1d96dqMPCpDff/HNkGCPAUEQMrQcAFMzixdJddzlGehVBpmzap1g10o/qHP2jEm/4Qzc91ECx19TMfafchte3aSOtW0fwKecYWg8A8B1jx0rPPVesQwTIqLb26rdlnyusd/eC7bRsmSN8PfqotH27YwmM555zzAQNlBDCEAAgb4sWFTsIZRV25njhdggPl2bMKLHzA9n5/DxDFy5c0D/+8Q/Fx8crPDxcl112mZ588kllZh8yCQAoeXa7NHx4yR6zZh6PxQAv8PmWoWeffVazZs3SG2+8oSZNmmjTpk0aOnSoIiIi9NBDD3m7egDgXwq55IR9TaoCjx0rmXPbbI7RXu3alczxgBLi82Hoiy++UO/evdWzZ09JUt26dbVw4UJt2rTJyzUDAD9htzvm25k1y7Hg6cmTf753cch71sVH0/elK2X6D3pvqV2hv+7Q7JKog3P+n2nT6PQMn+PzYaht27aaNWuWfvnlF11++eX65ptv9Nlnn2natGnerhoA+L7kZOm++6TffvP8/v79Uv/+OjTxZb37TYLeS62sdceb6ryukyR10IWSqUf2+X8AH+LzYeixxx5TWlqaEhISFBgYKLvdrsmTJ+u2227LdZ+MjAxlZGS4Xqenp5dFVQHAu5wtQM5ZlwMDpUmT8t7HGGXKpnNPTNZo7VSmHK029YN3KfGK3Uq8rZLMtFqy5TOPkJu4OGnqVKlqVeb/gV/w+TD0zjvv6K233tKCBQvUpEkTbd26VaNGjVJMTIwGDx7scZ+kpCRNyu8/AABQXtjt0uTJjhFfp04VenfnkPcHK76uuM6XK3FEbV3ePV5SXUeBy/7jWM8rr2npbr1V6t2b4AO/5POTLsbFxWncuHEaMWKEa9vTTz+tt956Sz///LPHfTy1DMXFxTHpIgD/l70D9LFj0v335/4YrDAWLJBya3VPTpYeesh9iY2wMGnAAOmVV6SQkOKfH8iGSRcvOnPmjAIC3GcACAwMzHNofWhoqEJDQ0u7agBQds6dk4YNc8z5c/p06ZwjryHv/fo5Wn4KMRIN8Bc+H4YSExM1efJk1a5dW02aNNGWLVs0depU/fWvf/V21QCgdGRv/Xn/fUcfnNKcXy0uLv8h74GBjpXfgXLG58PQiy++qMcff1zDhw/XkSNHFBMTo2HDhumf//ynt6sGACXP0+Oo0mazMeQdlubzfYZKAgu1AvApuU18mJycf0flkhYVJc2Zw5B3+CT6DAFAeeSp5Sc21vEYbMyYsgtClSo5Fj+dMIEWIVgeYQgAykpuLT/79ztGZZWmJ55wtEhJjn4/HTsSgoCLChWG9u7dq7i4uNKqCwD4v9wegdntjhYhTy0/xshIspVGfXgMBuSrUKvWJyQk6PHHH9fp0hrWCQD+LDlZqltX6tRJuv12xz/r1nVsX78+z07RJRqEqlRxtEB9/LF0+DBBCMhHocJQSkqKVq9erQYNGmju3LmlVScA8D/OR2DZA8/+/TL9++vglNcLdJhC9xiqVEl6911p7VrHpIlr10rHjzvmI+rShUdhQAEUaTTZm2++qQkTJqhq1ar697//rY4+Pu8Eo8kAlIi8HoHVrZtry0+mbDqqqorW0YKdx2bLvyN1QIB0yy3S/PkEHpRbZfX3u8hD68+ePaukpCS98MIL6tatm5577jnVr1+/pOtXIghDAAote/A5etQx2iv7KLDp06XISMcjsXyk2S5RFfO750diNtufo8pGj3Y/T1ycY92xgwelX3+V6tWThg9nCQyUez4/tN4Yo27duunkyZP6z3/+ow8//FAjRozQxIkTVbly5ZKsIwCUrQJOfGj275du7q/Pa9ystgU4bMRDQxzhSXJv+bFdjEfTpjn69/Tty7IXQBkqVMvQrFmztHHjRm3cuFE//fSTAgMD1axZM7Vu3VotWrTQ/Pnz9csvv2jp0qW6+uqrS7PehULLEIB8nT3rmHfns8+kb74p8G6FegTm7M+TPWjFxf0ZhAC4+ORjsri4OLVu3dr1c/XVV+dYEHXKlClasGCBvv/++xKvbFERhgDkqU8fafnyYh3ij7BLFJqRJpun/6Q6H4Ht3PlnHyNafoB8+WQYKojDhw8rJiZGdufkXj6AMAQgVyUQhCRJo0bl/Qhs8WJafoBCKqu/34UaWl8Q1atX15o1a0r6sABQeHa7lJoqLVzo+Gf2/0k7e7ZkgpAk9e7tCDy1arlvj40lCAE+rsSX47DZbOrQoUNJHxYACsb5CGr5csew86NZ+vI4R3/16yeTaXSi/72KLO75nI/AnI+6evfmERjgZ1ibDID/yt735tixnMPSs3CM/rpZr9d5UlP236WXLhzVjcU5f9ZRYM7AExjoWPcLgN8gDAHwL1lbft56yxGACshmjDJlU9fdr+g+/V3/U3zx6hIbyygwoBwgDAHwD3a7NHmy4zHX8eNFPkyAjGprr9bdNlstJj8lXTa7YDvGxTkmRKxalUdgQDlDGALgm7I+Atu+vdghKLu2iZdK8dUcfXzy6kTdsqX0/PMEH6AcIwwB8B3OiQ/Xr3csO3H6dOmdq2ZNxz+XLct9eH3v3o73AZRrhCEAvqGk5vvJT9bRX07Llv0ZxLZvlxo0cKwFFh5e+vUB4HWEIQDeV5ZBSHIf/eUUHi7NmFH6dQDgcwhDAEqPs9/P/v3S4cPSb79JAQGOoecdOzoCSUlOfJgfRn8B8IAwBKB05LXy+9NPS1FR0pw5sqekqFS7JVerJt1xh6P/D52gAXhAGAJQ8pKTpf793dfoysb89pt08836UU10RUmdNy5OeuEFRwBi+DuAAiIMAShZdrujRSifNaBtkoykutpVvPNVqSL99a+0/AAoMsIQgJK1fn2uy2FkZ5NUWadlLv57oURGOkLXhAkEIADFQhgCkL/sa4Dl0gJzYufv+ilpna4v5OFtLVtKX3+dd6HYWOneex3D3nn8BaAEEYYA5M1TR+iLq7+bvv30y6r/6b2Ze7VyfYQ+S2uqtuqg1MKe4/nnHaO8PI0qYwZoAKWMMAQgd7l0hDb7HKu/PxD4smba75d0meu930Jq6ndVU8S5owV79OWcALFjRyY+BOAVNmPy6eVYDqSnpysiIkJpaWmqUqWKt6sD+Ae7XapbN9f+P5myaZ9idbm2qX3kj0rsdEo9H4jXZR1rF2g0mcuSJcz7A8Cjsvr7TcsQgBxMptGuKQsVn0dHaOfq78cXrVWF/je5v9mvn7R4ce7zDEmueYYIQgC8jTAElHcF7Pz8R1qGUmd8r5XvnNbKHy/TdfZALSzA4SucT/P8Rr9+juHu+c1ADQBeRhgCyrM8Oj+rXz8d+vaI3v/3Nq38KEQpB5votFq6il2uqIKdw7n6uyeBgY7QAwA+jDAElAfnzkkzZ0q//irVqycNHy6tXJln5+fHQv+t5zJGSaruei8m4KB6Xb5dvfqHqfPw1tK1sY5WHU99fzyt/g4AfogO1IC/GztWmjrV8TjMKSBAqlBBOnXK4y7Ozs/x2qmrKmxT4rWH1eueGrrytgTZArKMAXN2hJbcA5Fz9ffFi+nzA6DU0IEawJ9y6/czdqxj+Hk2JjNTtlyCkPRn5+cjryxX1D39JDX2XDC3jtCs/g6gHCEMAb7KGYCWL5fmz5eOHv3zvdhYx4KkU6d63LWgS1tEVczIv1DWjtAsfgqgHCIMAb7IU8fnrPbvl7n11sKv55VdXp2fs6IjNIByjDAEeIvdLqWmOn6kP4ebL1+e/4SFxe3qR+dnAHAhDAHesGiRdM89Unr6n9ueftqxErvNVqCwU+RWIWfn52nTeNQFAJICvF2Bgti/f7/uvPNORUVFqUKFCmrRooW+zm+Fa8BXjR0rDRjgHoScjh93TExYUgICpFq13LfFxjIKDACy8PmWoRMnTqhNmzbq1KmTPvzwQ1WvXl2//vqrLrnkEm9XDfAsrxmfFy/2OPqr1Dz8sJSUROdnAMiDz88zNG7cOH3++edav359kY/BPEMoM3nN+Ny7tyOMZB0VVhzOfj8DBjgeeWWdZygwUBozRvrXv0rmXADgBWX199vnw1Djxo3VvXt37du3T+vWrVOtWrU0fPhw3Xvvvbnuk5GRoYyMP4cMp6enKy4ujjCE0pXbSu0X++icHvmYKk5/pmTOlX3SQ08zUIeElMy5AMBLCEMXhYWFSZLGjBmjW265Rf/97381atQozZ49W3fddZfHfSZOnKhJkybl2E4YQrHl9gjMbpfq1s11KHymbDquS1VVx0umHnFxTHoIoNwjDF0UEhKiq6++Whs2bHBtGzlypDZu3KgvvvjC4z60DKFU5PUILDJS6tSp5M4VGelYTiPruapVk+64w/G4jX4/ACyA5Tguqlmzpho3dl8qoFGjRlqyZEmu+4SGhio0NLS0q4byylPrT25z/+zfL9O/v35s3F9NCnLsyEjHiLH8vPIKsz4DQBnx+TDUpk0bbdu2zW3bL7/8ojp16nipRijXcmv9OXvW89w/xsjIpqo/pBbs+A89JE2cmPs8QpUqSW+88efjL2Z9BoBS5/NhaPTo0br++us1ZcoUDRgwQP/97381Z84czZkzx9tVgz87e1Z69FHpl1+kihWlPn2k3bulJ57IWTa3JTEuCpBRtI7qdHCEKpxPl00ego5z5NeECVLTpjkDV5Uq0ujR0uOP0/oDAGXM5/sMSdLKlSs1fvx4bd++XfHx8RozZkyeo8myY2g9XOx2qX17KUsftBIzapSj/5Dk3vKTfeSXsx48AgOAPNGBugQRhiDJ8Qhs4EDp/PnSOf7atY7+QNlbfRj5BQBFQgdqoCQlJ0s331w6x8666GlgIB2fAcDPEIZQ/tnt0siRpXNsT4ueBgbS8RkA/AhhCP6pEH1u7GtSFbh/f/HOZ7M5hsWHhUlZjxUbyyMwAPBzhCH4h6zhZ/t2xzw8niY/vBhK0vela/W0H7Vy2QWF/rpDs4tzbmfrz5w5PAIDgHKIMATflF/4yW7/fpmb++v9aydp2o6e+vR4U51Xa0lSB10oXl2yt/7wCAwAyhXCEHyL3S5Nnuxo5SnITM1OFyc/bPbfV7RWf1emAnV58P/U64o96nVbJZl/x8h24EDBjmWzOSZGbNCA1h8AsADCELwrewtQYUNQFgEyqq29Wnj9i2rxz966vPtlki5zvHnZiwUbTcYweACwHMIQvMfT0hclYMAD0VL3ePeN/fpJS5ZI990n/fab+3shIY7tN99MKxAAWBBhCKWvMAufloSaNT1v79fP0QE6NdXxIzn6/3TsSAACAAsjDKF0nDsnvfii9O670vffS2fO/PleXgufFkfWyQ9zExgodeni+AEAQIQhlCS73dHi8o9/SF9+mXu5En4sJsnz5IcAABRAgLcrgHIiOVmKjpZuuCHvIFRaYmPdF0IFAKCAaBlC8ZXmul+5iY2V7r2X4e8AgGIjDKF47HbHiLDSFhnpOA/hBwBQwghDKJ7160unD5BTVJRjkdUJEwg/AIBSQRhCkezbeFArp21X+srPNLaoB4mK8rzwKY+/AABliDCEAsm8kKmNb/6klXOPauXXNbT1bIKkmuqgzKKFIZuNhU8BAD6BMGQ1niZAzCV8nDx4SinTftDKpef1/q8NdSSzies9mzLVutIPuvG6Czr3dbSCjx+WraB1iIpyBCEWPgUA+ADCkJV4Wv4iNtaxHtjFYLJz/T6t/M//tHJtRaX+1lTn1MpVtLLSdWPs9+rVw64eoxupWqMrJF0hJc8s2Giy0FDp73+n/w8AwKfYjCmN9RB8S3p6uiIiIpSWlqYqVap4uzrekZzscfkLY7NJRnrr8ol6dvdt+iGjgdv79YJ2K7HpTvW6rbLaDb9CIZVCcj++p3W/goOl665zBKAuXQhBAIACK6u/34Qhf5f1sVf16o5tR464PwKz26W6dXMd9ZUpm/YpVvHaKZuM2kZ8p15t09Trb3Fq2OMy2QIK+ADMOQM1634BAEpAWf395jGZP8tv1XfnI7DIyDyHvwfIqLb2anWfmbry+TsVWe/KotWHdb8AAH6IMORPsrYCbd8uTZyY52KnZt9+6eabNc82VEMLcPguA6pK9S4tseoCAOAPCEP+wG6XJk92tPIcP17g3WwyypRNN5mVBduhZs0iVhAAAP9FGPJ1uXVMLqAAGUXrqDIjoxRw4rjnliSbzfFIrV27YlYWAAD/w6r1vsw5AqyIQSirgLsGOf7Flq0ztPP1tGl0dAYAWBJhyFc5F0AtqcF+vXtLixdLtWq5b4+NdWx3ToAIAIDF8JjMB5lMo1+fnK/6JbEAatZHYIGBLH8BAEA2hCEfcfbEH1rzn++18t0zWrmtvtrag7WwuAf19AgsMJDlLwAAyIIw5EUHNh/SymnbtXJ1qD4+3FRndbXrveMqgSHusbGOIMQjMAAAckUYKkOZFzK1eeE2rXztsN7bGK3NZxpJquF6PzbwgBIbblevW8LVaXgbqWWstH9//v2GJk2SGjTIfQZqAACQK8JQKTt95LQ+nv6D3lucofd3XK5DmY0kNZLkWPn92oo/qlfrY0q8t4aa3dJQtoCYP3eePt0xmsxm8xyIsq/+DgAACo0wVBhZZ4DOo+Vl94b9ev8/O/TeJxW19lhTZeha13uVdFLdYr5Xr+4XdNOYBEU3bZr7+fr1c4z0yr7kRlSUNHIkq78DAFACWKjVE0+hZ/nynKHk4tpf9l699d83ftJ7845p5eYYfffH5W6Hqxu0V4mNf1XiwEpqP+IKhVYJLdwFFDCEAQBQnrBqfQkq1IfpafHTqCiPEx86P7ihmqs3NMS1PUB2XV/le/W6/oQS/xarRr3qFXzldwAAIIlV673DOeNz9nyYywzQNkmZsulJ/VMrlKhucduU2DNTN45upKjLm5d+fQEAQLERhpyKOONzgIxqa6+OfLhFQTfeUEqVAwAApYXlOJzWr3d/NFZIQSeOlmBlAABAWfG7MJSUlCSbzaZRo0aVyPFMptHPH+7UyjFrinegmjVLpD4AAKBs+VUY2rhxo+bMmaNmzZoV6zjnTp3TJ89v0eir1qlB2B41uilez2/pXLSD2WxSXJxjhBcAAPA7ftNn6NSpU7rjjjv0yiuv6Omnny7SMRY+9KU+/riyPtrXRCd1pWt7iDIUHhmuk2erqdLZY7KpgP2GPK39BQAA/IrftAyNGDFCPXv21A035N9JOSMjQ+np6W4/knT/vNZavO86nVQVVbcd1dD667Xk0S91bP85ffhbK1V+a5Yj39iyDYN3vo6Kct8eG+uYFJEZoAEA8Ft+0TL09ttva/Pmzdq4cWOByiclJWnSpEk5tl8R+ot6X31SiX+tpqvvaqyAoGyPtnKb8dm54Gnv3kx+CABAOePzky7u3btXV199tVavXq3mzR1z93Ts2FEtWrTQtGnTPO6TkZGhjIwM1+v09HTFxcUVbwZqQg8AAGWKGagvWrZsmfr27avALGHEbrfLZrMpICBAGRkZbu95UlYfJgAAKDnMQH1Rly5d9N1337ltGzp0qBISEvTYY4/lG4QAAADy4vNhqHLlymqabWX3ihUrKioqKsd2AACAwvKb0WQAAAClwedbhjxJTU31dhUAAEA5QcsQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNMIQAACwNJ8PQ0lJSbrmmmtUuXJlVa9eXX369NG2bdu8XS0AAFBO+HwYWrdunUaMGKEvv/xSKSkpunDhgrp166bTp097u2oAAKAcsBljjLcrURhHjx5V9erVtW7dOrVv375A+6SnpysiIkJpaWmqUqVKKdcQAACUhLL6+x1UakcuJWlpaZKkyMjIXMtkZGQoIyPD9To9Pb3U6wUAAPyTzz8my8oYozFjxqht27Zq2rRpruWSkpIUERHh+omLiyvDWgIAAH/iV4/JRowYoffff1+fffaZYmNjcy3nqWUoLi6Ox2QAAPgRHpNl8+CDD2rFihX69NNP8wxCkhQaGqrQ0NAyqhkAAPBnPh+GjDF68MEHtXTpUqWmpio+Pt7bVQIAAOWIz4ehESNGaMGCBVq+fLkqV66sQ4cOSZIiIiIUHh7u5doBAAB/5/N9hmw2m8ftc+fO1ZAhQwp0DIbWAwDgf+gzdJGPZzUAAODn/GpoPQAAQEkjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEsjDAEAAEvzmzA0c+ZMxcfHKywsTC1bttT69eu9XSUAAFAO+EUYeueddzRq1ChNmDBBW7ZsUbt27dSjRw/t2bPH21UDAAB+zmaMMd6uRH5atWqlq666Si+//LJrW6NGjdSnTx8lJSXlu396eroiIiKUlpamKlWqlGZVAQBACSmrv98+3zJ07tw5ff311+rWrZvb9m7dumnDhg1eqhUAACgvgrxdgfwcO3ZMdrtd0dHRbtujo6N16NAhj/tkZGQoIyPD9TotLU2SI2ECAAD/4Py7XdoPsXw+DDnZbDa318aYHNuckpKSNGnSpBzb4+LiSqVuAACg9Pz222+KiIgoteP7fBiqWrWqAgMDc7QCHTlyJEdrkdP48eM1ZswY1+vff/9dderU0Z49e0r1w/Q16enpiouL0969ey3VV4rr5rqtgOvmuq0gLS1NtWvXVmRkZKmex+fDUEhIiFq2bKmUlBT17dvXtT0lJUW9e/f2uE9oaKhCQ0NzbI+IiLDUL5FTlSpVuG4L4bqtheu2Fqted0BA6XZx9vkwJEljxozRoEGDdPXVV+u6667TnDlztGfPHt1///3erhoAAPBzfhGGbr31Vv3222968skndfDgQTVt2lQffPCB6tSp4+2qAQAAP+cXYUiShg8fruHDhxdp39DQUD3xxBMeH52VZ1w3120FXDfXbQVcd+let19MuggAAFBafH7SRQAAgNJEGAIAAJZGGAIAAJZGGAIAAJbml2Fo5syZio+PV1hYmFq2bKn169fnWX7dunVq2bKlwsLCdNlll2nWrFk5yixZskSNGzdWaGioGjdurKVLl5ZW9YusMNednJysrl27qlq1aqpSpYquu+46ffTRR25l5s2bJ5vNluPnjz/+KO1LKZTCXHdqaqrHa/r555/dypW373vIkCEer7tJkyauMv7wfX/66adKTExUTEyMbDabli1blu8+5eH+Lux1l5f7u7DXXV7u78Jed3m5v5OSknTNNdeocuXKql69uvr06aNt27blu19Z3ON+F4beeecdjRo1ShMmTNCWLVvUrl079ejRQ3v27PFYfufOnbrpppvUrl07bdmyRX//+981cuRILVmyxFXmiy++0K233qpBgwbpm2++0aBBgzRgwAB99dVXZXVZ+SrsdX/66afq2rWrPvjgA3399dfq1KmTEhMTtWXLFrdyVapU0cGDB91+wsLCyuKSCqSw1+20bds2t2tq0KCB673y+H1Pnz7d7Xr37t2ryMhI3XLLLW7lfP37Pn36tJo3b64ZM2YUqHx5ub8Le93l5f4u7HU7+fv9XdjrLi/397p16zRixAh9+eWXSklJ0YULF9StWzedPn06133K7B43fubaa681999/v9u2hIQEM27cOI/lx44daxISEty2DRs2zLRu3dr1esCAAebGG290K9O9e3czcODAEqp18RX2uj1p3LixmTRpkuv13LlzTURERElVsVQU9rrXrl1rJJkTJ07kekwrfN9Lly41NpvN7Nq1y7XNH77vrCSZpUuX5lmmvNzfWRXkuj3xx/s7q4Jcd3m5v7MqyvddHu5vY4w5cuSIkWTWrVuXa5myusf9qmXo3Llz+vrrr9WtWze37d26ddOGDRs87vPFF1/kKN+9e3dt2rRJ58+fz7NMbscsa0W57uwyMzN18uTJHIvdnTp1SnXq1FFsbKx69eqV4/8svak4133llVeqZs2a6tKli9auXev2nhW+79dee0033HBDjlnaffn7LorycH+XBH+8v4vDn+/vklBe7u+0tDRJynMR1rK6x/0qDB07dkx2uz3HavXR0dE5VrV3OnTokMfyFy5c0LFjx/Isk9sxy1pRrju7F154QadPn9aAAQNc2xISEjRv3jytWLFCCxcuVFhYmNq0aaPt27eXaP2LqijXXbNmTc2ZM0dLlixRcnKyGjZsqC5duujTTz91lSnv3/fBgwf14Ycf6p577nHb7uvfd1GUh/u7JPjj/V0U5eH+Lq7ycn8bYzRmzBi1bdtWTZs2zbVcWd3jfrMcR1Y2m83ttTEmx7b8ymffXthjekNR67hw4UJNnDhRy5cvV/Xq1V3bW7durdatW7tet2nTRldddZVefPFF/ec//ym5ihdTYa67YcOGatiwoev1ddddp7179+r5559X+/bti3RMbylqHefNm6dLLrlEffr0cdvuL993YZWX+7uo/P3+LozydH8XVXm5vx944AF9++23+uyzz/ItWxb3uF+1DFWtWlWBgYE50t6RI0dypEKnGjVqeCwfFBSkqKioPMvkdsyyVpTrdnrnnXd09913691339UNN9yQZ9mAgABdc801PvN/EsW57qxat27tdk3l+fs2xuj111/XoEGDFBISkmdZX/u+i6I83N/F4c/3d0nxt/u7OMrL/f3ggw9qxYoVWrt2rWJjY/MsW1b3uF+FoZCQELVs2VIpKSlu21NSUnT99dd73Oe6667LUX716tW6+uqrFRwcnGeZ3I5Z1opy3ZLj/xiHDBmiBQsWqGfPnvmexxijrVu3qmbNmsWuc0ko6nVnt2XLFrdrKq/ft+QYrbFjxw7dfffd+Z7H177voigP93dR+fv9XVL87f4uDn+/v40xeuCBB5ScnKw1a9YoPj4+333K7B4vcFdrH/H222+b4OBg89prr5kff/zRjBo1ylSsWNHVq37cuHFm0KBBrvL/+9//TIUKFczo0aPNjz/+aF577TUTHBxsFi9e7Crz+eefm8DAQPPMM8+Yn376yTzzzDMmKCjIfPnll2V+fbkp7HUvWLDABAUFmZdeeskcPHjQ9fP777+7ykycONGsWrXK/Prrr2bLli1m6NChJigoyHz11Vdlfn25Kex1//vf/zZLly41v/zyi/n+++/NuHHjjCSzZMkSV5ny+H073XnnnaZVq1Yej+kP3/fJkyfNli1bzJYtW4wkM3XqVLNlyxaze/duY0z5vb8Le93l5f4u7HWXl/u7sNft5O/399/+9jcTERFhUlNT3X5vz5w54yrjrXvc78KQMca89NJLpk6dOiYkJMRcddVVbsPyBg8ebDp06OBWPjU11Vx55ZUmJCTE1K1b17z88ss5jrlo0SLTsGFDExwcbBISEtxuLl9RmOvu0KGDkZTjZ/Dgwa4yo0aNMrVr1zYhISGmWrVqplu3bmbDhg1leEUFU5jrfvbZZ029evVMWFiYufTSS03btm3N+++/n+OY5e37NsaY33//3YSHh5s5c+Z4PJ4/fN/OodO5/d6W1/u7sNddXu7vwl53ebm/i/J7Xh7ub0/XLMnMnTvXVcZb97jtYgUBAAAsya/6DAEAAJQ0whAAALA0whAAALA0whAAALA0whAAALA0whAAALA0whAAALA0whAAALA0whAAALA0whAAALA0whAAv7Rw4UKFhYVp//79rm333HOPmjVrprS0NC/WDIC/YW0yAH7JGKMWLVqoXbt2mjFjhiZNmqRXX31VX375pWrVquXt6gHwI0HergAAFIXNZtPkyZPVv39/xcTEaPr06Vq/fj1BCECh0TIEwK9dddVV+uGHH7R69Wp16NDB29UB4IfoMwTAb3300Uf6+eefZbfbFR0d7e3qAPBTtAwB8EubN29Wx44d9dJLL+ntt99WhQoVtGjRIm9XC4Afos8QAL+za9cu9ezZU+PGjdOgQYPUuHFjXXPNNfr666/VsmVLb1cPgJ+hZQiAXzl+/LjatGmj9u3ba/bs2a7tvXv3VkZGhlatWuXF2gHwR4QhAABgaXSgBgAAlkYYAgAAlkYYAgAAlkYYAgAAlkYYAgAAlkYYAgAAlkYYAgAAlkYYAgAAlkYYAgAAlkYYAgAAlkYYAgAAlkYYAgAAlvb/TOduonBuf1oAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 7,
+ "id": "8302f583",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"\n",
"# Importing various packages\n",
@@ -1768,7 +2057,7 @@
"# the number of datapoints\n",
"n = 100\n",
"x = 2*np.random.rand(n,1)\n",
- "y = 4+3*x#+np.random.randn(n,1)\n",
+ "y = 4+3*x+np.random.randn(n,1)\n",
"\n",
"X = np.c_[np.ones((n,1)), x]\n",
"# Hessian matrix\n",
@@ -1781,8 +2070,8 @@
"print(beta_linreg)\n",
"beta = np.random.randn(2,1)\n",
"\n",
- "eta = 0.1 #1.0/np.max(EigValues)\n",
- "Niterations = 400\n",
+ "eta = 1.0/np.max(EigValues)\n",
+ "Niterations = 1000\n",
"\n",
"for iter in range(Niterations):\n",
" gradient = (2.0/n)*X.T @ (X @ beta-y)\n",
@@ -1805,8 +2094,10 @@
},
{
"cell_type": "markdown",
- "id": "efda8de7",
- "metadata": {},
+ "id": "52976521",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## And a corresponding example using **scikit-learn**"
]
@@ -1814,8 +2105,11 @@
{
"cell_type": "code",
"execution_count": 8,
- "id": "e4e5b556",
- "metadata": {},
+ "id": "29314f28",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Importing various packages\n",
@@ -1838,8 +2132,10 @@
},
{
"cell_type": "markdown",
- "id": "14596669",
- "metadata": {},
+ "id": "d5a8fb29",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Gradient descent and Ridge\n",
"\n",
@@ -1848,8 +2144,10 @@
},
{
"cell_type": "markdown",
- "id": "e5c564c8",
- "metadata": {},
+ "id": "8b253c73",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C_{\\text{ridge}}(\\beta) = \\frac{1}{n}||X\\beta -\\mathbf{y}||^2 + \\lambda ||\\beta||^2, \\ \\lambda \\geq 0.\n",
@@ -1858,16 +2156,20 @@
},
{
"cell_type": "markdown",
- "id": "50689ad4",
- "metadata": {},
+ "id": "f829349d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"In order to minimize $C_{\\text{ridge}}(\\beta)$ using GD we adjust the gradient as follows"
]
},
{
"cell_type": "markdown",
- "id": "1bdd6193",
- "metadata": {},
+ "id": "29b29b1e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\nabla_\\beta C_{\\text{ridge}}(\\beta) = \\frac{2}{n}\\begin{bmatrix} \\sum_{i=1}^{100} \\left(\\beta_0+\\beta_1x_i-y_i\\right) \\\\\n",
@@ -1878,16 +2180,20 @@
},
{
"cell_type": "markdown",
- "id": "725c4b2e",
- "metadata": {},
+ "id": "7da5e43a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We can easily extend our program to minimize $C_{\\text{ridge}}(\\beta)$ using gradient descent and compare with the analytical solution given by"
]
},
{
"cell_type": "markdown",
- "id": "a8c70cc2",
- "metadata": {},
+ "id": "f26795de",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_{\\text{ridge}} = \\left(X^T X + n\\lambda I_{2 \\times 2} \\right)^{-1} X^T \\mathbf{y}.\n",
@@ -1896,8 +2202,10 @@
},
{
"cell_type": "markdown",
- "id": "d4a00982",
- "metadata": {},
+ "id": "6db6ee02",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The Hessian matrix for Ridge Regression\n",
"The Hessian matrix of Ridge Regression for our simple example is given by"
@@ -1905,8 +2213,10 @@
},
{
"cell_type": "markdown",
- "id": "c21352ee",
- "metadata": {},
+ "id": "25716dbe",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{H} \\equiv \\begin{bmatrix}\n",
@@ -1918,8 +2228,10 @@
},
{
"cell_type": "markdown",
- "id": "437d224b",
- "metadata": {},
+ "id": "de2918c3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"This implies that the Hessian matrix is positive definite, hence the stationary point is a\n",
"minimum.\n",
@@ -1930,40 +2242,23 @@
},
{
"cell_type": "markdown",
- "id": "e46c76c4",
- "metadata": {},
+ "id": "0b0dabd7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Program example for gradient descent with Ridge Regression"
]
},
{
"cell_type": "code",
- "execution_count": 11,
- "id": "1e7aff0e",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Eigenvalues of Hessian Matrix:[0.29809212 4.62003583]\n",
- "[[4.02315947]\n",
- " [2.88696751]]\n",
- "[[4.02316137]\n",
- " [2.88696598]]\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABUb0lEQVR4nO3dd3gU1foH8O+kh5JgQg2pFzE06UWFmISmiAgERBEwcC0ooAREhatewEL02kABQb2K/iQRIUFQUUFIQpAi1S6IlxIgFEETigSyOb8/1l2y2Ta72Z2d2fl+nmcfyGR29syWzLvnvOc9khBCgIiIiEinAnzdACIiIiJfYjBEREREusZgiIiIiHSNwRARERHpGoMhIiIi0jUGQ0RERKRrDIaIiIhI1xgMERERka4xGCIiIiJdYzBEPvHdd9/hnnvuQYsWLRAeHo7w8HC0bNkS48ePx44dOxRrx6xZsyBJksW2xMREjB071quPu3nzZsyaNQt//vlnrY4zduxYJCYmeqRNanHs2DHMmjULe/bs8XVTVKOwsBCSJKGwsNBjxzx48CAGDhyIqKgoSJKErKwsjx3blsTEREiSZL7VrVsXnTt3xvz581FzIQRXztcfPwOkvCBfN4D0Z/HixZg0aRKSk5MxefJktG3bFpIk4eeff0Zubi66deuG/fv3o0WLFj5p38qVKxEREeHVx9i8eTNmz56NsWPHokGDBl59LK05duwYZs+ejcTERHTs2NHXzfFbU6ZMwbZt2/DOO++gadOmaNasmdcfs2fPnnjppZcAGF/nV155BQ899BDKy8vxr3/9y7xf586dsWXLFrRp08brbSICGAyRwr7++mtMmDABAwcOxIoVKxASEmL+Xe/evTFx4kQsX74c4eHhDo9z4cIF1KlTxytt7NSpk1eOS6QmP/zwA7p3744hQ4Z45HgGgwGVlZUIDQ21u0+DBg1w3XXXmX/u27cv4uPjsXjxYotgKCIiwmI/Im/jMBkpas6cOQgMDMTixYstAqHqbr/9dsTExJh/Hjt2LOrVq4fvv/8e/fv3R/369dGnTx8AwLp16zB48GDExsYiLCwMV199NcaPH4/ff//d6rifffYZOnbsiNDQUCQlJZm/odZka5isvLwc06ZNQ1JSEkJCQtC8eXNkZWXh/PnzFvtJkoRJkybh//7v/9C6dWvUqVMHHTp0wKeffmreZ9asWXj00UcBAElJSeZhA2dDAkuWLEFycjJCQ0PRunVrvP/++zb3u3TpEp599lm0atUKoaGhaNSoEcaNG4dTp05Z7LdhwwakpaUhOjoa4eHhiI+Px7Bhw3DhwgXzPhUVFXj66afRunVrhIWFITo6Gunp6di8ebN5HyEEFi5ciI4dOyI8PBxXXXUVhg8fjv/9738Wj5eWloZ27dph+/btSElJQZ06dfCPf/wDzz//PKqqqgAYh0e6desGABg3bpz5uZk1a5bD5+b48eMYP348YmNjERISgqSkJMyePRuVlZXmNt5yyy2Ijo7G4cOHzfe7cOEC2rZti9atW5tfy/3792PcuHFo2bIl6tSpg+bNm2PQoEH4/vvvLR7TNJSTk5ODxx9/HM2aNUO9evUwaNAgnDhxAmfPnsX999+Phg0bomHDhhg3bhzOnTtncQzT+2Xx4sW45pprEBoaijZt2uDDDz90eL4mO3bswG233YaoqCiEhYWhU6dO+Oijjxzex9Tu/fv34/PPPzc/xwcPHgQAHD58GKNHj0bjxo3N77WXX37Z/BoBxiE2SZLwn//8B88++yySkpIQGhqKgoICWe02iYiIwDXXXIMTJ07YbGPNz4Tcz8CRI0cwfPhw1K9fHw0aNMCoUaOwfft2SJKEJUuWWOzrznNIfkgQKaSyslKEh4eL66+/3qX7ZWZmiuDgYJGYmCiys7PF+vXrxZdffimEEOKNN94Q2dnZYvXq1aKoqEi89957okOHDiI5OVlcunTJfIyvvvpKBAYGil69eon8/HyxfPly0a1bNxEfHy9qfgwSEhJEZmam+efz58+Ljh07ioYNG4pXXnlFfPXVV2LevHkiMjJS9O7dW1RVVZn3BSASExNF9+7dxUcffSTWrFkj0tLSRFBQkPjtt9+EEEKUlJSIhx56SAAQ+fn5YsuWLWLLli2irKzM7nPw7rvvCgBi8ODB4pNPPhEffPCBuPrqq0VcXJxISEgw72cwGMTNN98s6tatK2bPni3WrVsn3n77bdG8eXPRpk0bceHCBSGEEAcOHBBhYWGiX79+4uOPPxaFhYVi6dKlYsyYMeKPP/4QQghx+fJlkZ6eLoKCgsS0adPEmjVrxOrVq8W//vUvkZuba37M++67TwQHB4tHHnlEfPHFFyInJ0e0atVKNGnSRBw/fty8X2pqqoiOjhYtW7YUixYtEuvWrRMTJkwQAMR7770nhBCirKzMfK5PPvmk+bkpKSmx+9yUlpaan4fFixeLr776SjzzzDMiNDRUjB071rzf77//LmJjY0WPHj3M743MzEwRHh4uvvvuO/N+RUVF4pFHHhErVqwQRUVFYuXKlWLIkCEiPDxc/PLLL+b9CgoKBACRkJAgxo4dK7744guxaNEiUa9ePZGeni769esnpk2bJtauXSteeOEFERgYKB566CGLtgMQcXFxok2bNiI3N1esXr1a3HzzzQKAWL58udVjFRQUmLdt2LBBhISEiJSUFLFs2TLxxRdfiLFjxwoA4t1337X7fJWVlYktW7aIpk2bip49e5qf44sXL4qTJ0+K5s2bi0aNGolFixaJL774QkyaNEkAEA8++KD5GAcOHBAARPPmzUV6erpYsWKFWLt2rThw4IDdx01ISBADBw602Hb58mXRtGlTce2111pst3W+cj8D586dE1dffbWIiooSCxYsEF9++aWYMmWKSEpKsnpu3H0Oyf8wGCLFHD9+XAAQd955p9XvKisrxeXLl8236gFGZmamACDeeecdh8evqqoSly9fFocOHRIAxKpVq8y/69Gjh4iJiRF//fWXeVt5ebmIiopyGgxlZ2eLgIAAsX37dov9VqxYIQCINWvWmLcBEE2aNBHl5eUW5x0QECCys7PN21588UUBwOHFw8RgMIiYmBjRuXNni+fl4MGDIjg42OJCkJubKwCIvLw8i2Ns375dABALFy60aPuePXvsPu77778vAIi33nrL7j5btmwRAMTLL79ssb2kpESEh4eLxx57zLwtNTVVABDbtm2z2LdNmzbipptusmqr3IvR+PHjRb169cShQ4cstr/00ksCgPjxxx/N2zZt2iSCgoJEVlaWeOeddwQA8fbbbzs8fmVlpbh06ZJo2bKlmDJlinm76YI9aNAgi/2zsrIEAPHwww9bbB8yZIiIioqy2AZAhIeHWwSNlZWVolWrVuLqq6+2eqzqwUGrVq1Ep06dxOXLly2Oeeutt4pmzZoJg8Hg8LxsBSfTp0+3+Ro9+OCDQpIksXfvXiHElWCoRYsWFl86nD3eLbfcYv6MHzp0yBxIf/rppxb71jxfVz4DCxYsEADE559/bnHM8ePHW72vavsckv/gMBmpQpcuXRAcHGy+vfzyy1b7DBs2zGrbyZMn8cADDyAuLg5BQUEIDg5GQkICAODnn38GAJw/fx7bt29HRkYGwsLCzPetX78+Bg0a5LRtn376Kdq1a4eOHTuisrLSfLvppptsduWnp6ejfv365p+bNGmCxo0b49ChQ7Kei5r27t2LY8eO4a677rKY+ZaQkIAbbrjBqq0NGjTAoEGDLNrasWNHNG3a1NzWjh07IiQkBPfffz/ee+89qyEtAPj8888RFhaGf/7zn3bb9umnn0KSJIwePdri8Zo2bYoOHTpYPTdNmzZF9+7dLba1b9/e7efG1Ib09HTExMRYtGHAgAEAgKKiIvO+PXv2xHPPPYe5c+fiwQcfxOjRo3HPPfdYHK+yshJz5sxBmzZtEBISgqCgIISEhODXX381v6equ/XWWy1+bt26NQBg4MCBVtvPnDljNVTWp08fNGnSxPxzYGAg7rjjDuzfvx9Hjhyxec779+/HL7/8glGjRpnbbLrdcsstKC0txd69ex0+b7Zs2LABbdq0sXqNxo4dCyEENmzYYLH9tttuQ3BwsOzjr1mzxvwZT0hIwFtvvYXXX3/d6rmqyZXPQFFREerXr4+bb77ZYvvIkSMtfvbWc0jaxGCIFNOwYUOEh4fbvPDl5ORg+/btWL16tc371qlTx2qGV1VVFfr374/8/Hw89thjWL9+Pb755hts3boVAPDXX38BAP744w9UVVWhadOmVse1ta2mEydO4LvvvrMI1oKDg1G/fn0IIazyk6Kjo62OERoaam6Pq06fPm23rTW3nThxAn/++SdCQkKs2nv8+HFzW1u0aIGvvvoKjRs3xsSJE9GiRQu0aNEC8+bNMx/r1KlTiImJQUCA/T8TJ06cgBACTZo0sXq8rVu3ev25MbXhk08+sXr8tm3bAoBVG0aNGoWQkBBUVFSYc7eqmzp1Kp566ikMGTIEn3zyCbZt24bt27ejQ4cONtsZFRVl8bMpF87e9osXL1psd/S6ml57W+cMANOmTbM67wkTJtg8bzlOnz5tc1aZKYevZntcnYHWq1cvbN++HVu3bsX//d//ITExEZMmTcKmTZuctguQ9xk4ffq0RXBpUnObt55D0ibOJiPFBAYGonfv3li7di1KS0st/pCaptCakjhrqlkLCDDOhvn222+xZMkSZGZmmrfv37/fYr+rrroKkiTh+PHjVsewta0mUxD3zjvv2P29N5kCCDntb9iwIaKjo/HFF1/YPFb1HquUlBSkpKTAYDBgx44deP3115GVlYUmTZrgzjvvRKNGjbBp0yZUVVXZDYgaNmwISZJQXFxscxaRo5lFntKwYUO0b98ezz33nM3fV0/GNxgMGDVqFK666iqEhobinnvuwddff22RzP/BBx/g7rvvxpw5cyyO8/vvv3ulDIKj19VW8Ahcec/NmDEDGRkZNvdJTk52uS3R0dEoLS212n7s2DGLxzWx9bl0JDIyEl27dgUA9OjRAz169ECHDh0wYcIE7Nmzx+77zJXPQHR0NL755hun+3nrOSRtYs8QKWrGjBkwGAx44IEHcPny5Vody/SHuOYFd/HixRY/161bF927d0d+fr7Ft/KzZ8/ik08+cfo4t956K3777TdER0eja9euVjd3Cr6Z2iynRyQ5ORnNmjVDbm6uRXG6Q4cOWczqMrX19OnTMBgMNttq6497YGAgevTogQULFgAAdu3aBQAYMGAALl68aDX7pubjCSFw9OhRm4937bXXOj2/mlx5bkxt+OGHH9CiRQubbageDM2cORPFxcVYunQpli1bhm+//daqd0iSJKv31GeffYajR4+6fC5yrF+/3mI2lcFgwLJly9CiRQvExsbavE9ycjJatmyJb7/91uY5d+3a1SLwlatPnz746aefzO8Bk/fffx+SJCE9Pd3lYzrSsmVLPPbYY/j++++xbNkyu/u58hlITU3F2bNn8fnnn1tsrzlDz1vPIWkTe4ZIUT179sSCBQvw0EMPoXPnzrj//vvRtm1bBAQEoLS0FHl5eQAgq+hhq1at0KJFC0yfPh1CCERFReGTTz7BunXrrPZ95plncPPNN6Nfv3545JFHYDAY8MILL6Bu3bo4c+aMw8fJyspCXl4ebrzxRkyZMgXt27dHVVUVDh8+jLVr1+KRRx5Bjx49XHoeTEHCvHnzkJmZieDgYCQnJ9v84xsQEIBnnnkG9957L4YOHYr77rsPf/75J2bNmmU1RHDnnXdi6dKluOWWWzB58mR0794dwcHBOHLkCAoKCjB48GAMHToUixYtwoYNGzBw4EDEx8fj4sWL5p6vvn37AjDmWLz77rt44IEHsHfvXqSnp6Oqqgrbtm1D69atceedd6Jnz564//77MW7cOOzYsQM33ngj6tati9LSUmzatAnXXnstHnzwQZeeG1NV8qVLl6J169aoV68eYmJiLIKa6p5++mmsW7cON9xwAx5++GEkJyfj4sWLOHjwINasWYNFixYhNjYW69atQ3Z2Np566ilzaYbs7GxMmzYNaWlpGDp0KABjcLVkyRK0atUK7du3x86dO/Hiiy/aDUxqq2HDhujduzeeeuop1K1bFwsXLsQvv/zidHr94sWLMWDAANx0000YO3YsmjdvjjNnzuDnn3/Grl27sHz5cpfbMmXKFLz//vsYOHAgnn76aSQkJOCzzz7DwoUL8eCDD+Kaa65x9zTtmjZtGhYtWoTZs2djxIgRCAwMtNrHlc9AZmYmXn31VYwePRrPPvssrr76anz++ef48ssvzccy8cZzSBrlw+Rt0rE9e/aIcePGiaSkJBEaGirCwsLE1VdfLe6++26xfv16i30zMzNF3bp1bR7np59+Ev369RP169cXV111lbj99tvF4cOHBQAxc+ZMi31Xr14t2rdvL0JCQkR8fLx4/vnnxcyZM53OJhPCOF33ySefFMnJySIkJERERkaKa6+9VkyZMsViJhAAMXHiRKt22jrmjBkzRExMjAgICLCaKWTL22+/LVq2bClCQkLENddcI9555x2RmZlpMZNGCON05Zdeekl06NBBhIWFiXr16olWrVqJ8ePHi19//VUIYZwFNnToUJGQkCBCQ0NFdHS0SE1NFatXr7Y41l9//SX+/e9/mx83Ojpa9O7dW2zevNliv3feeUf06NFD1K1bV4SHh4sWLVqIu+++W+zYscO8T2pqqmjbtq3Vedk6h9zcXNGqVSsRHBxs87Ws6dSpU+Lhhx8WSUlJIjg4WERFRYkuXbqIJ554Qpw7d04cO3ZMNG7cWPTu3dtihlBVVZUYNGiQaNCggXlm3x9//CHuuece0bhxY1GnTh3Rq1cvUVxcLFJTU0Vqaqr5vqYZT9WnwAtxZQp4zdmHpvfaqVOnzNtM75eFCxeKFi1aiODgYNGqVSuxdOlSi/vamk0mhBDffvutGDFihGjcuLEIDg4WTZs2Fb179xaLFi1y+HwJYXs2mRBCHDp0SNx1110iOjpaBAcHi+TkZPHiiy9aPG+m2WQvvvii08dx9nhCXJkBZiqxYO985X4GDh8+LDIyMkS9evVE/fr1xbBhw8SaNWusZpkKUbvnkPyHJESNRWGIiEgRkiRh4sSJmD9/vq+b4vfmzJmDJ598EocPH/ZaLx9pF4fJiIjIr5iCy1atWuHy5cvYsGEDXnvtNYwePZqBENnEYIiIiPxKnTp18Oqrr+LgwYOoqKhAfHw8Hn/8cTz55JO+bhqpFIfJiIiISNc4tZ6IiIh0jcEQERER6RqDISIiItI1XSRQV1VV4dixY6hfv77L5eOJiIjIN4QQOHv2rNN1EmtLF8HQsWPHEBcX5+tmEBERkRtKSkq8WhZBF8GQaYmDkpISWcs8EBERke+Vl5cjLi7O6+vE6SIYMg2NRUREMBgiIiLSGG+nuDCBmoiIiHSNwRARERHpGoMhIiIi0jUGQ0RERKRrDIaIiIhI1xgMERERka4xGCIiIiJdYzBEREREusZgiIiIiHSNwRARERHpGoMhIiIi0jUGQ0RERKRrDIaIiIhI1xgMERERka4xGCIiIiJdYzBEREREusZgiIiIiHTN58HQxo0bMWjQIMTExECSJHz88cd29x0/fjwkScLcuXMVax8RERH5N58HQ+fPn0eHDh0wf/58h/t9/PHH2LZtG2JiYhRqGREREelBkK8bMGDAAAwYMMDhPkePHsWkSZPw5ZdfYuDAgQq1jIiIiPTA5z1DzlRVVWHMmDF49NFH0bZtW183h4iIiPyMz3uGnHnhhRcQFBSEhx9+WPZ9KioqUFFRYf65vLzcG00jIiIiP6DqnqGdO3di3rx5WLJkCSRJkn2/7OxsREZGmm9xcXFebCURERFpmaqDoeLiYpw8eRLx8fEICgpCUFAQDh06hEceeQSJiYl27zdjxgyUlZWZbyUlJco1moiIiDRF1cNkY8aMQd++fS223XTTTRgzZgzGjRtn936hoaEIDQ31dvOIiIjID/g8GDp37hz2799v/vnAgQPYs2cPoqKiEB8fj+joaIv9g4OD0bRpUyQnJyvdVCIiIvJDPg+GduzYgfT0dPPPU6dOBQBkZmZiyZIlPmoVERER6YXPg6G0tDQIIWTvf/DgQe81hoiIiHRH1QnURERERN7GYIiIiIh0jcEQERER6RqDISIiItI1BkNERESkawyGiIiISNcYDBEREZGuMRgiIiIiXWMwRERERLrGYIiIiIh0jcEQERER6RqDISIiItI1BkNERESkawyGiIiISNcYDBEREZGuMRgiIiIiXWMwRERERLrGYIiIiIh0jcEQERER6RqDISIiItI1BkNERESkawyGiIiISNcYDBEREZGuMRgiIiIiXWMwRERERLrGYIiIiIh0jcEQERER6RqDISIiItI1BkNERESkawyGiIiISNcYDBEREZGuMRgiIiIiXWMwRERERLrGYIiIiIh0jcEQERER6RqDISIiItI1BkNERESkawyGiIiISNcYDBEREZGu+TwY2rhxIwYNGoSYmBhIkoSPP/7Y/LvLly/j8ccfx7XXXou6desiJiYGd999N44dO+a7BhMREZFf8XkwdP78eXTo0AHz58+3+t2FCxewa9cuPPXUU9i1axfy8/Oxb98+3HbbbT5oKREREfkjSQghfN0IE0mSsHLlSgwZMsTuPtu3b0f37t1x6NAhxMfHyzpueXk5IiMjUVZWhoiICA+1loiIiLxJqet3kNeO7CVlZWWQJAkNGjSwu09FRQUqKirMP5eXlyvQMiIiItIinw+TueLixYuYPn067rrrLocRYnZ2NiIjI823uLg4BVtJREREWqKZYOjy5cu48847UVVVhYULFzrcd8aMGSgrKzPfSkpKFGolERERaY0mhskuX76MESNG4MCBA9iwYYPTccPQ0FCEhoYq1DoiIiLSMtUHQ6ZA6Ndff0VBQQGio6N93SQiIiLyIz4Phs6dO4f9+/ebfz5w4AD27NmDqKgoxMTEYPjw4di1axc+/fRTGAwGHD9+HAAQFRWFkJAQXzWbiIiI/ITPp9YXFhYiPT3dantmZiZmzZqFpKQkm/crKChAWlqarMfg1HoiIiLt0c3U+rS0NDiKx1RUBomIiIj8kGZmkxERERF5A4MhIiIi0jUGQ0RERKRrDIaIiIhI1xgMERERka4xGCIiIiJdYzBEREREusZgiIiIiHSNwRARERHpGoMhIiIi0jUGQ0RERKRrDIaIiIhI1xgMERERka4xGCIiIiJdYzBEREREuhbk6wYQERFplsEAFBcDpaVAs2ZASgoQGOjrVpGLGAwRERG5Iz8fmDwZOHLkyrbYWGDePCAjw3ftIpdxmIyIiMhV+fnA8OGWgRAAHD1q3J6f75t2kVsYDBEREbnCYDD2CAlh/TvTtqws436kCQyGiIiIXFFcbN0jVJ0QQEmJcT/SBOYMERERyWFKls7Lk7d/aal320Mew2CIiIjIGVvJ0s40a+a99pBHMRgiIiJyxJQsbStHyBZJMs4qS0nxbrvIY5gzREREZI+jZGlbJMn479y5rDekIQyGiIiI7HGWLF1TbCywYgXrDGkMh8mIiIjskZsEPWkSMGwYK1BrFIMhIiIie+QmQQ8bBqSlebUp5D0MhoiIiOxJSTEOfR09ajtviMnSjmlk7TbmDBEREdkTGGhcawy4khxtwmRpx/LzgcREID0duOsu47+JiapcqoTBEBERkSMZGcak6ObNLbczWdo+ja3dJgkhd76gdpWXlyMyMhJlZWWIiIjwdXOIiEiLNDLk43MGg7EHyN4sPNPQ4oEDTp8/pa7fzBkiIiKSIzBQ+SRpLQZgrqzdppKkcwZDREREamRrCZDYWGMOk5qH5uSWI1DR2m3MGSIiIlIbjeXcWJBbjkBFa7cxGCIiIlITR0uAmLZlZRn3UyNTOYKas+9MJAmIi1NVOQIGQ0RERGriSs6NGmmwHAGDISIiIjXRUs6NwQAUFgK5ucZ/Tb1VzsoRDB5s+34+wgRqIiIiNdFKzo29BO/77gNatjS277ffgM2bLWfDrVplPfXex4nhPu8Z2rhxIwYNGoSYmBhIkoSPP/7Y4vdCCMyaNQsxMTEIDw9HWloafvzxR980loiIyNu0kHNjL8H7yBFg5swrFadbtADOnAFGjjROo1+1SpWJ4T4Phs6fP48OHTpg/vz5Nn//n//8B6+88grmz5+P7du3o2nTpujXrx/Onj2rcEuJiIgUoPacG0cJ3jVVD3JUnBju82BowIABePbZZ5Fho2tMCIG5c+fiiSeeQEZGBtq1a4f33nsPFy5cQE5Ojg9aS0RE5EWmHJyKCmDWLCAmxvL3algCxFmCd3XVg5zCQtUmhqs6Z+jAgQM4fvw4+vfvb94WGhqK1NRUbN68GePHj7d5v4qKClRUVJh/Li8v93pbiYiIasVeDs7s2VdycNRQgdrVxG1TkFNY6J3je4DPe4YcOX78OACgSZMmFtubNGli/p0t2dnZiIyMNN/i4uK82k4iIqJacVRkcdYsIDTUmHPj60AI8H7itg8Sw1UdDJlINcZMhRBW26qbMWMGysrKzLeSkhJvN5GIiMg9Ks6lsclZgrc9DRoAUVH2f+/DxHBVB0NNmzYFAKteoJMnT1r1FlUXGhqKiIgIixsREemQvTo4aqK1IouOErztCQgApk0zziyzxceJ4aoOhpKSktC0aVOsW7fOvO3SpUsoKirCDTfc4MOWERGR6uXnG+vZpKdfmeqdmKi+db20VGTRxF5RRXuqqhz/3seJ4T5PoD537hz2799v/vnAgQPYs2cPoqKiEB8fj6ysLMyZMwctW7ZEy5YtMWfOHNSpUwd33XWXD1tNRESqZsrBqTn0ZJrq7esZWdVppchiTRkZxkrSxcXGQO3XX4E33zQ+x3JFRwPLlvk8H0oSQk6hAO8pLCxEenq61fbMzEwsWbIEQgjMnj0bixcvxh9//IEePXpgwYIFaNeunezHKC8vR2RkJMrKyjhkRkTk7wwG6wrH1UmSsSfiwAF1JCSb2nv0qO28IbW115FLl4xtPXVK/n0KCozBkA1KXb99HgwpgcEQEZGOFBYah8SccXARVpypJwuwDIhMuTRq6slyRO5zX11OjrFCtQ1KXb9VnTNERETkMn/KwVFDkUVXuPGcVjW2PyFKKT7PGSIiIj9kMFzJJVG6WKC/5OCopciiK1x4Tqsg4QhiEZ+W6sUGycNgiIiIPMteJWWlViWXk6/i64VO7QkMVM/QnRuqru+Jo4hFcxxFAOxn4VQBkCAQnzdXFcEeh8mIiMhzHFVSVmJVcoMBmDrV+X6vvKLsRVgL9Y7cNKHdRkiSMb0pMCwYk2GsQVQF+zWIAuLiIOXlqWb4j8EQERF5hhoqKctdRLRhQ++1oSat1DuSSVQJc/AjScAbP95o8fuVyMBwrMBR1Mh/atTI+PoXFBhnxqkkEAIYDBERkaeooZKy2pKnfd1T5iGjk742Bz8BgY6rTmffVIh8kYG4yoPGwCcnx/hvaSnw6qs+rylkC3OGiIj0yBsJzmoIRNSUPO2sp0ySjD0lgwerLjgQVaJG0NPT4f5VBgEpwLR/mvEfDeU/sWeIiEhvvDVso4ZAxNkiokouBqqGnjIXPNS+SHbvz21Nt0EImG9XAiFtYjBERKQn3hy2UUMg4mgRUaUXA1VDT5kT1XN/5n/veIp7ZYXBHPysKu2hUAuVwWCIiEgvvJ3grJZARC0FDNXQU1bDrU2+sQiAHOkV8a1F709giLqG8jyJy3EQEemFUstU2KozFBdnDISUnEHki8KP1R+zcWMgMxM4dsyna445C3qqu3T+MoLrBHutLa5S6vrNBGoiIr1QathGLZWUlU7gtRUERkdfSZa2teaYF3rK+kTtwoY/Osve3zJOU08gpCQGQ0REeqHksI2GZhJ5hCkXq2YP0Jkzxn+jooDTp69sj431aE+ZZe/PlUAoAAakoBjNUIpSNEMxUlB+4iLqNq7rkcf1FwyGiIj0wpTgfPSo42EbNS5T4UvOhtvkTKEPDwe++go4edIjPWV3JX6N3EOOp7sPRT7mYTLiUGNZlE0KLYuiIUygJiLSC7UkOGuJnDIEcqbQHzlifF5HjnS76GD1xGdngVD56+8hXxpuGQgBmiv2qBQGQ0REeqKWmVZaILcMgZdyse5NLpY98wu4MutLVBpQ/4UnfbssisYwGCIi0puMDODgQculElS2VpRPGQzA+vXAfffJCyg8mItVPfj57z7Hw5VHtpdaTH0301ixRzVgzhARkR7pLcFZLlszwmypHlDUIhcrq1MR5u1xXOyw5sNeYSe40kCxR7VhMEREVJ0vatOQOtibEeZIaemVXKzhw2VNobcc8nIcCP2y5n9IHvAP+e0B5PdUNW5srD3F9zqLLhIRmdnqFYiNNV7oOIRkyd+CRoPBmBjtrEeopuoFKu0Um3wv5F6M/e3fsg9Z66uy6Vwc9VRFRRlnuKn8va7U9ZvBEBERYL9XwPQ1nsnFV/hj0Ci3OreJverRfweJI9Ov1PWpguMgcevbP6DHPe3ca7c9pvczYN1TZe+yr8L3ulLXbyZQExF5e80uf+LNhV59yZX8GRtDX8/2LTQmPgcFQkpPw4cYiSKk2Q2Eqic+ezwQAuzPGmze3FgV216jAF2+1xkMERFx9o08/hw0ulJ1++8yBNKwDPPMr6fWpzm8y6p/bbM988ubbM0aXLLEshJ2TTp9rzOBmoiIs2/kkRs0FhYCffoo1iyPcDYjDMBfwREYeDkfRSVpqBrmPD/K8jA9PNNOV9WcNZibK+9+Onuvs2eIiEjJNbu0TO4FcsQIYPlyY1CUm2v8V+29RQ6qc1dBQhUkjLr8LgrQx+7Q15tjNirf++MqT7/XDQZtvc52MIGaiEjO7BtbybJ642qScXUaSbBee+OzaF282GIZi8OIQxbmYiWs2+61K6i3Zus5e68DxpyiEyecP54CifRMoCYiUgrX7JLHNJQkZ22ImlScYF296vNNxU8iEQeRhgKMRA7SUIAkHDAHQo/3KPR+74+c9dDcZXqvO2r86dPAqlXO2+hHifTsGSIiMrFTJwZz56q+R0Mx+fnAsGHu3VclPWzLp27BiFevl72/oldJJUo8GAxAkyb2E6mdvU7OajJ58HVmnSEPYjBERLL5WzFBT3N2IZWjeqFChbjSmdUvaifWnu7ivcbYo1SQIXe4097rVNv7u/AZU+r6zdlkRETVcc0ux4qLaxcIAYrMVFr3/E70nyE/oLHsFvBBIAS4VuKhNu/R2s6erM39VVqwk8EQERHJ54lAxkuz8ix7fxwHNC2DD2DfpSSvtMNtSpV4qO2MMnfvb28I0JRn5MPK10ygJiIi+WoTyEiSMQfLxurt7tj5wc8Wyc/OVE98VjwQkjMFXakSD84S4Z29Tu7cX+UFOxkMERGRfO7OKHNlVp6DwKF68NN1TGunD6uKuj9yZ4fVNkiRq7azJ925v8qrvDMYIiIi+RxdCE2mTTNe1Kv7ewkLp8MgNgKHkqBEZEj5Lvf+qGJ6kCtT0JUs8WBv7TK5r5Or91d5lXfOJiMiItc5K0Pgzqy8vwOHKiEsvqlXwRgIDMcKZQsfusLW+QLuzQ5TssRDbWdPyr2/mzPQOLXegxgMEZHf8mUpAA899okfTiHm2igcRCKa44jNIYsqSDiCWCThAAxCZaUO7M2Quu8+YOZM5/e3NQXd26+r0u8bN6u8c2o9ERE55utpyrUoQ2A5CtQIqSi0WAKjpgAIxKMEhq8KgcJA9dSBcjRDSk4gBNgeGvJmiQdfvG9MQ4DDhxtf/OrPlwqqvDNniIhIizS2HEL5kXKHM7+awYVFYL2xTIU75MyQkkPJBYB9+b6pbZ6SF6l+mKyyshKzZs3C0qVLcfz4cTRr1gxjx47Fk08+iYAAebEch8mIyK8ouBxCbbgy4SwVhSiEG4vAenKZClfVZuFaQLnXyTQkdvQoMGUKcOqUOtrDCtTyvfDCC1i0aBHee+89tG3bFjt27MC4ceMQGRmJyZMn+7p5RKQELpFhSalKxS6qKK9AncggpKAYzVCKVDRDMVJQBduvlcVXcUMKkBjreDV1eweRJGONmsGDlX1fuDLzyVdDQ7aGxOxR6n2jwirvqg+GtmzZgsGDB2PgwIEAgMTEROTm5mLHjh0+bhkRKcLWH/NGjYBRo4wXPy0FRp4K6lQ0Tbl6789QfIaDmGyR+1OCWEzGPPMsMLtxjqOcEmd8FPzJHt6aPRt46y3rHB1vLwBsL5/JGR9Nb/cl1ecM9erVC+vXr8e+ffsAAN9++y02bdqEW265xcctIyKvs5ffcOqU8UJiL2dETrXfmty5jyvkFt6TQ6lKxTZUVVbZzP0ZinyswHA0r5EEHSsdRb40HCIv3/k12V5OSXS0vMYpfRGXWyTxiSeAgweNs8Zycoz/Hjjg3UDIUT6TM0rmMKmFULmqqioxffp0IUmSCAoKEpIkiTlz5ji8z8WLF0VZWZn5VlJSIgCIsrIyhVpNRLVWWSlEbGzNGnq2b5IkRF6e8X55edb3i4298ntb3LmPK/LyjG201e7qbZfL9NzYOqbpuHFxxv08wNnTH4BKcRixwuDo9XGlPZWVQhQUCJGTY/z3q6/kvQ8KCjxyvi61MSvryjl64rX1lIICec+ZF983nlBWVqbI9Vv1wVBubq6IjY0Vubm54rvvvhPvv/++iIqKEkuWLLF7n5kzZwoAVjcGQ0Qa4sofc9Mf8eXLXQ86PB2o1OQsqHP3AmRqtxcuwlWGKpeuoZVrvRysKBz8OWUreA4MtPw5Ls53gZAQxkDS1UDIl8GbHQyG/hYbGyvmz59vse2ZZ54RycnJdu/DniEiP+DqH3NAiEaNXAs6vBWoVCc3qHMnULB1UXbzIuzqU21B7muVk+P6OVY/Vy8Ff261w945ZmUZX0tf96642jPk6+DNDqWCIdXnDF24cMFqCn1gYCCqqqrs3ic0NBQREREWNyLSGHfyFuxNGQaMf/JrLgSpxOKR3kx2zsioVS6KKyu+//XHRYurpwUlcpjUUKPGYADuv9/GE/A3SQLy8tSR1C9nQd1GjYAPPlAmh0nlVD+bbNCgQXjuuecQHx+Ptm3bYvfu3XjllVfwz3/+09dNIyJvMv0xd3WqtTPVgw4lZmV5O1BwYZqyqwvNWz7tYVf+W3NW3IkTzg/midXWMzKMMwh9VWbhueeA06ft/7568OzrqeNyKj4vWqTrAKg61QdDr7/+Op566ilMmDABJ0+eRExMDMaPH49///vfvm4aEXmTK1OtJQlo2NBxz5BJ9aDDE4GKs+nyzoI6U6G72gYKdrgSAP15qAyR8ZGOd7JV6kBOMPLyy54JWnxVo8ZguLKivDNqmZpu6k2ztfSGt6f1a4zqK1B7AitQE2mYs6Jxpqv9smXA1KmuLQTp5uKRDttma40nU4kAwPY3dA8O89Su98cJd+vWALYXI9USV6pNq+1cNVy0VKnrt+pzhohI56rnxWRlGXuAqjPljNx++5Vv7jUjAnvVfk29T67cx8SVNZ68nO/iSu7Psd0n7Of+OFKbujWAenpL3CW3/dHRXuvlc5upN23kSOO/GgmElKT6YTIiIrPu3YFbbzX+/+RJ62+57gwLuHMfZwt02loewlG+i4vf3GvX+9PEtTubOEs2d0brhfzktv/hhxlsaJBLw2QlJSWIi4vzZnu8gsNkRBomdyiqOneGBVy5j9whEznDJTLPz5UAaN/ag2jZL1H+HeTIzTVWznaVShaNrTVnQ6qAsVfoxAltn6fKqHKh1latWmHq1KmYPn066tat6602EREZ2ctRMQ1F2RticifJ1pX7eGoWmoPzqxo2HMOxwrymlzOWh0iU1z5XuNOzo9RipJ7iKCCWk9D/5pvaOE+y4lLO0Lp167B27Vq0bNkS7777rrfaRETkfCgKMA5FeXoNMTk8NQvNyfnNRRYCYPv8trz1g3u5P+6SU7emZiCgZA2g2pKzdpy93K+4OGN9IS2cJ9nk1myy999/H0888QQaNmyIV199FWlqypq3gcNkRDKpadaJJ4eiPK22s9AA2eeXhgIUIQ2AQkGPI85mxX30kTHBXQ3vH1fY66GzN9tPTZ8TP6fq2WR333039u3bh0GDBmHgwIEYOnQo9u/f7+m2EZGSPLmquicoURDRXbWYhWaa9TUyXV67C3NKlev9ccbZrLjhw7U3a8mdHkjOzvI7bk+tF0Kgf//+uP/++7F69Wq0a9cOjzzyCM6ePevJ9hGRElyZJq4UJZZ4qA2Z0+W71v3J5tT3Uqj8/Oyp5RIgqqPEkiykei4Nky1atAjbt2/H9u3b8fPPPyMwMBDt27fHddddh44dO2Lp0qXYt28fVq5cia5du3qz3S7hMBmRA6YhH0dFDX0xG8gTQ1FKsDFkIgU5b08ADDiIRMTiCGxm4ajl/Pyd3FlyOTnGniBSlCqHyZ577jmUl5cjMzMThYWFKCsrwzfffIPXXnsN//znP7F+/Xo8+OCDGDt2rJeaS0Qep9ZvxrUpiKikwECMGhsM6a6RkNLTZAVCQgAGEYi4vHmQbFVLVNP5+Tu190CSIjy+HMeJEycQExMDgy9meNjBniEiB+R+M87KAl591evNsWKrDk9cnM/XVnKl7s/TvQvx1Po0279U6fnphlZ6IHVKqeu3x4MhIQQ2btyI1NRUTx62VhgMETngyppLvpo+7M3ZOzKPPf26QrywLU32YV1e6oKzk3xHwbXjyDWaDYbUiMEQkQNyKuuaxMX51zdkJ9WfXen9mXRtEV7/Tj1fAslF7KFTJQZDHsRgyE/w27P35OcDw4bJ21dtK3K7y05tmaq/05nlVH/2/7+eOsO/MaqjygRqIp9RWw0cf5ORYcwJkkPrq48DDmvLBMB+9efbmm5TtuozKYv1g3SLwRCpnxpr4PijwYPl7afxWTXLJm9GWpDjGXQBEIhHCVJQbBH8rCrtoWBLiUgpDIZI3dS8PpW/cbb2lCQZcyhSUpRtlwdUL3p452s3oBnkV38mIv/HYIjUTa01cPyRVur6yFA4d4/Nqs8mmq3+TEReEeTrBhA5TFpU8/pU/qDmcz94sHEasa0ZViqfVWMZ9HR0uO8udDKek7PaMhrsBSMi1zEYIt9yMrWZ1WG9yNFzf/Cg6mfVfPvRXnS8I9lqewAMSEExmqEUpWiGYqSgCoE1Yp5IIH+eMedMkmzXltFILxgR1R6n1pPv2JnabFHobPBgVof1BjnPvQp7gZzV/RmKfMzDZMTBTnBdE2vLEKka6wx5EIMhFXJlcdBVq1gd1pPUujCrDb9tOISr+yTI2nco8pEvuRHgsbYMkWoxGPIgBkMqJHcJCFOBP36D9xxXn3uFuVL1Gfg79tFQgEdE8il1/WbOEPmGq4nRGRnGITN+g689lSWln/zhJJpc21j2/ja/vrky69Afqmd7C3vJSKcYDJFvuJMYbaoOS7WjgqR0y94f54GQ0/5rlQV4muRsMgORH2OdIfINPy7wp3o+eO7PHjvrsO5PTdWrPssayFdBgKdprPJOOsdgiHzDjwr8aY67z73BYMw3ys01/uuk6nf14CeieX2nzarVml8Mrt3HKu9EDIbIhzIyjDN8mje33B4byxli3ubqcy9jodxL5y6Zg59AyYBUFOJO5CIVhVYLngJu9P44oqbg2sWg0edY5Z2Is8lIBZi06TtynnsHNYmqBDAcK7ASV4InW7V+ShCLyZiHfOHlANfXsw61mHeTm2sMcJ3JyTGu5k6kIE6t9yAGQ0RucjJlvQoSjiAWSTiAKgRiKPKxAsMBCMtuZyVrQvkquNZoIUu1l1ogfWMw5EEMhojcJPNCmYYCFCMFB5FoWf25On+u9aPlOkemtrPKO6mQUtdv5gyRJa3lO5DHiSphzv0ZmS5vKnphTikMBcX2AyHAv3NPtJx3o6Z8KyIfYTBUW/4UPMhIkiX/VH3mV0DglQtiKVyYsq7nWj+ePnel/65wMgPpHIsu1oYWkyXtsZfvYKozwj+IfkdOvZ9ipKAEsYiVjkJyNISSkiK/18Mfa/14ss6Rr/6usMo76Rhzhtyl1WRJW7Sc76AVKpgx59aaX8CV97rFRli/1/Wce+Kpc/envytEHsCcITXztyJlWs530AIfDj+6UvX5931nbNf9kTuEoufcE0+cu7/9XSHSEAZD7vC34EHPuR7epvAyB9WDH1eXvYhuGWV/x4wM4OBB4/TqnBzjvwcOWPdS6Dn3pLbn7m9/V4g0hDlD7vC34IHrOnmHs2/6kmT8pj94cK16S1wZ/jq0+Sjir2/ufEdb5C6Uq+fck9qcu7/9XSHSEAZD7vC34MG0rpOzfAeu6+QaV77pu1DMzu3cHwCAm4GQq+QGTv7I3XP3t78rRBqiiWGyo0ePYvTo0YiOjkadOnXQsWNH7Ny503cN8rdFIeXkO7z8svGi7Q8lBJTiwW/6rgx9/bhqv+trfvlTiQit8re/K0Qaovpg6I8//kDPnj0RHByMzz//HD/99BNefvllNGjQwHeN8sdEUUf5DtOmAVOnsv6Qq2rxTb+udN7t3J82t13tWjtZX0od3Pm7wiCWyDOEyj3++OOiV69etTpGWVmZACDKyso81Kq/5eUJERtrufh2XJxxu1ZVVgpRUCBETo7x3+XLhZCkmguMG7dJkrbP1dsqK43vD1vPn+k5jIsz7ids72LvtuWt7z3Txrw8vr5qI/fviq39YmP5mpFf8dr1uwbV1xlq06YNbrrpJhw5cgRFRUVo3rw5JkyYgPvuu8/ufSoqKlBRUWH+uby8HHFxcd6pU6CC+jFe40/1h3y9eCdgVaNHCGBYjRXfHfH4J9WfXl9/4+z9ynpEpBOKrS3q1VDLA0JDQ0VoaKiYMWOG2LVrl1i0aJEICwsT7733nt37zJw5UwCwunk7svQ7BQXyuikKCnzdUsd8/Q3axuMfQpwYijyHT+tns77xbrtq+/rW7EX8u4eLvMzU42jv9arR40ikZUr1DKl+NllVVRW6du2KOXPmAAA6deqEH3/8EW+88Qbuvvtum/eZMWMGpk6dav7Z1DNELvKHqb4+XmZkeOwW5B3NQAAGIwXFaIZSlKIZipGCKlj3tlg2s5vX2gWgdq+vPy1FozVemqVIpGeqT6Bu1qwZ2rRpY7GtdevWOHz4sN37hIaGIiIiwuJGbtD6VF9XK/p6KBm1euJz3tHrAQBVCEQR0vAhRqIIaeZA6IMHv3Z95penuPv6KlxIkmrwhy8pRCqj+mCoZ8+e2Lt3r8W2ffv2ISEhwUct0hGtT/V15Rt0LWZUZXUqcnvm16iFPWWfjse58/pyyQjf0/qXFCIVUn0wNGXKFGzduhVz5szB/v37kZOTgzfffBMTJ070ddP8n9ZLCMj9Zrxqlcs9HdWDn3l7Uh0e/pXbCn3X++OIO68vl4zwPa1/SSFSIdUHQ926dcPKlSuRm5uLdu3a4ZlnnsHcuXMxatQoXzdNH7S81pTcb8YffOC0p+M/N693u/dnyqo0uS1WnquvL4dofE/rX1KIVEj1U+s9QbGpef5MiyUETFPHHS0z0rAhcOqU00OloQBFSLP7+0e7FeI/39j/verJfX0LC41DiM4UFDB519tsJbHHxRkDITV/SSFygVLXbwZD5DumhOXCQuPPaWnGmyeDLAd1fgAYLyZz5zo9zEjk4EOMtNjm/58cG+QEmKxNpBwtfkkhcoFS12/VD5ORn8rPB5o0Afr2BZ591njr29e4zZOzkZwMA6XNHSzrMKVohpEJPpz5pRYcolEX06KwI0d6/osEkY6wZ4iUl58PDBvmeJ+8PM929f/9DfrHt7dg4tLrzXV+AmDAQSSiOY4iAOzpkI1DNESkAA6TeRCDIRUxGICEBOMwiyOxscDBgx4JQJwlPA9FPlbAOJRmERBxaQPHOERDRF7GYTLyT8XFzgMhwNjj4Ob07G8/2uvSzK98kYGAvBUIiNXgjDlf8vUQDVdsJyIPUf1yHORnXJly7cK+lkFPssN9r6/3PTafvdZyY0YGMHgwezq0gsuBEJEHMRgiZblSFdfBvv8rOIQWveVXIbccDL7W9k6mng5/4M9DWD5eb46I/A9zhkhZtcgZkjPkZZIc/D/8cukf7rVR6/y518Q0td9eFWwmvBP5FeYMkX8KDARee835fvPm4dS+P9yu+qzrQMifF1HlciBE5AUMhpTir8me7pxXRoZx6nx0tNWvTiEaGciDNCwDjds0dHiYMPxlDH4qDRAFLrbBH+lhEVUuB0JEXsCcISWsWAFMmGC57IM/DFvUZjjm74Tlv/I/x4sjtgEACpGGIqShCvaHNyyv8+H+PSTkKld6TVzJjVJT/hFXbCcibxA6UFZWJgCIsrIy5R/80Uerj95Y3iRJiLw85dvkCXl5xvbbOicn52Xv6bB1C8FFr7TBL+XkyHtSc3LkHzMvT4jYWMv7x8b67rmtrDQ+vq3X3fTax8UZ9yMizVPq+s1hMk+xNVy0fDnw4ov27yOENoctXByOqSivcDv3p0KEeqQNLtPisKane03UmH/E5UCIyAsYDHlCfr5xhkt6OnDXXcZ/ExKAe+91fl8tJnvKHI5JCyqGJAFhkXYCmr+F44Lra355M5HW1uvZuDHw9NPqDopSUoxDhPYiTkkyLpmRkuL8WJcuAePHqzP/yMl6c7obHiWiWmMwVFuOvj2Xl8s7htaSPWW2NwZHkYpC3IlcpKIQAbhy4awe/FwQdbzWBpefW3uv55kzwMyZnl9I1pM81WuSn28MLH7/3f4+vp61lZFhLL1QUADk5Bj/PXCAgRARuYUJ1LXhaKjGFVpL9pTZ3rmYgsbwUtK4NxJp5byep0+ru7CfqdfEVlK5nEVU7RU0tMeXgbw/FckkIp9i0cXaKCw0DqHURqNGxguKBnIcRJVAQKDkdKX3KgCmfgmL/glPLnxqKr539KjtC7c7xfdceT3j4tRd2M+dGWDOChraUlDAgISIvIZFF7XAE9+KFy5U7wUVQGxgqTnxOSDQGMxUIRCTMe/v/1vnpwTAGARZ/caTuSbeSKR15fX0xBCRN5O03VlE1VkeVnWu5B8REakcg6HaqO3w1qOPGockVKb6zK+jVbbPcSUyMBwrINVMYm3UyPHBPZlr4ulEWldfz9oEw7aStBMT5eUjeSuIcvV8OGuLiPwEc4ZqwzR7x9FQTVQUEB5u+Y27USNgwQLg9tuVa6sD/aJ34qszXWTvf+VUMwBDjZXejx4FRo92fhBP5Zp4crV50+spt3fE3WC4NguNerPIpNzzadQIWLRInTlTRERuYM5QbTlKOJUk44XNUxdrD3Jl0dOKs5cQUi/E9i9r5qYYDEDfvs4PqtZck/x8YNgwx/vUZjHQ2iw0au+95qlcLGd5WIAxEDpyBAix834gIvIg5gxpRUYGMG2a9YUrMNC4PSPDvfwND5t+XaHbhQ/tBkK2hnoyM41rjnmi1o0vOFg3DUDtC/u5Wx9JiXXHnOVhSZKxR4iBEBH5GQZDtZWfD7z0kvVFqKrKuN2HNWmqBz8vbEtzuO+F03+5VvjQXj2eY8eM08+F0G6F4IwM4MQJYPZs4zBndbUt7OdufSSlVmtnQUMi0iEGQ7WhslXCnx/gfu9PeFS4/Adydt6SZOxZ0fIFNTAQ+Pe/gZMnPVvYz936SEqu1s6ChkSkM0ygrg1vrRLuAsugx/FjnC09h3pN69X+QeWc9+nTwFdfGYMKFeVKuczThf3kJN3HxloPIyq9WjsLGhKRjrBnqDaU/Lb+tzdGbnS798cjgRAg/3xOnvR5rpTquFsfyZPrjhERkQUGQ7Wh0Lf16sHPpA972l3vCwDO/O9P1xc9dZXSvRT+xp28HK7WTkTkNZxaXxveWBICwNIJX2P0Gz2ttg9FPuZhMuJwZYiqBLGIy/PQel9yeem8fcadpSt89bi26gzFxclbd4yISGOUmlqv72DIExdB06wqwDIwsFX7xcHjORvyGop8rMBwAMKyO8+T6325wpXzVjNvFjH0Fl8Fb0REClMqGILQgbKyMgFAlJWVXdmYlydEbGz1ESXjz3l5rj+ArWPFxVkey8Y+hxErhiJPWGb2WN8CUGl9/Oo3STI+XmVl7Z8sT5+3muXlGZ87W8+nJGnnPIiI/JTN67cX6LNnyBuVfB19W//78aqEZa+OaZHT4ViBlbB8vIObjiChZ6zxB7mrqfuiqrOrvRRq6dWoTSVoIiJShFI9Q/qbWi+nRk5WlnEJDVcugjamIh/echRJNzTFQUxG85rDWwACIFAFCXORhVUYDIOo/nixV/7rg1lrsrkyBVtNQ1IqKItARETqoL/ZZF6u5Nv7qt3mmV8JNzRHCooRhyN2n+gACMSjBIYCB4/nD7O37FWsNi1OqnSlbjUHmEREpCj9BUMevgheOncJ98SvMwdABX92svh9M3jg8TxZY8ZgMA675eYa/1WiOrbKKnUD8I8Ak4iIPEJ/wZAHLoLf5+1D36hdkCQgtH4I3inpZ3ffhe/IXObCUbs8VWPG1sKqiYne75VRal0tV7CIIRER/U1/wZAbF8HLFy5j7tAic+9P++HXYP0fnW3e/ZZG2y2mJl119yDPXHRru4CmL4ep1DgkxSKGRET0N/0FQzIvgj99dgA3N9wBSQJC6gZjysepNg/3n1sKcbGswhz8fHaym/3Hq8nVi667C2j6ephKrUNSXKGdiIig56KLNmY2nQ1vjMy/3rCa5l5dauRuvPpGODqNbCW/Afn5wP33GxcvrS46GnjzTe9fdH09NV/tFavVMt2fiIgsKDW1XnM9Q9nZ2ZAkCVlZWbU70N+9LDndXsVI5CANBWjw1zGbgdCc/gX468xfEAIo/LOT64HQ8OHWgRBge5s3+HqYSu1DUqbyAFxQlohIlzQVDG3fvh1vvvkm2rdv75HjHfj6GEZtz8KHGIkipKEKxotgz/rfYvt7P5mHvmZ8mY6wq2QmQlfnaHgKuFLTyNuzqNQwTCVnSMoXM92IiEj3NBMMnTt3DqNGjcJbb72Fq666yiPHjL8uBtO6FgIAnk4vwPlTFyAEsKm8A7re3ab2D6CWWVTOksYBY2/IqVPebYejnCdfzXQjIiLd00wwNHHiRAwcOBB9+/Z1um9FRQXKy8stbmbVemkCQwLx4vY0CAE8tSEddRrW8WyjfT08ZeIoidvEYADuuMP7wYetISm1FWQkIiJd0UQw9OGHH2LXrl3Izs6WtX92djYiIyPNt7i4OOMvlM4VV8PwlElGBvDRR87zYZQufuhsppsQwH33AevXc9iMiIi8QvXBUElJCSZPnowPPvgAYWFhsu4zY8YMlJWVmW8lJSXGX5gKBSlFbYX9GjZ0HFD4ovihs6FEADhzBujbl8NmRETkFaoPhnbu3ImTJ0+iS5cuCAoKQlBQEIqKivDaa68hKCgIBhsX99DQUERERFjcfEJts6jUMmzn7mNx2IyIiLxA9cFQnz598P3332PPnj3mW9euXTFq1Cjs2bMHgWqfBq2mwn5qGrZz57F8tY4ZERH5NU0WXUxLS0PHjh0xd+5cWfsrVbTJITUU9lNj8UNnbbLHWwUiiYhINVh00d+oobCf2obtnLXJESWH8oiIyK9pMhgqLCyU3StENahp2M5ZmxxReh0zIiLyW5ocJnOVKobJ1EYNw3a22lRYCIwYYZxBZouv1zEjIiLFKHX9DvLakUndTMN2ahIYCPTpA7z1lnHWGGCZR6SGdcyIiMjvaHKYjPycGofyiIjIb7FniNQpIwMYPFh9Q3lEROR3GAyReqlxKI+IiPwOh8mIiIhI1xgMERERka4xGCIiIiJdYzBEREREusZgiIiIiHSNwRARERHpGoMhIiIi0jUGQ0RERKRrDIaIiIhI11iB2pfUuHI8ERGRzjAY8pX8fGDyZODIkSvbYmOBefO4ECkREZGCOEzmC/n5wPDhloEQABw9atyen++bdhEREekQgyGlGQzGHiEhrH9n2paVZdyPiIiIvI7BkNKKi617hKoTAigpMe5HREREXsdgSGmlpZ7dj4iIiGqFwZDSmjXz7H5ERERUKwyGlJaSYpw1Jkm2fy9JQFyccT8iIiLyOgZDSgsMNE6fB6wDItPPc+ey3hAREZFCGAz5QkYGsGIF0Ly55fbYWON21hkiIiJSDIsu+kpGBjB4MCtQExER+RiDIV8KDATS0nzdCiIiIl3jMBkRERHpGoMhIiIi0jUGQ0RERKRrDIaIiIhI1xgMERERka4xGCIiIiJdYzBEREREusZgiIiIiHSNwRARERHpGoMhIiIi0jUGQ0RERKRrDIaIiIhI11QfDGVnZ6Nbt26oX78+GjdujCFDhmDv3r2+bhYRERH5CdUHQ0VFRZg4cSK2bt2KdevWobKyEv3798f58+d93TQiIiLyA5IQQvi6Ea44deoUGjdujKKiItx4442y7lNeXo7IyEiUlZUhIiLCyy0kIiIiT1Dq+h3ktSN7SVlZGQAgKirK7j4VFRWoqKgw/1xeXu71dhEREZE2qX6YrDohBKZOnYpevXqhXbt2dvfLzs5GZGSk+RYXF6dgK4mIiEhLNDVMNnHiRHz22WfYtGkTYmNj7e5nq2coLi6Ow2REREQawmGyGh566CGsXr0aGzdudBgIAUBoaChCQ0MVahkRERFpmeqDISEEHnroIaxcuRKFhYVISkrydZOIiIjIj6g+GJo4cSJycnKwatUq1K9fH8ePHwcAREZGIjw83MetIyIiIq1Tfc6QJEk2t7/77rsYO3asrGNwaj0REZH2MGfobyqP1YiIiEjjNDW1noiIiMjTGAwRERGRrjEYIiIiIl1jMERERES6xmCIiIiIdI3BEBEREekagyEiIiLSNQZDREREpGsMhoiIiEjXGAwRERGRrjEYIiIiIl1jMERERES6xmCIiIiIdI3BEBEREekagyEiIiLSNQZDREREpGsMhoiIiEjXGAwRERGRrjEYIiIiIl1jMERERES6xmCIiIiIdI3BEBEREekagyEiIiLSNQZDREREpGsMhoiIiEjXGAwRERGRrjEYIiIiIl1jMERERES6xmCIiIiIdI3BEBEREekagyEiIiLSNQZDREREpGsMhoiIiEjXGAwRERGRrjEYIiIiIl1jMERERES6xmCIiIiIdI3BEBEREemaZoKhhQsXIikpCWFhYejSpQuKi4t93SQiIiLyA5oIhpYtW4asrCw88cQT2L17N1JSUjBgwAAcPnzY100jIiIijZOEEMLXjXCmR48e6Ny5M9544w3zttatW2PIkCHIzs52ev/y8nJERkairKwMERER3mwqEREReYhS12/V9wxdunQJO3fuRP/+/S229+/fH5s3b/ZRq4iIiMhfBPm6Ac78/vvvMBgMaNKkicX2Jk2a4Pjx4zbvU1FRgYqKCvPPZWVlAIwRJhEREWmD6brt7UEs1QdDJpIkWfwshLDaZpKdnY3Zs2dbbY+Li/NK24iIiMh7Tp8+jcjISK8dX/XBUMOGDREYGGjVC3Ty5Emr3iKTGTNmYOrUqeaf//zzTyQkJODw4cNefTLVpry8HHFxcSgpKdFVrhTPm+etBzxvnrcelJWVIT4+HlFRUV59HNUHQyEhIejSpQvWrVuHoUOHmrevW7cOgwcPtnmf0NBQhIaGWm2PjIzU1ZvIJCIiguetIzxvfeF564tezzsgwLspzqoPhgBg6tSpGDNmDLp27Yrrr78eb775Jg4fPowHHnjA100jIiIijdNEMHTHHXfg9OnTePrpp1FaWop27dphzZo1SEhI8HXTiIiISOM0EQwBwIQJEzBhwgS37hsaGoqZM2faHDrzZzxvnrce8Lx53nrA8/bueWui6CIRERGRt6i+6CIRERGRNzEYIiIiIl1jMERERES6xmCIiIiIdE2TwdDChQuRlJSEsLAwdOnSBcXFxQ73LyoqQpcuXRAWFoZ//OMfWLRokdU+eXl5aNOmDUJDQ9GmTRusXLnSW813myvnnZ+fj379+qFRo0aIiIjA9ddfjy+//NJinyVLlkCSJKvbxYsXvX0qLnHlvAsLC22e0y+//GKxn7+93mPHjrV53m3btjXvo4XXe+PGjRg0aBBiYmIgSRI+/vhjp/fxh8+3q+ftL59vV8/bXz7frp63v3y+s7Oz0a1bN9SvXx+NGzfGkCFDsHfvXqf3U+IzrrlgaNmyZcjKysITTzyB3bt3IyUlBQMGDMDhw4dt7n/gwAHccsstSElJwe7du/Gvf/0LDz/8MPLy8sz7bNmyBXfccQfGjBmDb7/9FmPGjMGIESOwbds2pU7LKVfPe+PGjejXrx/WrFmDnTt3Ij09HYMGDcLu3bst9ouIiEBpaanFLSwsTIlTksXV8zbZu3evxTm1bNnS/Dt/fL3nzZtncb4lJSWIiorC7bffbrGf2l/v8+fPo0OHDpg/f76s/f3l8+3qefvL59vV8zbR+ufb1fP2l893UVERJk6ciK1bt2LdunWorKxE//79cf78ebv3UewzLjSme/fu4oEHHrDY1qpVKzF9+nSb+z/22GOiVatWFtvGjx8vrrvuOvPPI0aMEDfffLPFPjfddJO48847PdTq2nP1vG1p06aNmD17tvnnd999V0RGRnqqiV7h6nkXFBQIAOKPP/6we0w9vN4rV64UkiSJgwcPmrdp4fWuDoBYuXKlw3385fNdnZzztkWLn+/q5Jy3v3y+q3Pn9faHz7cQQpw8eVIAEEVFRXb3UeozrqmeoUuXLmHnzp3o37+/xfb+/ftj8+bNNu+zZcsWq/1vuukm7NixA5cvX3a4j71jKs2d866pqqoKZ8+etVrs7ty5c0hISEBsbCxuvfVWq2+WvlSb8+7UqROaNWuGPn36oKCgwOJ3eni9//vf/6Jv375WVdrV/Hq7wx8+356gxc93bWj58+0J/vL5LisrAwCHi7Aq9RnXVDD0+++/w2AwWK1W36RJE6tV7U2OHz9uc//Kykr8/vvvDvexd0yluXPeNb388ss4f/48RowYYd7WqlUrLFmyBKtXr0Zubi7CwsLQs2dP/Prrrx5tv7vcOe9mzZrhzTffRF5eHvLz85GcnIw+ffpg48aN5n38/fUuLS3F559/jnvvvddiu9pfb3f4w+fbE7T4+XaHP3y+a8tfPt9CCEydOhW9evVCu3bt7O6n1GdcM8txVCdJksXPQgirbc72r7nd1WP6grttzM3NxaxZs7Bq1So0btzYvP26667DddddZ/65Z8+e6Ny5M15//XW89tprnmt4Lbly3snJyUhOTjb/fP3116OkpAQvvfQSbrzxRreO6SvutnHJkiVo0KABhgwZYrFdK6+3q/zl8+0urX++XeFPn293+cvne9KkSfjuu++wadMmp/sq8RnXVM9Qw4YNERgYaBXtnTx50ioqNGnatKnN/YOCghAdHe1wH3vHVJo7522ybNky3HPPPfjoo4/Qt29fh/sGBASgW7duqvkmUZvzru66666zOCd/fr2FEHjnnXcwZswYhISEONxXba+3O/zh810bWv58e4rWPt+14S+f74ceegirV69GQUEBYmNjHe6r1GdcU8FQSEgIunTpgnXr1llsX7duHW644Qab97n++uut9l+7di26du2K4OBgh/vYO6bS3DlvwPiNcezYscjJycHAgQOdPo4QAnv27EGzZs1q3WZPcPe8a9q9e7fFOfnr6w0YZ2vs378f99xzj9PHUdvr7Q5/+Hy7S+ufb0/R2ue7NrT++RZCYNKkScjPz8eGDRuQlJTk9D6KfcZlp1qrxIcffiiCg4PFf//7X/HTTz+JrKwsUbduXXNW/fTp08WYMWPM+//vf/8TderUEVOmTBE//fST+O9//yuCg4PFihUrzPt8/fXXIjAwUDz//PPi559/Fs8//7wICgoSW7duVfz87HH1vHNyckRQUJBYsGCBKC0tNd/+/PNP8z6zZs0SX3zxhfjtt9/E7t27xbhx40RQUJDYtm2b4udnj6vn/eqrr4qVK1eKffv2iR9++EFMnz5dABB5eXnmffzx9TYZPXq06NGjh81jauH1Pnv2rNi9e7fYvXu3ACBeeeUVsXv3bnHo0CEhhP9+vl09b3/5fLt63v7y+Xb1vE20/vl+8MEHRWRkpCgsLLR43164cMG8j68+45oLhoQQYsGCBSIhIUGEhISIzp07W0zLy8zMFKmpqRb7FxYWik6dOomQkBCRmJgo3njjDatjLl++XCQnJ4vg4GDRqlUriw+XWrhy3qmpqQKA1S0zM9O8T1ZWloiPjxchISGiUaNGon///mLz5s0KnpE8rpz3Cy+8IFq0aCHCwsLEVVddJXr16iU+++wzq2P62+sthBB//vmnCA8PF2+++abN42nh9TZNnbb3vvXXz7er5+0vn29Xz9tfPt/uvM/94fNt65wBiHfffde8j68+49LfDSQiIiLSJU3lDBERERF5GoMhIiIi0jUGQ0RERKRrDIaIiIhI1xgMERERka4xGCIiIiJdYzBEREREusZgiIiIiHSNwRARERHpGoMhIiIi0jUGQ0SkSbm5uQgLC8PRo0fN2+699160b98eZWVlPmwZEWkN1yYjIk0SQqBjx45ISUnB/PnzMXv2bLz99tvYunUrmjdv7uvmEZGGBPm6AURE7pAkCc899xyGDx+OmJgYzJs3D8XFxQyEiMhl7BkiIk3r3LkzfvzxR6xduxapqam+bg4RaRBzhohIs7788kv88ssvMBgMaNKkia+bQ0QaxZ4hItKkXbt2IS0tDQsWLMCHH36IOnXqYPny5b5uFhFpEHOGiEhzDh48iIEDB2L69OkYM2YM2rRpg27dumHnzp3o0qWLr5tHRBrDniEi0pQzZ86gZ8+euPHGG7F48WLz9sGDB6OiogJffPGFD1tHRFrEYIiIiIh0jQnUREREpGsMhoiIiEjXGAwRERGRrjEYIiIiIl1jMERERES6xmCIiIiIdI3BEBEREekagyEiIiLSNQZDREREpGsMhoiIiEjXGAwRERGRrjEYIiIiIl37fzeNlVyAlJr1AAAAAElFTkSuQmCC\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 9,
+ "id": "bf4b62f5",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"from random import random, seed\n",
"import numpy as np\n",
@@ -2019,8 +2314,10 @@
},
{
"cell_type": "markdown",
- "id": "aa8914a1",
- "metadata": {},
+ "id": "03734441",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Using gradient descent methods, limitations\n",
"\n",
@@ -2039,8 +2336,10 @@
},
{
"cell_type": "markdown",
- "id": "9848d0a1",
- "metadata": {},
+ "id": "ac3529e1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Improving gradient descent with momentum\n",
"\n",
@@ -2049,57 +2348,13 @@
},
{
"cell_type": "code",
- "execution_count": 12,
- "id": "c4ad0253",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- ">0 f([0.74724774]) = 0.55838\n",
- ">1 f([0.59779819]) = 0.35736\n",
- ">2 f([0.47823856]) = 0.22871\n",
- ">3 f([0.38259084]) = 0.14638\n",
- ">4 f([0.30607268]) = 0.09368\n",
- ">5 f([0.24485814]) = 0.05996\n",
- ">6 f([0.19588651]) = 0.03837\n",
- ">7 f([0.15670921]) = 0.02456\n",
- ">8 f([0.12536737]) = 0.01572\n",
- ">9 f([0.10029389]) = 0.01006\n",
- ">10 f([0.08023512]) = 0.00644\n",
- ">11 f([0.06418809]) = 0.00412\n",
- ">12 f([0.05135047]) = 0.00264\n",
- ">13 f([0.04108038]) = 0.00169\n",
- ">14 f([0.0328643]) = 0.00108\n",
- ">15 f([0.02629144]) = 0.00069\n",
- ">16 f([0.02103315]) = 0.00044\n",
- ">17 f([0.01682652]) = 0.00028\n",
- ">18 f([0.01346122]) = 0.00018\n",
- ">19 f([0.01076897]) = 0.00012\n",
- ">20 f([0.00861518]) = 0.00007\n",
- ">21 f([0.00689214]) = 0.00005\n",
- ">22 f([0.00551372]) = 0.00003\n",
- ">23 f([0.00441097]) = 0.00002\n",
- ">24 f([0.00352878]) = 0.00001\n",
- ">25 f([0.00282302]) = 0.00001\n",
- ">26 f([0.00225842]) = 0.00001\n",
- ">27 f([0.00180673]) = 0.00000\n",
- ">28 f([0.00144539]) = 0.00000\n",
- ">29 f([0.00115631]) = 0.00000\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABT/ElEQVR4nO3dd1xV5eMH8M+5iyVDNggibhMn5DZHilo5WtpylJr2bZi2NMvUhvVr2dIsNRtqZFpamUal5kzFrbiZAiIoe9/7/P64cBMB5SLw3Hv5vF8vXn49nAuf8z3g/fSc5zxHEUIIEBEREUmikh2AiIiIGjaWESIiIpKKZYSIiIikYhkhIiIiqVhGiIiISCqWESIiIpKKZYSIiIikYhkhIiIiqTSyA1SHwWBAUlISnJ2doSiK7DhERERUDUIIZGdnw9/fHypV1eMfVlFGkpKSEBgYKDsGERER1UBCQgICAgKq/LxVlBFnZ2cAxoNxcXGRnIaIiIiqIysrC4GBgab38apYRRkpuzTj4uLCMkJERGRlbjTFghNYiYiISCqWESIiIpKKZYSIiIikYhkhIiIiqVhGiIiISCqWESIiIpKKZYSIiIikYhkhIiIiqVhGiIiISCqzy8g///yD4cOHw9/fH4qi4Oeff77ha7Zt24bQ0FDY29ujefPm+Pzzz2uSlYiIiGyQ2WUkNzcXnTp1wqefflqt/WNiYnDHHXegb9++OHjwIF5++WU888wzWLt2rdlhiYiIyPaY/WyaYcOGYdiwYdXe//PPP0fTpk2xcOFCAEC7du2wf/9+vPfee7j33nvN/fZERERkY+p8zsju3bsRHh5ebtuQIUOwf/9+FBcXV/qawsJCZGVllfuoC3+euIj/rYzCuUs5dfL1iYiILN3qvfGY/8sJnL6YLS1DnZeRlJQU+Pj4lNvm4+ODkpISpKWlVfqaBQsWwNXV1fQRGBhYJ9lW/huHjUdTsP5QUp18fSIiIkv3ze44LN8Zg6i4K9Iy1MvdNNc+OlgIUen2MrNmzUJmZqbpIyEhoU5yjezcBACw4dAFUyYiIqKG4vTFbEQnZ0GrVjAsxFdaDrPnjJjL19cXKSkp5balpqZCo9HAw8Oj0tfY2dnBzs6urqNh8C0+cNCqEZueh8OJmegc6Fbn35OIiMhSrD90AQDQr7U33Bx10nLU+chIz549ERkZWW7bH3/8gbCwMGi12rr+9tflZKfB4FuMl5DKTggREVFDIIQwTVMY1cVfahazy0hOTg4OHTqEQ4cOATDeunvo0CHEx8cDMF5iGTdunGn/qVOnIi4uDjNmzEB0dDSWL1+OZcuW4fnnn6+dI7hJIzsbT8Avh5NRojdITkNERFQ/DsRfQeKVfDjp1Li9rc+NX1CHzC4j+/fvR5cuXdClSxcAwIwZM9ClSxfMmTMHAJCcnGwqJgAQHByMjRs3YuvWrejcuTNef/11fPzxxxZzW+9trb3Q2FGLtJxC7D6fLjsOERFRvSgbFRkS4gsHnVpqFrPnjPTv3/+6kz1XrFhRYVu/fv1w4MABc79VvdCqVbijgx9W/huP9YeS0LeVl+xIREREdapYb8BvR5IB/Hczh0x8Ng3+OxGbjqWgoFgvOQ0REVHd2nE2Dem5RfBspEPvFpXfTFKfWEYAhAU1RhM3B+QUluDvk6my4xAREdWpDaWXaO7q6A+NWn4VkJ/AAqhUCoZ3Mk5k/fkg76ohIiLblVdUgs3HjUtujOgs9y6aMiwjpcpua9p66hIy8ypfpp6IiMja/RmdirwiPZq6O6KLhayvxTJSqq2vC9r4OKNIb8Cm48my4xAREdWJDaXrao3s7F/lSuj1jWXkKiO7lF2q4bNqiIjI9lzJLcLWU5cA/LfOliVgGbnK8I7GE7MnJh0pmQWS0xAREdWujceSUWIQaO/vgpbezrLjmLCMXCXQ3RFhQY0hBPDLYY6OEBGRbVlfOvJvSaMiAMtIBSO7GNccWX+Yd9UQEZHtuJCRj72xl6EoMN1BailYRq5xZwc/aFQKjl3IwtnUHNlxiIiIakXZiH/3YHf4uTpITlMey8g13J10uK21cUn4DXySLxER2YiydbRGWcDy79diGalE2bW09YeTrvscHiIiImtwKiUbJ1OyoVUrGBbiJztOBSwjlRjUzgcOWjXi0vNwKCFDdhwiIqKbsr50pL9/G2+4Omolp6mIZaQSTnYahLf3AfDfI5aJiIiskRDC9F5miZdoAJaRKpVdqvn1SBJK9AbJaYiIiGomKu4KLmTko5GdBre385Ydp1IsI1Xo28oLjR21SMspwq5z6bLjEBER1UjZqMiQ9r6w16olp6kcy0gVtGoV7uxonOTDSzVERGSNivUG/HbU+Lw1S1vo7GosI9cxsvTa2ubjKSgo1ktOQ0REZJ4dZ9JwObcIno106NXCQ3acKrGMXEdo08Zo4uaAnMIS/BWdKjsOERGRWcruormroz80ast9y7fcZBZApVIwonRY62cugEZERFYkr6gEf5y4CMCyL9EALCM3VHYb1NZTqcjMK5achoiIqHoiT1xEXpEeQR6O6BzoJjvOdbGM3EAbX2e09XVGsV7g92PJsuMQERFVy4bSmy9GdvKHoiiS01wfy0g18FINERFZk8u5Rdh2+hIAYISFLnR2NZaRahhR+qjlf2MuIzkzX3IaIiKi69t4NBklBoGQJi5o6d1IdpwbYhmphoDGjri1WWMI8d8jmImIiCxV2V00IztZ/qgIwDJSbWVrjnABNCIismSJV/KwL/YKFAUY3smy76IpwzJSTXd08INGpeB4UhbOpmbLjkNERFSpXw4bb7boEewBX1d7yWmqh2WkmtyddLittRcAjo4QEZHlMl2isfC1Ra7GMmKGshO7/lAShBCS0xAREZV3MiULJ1OyoVOrMCzET3acamMZMcPgW3zgqFMj/nIeDiZkyI5DRERUTtnIff82XnB11EpOU30sI2Zw1GkQfosPgP8WkyEiIrIEBoMwvTeN6mIdd9GUYRkxU9ldNb8eSUKJ3iA5DRERkdGB+Cu4kJGPRnYaDGzrLTuOWVhGzNSnlSfcnXRIyynCznPpsuMQEREB+G+V8KEhvrDXqiWnMQ/LiJm0ahXu7GCcFLSey8MTEZEFKNYb8NsR4y291nQXTRmWkRooO9Gbj6Ugv0gvOQ0RETV0289cwpW8Yng2skPP5h6y45iNZaQGQoMaI6CxA3KL9Pjr5EXZcYiIqIEru4tmeCc/aNTW99ZufYktgKIopofn/XyQd9UQEZE8uYUl+OO48T+MR1rBE3orwzJSQ2W3TW07nYqMvCLJaYiIqKH6M/oi8ov1aObhiE4BrrLj1AjLSA219nFGW19nFOsFfj+WIjsOERE1UGWXaEZ0bgJFUSSnqRmWkZtQNhz280HeVUNERPXvcm4R/jl9CYB13kVThmXkJowoPfF7Yy8jKSNfchoiImpofjuajBKDQIcmrmjh1Uh2nBpjGbkJTdwc0K2ZO4QAfjnMiaxERFS/1h+0vif0VoZl5CaN7PLfk3yJiIjqS8LlPOyPuwJFAYZ3Yhlp0O4I8YNGpeBEchbOXMyWHYeIiBqIX44Y/yO4Z3MP+LjYS05zc1hGblJjJx36tfYCwNERIiKqP+tL17my9ks0AMtIrRhZuubI+sMXIISQnIaIiGzdyZQsnLqYDZ1ahaEhfrLj3DSWkVowqJ03HHVqJFzOx4H4DNlxiIjIxpWt/j2grRdcHbSS09w8lpFa4KjTYEh7XwDABj7Jl4iI6pDBIEx3cI6y0uXfr8UyUkvK1hz59UgySvQGyWmIiMhWRcVfwYWMfDjbaTCgrbfsOLWCZaSW9GnpCXcnHdJzi7DjbJrsOEREZKPKVv0eEuILe61acprawTJSS7RqFe7qaJxEtIF31RARUR0oKjHgt6PJAGznEg3AMlKrym6v2nw8BflFeslpiIjI1mw/cwkZecXwcrZDzxYesuPUGpaRWtS1aWMENHZAbpEef0ZflB2HiIhsTNl6VsM7+kOtss4n9FaGZaQWKYpiGh1Zz7tqiIioFuUWliDyhPE/dG1hobOrsYzUspGl1/C2nrqEK7lFktMQEZGtiDxxEfnFejTzcETHAFfZcWoVy0gta+3jjHZ+LigxCPx+LEV2HCIishFlI+4jOzeBotjOJRqAZaROlA2f/cxLNUREVAvScwrxzxnjshG2dokGqGEZWbRoEYKDg2Fvb4/Q0FBs3779uvuvXLkSnTp1gqOjI/z8/PDoo48iPT29RoGtwYhO/lAUYG/MZcSl58qOQ0REVu7nQ0nQGwQ6BriiuVcj2XFqndllJCIiAs8++yxmz56NgwcPom/fvhg2bBji4+Mr3X/Hjh0YN24cJk6ciOPHj2PNmjXYt28fJk2adNPhLZW/mwP6tjI+yTdiX4LkNEREZM2EEPh+r/E99v6wQMlp6obZZeSDDz7AxIkTMWnSJLRr1w4LFy5EYGAgFi9eXOn+e/bsQbNmzfDMM88gODgYffr0wZQpU7B///6bDm/JHrzV+AOzJioRxVwenoiIauhA/BWcSc2Bg1Ztk5doADPLSFFREaKiohAeHl5ue3h4OHbt2lXpa3r16oXExERs3LgRQghcvHgRP/74I+68884qv09hYSGysrLKfVib29v5wLORDpeyC/H3yVTZcYiIyEqt3mscYb+rox9c7K3/Cb2VMauMpKWlQa/Xw8fHp9x2Hx8fpKRUfudIr169sHLlSowZMwY6nQ6+vr5wc3PDJ598UuX3WbBgAVxdXU0fgYHWNyyl06hwb2gAAJiG14iIiMyRVVCMX48YFzp7oFtTyWnqTo0msF57S5EQosrbjE6cOIFnnnkGc+bMQVRUFDZt2oSYmBhMnTq1yq8/a9YsZGZmmj4SEqxz3sUDtxp/cLadvoQLGfmS0xARkbVZfygJBcUGtPZphK5N3WTHqTMac3b29PSEWq2uMAqSmppaYbSkzIIFC9C7d2+88MILAICOHTvCyckJffv2xRtvvAE/P78Kr7Gzs4OdnZ050SxSsKcTejR3x57zl/HDvgRMH9xadiQiIrISQgis/tc4sv7ArU1tbm2Rq5k1MqLT6RAaGorIyMhy2yMjI9GrV69KX5OXlweVqvy3UauNjzwWQpjz7a3Sg6XDamv2J0BvsP3jJSKi2nH0QiZOJGdBp1Hhnq6284Teyph9mWbGjBlYunQpli9fjujoaEyfPh3x8fGmyy6zZs3CuHHjTPsPHz4c69atw+LFi3H+/Hns3LkTzzzzDLp16wZ/f9ucFXy1Ie194eaoRVJmAf45fUl2HCIishJlE1eHhfjCzVEnOU3dMusyDQCMGTMG6enpmD9/PpKTkxESEoKNGzciKCgIAJCcnFxuzZEJEyYgOzsbn376KZ577jm4ublh4MCBeOedd2rvKCyYvVaNu7s0wVc7Y7F6bzwGtPWWHYmIiCxcbmEJNpSu4l02/9CWKcIKrpVkZWXB1dUVmZmZcHFxkR3HbKcvZiP8w3+gVinYPXMgvF3sZUciIiILFrEvHi+tPYpgTyf8/Vw/q50vUt33bz6bph609nFG16Zu0BsE1kQlyo5DREQWruwSzZhbA622iJiDZaSelN0fHrEvAQZOZCUioiqcTMnCoYQMaFQK7u0aIDtOvWAZqSd3dfSDs50G8ZfzsPu87T4kkIiIbs73paMig2/xgZez9S9zUR0sI/XEUafBiNJnCqzmiqxERFSJgmI91h0wXs635RVXr8UyUo/K1hz54/hFXM4tkpyGiIgsze/HkpFVUIImbg7o29JTdpx6wzJSj0KauCKkiQuK9AZT8yUiIipz9cRVlcr2J66WYRmpZ2X3i6/eG98gVqAlIqLqOXcpB3tjLkOlAPeHNYyJq2VYRurZyM7+cNCqce5SLvbHXZEdh4iILETEPuOoyIA23vBzdZCcpn6xjNQzZ3st7upofDggJ7ISEREAFJUYsDaq4U1cLcMyIkHZD9rGo8nIzC+WnIaIiGSLPHER6blF8HGxw4A2XrLj1DuWEQm6NnVDGx9nFBQbsL702QNERNRwfb/POFJ+f2ggNOqG99bc8I7YAiiKgge6BQIwzpzmRFYiooYr4XIetp9JA2C8i6YhYhmR5O4uTaDTqBCdnIUjiZmy4xARkSRlE1f7tvJEoLuj5DRysIxI4uaowx0hvgD+G54jIqKGpURvwJooYxkpW/qhIWIZkahsIuuGQ0nILSyRnIaIiOrbllOXcDGrEB5OOgy+xUd2HGlYRiTqHuyO5p5OyC3S45fDSbLjEBFRPfu+dImHe0MDoNM03LfkhnvkFkBRFNNkJa45QkTUsCRn5mPLqVQADXfiahmWEcnuDQ2AVq3gcGImTiRlyY5DRET15Id9iTAIoFuwO1p4NZIdRyqWEck8G9mZrhNyIisRUcOgNwj8sN84cfXBbg17VARgGbEIZTOofzp4AflFeslpiIiorm0/cwkXMvLhYq/BsBA/2XGkYxmxAH1aeiKgsQOyC0qw8Wiy7DhERFTHvt9rHBW5p2sA7LVqyWnkYxmxACqVgjFhxmE6XqohIrJtl7IL8Wf0RQAwrcbd0LGMWIj7wwKhUoB9sVdwNjVbdhwiIqojP0YlosQg0KWpG9r6usiOYxFYRiyEr6s9Brb1BvDf8B0REdkWIQQiSkfAH2zAK65ei2XEgpRNZF17IBGFJZzISkRka3afT0dseh4a2WlwVydOXC3DMmJB+rfxgo+LHa7kFeOP4xdlxyEiolpWNvI9orM/HHUayWksB8uIBdGoVRjNiaxERDbpSm4RNh1LAcBLNNdiGbEwo8MCoSjAzrPpiEvPlR2HiIhqybqDF1CkN6C9vws6BLjKjmNRWEYsTKC7I/q09AQAROzjRFYiIlsghDA9FK/sie30H5YRC/Rg6Q/qmqhEFOsNktMQEdHNOhB/BWdSc+CgVWNkZ3/ZcSwOy4gFGtTOBx5OOlzKLsTfJ1NlxyEiopu0unTi6p0d/eBir5WcxvKwjFggnUaF+8ICAMA0rEdERNYpq6AYvx5JAsCH4lWFZcRCla05su30JSRl5EtOQ0RENbX+UBIKig1o5d0IXZs2lh3HIrGMWKhgTyf0aO4Og4DpMdNERGR9rp64qiiK5DSWiWXEgpVNZP1hXwL0BiE5DRERmetoYiaOJ2VBp1bhni5NZMexWCwjFmxIe1+4OWqRlFmAf85ckh2HiIjMtLp0AcuhIb5o7KSTnMZysYxYMHutGneXNmlOZCUisi65hSXYcMg4cfUBTly9LpYRC1d2qebP6FSkZhVITkNERNX165Ek5BSWoJmHI3o295Adx6KxjFi41j7O6NrUDXqDwJqoRNlxiIiomsrWFhlzKyeu3gjLiBUoWzo4Yl8CDJzISkRk8U6mZOFQQgY0KgX3hQbIjmPxWEaswF0d/eBsp0H85TzsPp8uOw4REd3A96WjIoPa+cDL2U5yGsvHMmIFHHUajCh9lsFqTmQlIrJoBcV6rDtgvKzOiavVwzJiJcomsv5x/CIu5xZJTkNERFX5/VgysgpK0MTNAX1becmOYxVYRqxESBNXhDRxQZHeYGrcRERkecomro4OC4RaxYmr1cEyYkXKnlezem88hOBEViIiS3PuUg72xlyGSgFG38qJq9XFMmJFRnb2h4NWjXOXcrE/7orsOEREdI2IfcZRkf5tvOHn6iA5jfVgGbEizvZaDO/kBwD4Znec5DRERHS1/CI91pQ+2PSBWzlx1RwsI1ZmfK9mAICNR5NxISNfbhgiIjL58UAiruQVI9DdAbe385Edx6qwjFiZ9v6u6NXCA3qDwIqdMbLjEBERAINBYPkO47/Jj/UO5sRVM7GMWKFJfYMBGBfVyS4olpyGiIj+OpmKmLRcONtrcH8YL9GYi2XECvVv7Y0WXk7ILiwxTZYiIiJ5lm4/DwB4qHtTNLLTSE5jfVhGrJBKpWBin+YAgK92xqJEb5CciIio4TqamIl/Yy5Do1IwoXReH5mHZcRK3dO1CTycdLiQkY9Nx1NkxyEiarCW7jCOitzV0Y+389YQy4iVsteq8UiPIADAl9tjuAgaEZEESRn5+PVIMgBgUt/mktNYL5YRKza2ZxB0GhUOJ2QgiougERHVu693xUJvEOjR3B0hTVxlx7FaLCNWzLORHe7p0gQA8GXp5CkiIqofOYUlWFX6JPXJHBW5KTUqI4sWLUJwcDDs7e0RGhqK7du3X3f/wsJCzJ49G0FBQbCzs0OLFi2wfPnyGgWm8ib2Md7m+8eJi4hLz5Wchoio4fhhXwKyC0rQ3MsJA9p4y45j1cwuIxEREXj22Wcxe/ZsHDx4EH379sWwYcMQHx9f5WtGjx6Nv/76C8uWLcOpU6ewevVqtG3b9qaCk1ErH2f0b+MFIWBacIeIiOpWid6A5aULT07sEwwVFzm7KYowc+Zj9+7d0bVrVyxevNi0rV27dhg1ahQWLFhQYf9NmzbhgQcewPnz5+Hu7l6jkFlZWXB1dUVmZiZcXFxq9DVs2Y4zaXhk2b9w0KqxZ9btcHXUyo5ERGTTNh5Nxv9WHkBjRy12zbwdDjq17EgWqbrv32aNjBQVFSEqKgrh4eHltoeHh2PXrl2VvmbDhg0ICwvD//3f/6FJkyZo3bo1nn/+eeTnV/1clcLCQmRlZZX7oKr1bumBtr7OyC/WY+VePkCPiKiulc3TG9sjiEWkFphVRtLS0qDX6+HjU/4BQD4+PkhJqXyti/Pnz2PHjh04duwYfvrpJyxcuBA//vgjnnzyySq/z4IFC+Dq6mr6CAzk0rrXoyiK6Zayr3fFoqiEi6AREdWVqLgrOBifAZ1ahUd6BsmOYxNqNIFVUcpfGxNCVNhWxmAwQFEUrFy5Et26dcMdd9yBDz74ACtWrKhydGTWrFnIzMw0fSQkcMnzGxnRyR/ezna4mFWIX48kyY5DRGSzypZ+H9XFH97O9pLT2AazyoinpyfUanWFUZDU1NQKoyVl/Pz80KRJE7i6/nf/dbt27SCEQGJiYqWvsbOzg4uLS7kPuj6dRoXxpcsQL+UiaEREdSI+PQ+bS1e9LnssB908s8qITqdDaGgoIiMjy22PjIxEr169Kn1N7969kZSUhJycHNO206dPQ6VSISAgoAaRqSoPd28KB60aJ5KzsPtcuuw4REQ2Z/nOGBgEcFtrL7TxdZYdx2aYfZlmxowZWLp0KZYvX47o6GhMnz4d8fHxmDp1KgDjJZZx48aZ9n/ooYfg4eGBRx99FCdOnMA///yDF154AY899hgcHLiGf21yc9ThvlBjwVvK23yJiGpVZn4xfthvnDYwqXSNJ6odZj/neMyYMUhPT8f8+fORnJyMkJAQbNy4EUFBxkk8ycnJ5dYcadSoESIjI/H0008jLCwMHh4eGD16NN54443aOwoyeaxPML77Nw5/n0zF2dRstPRmcyciqg2r98Yjr0iPNj7O6NvKU3Ycm2L2OiMycJ0R80z+Zj8iT1zEg92aYsE9HWTHISKyesV6A/q+swUpWQX4v/s6YnQY7/KsjjpZZ4SsQ9kzEtYdSER6TqHkNERE1u+3I8lIySqAZyM7jOzsLzuOzWEZsUG3NmuMjgGuKCwx4Ls9VS/TT0RENyaEwNIdxtt5x/cMgp2Gi5zVNpYRG3T1Imjf7olFQbFeciIiIuu15/xlHLuQBXutCg/34CJndYFlxEYNC/GFv6s90nKKsP7QBdlxiIis1rLSUZF7uwbA3UknOY1tYhmxUVq1Co/2Nt56xkXQiIhqJvbwKeRu+hO+WWmYyNt56wzLiA0b0y0Qjew0OJOag22nL8mOQ0RkXZYtQ9Mu7bD6+5ex6/PH0HxDhOxENotlxIa52Gsx5lbj7WdLt3MRNCKiaktMhHj8cahKR5VVwgBMmQJU8RgTujksIzZuQq9mUCnAjrNpiE7Okh2HiMg6nDkDxXDNE9D1euDsWTl5bBzLiI0LdHfEsA5+ADg6QkRUXQXNmqPCTDu1GmjZUkYcm8cy0gCUPUNhw+ELSM0qkJyGiMjybf33DJSrN6jVwJIlAB/wWidYRhqALk0bIyyoMYr1Al/vjpUdh4jIogkh4Pj2mwCA2B79gS1bgNhYYOJEqblsGctIA1G2CNrKf+ORV1QiOQ0RkeWK+m07+hzeBgDw+Ph9oH9/jojUMZaRBmLwLT4I8nBERl4x1kZxNjgRUVVK5s2HCgLHe4XD+dausuM0CCwjDYRapeCx0kXQlu2IgcHARdCIiK4Vs/VfdNv/NwDA453XJadpOFhGGpD7QgPgYq9BbHoe/oy+KDsOEZHFyXx5DlQQONDtdvj26SY7ToPBMtKAONlpTA954m2+RETlpe89gI67IwEA9q/Pk5ymYWEZaWDG92wGjUrB3tjLOJyQITsOEZHFuPTiq1BBYHfn/rglvLfsOA0Ky0gD4+tqjxGd/AEAS3dwdISICAAKDh9F639+BwDoX3lFcpqGh2WkAZrY1ziRdePRZFzIyJechohIvpQXZkMlBLaG9EXPuwfKjtPgsIw0QO39XdGrhQf0BoEVOzk6QkQNm+HYcTT981cAQMZzM6FWKTd4BdU2lpEGalLp6Mj3exOQXVAsOQ0RkTypL74ClRCIbNcbgx4Mlx2nQWIZaaD6t/ZGCy8nZBeWIGJfguw4RERyREfDe9N6AED8k8+hkZ1GcqCGiWWkgVKpFEzsY1wi/qudsSjRG27wCiIi25Mxaw5UQmBz656445GhsuM0WCwjDdg9XZvA3UmHCxn52HQ8RXYcIqL6FR0Nlw1rAQBHHn0Gfq4OkgM1XCwjDZi9Vo1HShdB+3J7DITgEvFE1HDkvToXKiGwqXVPDBs7THacBo1lpIEb2yMIOo0KhxMyEBV3RXYcIqL6cfIkHNatAQBsGT0FIU1cJQdq2FhGGjgvZzvc3bkJAC4RT0QNR/HceVCEwOZWPTD4Qc4VkY1lhEyLoG0+kYK49FzJaYiI6tjJk9D8EAEAWHvnYxjY1ltyIGIZIbT2cUa/1l4QwnhnDRGRLTO8/oZpVOS2MeFQcZEz6VhGCAAwua/xNt8f9icgPadQchoiojpy8iSU71cDAL66fSzu7RogORABLCNUqndLD4Q0cUFekR5L/jkvOw4RUZ0wvP46FIMBf7TqgR73DoKDTi07EoFlhEopioLnBrcBAHy9KxapWQWSExER1bJTp6B8/z0AYNntYzGxT7DkQFSGZYRM+rfxQmhQYxSWGPDZlrOy4xAR1aqyUZHIlt0x4IEhcLbXyo5EpVhGyERRFDwX3hoAsGpvPBKv5ElORERUS06dAlYb54qsGDwe43oGSQ5EV2MZoXJ6tfBErxYeKNYLfPo3R0eIyDbo578OVemoyKCHh8JRxwfiWRKWEaqgbHRkTVQiYtO47ggRWbnTp0130Hw3ZAIe7NZUciC6FssIVRAa5I4BbbygNwgs/PO07DhERDelZP780lGRbhgy9g7Ya3kHjaVhGaFKzSi9s2b94SScvpgtOQ0RUQ2dPg1V6VyRiGGP4f4writiiVhGqFIdAlwxtL0vhAA+jOToCBFZpyLTXJFuGDb+TmjVfNuzRDwrVKUZ4a2hKMDvx1Jw7EKm7DhEROY5cwaa1asAAGvvegyjujSRHIiqwjJCVWrt44yRnfwBAB9wdISIrEzhXONckT9b3Irhj46Ams+gsVgsI3Rd0wa1hlql4O+TqYiKuyI7DhFR9Zw5A+33xlGRDSMnY1iIr+RAdD0sI3RdwZ5OuK/0QVIfRJ6SnIaIqHoKrhoVGTFxBJ/Ma+FYRuiGnr69JbRqBTvPpmPXuTTZcYiIru/sWehK1xX5/Z4puL2dt+RAdCMsI3RDAY0dTYsEvf/HaQghJCciIqpa3px5UBn0+KvFrRg1eQQUhaMilo5lhKrlyQEtYadRISruCraeviQ7DhFR5c6ehV2EcVTkr/unoE9LT8mBqDpYRqhafFzsTQ+Wev+PUxwdISKLlDNnHtQGPf5uHoZRj9/NURErwTJC1Ta1Xws46dQ4diELm49flB2HiKi8s2fhUDoq8s+D/0O3YHfJgai6WEao2jwa2eGxPsEAjHfW6A0cHSEiy5H16lzTqMjdU+6WHYfMwDJCZpnUtzlc7DU4fTEHvx5Jkh2HiMjo3Dk4/fA9AGD3I0+hU6Cb3DxkFpYRMourgxaP39YcALDwzzMo0RskJyIiAjJmG0dFtjQPxT1TOSpibVhGyGwTegfD3UmHmLRcrDtwQXYcImrozp2D8xrjXJH9459GOz8XyYHIXCwjZLZGdho80a8FAOCjv86gsEQvORERNWTppaMiW5uH4p4n7pUdh2qAZYRq5JEeQfB2tsOFjHz8sC9BdhwiaqjOn4db6ajIkYnPooVXI8mBqCZYRqhGHHRqPDWwJQDgk7/PoqCYoyNEVP9SZ86B2qDHtuahuHvqPbLjUA2xjFCNjbk1EE3cHJCaXYjv9sTJjkNEDYw4dw4ea4130ERPfhaB7o6SE1FN1aiMLFq0CMHBwbC3t0doaCi2b99erdft3LkTGo0GnTt3rsm3JQtjp1Fj2u2tAACLtp5DbmGJ5ERE1JBcnPka1AY9/mkeipGcK2LVzC4jERERePbZZzF79mwcPHgQffv2xbBhwxAfH3/d12VmZmLcuHG4/fbbaxyWLM89XZugmYcjLucWYcWuWNlxiKiBEOfOwXOdcVTkzNTn4OfqIDkR3Qyzy8gHH3yAiRMnYtKkSWjXrh0WLlyIwMBALF68+LqvmzJlCh566CH07NmzxmHJ8mjUKkwf3BoAsGTbOWTmF0tOREQNQdJLc6Ax6LG9eShGcK6I1TOrjBQVFSEqKgrh4eHltoeHh2PXrl1Vvu6rr77CuXPn8Nprr1Xr+xQWFiIrK6vcB1muuzr6o7VPI2QVlGDp9vOy4xCRjTOcOw+fnyIAAHFPPQcvZzvJiehmmVVG0tLSoNfr4ePjU267j48PUlJSKn3NmTNnMHPmTKxcuRIajaZa32fBggVwdXU1fQQGBpoTk+qZWqVgRunoyPIdMUjPKZSciIhsWeKLr0Jj0GNn8664awrnitiCGk1gvfaRzEKISh/TrNfr8dBDD2HevHlo3bp1tb/+rFmzkJmZafpISOA6FpZuSHtftPd3QW6RHkv+4egIEdWNknPn4f+zcVTkwjMvws1RJzkR1QazyoinpyfUanWFUZDU1NQKoyUAkJ2djf379+Opp56CRqOBRqPB/PnzcfjwYWg0Gvz999+Vfh87Ozu4uLiU+yDLpigKng9vAwD4elcsUrMKJCciIluU8IJxVGRXi1AMm8K5IrbCrDKi0+kQGhqKyMjIctsjIyPRq1evCvu7uLjg6NGjOHTokOlj6tSpaNOmDQ4dOoTu3bvfXHqyKP3beKFrUzcUlhjw2ZazsuMQkY0pPncegeuNoyIXp78EZ3ut5ERUW6o3ieMqM2bMwNixYxEWFoaePXviiy++QHx8PKZOnQrAeInlwoUL+Oabb6BSqRASElLu9d7e3rC3t6+wnayfoih4fkgbPPTlv1i1Nx6Tb2uOgMZchIiIakfcc6+gpUGPPS26YshkPpnXlphdRsaMGYP09HTMnz8fycnJCAkJwcaNGxEUFAQASE5OvuGaI2S7erXwRK8WHth1Lh2f/HUW79zXUXYkIrIBhWfOodkvPwAALj83E446s9++yIIpQgghO8SNZGVlwdXVFZmZmZw/YgWi4i7j3sW7oVYp+HNGPwR7OsmORERW7uTIh9B2w2rsbRmKTtH/wk6jlh2JqqG67998Ng3VutAgdwxo4wW9QeCjP0/LjkNEVi7/zDm0/NU4KpL54iwWERvEMkJ1YsZg45016w8n4fTFbMlpiMianZ/xCjQGPfa1DEX/xzhXxBaxjFCd6BDgiqHtfSEE8GEkR0eIqGayT51F640/AgDyZr4MrZpvW7aIZ5XqzIzw1lAU4PdjKTh2IVN2HCKyQjHPvQKtoQRRrULR51GOitgqlhGqM619nDGykz8A4AOOjhCRmbK2bMctG9cAAApffgVqVcWVvsk2sIxQnZo2qDXUKgV/n0xFVNwV2XGIyFosWwbngbdBIwwQAHoUp8lORHWIZYTqVLCnE+7rGgAAeP3XEzAYLP5OciKSLTER4vHHUTYOogBQPTEVSEyUmYrqEMsI1bnpg1vDSafGoYQMROznQw+J6PrE6dNQDIbyG/V64CwfM2GrWEaozvm62mP6YONTm9/+/STScwolJyIiS3Z097GKG9VqoGXL+g9D9YJlhOrFhF7N0NbXGZn5xXhn00nZcYjIQuVdyYTXe28BAIRSeqFGrQaWLAECAiQmo7rEMkL1QqNW4c27jQ9H/GF/IvbHXpaciIgs0ZHHn4NfxkWkuHmj4MhxYMsWIDYWmDhRdjSqQywjVG9Cg9wxJiwQAPDKz8dQojfc4BVE1JDEbdmNsLVfAQBS3noPDiHtgP79OSLSALCMUL16aVhbuDlqcTIlGyt2xcqOQ0QWQuj1KJw4GRphQFTYQHR+YqzsSFSPWEaoXrk76TBzaFsAxmXikzPzJSciIktw6NX/Q+uY48jROcBvxRLZcaiesYxQvRsdFoiuTd2QW6THG79Gy45DRJJlnYtHyw/fBAAcnPI8/NvzrpmGhmWE6p1KpeCNUR2gUoDfjiZj2+lLsiMRkURxE6bAuSAXpwLboPu7r8qOQxKwjJAUt/i7YEKvYADAa+uPoaBYLzkREclw/tsf0WHHJugVFQo/WQSdnVZ2JJKAZYSkmT64Fbyd7RCbnocl287LjkNE9UyfkwvH6c8AALYPexAdRw6UnIhkYRkhaZzttXj1rlsAAJ9tPYu49FzJiYioPkX/7wX4picjxcUT7ZculB2HJGIZIanu6uiHPi09UVRiwJz1xyEEH6RH1BBc/vcA2qz8AgAQ/fKb8PLzlJyIZGIZIakURcH8ke2hU6uw7fQlbDqWIjsSEdU1gwGZ4x6D1qDH7o59cdsLk2UnIslYRki65l6NMLVfcwDAvF9OIKewRHIiIqpL5xcsRPDpw8jROcDli0VQqxTZkUgylhGyCP8b0BKB7g5IySrAx3+dkR2HiOpI0YVkeL35GgBg28NPoX33EMmJyBKwjJBFsNeqMX+E8R+lZTticColW3IiIqoLcROmwjk/B9F+LdFn4TzZcchCsIyQxRjQ1htD2vtAbxB45eejnMxKZGPSfvoVrf7cAL2iQtI7C+Hq4iA7ElkIlhGyKHOGt4eDVo19sVew9sAF2XGIqLbk58Mw9QkAwOb+92LgI3dIDkSWhGWELEoTNwdMG9QKAPDWxmhk5BVJTkREtSF2+svwTk1EirMHWi35EIrCSav0H5YRsjgT+wSjlXcjXM4twv9tPiU7DhHdpILDR9Fk6acAgJ1Pz0GrVk0kJyJLwzJCFkerVuGNUcbJrKv3xuNg/BXJiYioxgwGpD38KLT6Emxv2xNDX31CdiKyQCwjZJG6N/fAPV2bQAjglZ+PQW/gZFYia5T68WIEHI9CrtYe+o8/gpM9H4RHFbGMkMWaNawdXOw1OJ6Uhe/2xMmOQ0RmEhcvwnH2ywCAX+5+HP0GhUlORJaKZYQslpezHV4Y2hYA8N7mU0jNLpCciIjMkTjpSTTKy0K0d3P0+vh1TlqlKrGMkEV7qFtTdAxwRXZhCd76LVp2HCKqprzfNyPw17UwQMGROe+gqY+L7EhkwVhGyKKpVQreHNUBigL8fCgJu86myY5ERDdSUID8SVMAAOt7jcTIx++WHIgsHcsIWbwOAa4Y2yMIAPDK+mMoKjFITkRE15P68mvwSIrDxUbu8P74Pdhr1bIjkYVjGSGr8Fx4G3g20uH8pVx8uf287DhEVAXDiWg0/vgDAMAvj72I3qEtJCcia8AyQlbB1UGL2Xe2AwB88vcZJFzOk5yIiCoQAmljH4NWX4JtLW/FXfOekZ2IrATLCFmNUZ2boHuwOwqKDZj3ywnZcYjoGjlfLIP3gT3I19gh+c334OvGB+FR9bCMkNVQFAVvjAqBRqXgz+iLiDxxUXYkIiqTlgblhRcAACuHTsB99/aRHIisCcsIWZVWPs6YfFtzAMDcDceRV1QiORERAUDaE8/AKTsD0V7N0OXDedCo+fZC1cefFrI6Tw9siSZuDriQkY9P/z4rOw5Rg6f/6294/rgaBij4e9p8hLb0kR2JrAzLCFkdR50Grw2/BQDw5fbzOJuaLTkRUQNWWIjsRycBAH4MuwMPPjtGciCyRiwjZJUG3+KD29t6o1gv8OrPxyEEH6RHJEP2vDfhlhCDS05uUL+9AO5OOtmRyAqxjJBVUhQFc0e0h51Ghd3n0/HVzljZkYgaHP3JU7B/920AwNdjpuPuASGSE5G1YhkhqxXo7oiX7zCuPbLg92gcTsiQG4ioIRECSQ9NgLakGNubh2LUW89CpeKD8KhmWEbIqo3rGYSh7X1RrBd4avUBZOYXy45E1CCc/eBzBB7cgwKNDrkffISWfBAe3QSWEbJqiqLgnfs6ItDdAQmX8zFz7RHOHyGqY+lxSfCYMwsA8NfoqRg6srfkRGTtWEbI6rk6aPHpg12hVSv4/VgKvt0TJzsSkc0yGASOPTIVjfMyEePbDAO+eFt2JLIBLCNkEzoFumHmMOP8kTd+jcaxC5mSExHZps3zP0W/Hb8AANRffAFHJy75TjePZYRsxmO9m2FQOx8U6Q14ctUBZBdw/ghRbYp9830MLX34nQDQNDVebiCyGSwjZDMURcF793dEEzcHxKXnYda6o5w/QlRLrpw+j6BXnkfZ/TIKAEyZAiQmSkxFtoJlhGyKm6MOHz/YBRqVgl+PJGP13gTZkYisnsEg8M8Lb6HCjbt6PXCWj2Sgm8cyQjYnNKgxXhjSBgAw75fjiE7OkpyIyLr9+N1mDN74XcVPqNVAy5b1H4hsDssI2aTJfZtjQBsvFJYY54/kFvLpvkQ1ceh4HEKnT4ZjSSEyg1oYCwhg/HPJEiAgQG5AsgksI2STVCoF74/uDF8Xe5y/lItXfj7G+SNEZsrILcSVB8aixeVEXHH3gcu/O4HYWGDLFuOfEyfKjkg2gmWEbJa7kw6fPNQFapWCnw5ewJr9nGhHVF1CCPwx8UUMOLYdxWoNdD+theLjYxwJ6d+fIyJUq2pURhYtWoTg4GDY29sjNDQU27dvr3LfdevWYfDgwfDy8oKLiwt69uyJzZs31zgwkTlubeaOGYNbAwDmbDiG0xezJScisg4bP/0e9/7wKQDg0uvvwOk2rrJKdcfsMhIREYFnn30Ws2fPxsGDB9G3b18MGzYM8fGV32/+zz//YPDgwdi4cSOioqIwYMAADB8+HAcPHrzp8ETV8US/FujbyhMFxQY8ufIA8oo4f4Toeo7vO4HuM/8HtTDg3NC74T9zuuxIZOMUYeaF9O7du6Nr165YvHixaVu7du0watQoLFiwoFpfo3379hgzZgzmzJlTrf2zsrLg6uqKzMxMuLjwYUxkvrScQtzx0XakZhfi/tAAvHt/J9mRiCxSZmYu4jt2Q4f4E0gMbIUm0QehODnJjkVWqrrv32aNjBQVFSEqKgrh4eHltoeHh2PXrl3V+hoGgwHZ2dlwd3evcp/CwkJkZWWV+yC6GZ6N7PDRA12gUoA1UYlYd4DzR4iuJYTAgfsnokP8CWTbO8H19w0sIlQvzCojaWlp0Ov18PHxKbfdx8cHKSkp1foa77//PnJzczF69Ogq91mwYAFcXV1NH4GBgebEJKpUzxYemHa7cf7IKz8fw9nUHMmJiCzLjnkfYUBkBAAg7bMv4dy+reRE1FDUaAKropRfh08IUWFbZVavXo25c+ciIiIC3t7eVe43a9YsZGZmmj4SEriKJtWOpwa2RK8WHsgr0uOpVQdQUKyXHYnIIpz5cxdC35oJADg07kkEP/ag5ETUkJhVRjw9PaFWqyuMgqSmplYYLblWREQEJk6ciB9++AGDBg267r52dnZwcXEp90FUG9QqBQsf6AzPRnY4mZKNeb8clx2JSLrsi2lweGA0HIsLcbx9d3RatlB2JGpgzCojOp0OoaGhiIyMLLc9MjISvXr1qvJ1q1evxoQJE7Bq1SrceeedNUtKVEu8ne3x0QOdoSjA6r0JWH/oguxIRNIIvR7n77wPAekXkOLmjYCN66BoNLJjUQNj9mWaGTNmYOnSpVi+fDmio6Mxffp0xMfHY+rUqQCMl1jGjRtn2n/16tUYN24c3n//ffTo0QMpKSlISUlBZmZm7R0FkZl6t/TE0wOMz9R4ed1RxKTlSk5EJMfhZ2ajU9Q2FKk1yPhmFVyb+suORA2Q2WVkzJgxWLhwIebPn4/OnTvjn3/+wcaNGxEUFAQASE5OLrfmyJIlS1BSUoInn3wSfn5+po9p06bV3lEQ1cC0Qa3RPdgduUV6PLmS80eo4Yn7YQM6LH4XALDn2bloO/x2yYmooTJ7nREZuM4I1ZWLWQUY9tF2XM4twtgeQXh9VIjsSET1Iu9sDIo6d4Fbbia29xmO3lt/hkrNJ4RQ7aqTdUaIbI2Piz0+GG1cAO3bPXHYeDRZciKiuicKCpA6bCTccjNxyq8l2v/0LYsIScWfPmrw+rfxxhP9WwAAXvrxCOLSOX+EbNu5Rx5Hs7NHkWHfCIXfR8Dd01V2JGrgWEaIADw3uDXCghoju7AET606iMISzh8h25T8yRK0XPstDFCwY86H6HhbV9mRiFhGiABAo1bh4we7wM1Ri6MXMrFg40nZkYhqXf6+/XCf8QwAYP2IibjjpYmSExEZsYwQlfJ3czDNH1mxKxabj1fvEQdEVuHKFeQNvxt2JUXY1epW9P32Y6hUN145m6g+sIwQXWVgWx88fltzAMALaw4j4XKe5EREtcBgQPLI0fC4mIgEVx9ov18FTxcH2amITFhGiK7xwpA26NLUDVkFJZj49T5cyS2SHYnopiQ8/wr8tv+JQrUWO97+HLd2bSk7ElE5LCNE19CqVfj0oa7wcbHD6Ys5GLv8X2TmF8uORVQjZ75biyYfvg0AWDX+JYx+fKTkREQVsYwQVaKJmwNWTuoBDycdjl3IwqNf7UVuYYnsWETVl5iI+EXL4D15PFQQ2NJ3BB78fC7UnCdCFohlhKgKLb0b4duJ3eHqoMWB+AxM+no/l4wn67BsGURQEJo+OQmuBblIdfdB919Xwl6rlp2MqFIsI0TXcYu/C75+rBsa2Wmw+3w6pn4XxTVIyLIlJkI8/jgUg8G0ySvjEhyzMuRlIroBlhGiG+gc6IblE26FvVaFracuYdrqQyjRG278QiIJLkUdKVdEABj/fvaspEREN8YyQlQN3YLd8eW4MOjUKmw6noLn1hyG3mDxz5ikBuZiZj62v72k4ifUaqAl76Ahy8UyQlRNfVt5YdHDXaFRKVh/KAmzfzoKK3joNTUQ6TmF2Hz/E7hnzwYAgFBKJ6qq1cCSJUBAgMR0RNfHMkJkhkG3+OCjB7pApQDf70vAvF9OsJCQdJl5xfjlgacxLvJrAMCVN9+BEh8PbNkCxMYCE7nsO1k2jewARNbmzo5+yC/uhOfXHMaKXbFw0Knx4pA2UBTeMkn1L7ugGOsefhaP/vYlACD91fnwePlF4yc5GkJWgiMjRDVwX2gA3hgVAgBYvPUcPv2bkwOp/uUX6fHjuBfx6M+LAABpL8yGx/xXJaciMh/LCFENPdIjCK/c2Q4A8H7kaSzdfl5yImpICor1WD1pNh5dsxAAkDrteXj+3xtyQxHVEMsI0U2Y1Lc5nhvcGgDwxm/R+G5PnORE1BAU6w1YPXUuHvv2HQBAypSn4f3h/0lORVRzLCNEN+mpgS3xRP8WAIBXfj6GH6MSJSciW6Y3CKx8+i2MX/4mACBpwuPwXfwRwDlLZMVYRohukqIoeHFIG0zo1QwA8OKPh/HrkSS5ocgmGQwCK597F2M/fw0qCFx4cAL8l3/OIkJWj2WEqBYoioI5d92CB24NhEEAz35/CH+euCg7FtkQIQRWv/wRHvpoFtTCgIR7HkKT75axiJBNYBkhqiUqlYI37+6AkZ39UWIQ+N/KA9h+5pLsWGQDhBD4Ye5i3P/uC9AIAxLuvBeBa74FVPwnnGwDf5KJapFapeD9+zthSHsfFOkNmPzNfuyNuSw7Flm5dW8vx6g3p0FnKEFc+AgEro9gESGbwp9molqmUavw8YNd0L+NFwqKDXhsxT4cSsiQHYus1IYPvsOdc/4HO30JYvsNQdCvPxqXeCeyISwjRHXATqPG54+EomdzD+QUlmDcsn9xIilLdiyyMhsXRWDQzEmwLylCbK+BaPbHBkCrlR2LqNaxjBDVEXutGkvHh6FrUzdkFZRg7LJ/cTY1W3YssgaJiTjwwuvo9+wEOBYXIiasL5r9vRHQ6WQnI6oTLCNEdcjJToMVj3VDSBMXpOcW4eGl/yIuPVd2LLJky5ZBNG2Kru/NgVNxAS4FNEezbZsAOzvZyYjqDMsIUR1zsdfim8e6o7VPI1zMKsRDX/6LCxn5smORJUpMhJg8GcpVT4L2TI6DcpmToMm2sYwQ1QN3Jx2+m9QdwZ5OuJCRj4e/3INzl3JkxyJLIgTOvTS3XBEBAEWvB87yQYxk21hGiOqJt7M9Vk7qjoDGDohNz8PwT3bgp4NcOp6Agtx87B06Bi1WLav4SbUaaNmy/kMR1SOWEaJ65O/mgHVP9ELP5h7IK9JjesRhvPjjYeQX6WVHI0lio2NwplNPdPtjDQxQcLLPEIiyW3fVamDJEiAgQG5IojrGMkJUz7xd7PHdpO6YdnsrKArww/5EjPxsB85c5J02Dc3fEZHQ9uyBDucOI8fOEce/WIm22zdBiY0FtmwBYmOBiRNlxySqc4oQ11ygtEBZWVlwdXVFZmYmXFxcZMchqjW7zqZhWsQhXMouhL1WhddHhuD+sEDZsaiO5RfpsWb2R7j3o9lwKi5AsncAtL/+As9bO8uORlSrqvv+zZERIol6tfTExmf6ok9LTxQUG/DCj0cw44dDyC0skR2N6sjZlEysGTEZ4957Dk7FBYjt2hveJw6ziFCDxjJCJJmXsx2+eawbng9vDZUCrDtwASM+3YHoZK7Yamt+2nEaZwfehXGbvwIAJE2Ygmb/boXaw11yMiK5WEaILIBKpeCpga2wenIP+LjY4dylXIz6bCdW/RsPK7iSSjeQW1iC+Yt+R+t7h2Fo9A6UqDXI+uxz+H/1OaDRyI5HJB3njBBZmPScQsz44TC2nb4EABjeyR9v3R0CZ3s+k8TqJCYi/t9DWPrLATyz5gN45mUiz80Ddht+hrpvH9npiOpcdd+/WUaILJDBIPDF9vN4d/Mp6A0CzTwc8elDXRHSxFV2NKomsXQpMGUKFIMBAoACIPeWDnD6/VegaVPZ8YjqBcsIkQ2IiruMp1cdRFJmAXRqFV69qx0e6REERVFkR6PryD0XC4dWLaASBtM2oShQTp4EWreWmIyofvFuGiIbEBrkjt+e6YtB7bxRpDfg1fXH8eSqA8gqKJYdjapwPCkTX01/t1wRAWBc5j0pSVIqIsvGMkJk4Ro76fDluDC8cmc7aFQKNh5NwZ0fb8fhhAzZ0egqQgis/vMYDg1/GE/9sqjiDlzWnahKLCNEVkBRFEzq2xw/PtELAY0dkHA5H/d9vgvLd8TwbhsLkFVQjE9nLULfewfi4QO/AQCK+/Q1FhCAy7oT3QDnjBBZmcz8Yrz442FsPn4RADD4Fh+8e19HuDnqJCdrmI4dj0PchCdw5/7fAQBZfoFw/mY5lEGDgMRE4xN3W7ZkEaEGiRNYiWyYEALf7I7Dm79Fo0hvQBM3B3zyUBd0bdpYdrQGQwiBP/9vGTq8MRO+OekwKArSJkyG98fvA40ayY5HZBFYRogagKOJmXhq9QHEpedBo1Lw4tA2mNSnOVQq3m1TZxITkbtjD06/+ym6HNgGAEjxbYpG336NRoP6y81GZGFYRogaiKyCYsxadxS/HUkGAHQOdMPjtzXHkPa+ULOU1Kq8RUvg8NQTxjtjABig4PhDkxHy5YdQHB0lpyOyPCwjRA2IEAKr9sZj/i8nUFhivKU00N0Bj/YKxuhbA9HIjkuO34y4tBzsf+sz3PPhTFxd74RKBSUujvNBiKrAMkLUAKVmF+Db3XH4bk8cruQZ1yJxttfgoW5NMaF3M/i5OkhOaD2EEIiKu4K/l/+M21Z8iB4JxyrfccsWoH//es1GZC1YRogasPwiPdYeSMTyHTE4n5YLANCoFNzZ0Q+T+zbnsvJVSUyE/uQpbBGu+GXLcQz/4TMMOrcPAFCk1kJrKDFdogFgvGU3NpYjI0RVYBkhIhgMAn+fTMXSHeex5/xl0/buwe6Y3Lc5Brb15mTXUgWffwHdk09AVfosGQHjQkx6lRrZD42F21vzgT/+AKZMAfT6/9YOmThRcnIiy8UyQkTlHE3MxLId5/HrkWSUGIy/9s09nfBYn2Dc2zUADjq15IRyXMjIx9r1u/HUo4Oguuafw4Jhd8J+4QflnyfDtUOIqo1lhIgqlZyZjxW7YrHq33hkF5QAABo7avFIjyCM7RkEb2d7yQnrWGIicOYMTjh548uTOVB+XIMndv6AVpcTKu7L+SBEN4VlhIiuK6ewBGv2J2D5zhgkXM4HAOjUKozs7I+JfYPR1tf2ftcMXy6FMnUKFIMBBgC5Ogc4FxmPXQDl7pThfBCim8cyQkTVojcI/HE8BV9uP48D8Rmm7X1beWJS3+a4rZUnFMUK55WUjoCgVSvkefsi8sctGP7wEKhQ/p+8Ym9faJ95CnB0BF54gfNBiGoRywgRmS0q7gqW7TiPTcdSUDqtBK19GmFSn+YY0dkf9loLnldyVfnA5s3A448DBgOEouCUTzCap8ZCZzBUfN2ffwK33/7f1+B8EKJaU6dlZNGiRXj33XeRnJyM9u3bY+HChejbt2+V+2/btg0zZszA8ePH4e/vjxdffBFTp06t9vdjGSGqXwmX87B8Zwx+2JeA3CI9AMDFXoO2vi4I9nRCsJcTgj2d0MLLCYHujrDTSCgpV5WP/F83wv7JJ6CUlg8IgcrGcngphqh+1VkZiYiIwNixY7Fo0SL07t0bS5YswdKlS3HixAk0bdq0wv4xMTEICQnB5MmTMWXKFOzcuRP/+9//sHr1atx77721ejBEVLsy84vx/d54rNgVi+TMgkr3USlAQGNHY0nxdEJzr7I/G8HPxf7Gtw5/8AGwaJHx8kiHDkCPHkDjqx745+EBBAejKDMTKSUaXExOR8nefei+5F2ohAEGKAAEVDc4Fv3SZVALAzB1Ki/FENWTOisj3bt3R9euXbF48WLTtnbt2mHUqFFYsGBBhf1feuklbNiwAdHR0aZtU6dOxeHDh7F79+5qfU+WESK5ivUGnEzOxvm0HMSk5SImLRfnLxn/zCksqfJ1dhqVqaT8V1YaobmnExo76QBvb+DSpRt+/7IRjWv/rLarR0B4KYao3lT3/dusB1YUFRUhKioKM2fOLLc9PDwcu3btqvQ1u3fvRnh4eLltQ4YMwbJly1BcXAytVlvhNYWFhSgsLCx3MEQkj1atQocAV3QIKL9yqxACl3IKEVNaTGLScnE+LRfnL+Ug/nIeCksMOJmSjZMp2RW+5v8OrscLly5Vq1QoVfxZJVXpOInB8N8ISFnxCAhgCSGyMGaVkbS0NOj1evj4+JTb7uPjg5SUlEpfk5KSUun+JSUlSEtLg5+fX4XXLFiwAPPmzTMnGhFJoCgKvJ3t4e1sj+7NPcp9rkRvwIWMfJxPyy1fVi7lICmzAAOObTdvdON6KisfQ4ZwBITIStToUZ7X3uYnhLjurX+V7V/Z9jKzZs3CjBkzTH/PyspCYGBgTaISkSQatQpBHk4I8nDCgDblP5dfpEeG/WGIebNvvpBcr3ywhBBZBbPKiKenJ9RqdYVRkNTU1AqjH2V8fX0r3V+j0cDDw6PS19jZ2cHOzs6caERkRRx0ajjMfRlYtLBac0YqUKuBt98GwsJYPohsgFllRKfTITQ0FJGRkbj77rtN2yMjIzFy5MhKX9OzZ0/88ssv5bb98ccfCAsLq3S+CBE1IKmpxrtpFi8GSkqATp2A7t0r3k3TrBmQmws4ORn/5KUXIptS41t7P//8c/Ts2RNffPEFvvzySxw/fhxBQUGYNWsWLly4gG+++QbAf7f2TpkyBZMnT8bu3bsxdepU3tpLRERk4+rkbhoAGDNmDNLT0zF//nwkJycjJCQEGzduRFBQEAAgOTkZ8fHxpv2Dg4OxceNGTJ8+HZ999hn8/f3x8ccfV7uIEBERkW3jcvBERERUJ6r7/n2jRQuJiIiI6hTLCBEREUnFMkJERERSsYwQERGRVCwjREREJBXLCBEREUnFMkJERERSsYwQERGRVCwjREREJJXZy8HLULZIbFZWluQkREREVF1l79s3WuzdKspIdnY2ACAwMFByEiIiIjJXdnY2XF1dq/y8VTybxmAwICkpCc7OzlAUpda+blZWFgIDA5GQkGCzz7yx9WPk8Vk/Wz9GWz8+wPaPkcdXc0IIZGdnw9/fHypV1TNDrGJkRKVSISAgoM6+vouLi03+gF3N1o+Rx2f9bP0Ybf34ANs/Rh5fzVxvRKQMJ7ASERGRVCwjREREJFWDLiN2dnZ47bXXYGdnJztKnbH1Y+TxWT9bP0ZbPz7A9o+Rx1f3rGICKxEREdmuBj0yQkRERPKxjBAREZFULCNEREQkFcsIERERSWXzZeTNN99Er1694OjoCDc3t2q9RgiBuXPnwt/fHw4ODujfvz+OHz9ebp/CwkI8/fTT8PT0hJOTE0aMGIHExMQ6OILru3LlCsaOHQtXV1e4urpi7NixyMjIuO5rFEWp9OPdd9817dO/f/8Kn3/ggQfq+GgqqsnxTZgwoUL2Hj16lNvHUs4fYP4xFhcX46WXXkKHDh3g5OQEf39/jBs3DklJSeX2k3UOFy1ahODgYNjb2yM0NBTbt2+/7v7btm1DaGgo7O3t0bx5c3z++ecV9lm7di1uueUW2NnZ4ZZbbsFPP/1UV/GrxZxjXLduHQYPHgwvLy+4uLigZ8+e2Lx5c7l9VqxYUenvZEFBQV0fSqXMOb6tW7dWmv3kyZPl9rOkc2jO8VX274miKGjfvr1pH0s6f//88w+GDx8Of39/KIqCn3/++YavsYjfQWHj5syZIz744AMxY8YM4erqWq3XvP3228LZ2VmsXbtWHD16VIwZM0b4+fmJrKws0z5Tp04VTZo0EZGRkeLAgQNiwIABolOnTqKkpKSOjqRyQ4cOFSEhIWLXrl1i165dIiQkRNx1113XfU1ycnK5j+XLlwtFUcS5c+dM+/Tr109Mnjy53H4ZGRl1fTgV1OT4xo8fL4YOHVoue3p6erl9LOX8CWH+MWZkZIhBgwaJiIgIcfLkSbF7927RvXt3ERoaWm4/Gefw+++/F1qtVnz55ZfixIkTYtq0acLJyUnExcVVuv/58+eFo6OjmDZtmjhx4oT48ssvhVarFT/++KNpn127dgm1Wi3eeustER0dLd566y2h0WjEnj176vRYqmLuMU6bNk288847Yu/eveL06dNi1qxZQqvVigMHDpj2+eqrr4SLi0uF300ZzD2+LVu2CADi1KlT5bJf/btkSefQ3OPLyMgod1wJCQnC3d1dvPbaa6Z9LOn8bdy4UcyePVusXbtWABA//fTTdfe3lN9Bmy8jZb766qtqlRGDwSB8fX3F22+/bdpWUFAgXF1dxeeffy6EMP5warVa8f3335v2uXDhglCpVGLTpk21nr0qJ06cEADK/UDs3r1bABAnT56s9tcZOXKkGDhwYLlt/fr1E9OmTautqDVS0+MbP368GDlyZJWft5TzJ0TtncO9e/cKAOX+QZVxDrt16yamTp1ablvbtm3FzJkzK93/xRdfFG3bti23bcqUKaJHjx6mv48ePVoMHTq03D5DhgwRDzzwQC2lNo+5x1iZW265RcybN8/09+r++1QfzD2+sjJy5cqVKr+mJZ3Dmz1/P/30k1AURcTGxpq2WdL5u1p1yoil/A7a/GUac8XExCAlJQXh4eGmbXZ2dujXrx927doFAIiKikJxcXG5ffz9/RESEmLapz7s3r0brq6u6N69u2lbjx494OrqWu0cFy9exG+//YaJEydW+NzKlSvh6emJ9u3b4/nnnzc9Pbm+3Mzxbd26Fd7e3mjdujUmT56M1NRU0+cs5fwBtXMOASAzMxOKolS4FFmf57CoqAhRUVHl/n8FgPDw8CqPZffu3RX2HzJkCPbv34/i4uLr7lPf5wqo2TFey2AwIDs7G+7u7uW25+TkICgoCAEBAbjrrrtw8ODBWstdXTdzfF26dIGfnx9uv/12bNmypdznLOUc1sb5W7ZsGQYNGoSgoKBy2y3h/NWEpfwOWsWD8upTSkoKAMDHx6fcdh8fH8TFxZn20el0aNy4cYV9yl5fH1JSUuDt7V1hu7e3d7VzfP3113B2dsY999xTbvvDDz+M4OBg+Pr64tixY5g1axYOHz6MyMjIWsleHTU9vmHDhuH+++9HUFAQYmJi8Oqrr2LgwIGIioqCnZ2dxZw/oHbOYUFBAWbOnImHHnqo3EOu6vscpqWlQa/XV/q7U9WxpKSkVLp/SUkJ0tLS4OfnV+U+9X2ugJod47Xef/995ObmYvTo0aZtbdu2xYoVK9ChQwdkZWXho48+Qu/evXH48GG0atWqVo/hempyfH5+fvjiiy8QGhqKwsJCfPvtt7j99tuxdetW3HbbbQCqPs/1fQ5v9vwlJyfj999/x6pVq8ptt5TzVxOW8jtolWVk7ty5mDdv3nX32bdvH8LCwmr8PRRFKfd3IUSFbdeqzj7VUd3jAyrmNDfH8uXL8fDDD8Pe3r7c9smTJ5v+d0hICFq1aoWwsDAcOHAAXbt2rdbXrkpdH9+YMWNM/zskJARhYWEICgrCb7/9VqF0mfN1zVFf57C4uBgPPPAADAYDFi1aVO5zdXkOr8fc353K9r92e01+H+tSTfOsXr0ac+fOxfr168uV0B49epSbZN27d2907doVn3zyCT7++OPaC15N5hxfmzZt0KZNG9Pfe/bsiYSEBLz33numMmLu16xrNc2yYsUKuLm5YdSoUeW2W9r5M5cl/A5aZRl56qmnbnhXQLNmzWr0tX19fQEY26Kfn59pe2pqqqkZ+vr6oqioCFeuXCn3X9epqano1atXjb7v1ap7fEeOHMHFixcrfO7SpUsVWmxltm/fjlOnTiEiIuKG+3bt2hVarRZnzpy56Tey+jq+Mn5+fggKCsKZM2cA1P35A+rnGIuLizF69GjExMTg77//vuGjv2vzHFbG09MTarW6wn8tXf27cy1fX99K99doNPDw8LjuPub8DNSWmhxjmYiICEycOBFr1qzBoEGDrruvSqXCrbfeavqZrS83c3xX69GjB7777jvT3y3lHN7M8QkhsHz5cowdOxY6ne66+8o6fzVhMb+DtTb7xMKZO4H1nXfeMW0rLCysdAJrRESEaZ+kpCRpE1j//fdf07Y9e/ZUe/Lj+PHjK9yBUZWjR48KAGLbtm01zmuumz2+MmlpacLOzk58/fXXQgjLOX9C1PwYi4qKxKhRo0T79u1Fampqtb5XfZzDbt26iSeeeKLctnbt2l13Amu7du3KbZs6dWqFyXPDhg0rt8/QoUOlTmA15xiFEGLVqlXC3t7+hpMJyxgMBhEWFiYeffTRm4laIzU5vmvde++9YsCAAaa/W9I5rOnxlU3UPXr06A2/h8zzdzVUcwKrJfwO2nwZiYuLEwcPHhTz5s0TjRo1EgcPHhQHDx4U2dnZpn3atGkj1q1bZ/r722+/LVxdXcW6devE0aNHxYMPPljprb0BAQHizz//FAcOHBADBw6Udmtvx44dxe7du8Xu3btFhw4dKtwWeu3xCSFEZmamcHR0FIsXL67wNc+ePSvmzZsn9u3bJ2JiYsRvv/0m2rZtK7p06WLxx5ednS2ee+45sWvXLhETEyO2bNkievbsKZo0aWKR508I84+xuLhYjBgxQgQEBIhDhw6Vu5WwsLBQCCHvHJbdNrls2TJx4sQJ8eyzzwonJyfTnQczZ84UY8eONe1fdlvh9OnTxYkTJ8SyZcsq3Fa4c+dOoVarxdtvvy2io6PF22+/bRG39lb3GFetWiU0Go347LPPqrzNeu7cuWLTpk3i3Llz4uDBg+LRRx8VGo2mXEm11OP78MMPxU8//SROnz4tjh07JmbOnCkAiLVr15r2saRzaO7xlXnkkUdE9+7dK/2alnT+srOzTe9zAMQHH3wgDh48aLrTzlJ/B22+jIwfP14AqPCxZcsW0z4AxFdffWX6u8FgEK+99prw9fUVdnZ24rbbbqvQhvPz88VTTz0l3N3dhYODg7jrrrtEfHx8PR3Vf9LT08XDDz8snJ2dhbOzs3j44Ycr3GJ37fEJIcSSJUuEg4NDpetOxMfHi9tuu024u7sLnU4nWrRoIZ555pkKa3XUB3OPLy8vT4SHhwsvLy+h1WpF06ZNxfjx4yucG0s5f0KYf4wxMTGV/kxf/XMt8xx+9tlnIigoSOh0OtG1a9dyIzHjx48X/fr1K7f/1q1bRZcuXYROpxPNmjWrtCCvWbNGtGnTRmi1WtG2bdtyb3QymHOM/fr1q/RcjR8/3rTPs88+K5o2bSp0Op3w8vIS4eHhYteuXfV4ROWZc3zvvPOOaNGihbC3txeNGzcWffr0Eb/99luFr2lJ59Dcn9GMjAzh4OAgvvjii0q/niWdv7IRnKp+3iz1d1ARonSmChEREZEEXGeEiIiIpGIZISIiIqlYRoiIiEgqlhEiIiKSimWEiIiIpGIZISIiIqlYRoiIiEgqlhEiIiKSimWEiIiIpGIZISIiIqlYRoiIiEgqlhEiIiKS6v8B0CYa08e5UzUAAAAASUVORK5CYII=\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 10,
+ "id": "278ad7da",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"from numpy import asarray\n",
"from numpy import arange\n",
@@ -2160,65 +2415,23 @@
},
{
"cell_type": "markdown",
- "id": "a33b2b43",
- "metadata": {},
+ "id": "82ee0ad4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Same code but now with momentum gradient descent"
]
},
{
"cell_type": "code",
- "execution_count": 13,
- "id": "7dfdf318",
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- ">0 f([0.74724774]) = 0.55838\n",
- ">1 f([0.54175461]) = 0.29350\n",
- ">2 f([0.37175575]) = 0.13820\n",
- ">3 f([0.24640494]) = 0.06072\n",
- ">4 f([0.15951871]) = 0.02545\n",
- ">5 f([0.1015491]) = 0.01031\n",
- ">6 f([0.0638484]) = 0.00408\n",
- ">7 f([0.03976851]) = 0.00158\n",
- ">8 f([0.02459084]) = 0.00060\n",
- ">9 f([0.01511937]) = 0.00023\n",
- ">10 f([0.00925406]) = 0.00009\n",
- ">11 f([0.00564365]) = 0.00003\n",
- ">12 f([0.0034318]) = 0.00001\n",
- ">13 f([0.00208188]) = 0.00000\n",
- ">14 f([0.00126053]) = 0.00000\n",
- ">15 f([0.00076202]) = 0.00000\n",
- ">16 f([0.00046006]) = 0.00000\n",
- ">17 f([0.00027746]) = 0.00000\n",
- ">18 f([0.00016719]) = 0.00000\n",
- ">19 f([0.00010067]) = 0.00000\n",
- ">20 f([6.05804744e-05]) = 0.00000\n",
- ">21 f([3.64373635e-05]) = 0.00000\n",
- ">22 f([2.19069576e-05]) = 0.00000\n",
- ">23 f([1.31664443e-05]) = 0.00000\n",
- ">24 f([7.91100141e-06]) = 0.00000\n",
- ">25 f([4.75216828e-06]) = 0.00000\n",
- ">26 f([2.85408468e-06]) = 0.00000\n",
- ">27 f([1.71384267e-06]) = 0.00000\n",
- ">28 f([1.02900153e-06]) = 0.00000\n",
- ">29 f([6.17748881e-07]) = 0.00000\n"
- ]
- },
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABVlklEQVR4nO3dd1gU18IG8Hd2l106iHRBxIYFK9jFLmpi+2JiS9TEcmOqxlSvKWpyY6rXNI2JmnJjjBpNTKIpJNHYoyL2rigdBKX33fP9AWxEQFkEzu7y/p6HJ9dhFt65A+7rmTNnFCGEABEREZEkKtkBiIiIqGFjGSEiIiKpWEaIiIhIKpYRIiIikoplhIiIiKRiGSEiIiKpWEaIiIhIKpYRIiIikkojO0B1GAwGJCQkwMnJCYqiyI5DRERE1SCEQFZWFnx9faFSVT3+YRFlJCEhAf7+/rJjEBERUQ3ExsbCz8+vys9bRBlxcnICUHIwzs7OktMQERFRdWRmZsLf39/4Pl4ViygjZZdmnJ2dWUaIiIgszO2mWHACKxEREUnFMkJERERSsYwQERGRVCwjREREJBXLCBEREUnFMkJERERSsYwQERGRVCwjREREJBXLCBEREUllchnZuXMnRo0aBV9fXyiKgu+///62r/nrr78QEhICW1tbNG/eHB9//HFNshIREZEVMrmM5OTkoFOnTvjwww+rtX90dDTuuusuhIWFISoqCv/+97/x5JNPYtOmTSaHJSIiIutj8rNpRowYgREjRlR7/48//hhNmzbFsmXLAABt27bFoUOH8M4772DcuHGmfnsiIiKyMnU+Z2Tfvn0IDw8vt23YsGE4dOgQioqKKn1NQUEBMjMzy33Uhd9PJePRtZG4eDW7Tr4+ERGRuVt3IAaLfzyFc8lZ0jLUeRlJSkqCl5dXuW1eXl4oLi5Gampqpa9ZsmQJXFxcjB/+/v51km3t31ew7XgSthxJqJOvT0REZO6+3HcFa/ZEI/LKdWkZ6uVumpsfHSyEqHR7mfnz5yMjI8P4ERsbWye5xnRuAgD44Ui8MRMREVFDcS45C6cTM2GjVjAi2FtaDpPnjJjK29sbSUlJ5balpKRAo9GgcePGlb5Gp9NBp9PVdTQMbecFOxs1Lqfl4mhcBjr7u9b59yQiIjIXW47EAwD6t/aEq71WWo46Hxnp1asXIiIiym377bffEBoaChsbm7r+9rfkoNNgaLuSS0hlJ4SIiKghEEIYpymM7eIrNYvJZSQ7OxtHjhzBkSNHAJTcunvkyBHExMQAKLnEMnXqVOP+s2fPxpUrVzBv3jycPn0aa9aswerVq/HMM8/UzhHcoTGdS07Aj0cTUaw3SE5DRERUPw7HXEfc9Tw4aNUY3Mbr9i+oQyaXkUOHDqFLly7o0qULAGDevHno0qULXn75ZQBAYmKisZgAQGBgILZt24YdO3agc+fOePXVV/H++++bzW29/Vp7oJG9DVKzC7DvUprsOERERPWibFRkWLA37LRqqVlMnjMyYMCAW072/Pzzzyts69+/Pw4fPmzqt6oXNmoV7urgg7V/x2DLkQSEtfKQHYmIiKhOFekN2HosEcA/N3PIxGfT4J8T8cuJJOQX6SWnISIiqlu7L6QiLacQ7o5a9GlR+c0k9YllBEBoQCM0cbVDdkEx/jyTIjsOERFRnfqh9BLNyI6+0KjlVwH5CcyASqVgVKeSiazfR/GuGiIisl65hcX49WTJkhujO8u9i6YMy0ipstuadpy9iozcypepJyIisnS/n05BbqEeTd3s0cVM1tdiGSnVxtsZQV5OKNQb8MvJRNlxiIiI6sQPpetqjensW+VK6PWNZeQGY7qUXarhs2qIiMj6XM8pxI6zVwH8s86WOWAZucGojiUnZn90GpIy8iWnISIiql3bTiSi2CDQ3tcZLT2dZMcxYhm5gb+bPUIDGkEI4MejHB0hIiLrsqV05N+cRkUAlpEKxnQpWXNky1HeVUNERNYjPj0PBy5fg6LAeAepuWAZucndHXygUSk4EZ+JCynZsuMQERHVirIR/x6BbvBxsZOcpjyWkZu4OWjRr3XJkvA/8Em+RERkJcrW0RprBsu/34xlpBJl19K2HE245XN4iIiILMHZpCycScqCjVrBiGAf2XEqYBmpxJC2XrCzUeNKWi6OxKbLjkNERHRHtpSO9A8I8oSLvY3kNBWxjFTCQadBeHsvAP88YpmIiMgSCSGM72XmeIkGYBmpUtmlmp+OJaBYb5CchoiIqGYir1xHfHoeHHUaDG7rKTtOpVhGqhDWygON7G2Qml2IvRfTZMchIiKqkbJRkWHtvWFro5acpnIsI1WwUatwd8eSST68VENERJaoSG/A1uMlz1szt4XObsQycgtjSq+t/XoyCflFeslpiIiITLP7fCqu5RTC3VGL3i0ay45TJZaRWwhp2ghNXO2QXVCMP06nyI5DRERkkrK7aEZ29IVGbb5v+eabzAyoVApGlw5rfc8F0IiIyILkFhbjt1PJAMz7Eg3AMnJbZbdB7TibgozcIslpiIiIqifiVDJyC/UIaGyPzv6usuPcEsvIbQR5O6GNtxOK9AI/n0iUHYeIiKhafii9+WJMJ18oiiI5za2xjFQDL9UQEZEluZZTiL/OXQUAjDbThc5uxDJSDaNLH7X8d/Q1JGbkSU5DRER0a9uOJ6LYIBDcxBktPR1lx7ktlpFq8Gtkj27NGkGIfx7BTEREZK7K7qIZ08n8R0UAlpFqK1tzhAugERGROYu7nouDl69DUYBRncz7LpoyLCPVdFcHH2hUCk4mZOJCSpbsOERERJX68WjJzRY9AxvD28VWcprqYRmpJjcHLfq19gDA0REiIjJfxks0Zr62yI1YRkxQdmK3HEmAEEJyGiIiovLOJGXiTFIWtGoVRgT7yI5TbSwjJhjazgv2WjViruUiKjZddhwiIqJyykbuBwR5wMXeRnKa6mMZMYG9VoPwdl4A/llMhoiIyBwYDML43jS2i2XcRVOGZcREZXfV/HQsAcV6g+Q0REREJQ7HXEd8eh4cdRoMauMpO45JWEZM1LeVO9wctEjNLsSei2my4xAREQH4Z5Xw4cHesLVRS05jGpYRE9moVbi7Q8mkoC1cHp6IiMxAkd6ArcdKbum1pLtoyrCM1EDZif71RBLyCvWS0xARUUO36/xVXM8tgrujDr2aN5Ydx2QsIzUQEtAIfo3skFOoxx9nkmXHISKiBq7sLppRnXygUVveW7vlJTYDiqIYH573fRTvqiEiInlyCorx28mSfxiPsYAn9FaGZaSGym6b+utcCtJzCyWnISKihur308nIK9KjWWN7dPJzkR2nRlhGaqi1lxPaeDuhSC/w84kk2XGIiKiBKrtEM7pzEyiKIjlNzbCM3IGy4bDvo3hXDRER1b9rOYXYee4qAMu8i6YMy8gdGF164g9cvoaE9DzJaYiIqKHZejwRxQaBDk1c0MLDUXacGmMZuQNNXO3QvZkbhAB+PMqJrEREVL+2RFneE3orwzJyh8Z0+edJvkRERPUl9louDl25DkUBRnViGWnQ7gr2gUal4FRiJs4nZ8mOQ0REDcSPx0r+EdyreWN4OdtKTnNnWEbuUCMHLfq39gDA0REiIqo/W0rXubL0SzQAy0itGFO65siWo/EQQkhOQ0RE1u5MUibOJmdBq1ZheLCP7Dh3jGWkFgxp6wl7rRqx1/JwOCZddhwiIrJyZat/D2zjARc7G8lp7hzLSC2w12owrL03AOAHPsmXiIjqkMEgjHdwjrXQ5d9vxjJSS8rWHPnpWCKK9QbJaYiIyFpFxlxHfHoenHQaDGzjKTtOrWAZqSV9W7rDzUGLtJxC7L6QKjsOERFZqbJVv4cFe8PWRi05Te1gGaklNmoVRnYsmUT0A++qISKiOlBYbMDW44kArOcSDcAyUqvKbq/69WQS8gr1ktMQEZG12XX+KtJzi+DhpEOvFo1lx6k1LCO1qGvTRvBrZIecQj1+P50sOw4REVmZsvWsRnX0hVplmU/orQzLSC1SFMU4OrKFd9UQEVEtyikoRsSpkn/oWsNCZzdiGallY0qv4e04exXXcwolpyEiImsRcSoZeUV6NGtsj45+LrLj1CqWkVrW2ssJbX2cUWwQ+PlEkuw4RERkJcpG3Md0bgJFsZ5LNADLSJ0oGz77npdqiIioFqRlF2Dn+ZJlI6ztEg1QwzKyfPlyBAYGwtbWFiEhIdi1a9ct91+7di06deoEe3t7+Pj44KGHHkJaWlqNAluC0Z18oSjAgehruJKWIzsOERFZuO+PJEBvEOjo54LmHo6y49Q6k8vI+vXrMXfuXCxYsABRUVEICwvDiBEjEBMTU+n+u3fvxtSpUzFjxgycPHkSGzduxMGDBzFz5sw7Dm+ufF3tENaq5Em+6w/GSk5DRESWTAiBbw6UvMfeF+ovOU3dMLmMLF26FDNmzMDMmTPRtm1bLFu2DP7+/lixYkWl++/fvx/NmjXDk08+icDAQPTt2xcPP/wwDh06dMfhzdmkbiU/MBsj41DE5eGJiKiGDsdcx/mUbNjZqK3yEg1gYhkpLCxEZGQkwsPDy20PDw/H3r17K31N7969ERcXh23btkEIgeTkZHz77be4++67q/w+BQUFyMzMLPdhaQa39YK7oxZXswrw55kU2XGIiMhCrTtQMsI+sqMPnG0t/wm9lTGpjKSmpkKv18PLy6vcdi8vLyQlVX7nSO/evbF27VpMmDABWq0W3t7ecHV1xQcffFDl91myZAlcXFyMH/7+ljcspdWoMC7EDwCMw2tERESmyMwvwk/HShY6m9i9qeQ0dadGE1hvvqVICFHlbUanTp3Ck08+iZdffhmRkZH45ZdfEB0djdmzZ1f59efPn4+MjAzjR2ysZc67mNit5Afnr3NXEZ+eJzkNERFZmi1HEpBfZEBrL0d0beoqO06d0Ziys7u7O9RqdYVRkJSUlAqjJWWWLFmCPn364NlnnwUAdOzYEQ4ODggLC8Nrr70GHx+fCq/R6XTQ6XSmRDNLge4O6NncDfsvXcOGg7F4amhr2ZGIiMhCCCGw7u+SkfWJ3Zpa3doiNzJpZESr1SIkJAQRERHltkdERKB3796VviY3NxcqVflvo1aXPPJYCGHKt7dIk0qH1TYeioXeYP3HS0REteN4fAZOJWZCq1Hhnq7W84Teyph8mWbevHlYtWoV1qxZg9OnT+Opp55CTEyM8bLL/PnzMXXqVOP+o0aNwubNm7FixQpcunQJe/bswZNPPonu3bvD19c6ZwXfaFh7b7ja2yAhIx87z12VHYeIiCxE2cTVEcHecLXXSk5Tt0y6TAMAEyZMQFpaGhYvXozExEQEBwdj27ZtCAgIAAAkJiaWW3PkwQcfRFZWFj788EM8/fTTcHV1xaBBg/Dmm2/W3lGYMVsbNf6vSxN8tucy1h2IwcA2nrIjERGRmcspKMYPpat4l80/tGaKsIBrJZmZmXBxcUFGRgacnZ1lxzHZueQshP93J9QqBfteGARPZ1vZkYiIyIytPxiD5zcdR6C7A/58ur/Fzhep7vs3n01TD1p7OaFrU1foDQIbI+NkxyEiIjNXdolmQjd/iy0ipmAZqSdl94evPxgLAyeyEhFRFc4kZeJIbDo0KgXjuvrJjlMvWEbqyciOPnDSaRBzLRf7LlnvQwKJiOjOfFM6KjK0nRc8nCx/mYvqYBmpJ/ZaDUaXPlNgHVdkJSKiSuQX6bH5cMnlfGtecfVmLCP1qGzNkd9OJuNaTqHkNEREZG5+PpGIzPxiNHG1Q1hLd9lx6g3LSD0KbuKC4CbOKNQbjM2XiIiozI0TV1Uq65+4WoZlpJ6V3S++7kBMg1iBloiIqufi1WwciL4GlQLcF9owJq6WYRmpZ2M6+8LORo2LV3Nw6Mp12XGIiMhMrD9YMioyMMgTPi52ktPUL5aReuZka4ORHUseDsiJrEREBACFxQZsimx4E1fLsIxIUPaDtu14IjLyiiSnISIi2SJOJSMtpxBezjoMDPKQHafesYxI0LWpK4K8nJBfZMCW0mcPEBFRw/XNwZKR8vtC/KFRN7y35oZ3xGZAURRM7O4PoGTmNCeyEhE1XLHXcrHrfCqAkrtoGiKWEUn+r0sTaDUqnE7MxLG4DNlxiIhIkrKJq2Gt3OHvZi85jRwsI5K42mtxV7A3gH+G54iIqGEp1huwMbKkjJQt/dAQsYxIVDaR9YcjCcgpKJachoiI6tv2s1eRnFmAxg5aDG3nJTuONCwjEvUIdENzdwfkFOrx49EE2XGIiKiefVO6xMO4ED9oNQ33LbnhHrkZUBTFOFmJa44QETUsiRl52H42BUDDnbhahmVEsnEhfrBRKzgal4FTCZmy4xARUT3ZcDAOBgF0D3RDCw9H2XGkYhmRzN1RZ7xOyImsREQNg94gsOFQycTVSd0b9qgIwDJiFspmUH8XFY+8Qr3kNEREVNd2nb+K+PQ8ONtqMCLYR3Yc6VhGzEDflu7wa2SHrPxibDueKDsOERHVsW8OlIyK3NPVD7Y2aslp5GMZMQMqlYIJoSXDdLxUQ0Rk3a5mFeD308kAYFyNu6FjGTET94X6Q6UABy9fx4WULNlxiIiojnwbGYdig0CXpq5o4+0sO45ZYBkxE94uthjUxhPAP8N3RERkXYQQWF86Aj6pAa+4ejOWETNSNpF10+E4FBRzIisRkbXZdykNl9Ny4ajTYGQnTlwtwzJiRgYEecDLWYfruUX47WSy7DhERFTLyka+R3f2hb1WIzmN+WAZMSMatQrjOZGViMgqXc8pxC8nkgDwEs3NWEbMzPhQfygKsOdCGq6k5ciOQ0REtWRzVDwK9Qa093VGBz8X2XHMCsuImfF3s0fflu4AgPUHOZGViMgaCCGMD8Ure2I7/YNlxAxNKv1B3RgZhyK9QXIaIiK6U4djruN8SjbsbNQY09lXdhyzwzJihoa09UJjBy2uZhXgzzMpsuMQEdEdWlc6cfXujj5wtrWRnMb8sIyYIa1GhXtD/QDAOKxHRESWKTO/CD8dSwDAh+JVhWXETJWtOfLXuatISM+TnIaIiGpqy5EE5BcZ0MrTEV2bNpIdxyyxjJipQHcH9GzuBoOA8THTRERkeW6cuKooiuQ05ollxIyVTWTdcDAWeoOQnIaIiEx1PC4DJxMyoVWrcE+XJrLjmC2WETM2rL03XO1tkJCRj53nr8qOQ0REJlpXuoDl8GBvNHLQSk5jvlhGzJitjRr/V9qkOZGViMiy5BQU44cjJRNXJ3Li6i2xjJi5sks1v59OQUpmvuQ0RERUXT8dS0B2QTGaNbZHr+aNZccxaywjZq61lxO6NnWF3iCwMTJOdhwiIqqmsrVFJnTjxNXbYRmxAGVLB68/GAsDJ7ISEZm9M0mZOBKbDo1Kwb0hfrLjmD2WEQswsqMPnHQaxFzLxb5LabLjEBHRbXxTOioypK0XPJx0ktOYP5YRC2Cv1WB06bMM1nEiKxGRWcsv0mPz4ZLL6py4Wj0sIxaibCLrbyeTcS2nUHIaIiKqys8nEpGZX4wmrnYIa+UhO45FYBmxEMFNXBDcxBmFeoOxcRMRkfkpm7g6PtQfahUnrlYHy4gFKXtezboDMRCCE1mJiMzNxavZOBB9DSoFGN+NE1eri2XEgozp7As7GzUuXs3BoSvXZcchIqKbrD9YMioyIMgTPi52ktNYDpYRC+Jka4NRnXwAAF/uuyI5DRER3SivUI+NpQ82ndiNE1dNwTJiYab1bgYA2HY8EfHpeXLDEBGR0beH43A9twj+bnYY3NZLdhyLwjJiYdr7uqB3i8bQGwQ+3xMtOw4REQEwGATW7C75O3l6n0BOXDURy4gFmhkWCKBkUZ2s/CLJaYiI6I8zKYhOzYGTrQb3hfISjalYRizQgNaeaOHhgKyCYuNkKSIikmfVrksAgMk9msJRp5GcxvKwjFgglUrBjL7NAQCf7bmMYr1BciIioobreFwG/o6+Bo1KwYOl8/rINCwjFuqerk3Q2EGL+PQ8/HIySXYcIqIGa9XuklGRkR19eDtvDbGMWChbGzUe6BkAAPh0VzQXQSMikiAhPQ8/HUsEAMwMay45jeViGbFgU3oFQKtR4WhsOiK5CBoRUb37Yu9l6A0CPZu7IbiJi+w4FotlxIK5O+pwT5cmAIBPSydPERFR/cguKMbXpU9Sn8VRkTtSozKyfPlyBAYGwtbWFiEhIdi1a9ct9y8oKMCCBQsQEBAAnU6HFi1aYM2aNTUKTOXN6Ftym+9vp5JxJS1HchoiooZjw8FYZOUXo7mHAwYGecqOY9FMLiPr16/H3LlzsWDBAkRFRSEsLAwjRoxATExMla8ZP348/vjjD6xevRpnz57FunXr0KZNmzsKTiVaeTlhQJAHhIBxwR0iIqpbxXoD1pQuPDmjbyBUXOTsjijCxJmPPXr0QNeuXbFixQrjtrZt22Ls2LFYsmRJhf1/+eUXTJw4EZcuXYKbm1uNQmZmZsLFxQUZGRlwdnau0dewZrvPp+KB1X/DzkaN/fMHw8XeRnYkIiKrtu14Ih5dexiN7G2w94XBsNOqZUcyS9V9/zZpZKSwsBCRkZEIDw8vtz08PBx79+6t9DU//PADQkND8dZbb6FJkyZo3bo1nnnmGeTlVf1clYKCAmRmZpb7oKr1adkYbbydkFekx9oDfIAeEVFdK5unN6VnAItILTCpjKSmpkKv18PLq/wDgLy8vJCUVPlaF5cuXcLu3btx4sQJfPfdd1i2bBm+/fZbPPbYY1V+nyVLlsDFxcX44e/PpXVvRVEU4y1lX+y9jMJiLoJGRFRXIq9cR1RMOrRqFR7oFSA7jlWo0QRWRSl/bUwIUWFbGYPBAEVRsHbtWnTv3h133XUXli5dis8//7zK0ZH58+cjIyPD+BEbyyXPb2d0J194OumQnFmAn44lyI5DRGS1ypZ+H9vFF55OtpLTWAeTyoi7uzvUanWFUZCUlJQKoyVlfHx80KRJE7i4/HP/ddu2bSGEQFxcXKWv0el0cHZ2LvdBt6bVqDCtdBniVVwEjYioTsSk5eLX0lWvyx7LQXfOpDKi1WoREhKCiIiIctsjIiLQu3fvSl/Tp08fJCQkIDs727jt3LlzUKlU8PPzq0Fkqsr9PZrCzkaNU4mZ2HcxTXYcIiKrs2ZPNAwC6NfaA0HeTrLjWA2TL9PMmzcPq1atwpo1a3D69Gk89dRTiImJwezZswGUXGKZOnWqcf/JkyejcePGeOihh3Dq1Cns3LkTzz77LKZPnw47O67hX5tc7bW4N6Sk4K3ibb5ERLUqI68IGw6VTBuYWbrGE9UOk59zPGHCBKSlpWHx4sVITExEcHAwtm3bhoCAkkk8iYmJ5dYccXR0REREBJ544gmEhoaicePGGD9+PF577bXaOwoymt43EF/9fQV/nknBhZQstPRkcyciqg3rDsQgt1CPIC8nhLVylx3Hqpi8zogMXGfENLO+PISIU8mY1L0pltzTQXYcIiKLV6Q3IOzN7UjKzMdb93bE+FDe5VkddbLOCFmGsmckbD4ch7TsAslpiIgs39ZjiUjKzIe7ow5jOvvKjmN1WEasULdmjdDRzwUFxQZ8tb/qZfqJiOj2hBBYtbvkdt5pvQKg03CRs9rGMmKFblwE7X/7LyO/SC85ERGR5dp/6RpOxGfC1kaF+3tykbO6wDJipUYEe8PXxRap2YXYciRedhwiIou1unRUZFxXP7g5aCWnsU4sI1bKRq3CQ31Kbj3jImhERDVz+ehZ5PzyO7wzUzGDt/PWGZYRKzahuz8cdRqcT8nGX+euyo5DRGRZVq9G0y5tse6bf2Pvx9PR/If1shNZLZYRK+Zsa4MJ3UpuP1u1i4ugERFVW1wcxL/+BVXpqLJKGICHHwaqeIwJ3RmWESv3YO9mUCnA7gupOJ2YKTsOEZFlOH8eiuGmJ6Dr9cCFC3LyWDmWESvn72aPER18AHB0hIiouvKbNUeFmXZqNdCypYw4Vo9lpAEoe4bCD0fjkZKZLzkNEZH52/vHISjAP4VErQZWrgT4gNc6wTLSAHRp2gihAY1QpBf4Yt9l2XGIiMyaMBjgvuRVAMD5IaOB7duBy5eBGTPkBrNiLCMNRNkiaGv/jkFuYbHkNERE5uvE59+i46WjKFDbwHv5MmDAAI6I1DGWkQZiaDsvBDS2R3puETZFcjY4EVGlhIDDqwsBAIdHToJzK64tUh9YRhoItUrB9NJF0FbvjobBwEXQiIhuFv/ZOjS/fBo5NrZo+uYi2XEaDJaRBuTeED8422pwOS0Xv59Olh2HiMi86PVQL3wZALDzrsloEtRMbp4GhGWkAXHQaYwPeeJtvkRE5WV89iW8Yy8iQ+cA31dfkh2nQWEZaWCm9WoGjUrBgcvXcDQ2XXYcIiLzUFQE8fIrAIAfhz2ATh2ayc3TwLCMNDDeLrYY3ckXALBqN0dHiIgAoPDT1XBNjMVVe1d4/vtZ2XEaHJaRBmhGWMlE1m3HExGfnic5DRGRZPn5KFq0GADw9eDJGNytueRADQ/LSAPU3tcFvVs0ht4g8Pkejo4QUcNmWLECDimJSHByR6O5T0CtUmRHanBYRhqomaWjI98ciEVWfpHkNEREkmRno/i1/wAAVg64H/f04bNnZGAZaaAGtPZECw8HZBUUY/3BWNlxiIjkeO89aK+lIbqRD+xnTYejTiM7UYPEMtJAqVQKZvQtuS762Z7LKNYbbvMKIiIrc/069G+9BQB4P+wBTO3XSnKghotlpAG7p2sTuDloEZ+eh19OJsmOQ0RUv955B+rMTJxxDwAmTICPi53sRA0Wy0gDZmujxgOli6B9uisaQnCJeCJqIJKTYVi2DADwbr8pmNGfc0VkYhlp4Kb0DIBWo8LR2HREXrkuOw4RUf1YsgSq3Fwc8WmNrPARCG7iIjtRg8Yy0sB5OOnwf52bAOAS8UTUQMTGQqxYAQB4J2wKZoa1kByIWEbIuAjar6eScCUtR3IaIqI69uqrUAoLsd8/GAmhfTCojafsRA0eywihtZcT+rf2gBAld9YQEVmt8+ch1qwBALzdbyqmhzWHioucSccyQgCAWWElt/luOBSLtOwCyWmIiOrIwoVQ9Hr82TwUl1p3wriufrITEVhGqFSflo0R3MQZuYV6rNx5SXYcIqLad+IExLp1AEruoJnWuxnstGrJoQhgGaFSiqLg6aFBAIAv9l5GSma+5ERERLXspZegCIGtQX0QH9gGM/oGyk5EpVhGyGhAkAdCAhqhoNiAj7ZfkB2HiKj2HDgAfP899IoKS/s+gNn9W8DJ1kZ2KirFMkJGiqLg6fDWAICvD8Qg7nqu5ERERLXkxRcBAN+1H4iMZi0xtVeA5EB0I5YRKqd3C3f0btEYRXqBD//k6AgRWYEdO4CICBSpNVjWZxIeG9gC9lo+EM+csIxQBWWjIxsj43A5leuOEJEFEwJYsAAA8E3HcOgDmmFS96aSQ9HNWEaogpAANwwM8oDeILDs93Oy4xAR1dzPPwN79yJfo8UHvSbgiUGtYGvDO2jMDcsIVWpe6Z01W44m4FxyluQ0REQ1YDAY54p80XUkbAP8cV8o1xUxRywjVKkOfi4Y3t4bQgD/jeDoCBFZoM2bgagoZGvt8XGPcZgzuBVs1HzbM0c8K1SleeGtoSjAzyeScCI+Q3YcIqLq0+uBl14CAKwOHQO3AF+M7dJEciiqCssIVam1lxPGdPIFACzl6AgRWZKvvgLOnEG6nRNWdR+LeUODoOYzaMwWywjd0pwhraFWKfjzTAoir1yXHYeI6PYKC4GFCwEAK3qMg18zH4wI9pabiW6JZYRuKdDdAfeWPkhqacRZyWmIiKph9Wrg8mVcdWyEL7qOxNNDW/PJvGaOZYRu64nBLWGjVrDnQhr2XkyVHYeIqGq5ucCrrwIAPug5HkHNvTG4rafkUHQ7LCN0W36N7I2LBL372zkIISQnIiKqwvLlQGIi4lw88U2n4XgmvDUUhaMi5o5lhKrlsYEtodOoEHnlOnacuyo7DhFRRZmZwBtvAADe6z0JXVp5oW9Ld8mhqDpYRqhavJxtjQ+Weve3sxwdISLzs2wZkJaGi25+2Bw8CE+HB3FUxEKwjFC1ze7fAg5aNU7EZ+LXk8my4xAR/SMtDXjnHQDAf/tORp823uge6CY5FFUXywhVW2NHHab3DQRQcmeN3sDRESIyE2+9BWRl4ZRnILa26Yunh7aWnYhMwDJCJpkZ1hzOthqcS87GT8cSZMchIgISE4EPPgAAvBM2BUPa+6CTv6vcTGQSlhEyiYudDf7VrzkAYNnv51GsN0hOREQN3uuvA3l5OOwbhD9bdMM8jopYHJYRMtmDfQLh5qBFdGoONh+Olx2HiBqyy5eBlSsBAG/3m4aRnXzR1sdZbiYyGcsImcxRp8Ej/VsAAN774zwKivWSExFRg7V4MVBUhN0BnfB3s454iqMiFollhGrkgZ4B8HTSIT49DxsOxsqOQ0QN0dmzwBdfAADe6TcV93T1QwsPR8mhqCZYRqhG7LRqPD6oJQDggz8vIL+IoyNEVM9efhkwGBDRsjtO+LXBnMGtZCeiGmIZoRqb0M0fTVztkJJVgK/2X5Edh4gakiNHgA0bAADvhk3BhG7+8Hezl5uJaqxGZWT58uUIDAyEra0tQkJCsGvXrmq9bs+ePdBoNOjcuXNNvi2ZGZ1GbfyXyPIdF5FTUCw5ERE1GC+9BAD4oW0/XPJtYRypJctkchlZv3495s6diwULFiAqKgphYWEYMWIEYmJibvm6jIwMTJ06FYMHD65xWDI/93RtgmaN7XEtpxCf770sOw4RNQT79gE//QS9So3/9r0fD/QIgI+LnexUdAdMLiNLly7FjBkzMHPmTLRt2xbLli2Dv78/VqxYccvXPfzww5g8eTJ69epV47BkfjRqlXH2+sq/LiIjr0hyIiKyegsWAAC+bT8ISV5N8ciAFpID0Z0yqYwUFhYiMjIS4eHh5baHh4dj7969Vb7us88+w8WLF/HKK69U6/sUFBQgMzOz3AeZr5EdfdHayxGZ+cVYteuS7DhEZM3++APYvh2Fahu832cSHurTDB5OOtmp6A6ZVEZSU1Oh1+vh5eVVbruXlxeSkpIqfc358+fxwgsvYO3atdBoNNX6PkuWLIGLi4vxw9/f35SYVM/UKsW44uGa3dFIyy6QnIiIrJIQxlGRtZ2HI9PT17giNFm2Gk1gvfmRzEKISh/TrNfrMXnyZCxatAitW1d/IZr58+cjIyPD+BEby3UszN2w9t5o7+uMnEI9Vu7k6AgR1YGffgL+/hv5Njos7zkeM8Oaw9VeKzsV1QKTyoi7uzvUanWFUZCUlJQKoyUAkJWVhUOHDuHxxx+HRqOBRqPB4sWLcfToUWg0Gvz555+Vfh+dTgdnZ+dyH2TeFEXBM+FBAIAv9l5GSma+5EREZFUMBuDFFwEAn3UdhSJPT0zv20xuJqo1JpURrVaLkJAQRERElNseERGB3r17V9jf2dkZx48fx5EjR4wfs2fPRlBQEI4cOYIePXrcWXoyKwOCPNC1qSsKig34aPsF2XGIyJps2AAcO4ZsnQM+7jEOs/u3gJOtjexUVEuqN4njBvPmzcOUKVMQGhqKXr164ZNPPkFMTAxmz54NoOQSS3x8PL788kuoVCoEBweXe72npydsbW0rbCfLpygKnhkWhMmf/o2vD8RgVr/m8GvERYiI6A4VFwOlN0Cs7DYWNh7umNorQHIoqk0ml5EJEyYgLS0NixcvRmJiIoKDg7Ft2zYEBJT8YCQmJt52zRGyXr1buKN3i8bYezENH/xxAW/e21F2JCKydF9+CZw7h+v2LlgTOgbPDmwBe63Jb19kxhQhhJAd4nYyMzPh4uKCjIwMzh+xAJFXrmHcin1QqxT8Pq8/At0dZEciIktVUAC0bg3ExOC1gdOxbehkbH92AHQatexkVA3Vff/ms2mo1oUEuGFgkAf0BoH3fj8nOw4RWbJPPgFiYpDi1Bj/63I3nhjcikXECrGMUJ2YN7TkzpotRxNwLjlLchoiskg5OcB//gMAeK/XBHh5NcK9IX6SQ1FdYBmhOtHBzwXD23tDCOC/ERwdIaIa+OADIDkZca7e2NBxKOYOaQUbNd+2rBHPKtWZeeGtoSjAzyeScCI+Q3YcIrIk6enAW28BAJb2mYSm3q4Y07mJ3ExUZ1hGqM609nLCmE6+AIClHB0hIlMsXQpcv46L7k3xfbsBmDc0CGpVxZW+yTqwjFCdmjOkNdQqBX+eSUHkleuy4xCRJbh6FfjvfwEAb/e9H22aNMKIYG/JoagusYxQnQp0d8C9XUsmnL360ykYDGZ/JzkRyfbGG0B2Nk54t8QvrXvj6fDWUHFUxKqxjFCde2poazho1TgSm471h/jQQyK6hfh4iI8+AgC80/cB9G7pjkFtPCWHorrGMkJ1ztvFFk8NLXlq8xs/n0FadoHkRERktl57DUpBAQ74tcOeVqFYPCa40qfCk3VhGaF68WDvZmjj7YSMvCK8+csZ2XGIyBxdugSxahUA4J1+U/Fw/5Zo6ekoORTVB5YRqhcatQr/+b+ShyNuOBSHQ5evSU5ERGZn4UIoxcXY2awLEjp2w2MDW8pORPWEZYTqTUiAGyaE+gMAXvz+BIr1BsmJiMhsnDoF8dVXAIB3+k3BotHtYaflsu8NBcsI1avnR7SBq70NziRl4fO9l2XHISIzIV5+GYoQ+KV1L3gNDsPgtl6yI1E9YhmheuXmoMULw9sAKFkmPjEjT3IiIpIuMhLKpk0wQMGHA6filVHtZCeiesYyQvVufKg/ujZ1RU6hHq/9dFp2HCKSrGj+AgDAlnb9cdfEIfBrZC85EdU3lhGqdyqVgtfGdoBKAbYeT8Rf567KjkREsuzeDZuIX1GsqPDdmJmY2be57EQkAcsISdHO1xkP9g4EALyy5QTyi/SSExFRvRMC2c88DwDY0DEcj8wYDq2Gb0sNEc86SfPU0FbwdNLhclouVv51SXYcIqpn+l9/g+Pfe1GgtsG5f81FrxaNZUciSVhGSBonWxu8NLJkotpHOy7gSlqO5EREVG+EwPV5zwEANoSOxKNT+ksORDKxjJBUIzv6oG9LdxQWG/DylpMQgg/SI2oIMtZthPvpY8ixsYXupX/D08lWdiSSiGWEpFIUBYvHtIdWrcJf567ilxNJsiMRUV3T65H7/L8BAD8OvA/jRoRIDkSysYyQdM09HDG7f8kM+kU/nkJ2QbHkRERUly689wl84i4iQ+eA9u++CrWKD8Jr6FhGyCw8OrAl/N3skJSZj/f/OC87DhHVkcK8Ati//hoA4O9x09EhOEByIjIHLCNkFmxt1Fg8uuRBeqt3R+NsUpbkRERUF/a99A580xKQ5uCKHv9dJDsOmQmWETIbA9t4Ylh7L+gNAi9+f5yTWYmsTHzSdQR98l8AQNzsuXDxbCQ5EZkLlhEyKy+Pag87GzUOXr6OTYfjZccholq05+lX4Z2VhtRGnuj46nOy45AZYRkhs9LE1Q5zhrQCALy+7TTScwslJyKi2rBv2x4M2/QJAKD43y9CsbOTnIjMCcsImZ0ZfQPRytMR13IK8davZ2XHIaI7VLjyU/S4OwwuBTkQALyddbIjkZlhGSGzY6NW4bWxJZNZ1x2IQVTMdcmJiKjG4uKgeWQ2VCiZA6YAwKOPAnFxUmOReWEZIbPUo3lj3NO1CYQAXvz+BPQGTmYlskTx+6OgEobyG/V64MIFOYHILLGMkNmaP6ItnG01OJmQia/2X5Edh4hMJITA8RVfVfyEWg20bFn/gchssYyQ2fJw0uHZ4W0AAO/8ehYpWfmSExGRKfat2ojwPzcCAISq9O1GrQZWrgT8/CQmI3PDMkJmbXL3pujo54KsgmK8vvW07DhEVE3ZMfFoPe8RqCBwcsS9UK5cAbZvBy5fBmbMkB2PzAzLCJk1tUrBf8Z2gKIA3x9JwN4LqbIjEdHtGAxIvGcS3LOv4bJnAFp8vbpkJGTAAI6IUKVYRsjsdfBzwZSeJc+veHHLCRQWG27zCiKSKWnh62gVuQsFahukrvoCtq7OsiORmWMZIYvwdHgQ3B21uHQ1B5/uuiQ7DhFVwfD3Abj/ZyEAYPPUZxA6qr/cQGQRWEbIIrjY2WDB3W0BAB/8eR6x13IlJyKiCjIykDvuPmgMevzWpg8GLn1JdiKyECwjZDHGdm6CHoFuyC8yYNGPp2THIaIbCYGCGbPgGB+DOGdPJL37IbxdueQ7VQ/LCFkMRVHw2thgaFQKfj+djIhTybIjEVGZNWug27QRxYoK7zy0EJOHdZSdiCwIywhZlFZeTpjVrzkAYOEPJ5FbWCw5ERHh1CnoH38CAPBOv6mY8tQEaNR8e6Hq408LWZwnBrVEE1c7xKfn4cM/uaQ0kVR5eRATJkCdn4edzbog/dE5CAlwk52KLAzLCFkce60Gr4xqBwD4dNclXEjJkpyIqAGbNw/KiRO46uCKhfc9j+fubic7EVkglhGySEPbeWFwG08U6QVe+v4khOCD9Ijq3bffAh9/DAB46u6n8a97e8HNQSs5FFkilhGySIqiYOHo9tBpVNh3KQ2f7bksOxJRw3L5MsTMmQCA5T3vRU7/gRgf6i85FFkqlhGyWP5u9vj3XSVrjyz5+TSOxqbLDUTUUBQVAZMmQcnIwGHfIKwYOA1v39sRKpUiOxlZKJYRsmhTewVgeHtvFOkFHl93GBl5RbIjEVm/l18G9u9Hps4BT45+DgvHdUZLTyfZqciCsYyQRVMUBW/e2xH+bnaIvZaHFzYd4/wRorr022/AG28AAJ4f/gR6DgrBuBA+/I7uDMsIWTwXOxt8OKkrbNQKfj6RhP/tvyI7EpF1SkqCmDIFAPBV5xG40G84Fo9pLzkUWQOWEbIKnfxd8cKIkvkjr/10GifiMyQnIrIyBgMwdSqUlBSccQ/A28P+hY/u7wp7rUZ2MrICLCNkNab3aYYhbb1QqDfgsa8PIyuf80eIas3bbwMREcjT6PD4mOexYFwIWntxngjVDpYRshqKouCd+zqiiasdrqTlYv7m45w/QlQb9u2DWLAAAPDKkIcRPKQn7gvlPBGqPSwjZFVc7bV4f1IXaFQKfjqWiHUHYmVHIrJs6ekQkyZB0evxQ9t+ODRoLF77vw5QFN7GS7WHZYSsTkhAIzw7LAgAsOjHkzidmCk5EZGFEgKYORPKlSu44uqNhXc9gQ/vD4GjjvNEqHaxjJBVmhXWHAODPFBQXDJ/JKeAT/clMtknnwCbNqFIpcYTo5/D0/d1QztfZ9mpyAqxjJBVUqkUvDu+M7ydbXHpag5e/P4E548QmeL4cYi5cwEAb/afhqbD+mNy96ZyM5HVYhkhq+XmoMUHk7tArVLwXVQ8Nh6Kkx2JyDLk5EBMmAAlPx/bm4fgj2GTseQezhOhulOjMrJ8+XIEBgbC1tYWISEh2LVrV5X7bt68GUOHDoWHhwecnZ3Rq1cv/PrrrzUOTGSKbs3cMG9oawDAyz+cwLnkLMmJiCzA3LlQTp9GsqMb5o96Gh88EAonWxvZqciKmVxG1q9fj7lz52LBggWIiopCWFgYRowYgZiYmEr337lzJ4YOHYpt27YhMjISAwcOxKhRoxAVFXXH4Ymq45H+LRDWyh35RQY8tvYwcgs5f4SoSt98A6xaBQMUzB35NB6d0AvBTVxkpyIrpwgTL6T36NEDXbt2xYoVK4zb2rZti7Fjx2LJkiXV+hrt27fHhAkT8PLLL1dr/8zMTLi4uCAjIwPOzpw8RaZLzS7AXe/tQkpWAe4L8cPb93WSHYnI/Fy6BNG5M5SsLLzfawJOPfIsVjzQlZdnqMaq+/5t0shIYWEhIiMjER4eXm57eHg49u7dW62vYTAYkJWVBTc3tyr3KSgoQGZmZrkPojvh7qjDexO7QKUAGyPjsPkw548QlVNYCDFxIpSsLBxs0g6bR83Am/d2ZBGhemFSGUlNTYVer4eXl1e57V5eXkhKSqrW13j33XeRk5OD8ePHV7nPkiVL4OLiYvzw9/c3JSZRpXq1aIw5g0vmj7z4/QlcSMmWnIjIjCxYAOXgQaTbOuLpsc/ivQe6wcWO80SoftRoAuvNTVkIUa32vG7dOixcuBDr16+Hp6dnlfvNnz8fGRkZxo/YWK6iSbXj8UEt0btFY+QW6vH414eRX6SXHYlIvp9/Bt55BwDw7F1zMW1if3Tyd5WbiRoUk8qIu7s71Gp1hVGQlJSUCqMlN1u/fj1mzJiBDRs2YMiQIbfcV6fTwdnZudwHUW1QqxQsm9gZ7o46nEnKwqIfT8qORCRXQgIMU6cCAD4LGQWMGYPpfZrJzUQNjkllRKvVIiQkBBEREeW2R0REoHfv3lW+bt26dXjwwQfx9ddf4+67765ZUqJa4ulki/cmdoaiAOsOxGLLkXjZkYjk0OshpkyBKjUVJz2b48sxj+JtzhMhCUy+TDNv3jysWrUKa9aswenTp/HUU08hJiYGs2fPBlByiWVqacsGSorI1KlT8e6776Jnz55ISkpCUlISMjIyau8oiEzUp6U7nhjYEgDw783HEZ2aIzkRkQRvvAHlzz+RY2OLuWOfx7vTesDVXis7FTVAJpeRCRMmYNmyZVi8eDE6d+6MnTt3Ytu2bQgICAAAJCYmlltzZOXKlSguLsZjjz0GHx8f48ecOXNq7yiIamDOkNboEeiGnEI9HlvL+SPUwOzeDfHKKwCAl8IfwX33D0bXpo0kh6KGyuR1RmTgOiNUV5Iz8zHivV24llOIKT0D8OrYYNmRiOretWswdOoMVVwsNrUfiK3PvIlVU0OhUvHyDNWuOllnhMjaeDnbYun4kgXQ/rf/CrYdT5SciKiOCQExfTpUcbG41MgXH417Cu/c14lFhKRiGaEGb0CQJx4Z0AIA8Py3x3AljfNHyIp99BGULVtQoNZg7tjn8eZDfeDmwHkiJBfLCBGAp4e2RmhAI2QVFOPxr6NQUMz5I2SFjhyB4emnAQBLBkzHsCl3oVuzqlfDJqovLCNEADRqFd6f1AWu9jY4Hp+BJdvOyI5EVLuys2EYPwGqwkJEtOyOS5Om45H+LWSnIgLAMkJk5OtqZ5w/8vney/j1ZPUecUBkER5/HKrz55Do2BhvjX8eSyd05jwRMhssI0Q3GNTGC//q1xwA8OzGo4i9lis5EVEt+N//gC++gF5RYe7oZ/HqjP5wd9TJTkVkxDJCdJNnhwWhS1NXZOYXY8YXB3E9p1B2JKKaO3cO+tmPAADe6zMJfabfg57NG0sORVQeywjRTWzUKnw4uSu8nHU4l5yNKWv+RkZekexYRKYrKEDuPfdBnZuD/f7BOD39cTxWuvIwkTlhGSGqRBNXO6yd2RONHbQ4EZ+Jhz47gJyCYtmxiEyS+thc2J88hmt2zlj7xOv44IFuUHOeCJkhlhGiKrT0dMT/ZvSAi50NDsekY+YXh7hkPFmM+C/Xw331xwCAlQ+9hDfnjICtjVpyKqLKsYwQ3UI7X2d8Mb07HHUa7LuUhtlfRXINEjJ7McfOwWH2LADAloHj8fi7c2Cv1UhORVQ1lhGi2+js74o1D3aDrY0KO85exZx1R1CsN8iORVSpxKOnURg+HK55WTjnF4T+m1bBydZGdiyiW2IZIaqG7oFu+HRqKLRqFX45mYSnNx6F3mD2z5ikBibjw4/h1bk9WiZHQwDwnXE/XBs5yY5FdFssI0TVFNbKA8vv7wqNSsGWIwlY8N1xWMBDr6mBuH72IpyefBQqlPxMKgAcX1sExMXJDUZUDSwjRCYY0s4L703sApUCfHMwFot+PMVCQtJl5Bbhj9kLoLr5Z1GvBy5ckBOKyAQsI0QmurujD966959l49/69SwLCUmTlVeI7yc/iXt3rK/4SbUaaMl1Rcj8sYwQ1cC9IX54bWwwAGDFjov48E/+65PqX15BMbaPfgjTtpTcwps9aGhJAQFK/rtyJeDnJzEhUfXwXi+iGnqgZwDyi/R4betpvBtxDnZaNWaGNZcdixqI/LwCHBh6L0bv+QkAkPjSq/BZ/GLJHJELF0pGRFhEyEKwjBDdgZlhzZFXqMe7Eefw2tbTsLVR44GeAbJjkZUrysnF6bAR6B+1E8WKCnFvvY9mzzxW8kk/P5YQsji8TEN0hx4f1BKPDGgBAHjx+xP4NpJ3L1Dd0V9PR3S3fugStRMFahucX/75P0WEyEKxjBDdIUVR8NywIDzYuxkA4Llvj+KnYwlyQ5FVMiQlI7FrL7Q+HYlsrR1OfbYBbWdPkR2L6I6xjBDVAkVR8PLIdpjYzR8GAcz95gh+P5UsOxZZEREdjWtde8Dv8hmk2rvi6Fdb0GXKWNmxiGoFywhRLVGpFPzn/zpgTGdfFBsEHl17GLvOX5Udi6yAOHEC2aE94Z54BXHOnjjy9Q/oc99Q2bGIag3LCFEtUqsUvHtfJwxr74VCvQGzvjyEA9HXZMciS7Z/Pwp694XTtRScdW+Kw1//iCFjwmSnIqpVLCNEtUyjVuH9SV0wIMgD+UUGTP/8II7EpsuORZbo119RNHAQbLMycNg3CJFffo/Rd3eXnYqo1rGMENUBnUaNjx8IQa/mjZFdUIypq//GqYRM2bHIkqxfD/3IUbDJz8OOwBBErtqIySO6yE5FVCdYRojqiK2NGqumhaJrU1dk5hdjyuq/cSElS3YssgQrVkBMmgR1cRF+aNsPRz74DLNGdJCdiqjOsIwQ1SEHnQafT++O4CbOSMspxP2r/saVtBzZschcCQG8+irw6KNQhMCXXe7GiTc+xJy7gmUnI6pTLCNEdczZ1gZfTu+B1l6OSM4swORP/0Z8ep7sWGRuDAZg7lzg5ZcBAO/1noRzL72O+aOCoSiK3GxEdYxlhKgeuDlo8dXMHgh0d0B8eh7u/3Q/Ll7Nlh2LzEVRETB1KvD++wCAV4Y8jJg5z2PxmA4sItQgsIwQ1RNPJ1usndkDfo3scDktF6M+2I3vorh0fIOXmwv9mDHA2rUoUqkxZ+TTSH3oX3jr3o5QqVhEqGFgGSGqR76udtj8SG/0at4YuYV6PLX+KJ779ijyCvWyo5EM6enIHzgY6p9/Rp5Gh1n3vIgmj83EexM6Q80iQg0IywhRPfN0tsVXM3tgzuBWUBRgw6E4jPloN84n806bBiUxERnde8P2wH5k6Bzw2LQ38NB/HsNzw9tAo+ZfzdSw8CeeSAK1SsFTQ1tj7Ywe8HDS4VxyNkZ9uBsbD8XKjkb1IP/0OaR16Q6X86eR7OiG157/GEuWzkb/1h6yoxFJwTJCJFHvlu7Y9mQY+rZ0R36RAc9+ewzzNhxBTkGx7GhUR2K270Vuj15onByHy64++OmjDXjjlfvh5WwrOxqRNCwjRJJ5OOnw5fTueCa8NVQKsPlwPEZ/uBunE7liq7XZvnozXEeEwy3rGs56N0fKtgjMmDqY80OowWMZITIDKpWCxwe1wrpZPeHlrMPFqzkY+9EefP13DIQQsuPRHcopKManL3yAXrMnwbkgB2dadoLbgT3o3qu97GhEZkERFvA3XWZmJlxcXJCRkQFnZ2fZcYjqVFp2AeZtOIq/zl0FAIzq5IvX/y8YTrY2kpORyeLiEPP3Efzwv18x+4fl0AgDonsORNOIH6F2dJCdjqjOVff9m2WEyAwZDAKf7LqEt389C71BoFlje3w4uSuCm7jIjkbVJFatAh5+GIrBYNx2dex4eGz4CrBhsaSGgWWEyApEXrmGJ76OQkJGPrRqFV4a2RYP9AzgqpxmLufiZdi1agGV+KeICEWBcvky0LSpvGBE9ay679+cM0JkxkIC3LD1yTAMaeuJQr0BL205ice+PozM/CLZ0agKJ+Ou48cZz5crIgCgCAFcuiQpFZF5YxkhMnONHLT4dGooXry7LTQqBduOJ+Hu93fhaGy67Gh0AyEEfvp8Kwp79sbEvzZU3EGtBlq2rP9gRBaAZYTIAiiKgplhzfHtI73h18gOsdfycO/He7FmdzTvtjEDmUlX8dewiRgxfTS6xJ9Bvq09Cu4ZV1JAgJL/rlwJ+PnJDUpkplhGiCxIZ39XbH0yDMPae6FIL7D4p1P41/8ikZ5bKDtawyQEYt77GEWtgjAgYgPUwoCLg+6G7vxZ6DZ9C1y+DGzfXvLfGTNkpyUyW5zASmSBhBD4ct8V/GfraRTqDWjiaocPJndB16aNZEdrMMTx40ieMgPeRw8CAK54+KNw2ftoNXms3GBEZoQTWImsmKIomNa7GTY90hsBje0Rn56H8R/vwyc7L8JgMPt/X1i2rCwUPPkUDJ27wPvoQeTa6LD5vsfgeu40iwhRDXFkhMjCZeYXYf7m49h6LBFAyaWcf/VrjmHtvbnMeG0SArlfroXhmWfgmJoMAPitdW9kvP4m7r2nD2+3JqoE1xkhakCEEPj6QAwW/3gKBcUlt5T6u9nhod6BGN/NH446jeSEli1+/2EUPPwYmh/bDwC47OqD5fc8iSmLHkEHPy5ER1QVlhGiBiglKx//23cFX+2/guu5JWuRONlqMLl7UzzYpxl8XOwkJ7QcQghEnY5DyvMvY9C2r6A1FKNAbYP1Qx+A40v/xt3dA6HTqGXHJDJrLCNEDVheoR6bDsdhze5oXErNAQBoVAru7uiDWWHNuaz8LRRficHBiP2I/DMSY7d+Br/MkmcEHenYB0VLlyF0UAgvyRBVE8sIEcFgEPjzTApW7b6E/ZeuGbf3CHTDrLDmGNTGEyrOKwEAZOUXIeqVd9H37X9DdcNfi9c8fJH39rtoMnUCwBJCZBKWESIq53hcBlbvvoSfjiWiuPSOm+buDpjeNxDjuvrBTtswLznEp+fhi53nkbP6C7z2w1LcWDeEokA5exZo1UpaPiJLxjJCRJVKzMjD53sv4+u/Y5CVXwwAaGRvgwd6BmBKrwB4OtlKTlg/jsam47tNu+CxcS3GHYuAd/a1ynfcvh0YMKBesxFZC5YRIrql7IJibDwUizV7ohF7LQ8AoFWrMKazL2aEBaKNt/X9rukNAn8euYJTH32B0D82o8+VY8bPFTq7wiYro+SBdmXU6pLVU7mMO1GNsIwQUbXoDQK/nUzCp7su4XBMunF7WCt3zAxrjn6t3C13wmZcHHD+PPICAvH7ntMoXPkpBkdGwDU/GwBgUBTk9hsIx8dmA6NHA199BTz8MKDX//M8GS7jTlRjLCNEZLLIK9exevcl/HIiCWULubb2csTMvs0xurMvbG0saF7J6tUQ//oXFIMBAig3FyTTwweq6dPh+MgsICCg/Ovi4oALF0qesMsREaI7UqdlZPny5Xj77beRmJiI9u3bY9myZQgLC6ty/7/++gvz5s3DyZMn4evri+eeew6zZ8+u9vdjGSGqX7HXcrFmTzQ2HIxFTqEeAOBsq0Ebb2cEujsg0MMBge4OaOHhAH83eznrbZSOeqBVK2S6eyH6ag5iYlORv3sP3P/8BQN+/ab8ZFQAcb0GwOPfz8J2xLB/nqhLRHWmzsrI+vXrMWXKFCxfvhx9+vTBypUrsWrVKpw6dQpNmzatsH90dDSCg4Mxa9YsPPzww9izZw8effRRrFu3DuPGjavVgyGi2pWRV4RvDsTg872XkZiRX+k+KgXwa2RfUlLcHdDco+y/jvBxtr39rcOBgSXzMpydgeefLxmRCAwEsrNL7mLx8wPi4lB45gziG/vhnNYVui8+Q793XoRKGGBQFOwIDIFzQQ46Jp6H1lBc9ffiZFSielVnZaRHjx7o2rUrVqxYYdzWtm1bjB07FkuWLKmw//PPP48ffvgBp0+fNm6bPXs2jh49in379lXre7KMEMlVpDfgTGIWLqVmIzo1B9GpObh0teS/2QVVv/nrNCpjSfmnrDiiubsDGjloq1y3o+yyikFRYXev4ei772eohIABCvY1DUbvmOOoquKku3vjetuOaLY7gpNRiSSr7vu3SQ+sKCwsRGRkJF544YVy28PDw7F3795KX7Nv3z6Eh4eX2zZs2DCsXr0aRUVFsLGxqfCagoICFBQUlDsYIpLHRq1CBz+XCs9hEULganYBokuLSXRqDi6l5uDS1WzEXMtFQbEBZ5KycCYpq8LX3L7iITQDKi0VZdtUwoCwvdv++TME+sQcrzzkc88Bs2fDtVkzuCoKsHp1xcmoLCJEZsmkMpKamgq9Xg8vL69y2728vJCUlFTpa5KSkirdv7i4GKmpqfDx8anwmiVLlmDRokWmRCMiCRRFgaeTLTydbNGjeeNynyvWGxCfnodLqTnly8rVbCRk5KNp5tUqRzfKfY/qBFGrgSeeKF82ZswAhg3jZFQiC1CjR3nefJufEOKWt/5Vtn9l28vMnz8f8+bNM/45MzMT/v7+NYlKRJJo1CoENHZAQGMHDAwq/7m8Qj0K1/rDNi62emXjRmo1sGQJMH/+7Uc9/PxYQogsgEllxN3dHWq1usIoSEpKSoXRjzLe3t6V7q/RaNC4ceNKX6PT6aDT6UyJRkQWxE6rBmJjbv+sF7UaeOCBkvU/bl77Y9IkjnoQWQmTyohWq0VISAgiIiLwf//3f8btERERGDNmTKWv6dWrF3788cdy23777TeEhoZWOl+EiBoQIcrfTfPCCyXlolkzICfnn6Lx2msViwdHPYishsmXaebNm4cpU6YgNDQUvXr1wieffIKYmBjjuiHz589HfHw8vvzySwAld858+OGHmDdvHmbNmoV9+/Zh9erVWLduXe0eCRFZpujo2+/D4kFk1UwuIxMmTEBaWhoWL16MxMREBAcHY9u2bQgoXcUwMTERMTExxv0DAwOxbds2PPXUU/joo4/g6+uL999/v9prjBAREZF143LwREREVCeq+/6tqsdMRERERBWwjBAREZFULCNEREQkFcsIERERScUyQkRERFKxjBAREZFULCNEREQkFcsIERERScUyQkRERFKZvBy8DGWLxGZmZkpOQkRERNVV9r59u8XeLaKMZGVlAQD8/f0lJyEiIiJTZWVlwcXFpcrPW8SzaQwGAxISEuDk5ARFUWrt62ZmZsLf3x+xsbFW+8wbaz9GHp/ls/ZjtPbjA6z/GHl8NSeEQFZWFnx9faFSVT0zxCJGRlQqFfzq8PHhzs7OVvkDdiNrP0Yen+Wz9mO09uMDrP8YeXw1c6sRkTKcwEpERERSsYwQERGRVA26jOh0OrzyyivQ6XSyo9QZaz9GHp/ls/ZjtPbjA6z/GHl8dc8iJrASERGR9WrQIyNEREQkH8sIERERScUyQkRERFKxjBAREZFUVl9G/vOf/6B3796wt7eHq6trtV4jhMDChQvh6+sLOzs7DBgwACdPniy3T0FBAZ544gm4u7vDwcEBo0ePRlxcXB0cwa1dv34dU6ZMgYuLC1xcXDBlyhSkp6ff8jWKolT68fbbbxv3GTBgQIXPT5w4sY6PpqKaHN+DDz5YIXvPnj3L7WMu5w8w/RiLiorw/PPPo0OHDnBwcICvry+mTp2KhISEcvvJOofLly9HYGAgbG1tERISgl27dt1y/7/++gshISGwtbVF8+bN8fHHH1fYZ9OmTWjXrh10Oh3atWuH7777rq7iV4spx7h582YMHToUHh4ecHZ2Rq9evfDrr7+W2+fzzz+v9HcyPz+/rg+lUqYc344dOyrNfubMmXL7mdM5NOX4Kvv7RFEUtG/f3riPOZ2/nTt3YtSoUfD19YWiKPj+++9v+xqz+B0UVu7ll18WS5cuFfPmzRMuLi7Ves0bb7whnJycxKZNm8Tx48fFhAkThI+Pj8jMzDTuM3v2bNGkSRMREREhDh8+LAYOHCg6deokiouL6+hIKjd8+HARHBws9u7dK/bu3SuCg4PFyJEjb/maxMTEch9r1qwRiqKIixcvGvfp37+/mDVrVrn90tPT6/pwKqjJ8U2bNk0MHz68XPa0tLRy+5jL+RPC9GNMT08XQ4YMEevXrxdnzpwR+/btEz169BAhISHl9pNxDr/55hthY2MjPv30U3Hq1CkxZ84c4eDgIK5cuVLp/pcuXRL29vZizpw54tSpU+LTTz8VNjY24ttvvzXus3fvXqFWq8Xrr78uTp8+LV5//XWh0WjE/v376/RYqmLqMc6ZM0e8+eab4sCBA+LcuXNi/vz5wsbGRhw+fNi4z2effSacnZ0r/G7KYOrxbd++XQAQZ8+eLZf9xt8lczqHph5fenp6ueOKjY0Vbm5u4pVXXjHuY07nb9u2bWLBggVi06ZNAoD47rvvbrm/ufwOWn0ZKfPZZ59Vq4wYDAbh7e0t3njjDeO2/Px84eLiIj7++GMhRMkPp42Njfjmm2+M+8THxwuVSiV++eWXWs9elVOnTgkA5X4g9u3bJwCIM2fOVPvrjBkzRgwaNKjctv79+4s5c+bUVtQaqenxTZs2TYwZM6bKz5vL+ROi9s7hgQMHBIByf6HKOIfdu3cXs2fPLretTZs24oUXXqh0/+eee060adOm3LaHH35Y9OzZ0/jn8ePHi+HDh5fbZ9iwYWLixIm1lNo0ph5jZdq1aycWLVpk/HN1/36qD6YeX1kZuX79epVf05zO4Z2ev++++04oiiIuX75s3GZO5+9G1Skj5vI7aPWXaUwVHR2NpKQkhIeHG7fpdDr0798fe/fuBQBERkaiqKio3D6+vr4IDg427lMf9u3bBxcXF/To0cO4rWfPnnBxcal2juTkZGzduhUzZsyo8Lm1a9fC3d0d7du3xzPPPGN8enJ9uZPj27FjBzw9PdG6dWvMmjULKSkpxs+Zy/kDauccAkBGRgYURalwKbI+z2FhYSEiIyPL/f8KAOHh4VUey759+yrsP2zYMBw6dAhFRUW33Ke+zxVQs2O8mcFgQFZWFtzc3Mptz87ORkBAAPz8/DBy5EhERUXVWu7qupPj69KlC3x8fDB48GBs37693OfM5RzWxvlbvXo1hgwZgoCAgHLbzeH81YS5/A5axIPy6lNSUhIAwMvLq9x2Ly8vXLlyxbiPVqtFo0aNKuxT9vr6kJSUBE9PzwrbPT09q53jiy++gJOTE+65555y2++//34EBgbC29sbJ06cwPz583H06FFERETUSvbqqOnxjRgxAvfddx8CAgIQHR2Nl156CYMGDUJkZCR0Op3ZnD+gds5hfn4+XnjhBUyePLncQ67q+xympqZCr9dX+rtT1bEkJSVVun9xcTFSU1Ph4+NT5T71fa6Amh3jzd59913k5ORg/Pjxxm1t2rTB559/jg4dOiAzMxPvvfce+vTpg6NHj6JVq1a1egy3UpPj8/HxwSeffIKQkBAUFBTgf//7HwYPHowdO3agX79+AKo+z/V9Du/0/CUmJuLnn3/G119/XW67uZy/mjCX30GLLCMLFy7EokWLbrnPwYMHERoaWuPvoShKuT8LISpsu1l19qmO6h4fUDGnqTnWrFmD+++/H7a2tuW2z5o1y/i/g4OD0apVK4SGhuLw4cPo2rVrtb52Ver6+CZMmGD838HBwQgNDUVAQAC2bt1aoXSZ8nVNUV/nsKioCBMnToTBYMDy5cvLfa4uz+GtmPq7U9n+N2+vye9jXappnnXr1mHhwoXYsmVLuRLas2fPcpOs+/Tpg65du+KDDz7A+++/X3vBq8mU4wsKCkJQUJDxz7169UJsbCzeeecdYxkx9WvWtZpm+fzzz+Hq6oqxY8eW225u589U5vA7aJFl5PHHH7/tXQHNmjWr0df29vYGUNIWfXx8jNtTUlKMzdDb2xuFhYW4fv16uX9dp6SkoHfv3jX6vjeq7vEdO3YMycnJFT539erVCi22Mrt27cLZs2exfv362+7btWtX2NjY4Pz583f8RlZfx1fGx8cHAQEBOH/+PIC6P39A/RxjUVERxo8fj+joaPz555+3ffR3bZ7Dyri7u0OtVlf419KNvzs38/b2rnR/jUaDxo0b33IfU34GaktNjrHM+vXrMWPGDGzcuBFDhgy55b4qlQrdunUz/szWlzs5vhv17NkTX331lfHP5nIO7+T4hBBYs2YNpkyZAq1We8t9ZZ2/mjCb38Fam31i5kydwPrmm28atxUUFFQ6gXX9+vXGfRISEqRNYP3777+N2/bv31/tyY/Tpk2rcAdGVY4fPy4AiL/++qvGeU11p8dXJjU1Veh0OvHFF18IIczn/AlR82MsLCwUY8eOFe3btxcpKSnV+l71cQ67d+8uHnnkkXLb2rZte8sJrG3bti23bfbs2RUmz40YMaLcPsOHD5c6gdWUYxRCiK+//lrY2tredjJhGYPBIEJDQ8VDDz10J1FrpCbHd7Nx48aJgQMHGv9sTuewpsdXNlH3+PHjt/0eMs/fjVDNCazm8Dto9WXkypUrIioqSixatEg4OjqKqKgoERUVJbKysoz7BAUFic2bNxv//MYbbwgXFxexefNmcfz4cTFp0qRKb+318/MTv//+uzh8+LAYNGiQtFt7O3bsKPbt2yf27dsnOnToUOG20JuPTwghMjIyhL29vVixYkWFr3nhwgWxaNEicfDgQREdHS22bt0q2rRpI7p06WL2x5eVlSWefvppsXfvXhEdHS22b98uevXqJZo0aWKW508I04+xqKhIjB49Wvj5+YkjR46Uu5WwoKBACCHvHJbdNrl69Wpx6tQpMXfuXOHg4GC88+CFF14QU6ZMMe5fdlvhU089JU6dOiVWr15d4bbCPXv2CLVaLd544w1x+vRp8cYbb5jFrb3VPcavv/5aaDQa8dFHH1V5m/XChQvFL7/8Ii5evCiioqLEQw89JDQaTbmSaq7H99///ld899134ty5c+LEiRPihRdeEADEpk2bjPuY0zk09fjKPPDAA6JHjx6Vfk1zOn9ZWVnG9zkAYunSpSIqKsp4p525/g5afRmZNm2aAFDhY/v27cZ9AIjPPvvM+GeDwSBeeeUV4e3tLXQ6nejXr1+FNpyXlycef/xx4ebmJuzs7MTIkSNFTExMPR3VP9LS0sT9998vnJychJOTk7j//vsr3GJ38/EJIcTKlSuFnZ1dpetOxMTEiH79+gk3Nzeh1WpFixYtxJNPPllhrY76YOrx5ebmivDwcOHh4SFsbGxE06ZNxbRp0yqcG3M5f0KYfozR0dGV/kzf+HMt8xx+9NFHIiAgQGi1WtG1a9dyIzHTpk0T/fv3L7f/jh07RJcuXYRWqxXNmjWrtCBv3LhRBAUFCRsbG9GmTZtyb3QymHKM/fv3r/RcTZs2zbjP3LlzRdOmTYVWqxUeHh4iPDxc7N27tx6PqDxTju/NN98ULVq0ELa2tqJRo0aib9++YuvWrRW+pjmdQ1N/RtPT04WdnZ345JNPKv165nT+ykZwqvp5M9ffQUWI0pkqRERERBJwnREiIiKSimWEiIiIpGIZISIiIqlYRoiIiEgqlhEiIiKSimWEiIiIpGIZISIiIqlYRoiIiEgqlhEiIiKSimWEiIiIpGIZISIiIqlYRoiIiEiq/wclH+JFCiVgUAAAAABJRU5ErkJggg==\n",
- "text/plain": [
- ""
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
+ "execution_count": 11,
+ "id": "abf96142",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
+ "outputs": [],
"source": [
"from numpy import asarray\n",
"from numpy import arange\n",
@@ -2287,8 +2500,10 @@
},
{
"cell_type": "markdown",
- "id": "709400b2",
- "metadata": {},
+ "id": "88461af3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Overview video on Stochastic Gradient Descent\n",
"\n",
@@ -2297,8 +2512,10 @@
},
{
"cell_type": "markdown",
- "id": "b20a0374",
- "metadata": {},
+ "id": "3aa7f3c9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Batches and mini-batches\n",
"\n",
@@ -2316,8 +2533,10 @@
},
{
"cell_type": "markdown",
- "id": "3cbbfa25",
- "metadata": {},
+ "id": "67c0758b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Stochastic Gradient Descent (SGD)\n",
"\n",
@@ -2346,8 +2565,10 @@
},
{
"cell_type": "markdown",
- "id": "aa788062",
- "metadata": {},
+ "id": "b4ef9e75",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Stochastic Gradient Descent\n",
"\n",
@@ -2361,8 +2582,10 @@
},
{
"cell_type": "markdown",
- "id": "a9169646",
- "metadata": {},
+ "id": "2c8ffc21",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"C(\\mathbf{\\beta}) = \\sum_{i=1}^n c_i(\\mathbf{x}_i,\n",
@@ -2372,8 +2595,10 @@
},
{
"cell_type": "markdown",
- "id": "40e7eda3",
- "metadata": {},
+ "id": "fbc49b1b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Computation of gradients\n",
"\n",
@@ -2383,8 +2608,10 @@
},
{
"cell_type": "markdown",
- "id": "2612121a",
- "metadata": {},
+ "id": "47107006",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\nabla_\\beta C(\\mathbf{\\beta}) = \\sum_i^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
@@ -2394,8 +2621,10 @@
},
{
"cell_type": "markdown",
- "id": "9191ca1c",
- "metadata": {},
+ "id": "ecb5ac17",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Stochasticity/randomness is introduced by only taking the\n",
"gradient on a subset of the data called minibatches. If there are $n$\n",
@@ -2406,8 +2635,10 @@
},
{
"cell_type": "markdown",
- "id": "91ca9074",
- "metadata": {},
+ "id": "4fe62968",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## SGD example\n",
"As an example, suppose we have $10$ data points $(\\mathbf{x}_1,\\cdots, \\mathbf{x}_{10})$ \n",
@@ -2426,8 +2657,10 @@
},
{
"cell_type": "markdown",
- "id": "b4b32b67",
- "metadata": {},
+ "id": "75c2c6b2",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\nabla_{\\beta}\n",
@@ -2439,8 +2672,10 @@
},
{
"cell_type": "markdown",
- "id": "c4b1dcc4",
- "metadata": {},
+ "id": "e2ebb692",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The gradient step\n",
"\n",
@@ -2449,8 +2684,10 @@
},
{
"cell_type": "markdown",
- "id": "f7b56002",
- "metadata": {},
+ "id": "812f02fc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\beta_{j+1} = \\beta_j - \\gamma_j \\sum_{i \\in B_k}^n \\nabla_\\beta c_i(\\mathbf{x}_i,\n",
@@ -2460,8 +2697,10 @@
},
{
"cell_type": "markdown",
- "id": "62572f80",
- "metadata": {},
+ "id": "e2dfe6e7",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $k$ is picked at random with equal\n",
"probability from $[1,n/M]$. An iteration over the number of\n",
@@ -2472,8 +2711,10 @@
},
{
"cell_type": "markdown",
- "id": "b8d2a032",
- "metadata": {},
+ "id": "4f2237b5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Simple example code"
]
@@ -2481,8 +2722,11 @@
{
"cell_type": "code",
"execution_count": 12,
- "id": "b19451bf",
- "metadata": {},
+ "id": "fb79c9b8",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np \n",
@@ -2503,8 +2747,10 @@
},
{
"cell_type": "markdown",
- "id": "30f707fb",
- "metadata": {},
+ "id": "2ee7aa93",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Taking the gradient only on a subset of the data has two important\n",
"benefits. First, it introduces randomness which decreases the chance\n",
@@ -2517,8 +2763,10 @@
},
{
"cell_type": "markdown",
- "id": "367c08e1",
- "metadata": {},
+ "id": "fb53dcf4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## When do we stop?\n",
"\n",
@@ -2536,8 +2784,10 @@
},
{
"cell_type": "markdown",
- "id": "b8c311e9",
- "metadata": {},
+ "id": "093d24d3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Slightly different approach\n",
"\n",
@@ -2554,8 +2804,10 @@
},
{
"cell_type": "markdown",
- "id": "c8c5cdc1",
- "metadata": {},
+ "id": "f31cb40f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Time decay rate\n",
"\n",
@@ -2571,8 +2823,11 @@
{
"cell_type": "code",
"execution_count": 13,
- "id": "544eb2dd",
- "metadata": {},
+ "id": "d451158b",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import numpy as np \n",
@@ -2603,8 +2858,10 @@
},
{
"cell_type": "markdown",
- "id": "3faaae92",
- "metadata": {},
+ "id": "bc39bab5",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Code with a Number of Minibatches which varies\n",
"\n",
@@ -2614,8 +2871,11 @@
{
"cell_type": "code",
"execution_count": 14,
- "id": "408e2ee7",
- "metadata": {},
+ "id": "a7d8b2f5",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Importing various packages\n",
@@ -2688,8 +2948,10 @@
},
{
"cell_type": "markdown",
- "id": "1e453a41",
- "metadata": {},
+ "id": "b298d58f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Replace or not\n",
"\n",
@@ -2701,8 +2963,10 @@
},
{
"cell_type": "markdown",
- "id": "689a1a4c",
- "metadata": {},
+ "id": "81a9f579",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Momentum based GD\n",
"\n",
@@ -2714,8 +2978,10 @@
},
{
"cell_type": "markdown",
- "id": "ba61a41e",
- "metadata": {},
+ "id": "0cd6d714",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n",
@@ -2724,8 +2990,10 @@
},
{
"cell_type": "markdown",
- "id": "5fff6bfd",
- "metadata": {},
+ "id": "c9c6fc5d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"
\n",
@@ -2740,8 +3008,10 @@
},
{
"cell_type": "markdown",
- "id": "1881c675",
- "metadata": {},
+ "id": "c607fe5d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have introduced a momentum parameter $\\gamma$, with\n",
"$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n",
@@ -2757,8 +3027,10 @@
},
{
"cell_type": "markdown",
- "id": "ed25754e",
- "metadata": {},
+ "id": "b3f80a69",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n",
@@ -2767,16 +3039,20 @@
},
{
"cell_type": "markdown",
- "id": "0d9564b0",
- "metadata": {},
+ "id": "1bfc8aba",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$."
]
},
{
"cell_type": "markdown",
- "id": "c65178dc",
- "metadata": {},
+ "id": "a6f7d10c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More on momentum based approaches\n",
"\n",
@@ -2789,8 +3065,10 @@
},
{
"cell_type": "markdown",
- "id": "4d475dff",
- "metadata": {},
+ "id": "1b69b8da",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"m {d^2 \\mathbf{w} \\over dt^2} + \\mu {d \\mathbf{w} \\over dt }= -\\nabla_w E(\\mathbf{w}).\n",
@@ -2799,16 +3077,20 @@
},
{
"cell_type": "markdown",
- "id": "cea4a37f",
- "metadata": {},
+ "id": "8b6e31ec",
+ "metadata": {
+ "editable": true
+ },
"source": [
"We can discretize this equation in the usual way to get"
]
},
{
"cell_type": "markdown",
- "id": "9dcb9aa5",
- "metadata": {},
+ "id": "04ccb04b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"m { \\mathbf{w}_{t+\\Delta t}-2 \\mathbf{w}_{t} +\\mathbf{w}_{t-\\Delta t} \\over (\\Delta t)^2}+\\mu {\\mathbf{w}_{t+\\Delta t}- \\mathbf{w}_{t} \\over \\Delta t} = -\\nabla_w E(\\mathbf{w}).\n",
@@ -2817,16 +3099,20 @@
},
{
"cell_type": "markdown",
- "id": "9500c863",
- "metadata": {},
+ "id": "03117758",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Rearranging this equation, we can rewrite this as"
]
},
{
"cell_type": "markdown",
- "id": "d9b58226",
- "metadata": {},
+ "id": "b0f6b81a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\Delta \\mathbf{w}_{t +\\Delta t}= - { (\\Delta t)^2 \\over m +\\mu \\Delta t} \\nabla_w E(\\mathbf{w})+ {m \\over m +\\mu \\Delta t} \\Delta \\mathbf{w}_t.\n",
@@ -2835,8 +3121,10 @@
},
{
"cell_type": "markdown",
- "id": "d6c27d85",
- "metadata": {},
+ "id": "38d7e2fb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Momentum parameter\n",
"\n",
@@ -2849,8 +3137,10 @@
},
{
"cell_type": "markdown",
- "id": "7415d191",
- "metadata": {},
+ "id": "04401e37",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\gamma= {m \\over m +\\mu \\Delta t }, \\qquad \\eta = {(\\Delta t)^2 \\over m +\\mu \\Delta t}.\n",
@@ -2859,8 +3149,10 @@
},
{
"cell_type": "markdown",
- "id": "cd444de6",
- "metadata": {},
+ "id": "9991596c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Thus, as the name suggests, the momentum parameter is proportional to\n",
"the mass of the particle and effectively provides inertia.\n",
@@ -2890,8 +3182,10 @@
},
{
"cell_type": "markdown",
- "id": "ac6dec18",
- "metadata": {},
+ "id": "06bf4bff",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t +\\gamma \\mathbf{v}_{t-1}) \\nonumber\n",
@@ -2900,8 +3194,10 @@
},
{
"cell_type": "markdown",
- "id": "5c0ed7f6",
- "metadata": {},
+ "id": "2165360e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"
\n",
@@ -2916,16 +3212,20 @@
},
{
"cell_type": "markdown",
- "id": "5223324b",
- "metadata": {},
+ "id": "ebe6e122",
+ "metadata": {
+ "editable": true
+ },
"source": [
"One of the major advantages of NAG is that it allows for the use of a larger learning rate than GDM for the same choice of $\\gamma$."
]
},
{
"cell_type": "markdown",
- "id": "54ad02fe",
- "metadata": {},
+ "id": "f475bfbc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Second moment of the gradient\n",
"\n",
@@ -2953,8 +3253,10 @@
},
{
"cell_type": "markdown",
- "id": "b8678992",
- "metadata": {},
+ "id": "4d7e384a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## RMS prop\n",
"\n",
@@ -2966,8 +3268,10 @@
},
{
"cell_type": "markdown",
- "id": "1fdf2408",
- "metadata": {},
+ "id": "b6071ded",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"
\n",
@@ -2982,8 +3286,10 @@
},
{
"cell_type": "markdown",
- "id": "634f0d79",
- "metadata": {},
+ "id": "94eac756",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{s}_t =\\beta \\mathbf{s}_{t-1} +(1-\\beta)\\mathbf{g}_t^2 \\nonumber\n",
@@ -2992,8 +3298,10 @@
},
{
"cell_type": "markdown",
- "id": "3382f2dc",
- "metadata": {},
+ "id": "ed51ae61",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\mathbf{g}_t \\over \\sqrt{\\mathbf{s}_t +\\epsilon}}, \\nonumber\n",
@@ -3002,8 +3310,10 @@
},
{
"cell_type": "markdown",
- "id": "fc84f4a7",
- "metadata": {},
+ "id": "421337f8",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\beta$ controls the averaging time of the second moment and is\n",
"typically taken to be about $\\beta=0.9$, $\\eta_t$ is a learning rate\n",
@@ -3018,8 +3328,10 @@
},
{
"cell_type": "markdown",
- "id": "759af7d5",
- "metadata": {},
+ "id": "d11b6a1b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## [ADAM optimizer](https://arxiv.org/abs/1412.6980)\n",
"\n",
@@ -3045,8 +3357,10 @@
},
{
"cell_type": "markdown",
- "id": "0911ded6",
- "metadata": {},
+ "id": "251b6047",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"
\n",
@@ -3061,8 +3375,10 @@
},
{
"cell_type": "markdown",
- "id": "b1fd1ff5",
- "metadata": {},
+ "id": "d9a7ac1c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{m}_t = \\beta_1 \\mathbf{m}_{t-1} + (1-\\beta_1) \\mathbf{g}_t \\nonumber\n",
@@ -3071,8 +3387,10 @@
},
{
"cell_type": "markdown",
- "id": "835b7553",
- "metadata": {},
+ "id": "ef72b020",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\mathbf{s}_t =\\beta_2 \\mathbf{s}_{t-1} +(1-\\beta_2)\\mathbf{g}_t^2 \\nonumber\n",
@@ -3081,8 +3399,10 @@
},
{
"cell_type": "markdown",
- "id": "70d7e199",
- "metadata": {},
+ "id": "4235c812",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\mathbf{m}}_t={\\mathbf{m}_t \\over 1-\\beta_1^t} \\nonumber\n",
@@ -3091,8 +3411,10 @@
},
{
"cell_type": "markdown",
- "id": "2b00e15c",
- "metadata": {},
+ "id": "41571087",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\mathbf{s}}_t ={\\mathbf{s}_t \\over1-\\beta_2^t} \\nonumber\n",
@@ -3101,8 +3423,10 @@
},
{
"cell_type": "markdown",
- "id": "e70ed9b3",
- "metadata": {},
+ "id": "12088e7b",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\boldsymbol{\\theta}_{t+1}=\\boldsymbol{\\theta}_t - \\eta_t { \\boldsymbol{\\mathbf{m}}_t \\over \\sqrt{\\boldsymbol{\\mathbf{s}}_t} +\\epsilon}, \\nonumber\n",
@@ -3111,8 +3435,10 @@
},
{
"cell_type": "markdown",
- "id": "774cfe10",
- "metadata": {},
+ "id": "6a9c8dcb",
+ "metadata": {
+ "editable": true
+ },
"source": [
"\n",
"
\n",
@@ -3126,8 +3452,10 @@
},
{
"cell_type": "markdown",
- "id": "90770cff",
- "metadata": {},
+ "id": "9da02b37",
+ "metadata": {
+ "editable": true
+ },
"source": [
"where $\\beta_1$ and $\\beta_2$ set the memory lifetime of the first and\n",
"second moment and are typically taken to be $0.9$ and $0.99$\n",
@@ -3143,8 +3471,10 @@
},
{
"cell_type": "markdown",
- "id": "9b07b4bf",
- "metadata": {},
+ "id": "743e88c6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"\\Delta \\theta_{t+1}= -\\eta_t { \\boldsymbol{m}_t \\over \\sqrt{\\sigma_t^2 + m_t^2 }+\\epsilon}.\n",
@@ -3153,8 +3483,10 @@
},
{
"cell_type": "markdown",
- "id": "69c90c53",
- "metadata": {},
+ "id": "7f91733d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Algorithms and codes for Adagrad, RMSprop and Adam\n",
"\n",
@@ -3165,8 +3497,10 @@
},
{
"cell_type": "markdown",
- "id": "354ca513",
- "metadata": {},
+ "id": "b4cd6366",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Practical tips\n",
"\n",
@@ -3183,8 +3517,10 @@
},
{
"cell_type": "markdown",
- "id": "8d7efffd",
- "metadata": {},
+ "id": "99202ef3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Automatic differentiation\n",
"\n",
@@ -3219,8 +3555,10 @@
},
{
"cell_type": "markdown",
- "id": "1c087a9c",
- "metadata": {},
+ "id": "4fa1d63c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f(x) = \\sin\\left(2\\pi x + x^2\\right)\n",
@@ -3229,16 +3567,20 @@
},
{
"cell_type": "markdown",
- "id": "3ec98868",
- "metadata": {},
+ "id": "b0ee20d3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"which has the following derivative"
]
},
{
"cell_type": "markdown",
- "id": "2afab27d",
- "metadata": {},
+ "id": "63e14055",
+ "metadata": {
+ "editable": true
+ },
"source": [
"$$\n",
"f'(x) = \\cos\\left(2\\pi x + x^2\\right)\\left(2\\pi + 2x\\right)\n",
@@ -3247,8 +3589,10 @@
},
{
"cell_type": "markdown",
- "id": "7c8893ac",
- "metadata": {},
+ "id": "66c4d23c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Using **autograd** we have"
]
@@ -3256,8 +3600,11 @@
{
"cell_type": "code",
"execution_count": 15,
- "id": "5413ef4a",
- "metadata": {},
+ "id": "2f87d79b",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3298,8 +3645,10 @@
},
{
"cell_type": "markdown",
- "id": "278b124a",
- "metadata": {},
+ "id": "3688ef67",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Using autograd\n",
"\n",
@@ -3313,8 +3662,11 @@
{
"cell_type": "code",
"execution_count": 16,
- "id": "853fe565",
- "metadata": {},
+ "id": "3ee251b9",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3338,8 +3690,10 @@
},
{
"cell_type": "markdown",
- "id": "e85a2d43",
- "metadata": {},
+ "id": "87c7c770",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Autograd with more complicated functions\n",
"\n",
@@ -3351,8 +3705,11 @@
{
"cell_type": "code",
"execution_count": 17,
- "id": "18ed31ce",
- "metadata": {},
+ "id": "fa38cdbe",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3392,16 +3749,20 @@
},
{
"cell_type": "markdown",
- "id": "32b4ee69",
- "metadata": {},
+ "id": "0989ade0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Note that the grad function will not produce the true gradient of the function. The true gradient of a function with two or more variables will produce a vector, where each element is the function differentiated w.r.t a variable."
]
},
{
"cell_type": "markdown",
- "id": "0c4d7bdf",
- "metadata": {},
+ "id": "0125622a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More complicated functions using the elements of their arguments directly"
]
@@ -3409,8 +3770,11 @@
{
"cell_type": "code",
"execution_count": 18,
- "id": "4b084e1f",
- "metadata": {},
+ "id": "d0d011ba",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3434,8 +3798,10 @@
},
{
"cell_type": "markdown",
- "id": "c969a4da",
- "metadata": {},
+ "id": "917f6fd6",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Note that in this case, when sending an array as input argument, the\n",
"output from Autograd is another array. This is the true gradient of\n",
@@ -3447,8 +3813,10 @@
},
{
"cell_type": "markdown",
- "id": "b2d25440",
- "metadata": {},
+ "id": "58ab78e1",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Functions using mathematical functions from Numpy"
]
@@ -3456,8 +3824,11 @@
{
"cell_type": "code",
"execution_count": 19,
- "id": "6181bda4",
- "metadata": {},
+ "id": "d9e1bc97",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3481,8 +3852,10 @@
},
{
"cell_type": "markdown",
- "id": "98e8aabf",
- "metadata": {},
+ "id": "6754db4a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## More autograd"
]
@@ -3490,8 +3863,11 @@
{
"cell_type": "code",
"execution_count": 20,
- "id": "60ac4f35",
- "metadata": {},
+ "id": "0ebc9386",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3512,8 +3888,10 @@
},
{
"cell_type": "markdown",
- "id": "c577e160",
- "metadata": {},
+ "id": "68ddedf0",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## And with loops"
]
@@ -3521,8 +3899,11 @@
{
"cell_type": "code",
"execution_count": 21,
- "id": "ae41b5ac",
- "metadata": {},
+ "id": "6003a8a3",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3554,8 +3935,11 @@
{
"cell_type": "code",
"execution_count": 22,
- "id": "68b924f3",
- "metadata": {},
+ "id": "80946969",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3571,8 +3955,10 @@
},
{
"cell_type": "markdown",
- "id": "5eb36e97",
- "metadata": {},
+ "id": "7b751a00",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Using recursion"
]
@@ -3580,8 +3966,11 @@
{
"cell_type": "code",
"execution_count": 23,
- "id": "cfd9ab29",
- "metadata": {},
+ "id": "706f1e09",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3615,16 +4004,20 @@
},
{
"cell_type": "markdown",
- "id": "c78802df",
- "metadata": {},
+ "id": "46e02aa3",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Note that if n is equal to zero or one, Autograd will give an error message. This message appears when the output is independent on input."
]
},
{
"cell_type": "markdown",
- "id": "7ac7ca67",
- "metadata": {},
+ "id": "1f5aaa8d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Unsupported functions\n",
"Autograd supports many features. However, there are some functions that is not supported (yet) by Autograd.\n",
@@ -3635,8 +4028,11 @@
{
"cell_type": "code",
"execution_count": 24,
- "id": "be76b7ff",
- "metadata": {},
+ "id": "8a81e157",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3654,16 +4050,20 @@
},
{
"cell_type": "markdown",
- "id": "6b5e54b1",
- "metadata": {},
+ "id": "2588e16c",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Here, Autograd tells us that an 'ArrayBox' does not support item assignment. The item assignment is done when the program tries to assign x[2] to the value 3. However, Autograd has implemented the computation of the derivative such that this assignment is not possible."
]
},
{
"cell_type": "markdown",
- "id": "d0a857aa",
- "metadata": {},
+ "id": "5589db0e",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## The syntax a.dot(b) when finding the dot product"
]
@@ -3671,8 +4071,11 @@
{
"cell_type": "code",
"execution_count": 25,
- "id": "6cffb59c",
- "metadata": {},
+ "id": "26173432",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3690,8 +4093,10 @@
},
{
"cell_type": "markdown",
- "id": "5dc38bbb",
- "metadata": {},
+ "id": "6cc52760",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Here we are told that the 'dot' function does not belong to Autograd's\n",
"version of a Numpy array. To overcome this, an alternative syntax\n",
@@ -3701,8 +4106,11 @@
{
"cell_type": "code",
"execution_count": 26,
- "id": "55d2236d",
- "metadata": {},
+ "id": "9ad63b9e",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -3723,8 +4131,10 @@
},
{
"cell_type": "markdown",
- "id": "e1949f73",
- "metadata": {},
+ "id": "ecd12f9d",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Recommended to avoid\n",
"The documentation recommends to avoid inplace operations such as"
@@ -3733,8 +4143,11 @@
{
"cell_type": "code",
"execution_count": 27,
- "id": "b55dcbf7",
- "metadata": {},
+ "id": "9bdfae28",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"a += b\n",
@@ -3745,8 +4158,10 @@
},
{
"cell_type": "markdown",
- "id": "bd2d55bf",
- "metadata": {},
+ "id": "0752edea",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Using Autograd with OLS\n",
"\n",
@@ -3758,8 +4173,11 @@
{
"cell_type": "code",
"execution_count": 28,
- "id": "a68465ca",
- "metadata": {},
+ "id": "571aa609",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Using Autograd to calculate gradients for OLS\n",
@@ -3815,8 +4233,10 @@
},
{
"cell_type": "markdown",
- "id": "d1352e31",
- "metadata": {},
+ "id": "1c57601a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Same code but now with momentum gradient descent"
]
@@ -3824,8 +4244,11 @@
{
"cell_type": "code",
"execution_count": 29,
- "id": "04d64f0f",
- "metadata": {},
+ "id": "da1775c9",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Using Autograd to calculate gradients for OLS\n",
@@ -3885,8 +4308,10 @@
},
{
"cell_type": "markdown",
- "id": "978f6eaa",
- "metadata": {},
+ "id": "0274af86",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## But noen of these can compete with Newton's method"
]
@@ -3894,8 +4319,11 @@
{
"cell_type": "code",
"execution_count": 30,
- "id": "d48d641e",
- "metadata": {},
+ "id": "c59ae866",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Using Newton's method\n",
@@ -3940,8 +4368,10 @@
},
{
"cell_type": "markdown",
- "id": "e6a4745e",
- "metadata": {},
+ "id": "e8b233d4",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Including Stochastic Gradient Descent with Autograd\n",
"In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**."
@@ -3950,8 +4380,11 @@
{
"cell_type": "code",
"execution_count": 31,
- "id": "781ec7f8",
- "metadata": {},
+ "id": "b38efede",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using SGD\n",
@@ -4031,8 +4464,10 @@
},
{
"cell_type": "markdown",
- "id": "ce7d63db",
- "metadata": {},
+ "id": "ce06f66f",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Same code but now with momentum gradient descent"
]
@@ -4040,8 +4475,11 @@
{
"cell_type": "code",
"execution_count": 32,
- "id": "51f7d9c5",
- "metadata": {},
+ "id": "640faf4f",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using SGD\n",
@@ -4115,8 +4553,10 @@
},
{
"cell_type": "markdown",
- "id": "ab30dcd4",
- "metadata": {},
+ "id": "c2c161c9",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Similar (second order function now) problem but now with AdaGrad"
]
@@ -4124,8 +4564,11 @@
{
"cell_type": "code",
"execution_count": 33,
- "id": "10f517b2",
- "metadata": {},
+ "id": "201d332f",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n",
@@ -4180,16 +4623,20 @@
},
{
"cell_type": "markdown",
- "id": "b91d052e",
- "metadata": {},
+ "id": "ac5ec82a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"Running this code we note an almost perfect agreement with the results from matrix inversion."
]
},
{
"cell_type": "markdown",
- "id": "65836a18",
- "metadata": {},
+ "id": "9dda24de",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## RMSprop for adaptive learning rate with Stochastic Gradient Descent"
]
@@ -4197,8 +4644,11 @@
{
"cell_type": "code",
"execution_count": 34,
- "id": "8c82fefc",
- "metadata": {},
+ "id": "8e8efb2f",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n",
@@ -4259,8 +4709,10 @@
},
{
"cell_type": "markdown",
- "id": "a0cda7e9",
- "metadata": {},
+ "id": "e84e816a",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## And finally [ADAM](https://arxiv.org/pdf/1412.6980.pdf)"
]
@@ -4268,8 +4720,11 @@
{
"cell_type": "code",
"execution_count": 35,
- "id": "8bd52f8a",
- "metadata": {},
+ "id": "177c74c7",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n",
@@ -4335,8 +4790,10 @@
},
{
"cell_type": "markdown",
- "id": "a975415e",
- "metadata": {},
+ "id": "c614be76",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## And Logistic Regression"
]
@@ -4344,8 +4801,11 @@
{
"cell_type": "code",
"execution_count": 36,
- "id": "5dc95a66",
- "metadata": {},
+ "id": "f4d36293",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import autograd.numpy as np\n",
@@ -4385,8 +4845,10 @@
},
{
"cell_type": "markdown",
- "id": "8e1dc4f6",
- "metadata": {},
+ "id": "3e0dd6bc",
+ "metadata": {
+ "editable": true
+ },
"source": [
"## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n",
"\n",
@@ -4402,8 +4864,11 @@
{
"cell_type": "code",
"execution_count": 37,
- "id": "27acc149",
- "metadata": {},
+ "id": "84c8f0d3",
+ "metadata": {
+ "collapsed": false,
+ "editable": true
+ },
"outputs": [],
"source": [
"import jax.numpy as jnp\n",
@@ -4418,25 +4883,7 @@
]
}
],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.9.10"
- }
- },
+ "metadata": {},
"nbformat": 4,
"nbformat_minor": 5
}
diff --git a/doc/src/week39/week39.do.txt b/doc/src/week39/week39.do.txt
index 8e170ce28..e1cd85b94 100644
--- a/doc/src/week39/week39.do.txt
+++ b/doc/src/week39/week39.do.txt
@@ -19,7 +19,7 @@ These sections summarize neatly what we have done till now and point to what is
!bblock Material for the lecture on Thursday September 28
* Repetition of Logistic regression equations and classification problems and discussion of Gradient methods. Examples on how to implement Logistic Regression and discussion of stochastic gradient descent
* Stochastic Gradient descent with examples and automatic differentiation
- * "Video of lecture":"https://youtu.be/"
+ * "Video of lecture":"https://youtu.be/bFRVuIJroHs"
* Whiteboard notes TBA at URL:"https://github.com/CompPhysics/MachineLearning/blob/master/doc/HandWrittenNotes/2023/NotesSep28.pdf"
* Readings and Videos:
* These lecture notes