FYS-STK4155/doc/LectureNotes/_build/jupyter_execute/exercisesweek41.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e3354581",
   "metadata": {
    "editable": true
   },
   "source": [
    "<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
    "doconce format html exercisesweek41.do.txt  -->\n",
    "<!-- dom:TITLE: Exercises week 41 -->"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d5fc499",
   "metadata": {
    "editable": true
   },
   "source": [
    "# Exercises week 41\n",
    "**October 9-13, 2023**\n",
    "\n",
    "Date: **Deadline is Sunday October 15 at midnight**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3ef3596",
   "metadata": {
    "editable": true
   },
   "source": [
    "# Overarching aims of the exercises this week\n",
    "\n",
    "The aim of the exercises this week is to get started with implementing\n",
    "gradient methods of relevance for project 2. This exercise will also\n",
    "be continued next week with the addition of automatic differentation.\n",
    "Everything you develop here will be used in project 2. \n",
    "\n",
    "In order to get started, we will now replace in our standard ordinary\n",
    "least squares (OLS) and Ridge regression codes (from project 1) the\n",
    "matrix inversion algorithm with our own gradient descent (GD) and SGD\n",
    "codes.  You can use the Franke function or the terrain data from\n",
    "project 1. **However, we recommend using a simpler function like**\n",
    "$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.\n",
    "You can obviously test your final codes against for example the Franke\n",
    "function. Automatic differentiation will be discussed next week.\n",
    "\n",
    "You should include in your analysis of the GD and SGD codes the following elements\n",
    "1. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression of the gradients\n",
    "\n",
    "2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate), again using the analytical expression of the gradients.\n",
    "\n",
    "3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n",
    "\n",
    "4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.\n",
    "\n",
    "5. Add RMSprop and Adam to your library of methods for tuning the learning rate.\n",
    "\n",
    "The lecture notes from weeks 39 and 40 contain more information and code examples. Feel free to use these examples.\n",
    "\n",
    "In summary, you should \n",
    "perform an analysis of the results for OLS and Ridge regression as\n",
    "function of the chosen learning rates, the number of mini-batches and\n",
    "epochs as well as algorithm for scaling the learning rate. You can\n",
    "also compare your own results with those that can be obtained using\n",
    "for example **Scikit-Learn**'s various SGD options.  Discuss your\n",
    "results. For Ridge regression you need now to study the results as functions of  the hyper-parameter $\\lambda$ and \n",
    "the learning rate $\\eta$.  Discuss your results.\n",
    "\n",
    "You will need your SGD code for the setup of the Neural Network and\n",
    "Logistic Regression codes. You will find the Python [Seaborn\n",
    "package](https://seaborn.pydata.org/generated/seaborn.heatmap.html)\n",
    "useful when plotting the results as function of the learning rate\n",
    "$\\eta$ and the hyper-parameter $\\lambda$ when you use Ridge\n",
    "regression.\n",
    "\n",
    "We recommend reading chapter 8 on optimization from the textbook of [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/). This chapter contains many useful insights and discussions on the optimization part of machine learning."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9940e21b",
   "metadata": {
    "editable": true
   },
   "source": [
    "# Code examples from week 39 and 40"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6559d6d2",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Code with a Number of Minibatches which varies, analytical gradient\n",
    "\n",
    "In the code here we vary the number of mini-batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "021b66d2",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[3.83505053]\n",
      " [3.09726322]]\n",
      "Eigenvalues of Hessian Matrix:[0.27717261 4.34154132]\n",
      "theta from own gd\n",
      "[[3.83505053]\n",
      " [3.09726322]]\n",
      "theta from own sdg\n",
      "[[3.77589027]\n",
      " [3.07286416]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABN1ElEQVR4nO3deXxTVfo/8E9oaQsIlRa6F6iKrLIIIggIfGURWYqVQUHZZlxZFIsLFR1ARxj4uaCiIurAqNO6UWjVcUEpUBW0bCKCIGPBUigIaFuWljY9vz9CQtMkzU1y19zP+/XKMLm5uTk3i/fpOc95jkUIIUBERERkUg20bgARERGRlhgMERERkakxGCIiIiJTYzBEREREpsZgiIiIiEyNwRARERGZGoMhIiIiMjUGQ0RERGRqDIaIiIjI1BgMEenQqlWrYLFYHLfQ0FDEx8fjtttuwy+//KJZu+bPnw+LxaLZ6xvBwYMHYbFY8Mwzz2jdFCKSKFTrBhCRZytXrkT79u1RUVGBb775Bk8//TTy8vLw888/o3nz5lo3j4goKDAYItKxzp07o2fPngCAgQMHwmq1Yt68eVi7di2mTp2qcetIS1VVVY5eQyIKDIfJiAzEHhgdO3bMsa2iogKzZ89Gt27dEBkZiaioKPTp0wc5OTkuz7dYLJgxYwbefvttdOjQAY0bN0bXrl3x8ccfu+z7ySefoFu3bggPD0dKSorHYZ+KigpkZGQgJSUFYWFhSExMxPTp0/Hnn3867demTRuMHDkSH3/8Mbp3745GjRqhQ4cOjtdetWoVOnTogCZNmqBXr17YunWr1/fDPpyYl5eH++67Dy1atEB0dDTS0tJw5MgRl3OfP3++yzHatGmDKVOmuBxz/fr1uOuuuxAdHY1mzZph0qRJOHPmDEpKSjBu3DhceumliI+Px0MPPYSqqiqX49bU1ODpp59Gq1atEBERgZ49e+Krr75y2e+XX37BhAkTEBMTg/DwcHTo0AEvv/yy0z4bNmyAxWLB22+/jdmzZyMxMRHh4eE4cOAAzp49i4ceeggpKSmIiIhAVFQUevbsiaysLK/vHxHZ8E8KIgMpLCwEAFx55ZWObZWVlTh16hQeeughJCYm4vz58/jyyy+RlpaGlStXYtKkSU7H+OSTT1BQUIAnn3wSl1xyCZYsWYKbb74Z+/btw2WXXQYA+Oqrr5Camoo+ffrg3XffhdVqxZIlS5yCMAAQQmDMmDH46quvkJGRgf79+2PXrl2YN28eNm/ejM2bNyM8PNyx/w8//ICMjAzMnTsXkZGRWLBgAdLS0pCRkYGvvvoKCxcuhMViwaOPPoqRI0eisLAQjRo18vq+3HnnnRgxYgQyMzNRVFSEhx9+GHfccQfWr1/v93t95513Ii0tDe+++y527NiBxx57DNXV1di3bx/S0tJw991348svv8TixYuRkJCA9PR0p+cvW7YMrVu3xtKlS1FTU4MlS5Zg+PDh2LhxI/r06QMA2LNnD6677jq0atUKzz77LOLi4vD555/j/vvvx4kTJzBv3jynY2ZkZKBPnz5Yvnw5GjRogJiYGKSnp+Ptt9/GP/7xD3Tv3h1nzpzB7t27cfLkSb/Pnch0BBHpzsqVKwUAsWXLFlFVVSXKy8vFZ599JuLi4sT1118vqqqqPD63urpaVFVVib/97W+ie/fuTo8BELGxsaKsrMyxraSkRDRo0EAsWrTIse3aa68VCQkJ4ty5c45tZWVlIioqStT+z8Znn30mAIglS5Y4vc57770nAIgVK1Y4trVu3Vo0atRIHD582LFt586dAoCIj48XZ86ccWxfu3atACByc3MlvU/Tpk1z2r5kyRIBQBw9etTp3OfNm+dyjNatW4vJkye7HHPmzJlO+40ZM0YAEM8995zT9m7duomrr77acb+wsFAA8Pj+DR482LFt2LBhIikpSZSWljodc8aMGSIiIkKcOnVKCCFEXl6eACCuv/56l/Z37txZjBkzxmU7EUnHYTIiHevduzcaNmyIpk2b4sYbb0Tz5s2Rk5PjkifywQcfoG/fvrjkkksQGhqKhg0b4s0338TevXtdjjlo0CA0bdrUcT82NhYxMTE4dOgQAODMmTMoKChAWloaIiIiHPs1bdoUo0aNcjqWveel9jATAPzlL39BkyZNXIaFunXrhsTERMf9Dh06ALDlQzVu3Nhlu71N3owePdrpfpcuXXx6vjsjR450um9v04gRI1y2u3sdT+/fpk2bYLVaUVFRga+++go333wzGjdujOrqasftpptuQkVFBbZs2eJ0zFtuucXldXr16oVPP/0Uc+bMwYYNG3Du3Dm/z5nIrBgMEenYW2+9hYKCAqxfvx733HMP9u7di/Hjxzvtk52djXHjxiExMRHvvPMONm/ejIKCAvz1r39FRUWFyzGjo6NdtoWHhzsuon/88QdqamoQFxfnsl/dbSdPnkRoaChatmzptN1isSAuLs5lqCYqKsrpflhYWL3b3bXfnbrnZB+aCyQw8KWt7trp6f07f/48Tp8+jZMnT6K6uhovvfQSGjZs6HS76aabAAAnTpxwen58fLzLMV988UU8+uijWLt2LQYNGoSoqCiMGTNG0xIMREbDnCEiHevQoYMjaXrQoEGwWq1444038OGHH2Ls2LEAgHfeeQcpKSl47733nGoAVVZW+vWazZs3h8ViQUlJictjdbdFR0ejuroav//+u1NAJIRASUkJrrnmGr/aoITw8HC374lSuTWe3r+wsDBccsklaNiwIUJCQjBx4kRMnz7d7TFSUlKc7rur8dSkSRMsWLAACxYswLFjxxy9RKNGjcLPP/8sz8kQBTn2DBEZyJIlS9C8eXP8/e9/R01NDQDbBTIsLMzpQllSUuJ2NpkU9tlc2dnZTj0e5eXl+Oijj5z2veGGGwDYArLaVq9ejTNnzjge14M2bdpg165dTtvWr1+P06dPK/J6nt6//v37IyQkBI0bN8agQYOwY8cOdOnSBT179nS5uevFq09sbCymTJmC8ePHY9++fTh79qzcp0UUlNgzRGQgzZs3R0ZGBh555BFkZmbijjvuwMiRI5GdnY1p06Zh7NixKCoqwlNPPYX4+Hi/h0qeeuop3HjjjRgyZAhmz54Nq9WKxYsXo0mTJjh16pRjvyFDhmDYsGF49NFHUVZWhr59+zpmk3Xv3h0TJ06U69QDNnHiRDzxxBP4+9//jgEDBmDPnj1YtmwZIiMjFXm9kJAQDBkyBOnp6aipqcHixYtRVlaGBQsWOPZ54YUX0K9fP/Tv3x/33Xcf2rRpg/Lychw4cAAfffSRpNlw1157LUaOHIkuXbqgefPm2Lt3L95++2306dPHKQ+LiDxjMERkMDNnzsSyZcvw5JNPYvz48Zg6dSqOHz+O5cuX41//+hcuu+wyzJkzB4cPH3a68PpiyJAhWLt2LR5//HHceuutiIuLw7Rp03Du3DmnY1osFqxduxbz58/HypUr8fTTT6NFixaYOHEiFi5c6DStXmsPP/wwysrKsGrVKjzzzDPo1asX3n//faSmpiryejNmzEBFRQXuv/9+HD9+HJ06dcInn3yCvn37Ovbp2LEjtm/fjqeeegqPP/44jh8/jksvvRRt27Z15A1583//93/Izc3F888/j7NnzyIxMRGTJk3C3LlzFTkvomBkEUIIrRtBREREpBXmDBEREZGpMRgiIiIiU2MwRERERKbGYIiIiIhMjcEQERERmRqDISIiIjI1U9QZqqmpwZEjR9C0aVO35eyJiIhIf4QQKC8vR0JCAho0UK7/xhTB0JEjR5CcnKx1M4iIiMgPRUVFSEpKUuz4pgiGmjZtCsD2ZjZr1kzj1hAREZEUZWVlSE5OdlzHlWKKYMg+NNasWTMGQ0RERAajdIoLE6iJiIjI1BgMERERkakxGCIiIiJTYzBEREREpsZgiIiIiEyNwRARERGZGoMhIiIiMjUGQ0RERGRqDIaIiIjI1BgMERERkakxGCIiIiJTYzBEREREpsZgiIiIiEyNwRARERGZGoMhIiIiMjUGQ0RERGRqDIaIiIjI1DQPhjZt2oRRo0YhISEBFosFa9eu9bjvPffcA4vFgqVLl6rWPiIiIgpumgdDZ86cQdeuXbFs2bJ691u7di2+++47JCQkqNQyIiIiMoNQrRswfPhwDB8+vN59iouLMWPGDHz++ecYMWKESi0jIiIiM9C8Z8ibmpoaTJw4EQ8//DA6deqkdXOIiIgoyGjeM+TN4sWLERoaivvvv1/ycyorK1FZWem4X1ZWpkTTiIiIKAjoumdo27ZteOGFF7Bq1SpYLBbJz1u0aBEiIyMdt+TkZAVbSUREREam62AoPz8fx48fR6tWrRAaGorQ0FAcOnQIs2fPRps2bTw+LyMjA6WlpY5bUVGReo0mIiIiQ9H1MNnEiRMxePBgp23Dhg3DxIkTMXXqVI/PCw8PR3h4uNLNIyIioiCgeTB0+vRpHDhwwHG/sLAQO3fuRFRUFFq1aoXo6Gin/Rs2bIi4uDi0a9dO7aYSERFRENI8GNq6dSsGDRrkuJ+eng4AmDx5MlatWqVRq4iIiMgsNA+GBg4cCCGE5P0PHjyoXGOIiIjIdHSdQE1ERESkNAZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIyNQZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZlaqNYNICIioiBltQL5+cDRo0B8PNC/PxASonWrXDAYIiIiIvllZwMPPAAcPnxxW1IS8MILQFqadu1yg8NkREREJK/sbGDsWOdACACKi23bs7O1aZcHDIaIiIhIPlarrUdICNfH7NtmzbLtpxMMhoiIiEg++fmuPUK1CQEUFdn20wkGQ0RERCSfo0fl3U8FDIaIiIhIPvHx8u6nAgZDREREJJ/+/W2zxiwW949bLEBysm0/nWAwRERERPIJCbFNnwdcAyL7/aVLdVVviMEQERERySstDfjwQyAx0Xl7UpJtu87qDLHoIhEREckvLQ1ITWUFaiIiIjKxkBBg4ECtW+EVh8mIiIjI1BgMERERkakxGCIiIiJTYzBEREREpsZgiIiIiEyNwRARERGZGoMhIiIiMjUGQ0RERGRqDIaIiIjI1BgMERERkakxGCIiIiJT0zwY2rRpE0aNGoWEhARYLBasXbvW8VhVVRUeffRRXHXVVWjSpAkSEhIwadIkHDlyRLsGExERUVDRPBg6c+YMunbtimXLlrk8dvbsWWzfvh1PPPEEtm/fjuzsbOzfvx+jR4/WoKVEREQUjCxCCKF1I+wsFgvWrFmDMWPGeNynoKAAvXr1wqFDh9CqVStJxy0rK0NkZCRKS0vRrFkzmVpLRERESlLr+h2q2JEVUlpaCovFgksvvdTjPpWVlaisrHTcLysrU6FlREREZESaD5P5oqKiAnPmzMGECRPqjRAXLVqEyMhIxy05OVnFVhIREZGRGCYYqqqqwm233Yaamhq88sor9e6bkZGB0tJSx62oqEilVhIREZHRGGKYrKqqCuPGjUNhYSHWr1/vddwwPDwc4eHhKrWOiIiIjEz3wZA9EPrll1+Ql5eH6OhorZtEREREQUTzYOj06dM4cOCA435hYSF27tyJqKgoJCQkYOzYsdi+fTs+/vhjWK1WlJSUAACioqIQFhamVbOJiIgoSGg+tX7Dhg0YNGiQy/bJkydj/vz5SElJcfu8vLw8DBw4UNJrcGo9ERGR8Zhmav3AgQNRXzymozJIREREFIQMM5uMiIiISAkMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIyNQZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIyNQZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIytVCtG0BEROTCagXy84GjR4H4eKB/fyAkROtWUZBiMERERPqSnQ088ABw+PDFbUlJwAsvAGlp2rWLghaHyYiISD+ys4GxY50DIQAoLrZtz87Wpl0U1BgMERGRPlitth4hIVwfs2+bNcu2H5GMGAwREZE+5Oe79gjVJgRQVGTbj0hGzBkiIiJ9OHpU3v1IGUGY3M5giIiI9CE+Xt79SH5BmtzOYTIiItKH/v1tF1aLxf3jFguQnGzbj9QXxMntDIaIiPTKagU2bACysmz/BnvicEiIrYcBcA2I7PeXLjX8kIwhBXlyO4MhIiI9ys4G2rQBBg0CJkyw/dumjaH/+vbKagWiomwX3eho58eSkoAPPzT0UIyhBXlyO3OGiIj0xj4cUfevcPtwRDAGBe5yUVq2BG6/HUhNDYokXd2rLzE6yJPb2TNERKQnQT4c4ZanXJQTJ2zDZqdOMRBSmreeSLmT23U2BMxgiIhIT4J8OMKFGYM/vZGSGC1ncrsOh4AZDBER6UmQD0e4MFvwpzdSg1FAnuR2nc5I0zwY2rRpE0aNGoWEhARYLBasXbvW6XEhBObPn4+EhAQ0atQIAwcOxE8//aRNY4mIlGa2WjtmC/70xpdgNC3Nlq+WmOi8j9Tkdh33AmoeDJ05cwZdu3bFsmXL3D6+ZMkSPPfcc1i2bBkKCgoQFxeHIUOGoLy8XOWWEhGpwGy1dswW/OmNr8FoWhpw8CCQlwdkZtr+LSyUltCv415AzWeTDR8+HMOHD3f7mBACS5cuxdy5c5F24Y3+97//jdjYWGRmZuKee+5Rs6lERMqz19oZO9YW+NT+KzoYa+3Yg7/iYvc9BhaL7fFgCf70xp9gNCQEGDjQ99fyMfAqO1yG7IXf+f46ftC8Z6g+hYWFKCkpwdChQx3bwsPDMWDAAHz77bcen1dZWYmysjKnGxGRYQQ6HGEkLLSoLTV7IiUGXmtfP4ZhLbaiRXIEpr56beCvK4Gug6GSkhIAQGxsrNP22NhYx2PuLFq0CJGRkY5bcnKyou0kIpJdIMMRRmOm4E9v1AxGvQReNbDgNyTjlryZ+OJkT1QhDFc0LAz8dSXQdTBkZ6nzxgkhXLbVlpGRgdLSUsetqKhI6SYSEcnPPhwxfrzt32DuHTFT8Kc3agWjFwIvAaDugGgNbNf0B/Ecejf9CYuHb8DP//0V206kyPPaXmieM1SfuLg4ALYeovha3WvHjx936S2qLTw8HOHh4Yq3j4iIZORvLgoFLi3NVunbUwXqAJUdLsNnz+5GzppYNBD/xkI8hmRcTKY+2SAGuwbMwKsvDUBMp5YXn6dSmouug6GUlBTExcVh3bp16N69OwDg/Pnz2LhxIxYvXqxx64iIiIKIzMHo4YKjyH12P3K+bIK8k11QhesuPNIX6zAE6Ynv44aepeg09Rq0HDkEN2jY86l5MHT69GkcOHDAcb+wsBA7d+5EVFQUWrVqhVmzZmHhwoVo27Yt2rZti4ULF6Jx48aYMGGChq0mIiJDqW/dLZKFqBHY9eF+5Cw/ipwtsdh+rgOAi6M67cJ+RWq33zB6ShR6/60TQsLu166xdWgeDG3duhWDBg1y3E9PTwcATJ48GatWrcIjjzyCc+fOYdq0afjjjz9w7bXX4osvvkDTpk21ajIRERmJu0Vgk5JsicPMSQpI1dkqbHppJ/a8/g1++TUUu0Rn5KM/ahACC2rQp+lupPY7hdSZrdBu+GUALtO6yW5ZhHBX2CG4lJWVITIyEqWlpWjWrJnWzSEiIl/527NjX/6h7qXOPgmHs9V8djH/R6DBoV9d8n9ONIjBDwNm4qqX7nHK//HrtVS6fjMYIiIiffO3Z8dqtS0A6qnqsb2gY2Ehh8y8cM3/CcPNyMaHGAtAOE9NlzHQZDAkIwZDREQGFUjPzoYNthXRvcnL4yy2Otzn/1zUPvQANluuQ2TV73Bb6EamQFOt67fmOUNERERueVvY02KxLeyZmur+gstFYH1SdbYKm17+Ebn/KUfO7stxyNoOQDsAcM3/aXQYGPS754PVXmfMAIEmgyEiItInXxb2dHfB5SKwXtXO//nvb53wp7ja8VgjnMWQuB+ReuN5jJjdHrGdu1x8YpbENcMMEmgyGCIiIn0KtGeHi8C65bn+D9DS8jtGXfEzRo8Nw5D0q9C4hYe1wYIs0GQwRERE+hToBde+7tbYsbbAp3ZAZKJFYO35P7mvHUXOlhhsO9sRtev/XNmwEKndDiF1qr3+j4TgMMgCTSZQExGRPtlng3m74HpL0nU3Gy052RYIBem0+qqzVch/dTdy3i5D7k+X4WD1xQXL3df/8YM9uR1wH2hyNpm+MBgiIjIouS64JqhA7Zr/c6njMdf8n8Dq/zgoHGgyGJIRgyEiIgMzYc+OVO7q/9i55v80VqYRCgaaDIZkxGCIiMjgTNCzI4X7/J+LXPN/jP0esc4QERGRncwrqitCoYDNNf+nnvo/wy8DkBLwa5oNgyEiInLFnhjfyLwYrGv+T3fHY/XW/yG/MBgiIiJnXOXdN56WDCkutm2XmOTtrf7PyCt+Rqq3+j/kF+YMERHRRVzl3TcBLAZrtvwffzCBWkYMhoh0ikMx+mLWVd4D+R76uBis1Po/o2e0Qvub/Kz/E0SYQE1EwY1DMfoT6FpgWvMnqAn0eyhxyZDNizdi2ZSGLvk/ETiHoXG7MHrYeYx8SIH8H/7BIQmDISJSn0w5FiQzI6/y7k9QI8f3UOKSIRmfDcBG9AWgYv4P/+CQjMNkRKQusw7FGIGPQz664U+ek1zfwwvHEcXFsLi5nNbAgsNIwrDQ9RjV/bB6+T9q534p1AOl1vW7gWJHJiJyx5ehGFKXffFN+wWzLovFVvlZT4tvWq223g93f9fbt82aZduvNhm+h1Vnq7B+6S68GXI3hLAFPrXVwAILBBo8loG9VVdgyfcD0fe+LsoHQv6+J/7KzrYFloMGARMm2P5t08a23SAYDBGRuow8FBPs7Ku8A64BkV5Xefc3qPHze1h2uAzvP/gtbm/zDWIuOYMbHuqOOw89gbH4EEeQ4LRvg+QkWFavRtLT90l7Lbmo+QeHvQeq7uvZhxoNEhAxZ4iI1CUxx0LyfiSvtDTbEIq7XBM9rgXmb3Dtw/fwcMFRfPTcL8hZ1xjrPdb/SUTUA3uBvdu0T1bOyZG2X6B/cHjrgbJYbD1Qqan6CqDdYDBEROqyD8UUF7v/j6g9V0NPQzFmk5Zmu4AZYRaSv8G1l++hgAVl4S0w5KZoFJyLB3Dx+a71f2p9V2MH+n4OcrJagf/8R9q+gf7BYfTZh7UwGCIiddmHYsaOtQU+tS9Eeh2KMaO6a4FZrbYEa70FR/4G1/V8D+25P1Mrl6MAVznq/4zua1v/y1b/J8D1v5Sa8p6fD/z+u/f9WrYM/A+OIBryZs4QEanPPhSTmOi8PSmJ0+r1SM8JsgHkOZX1GoxvRvwDJxu0cNp+GEmYgP+gOjYRb0zOx9EfT+Kbsi549NOB8hRCVPL9lBp43H574MFXEA15c2o9EWmHBeG05+0z8HWKtlafqbuaOsnJLnlOrvk/YWgAK/ojH1diPy5PrED7O3picHpXNIlpokw7lZzyrmZ5BHt5Am+9cgGUyeByHDJiMERE5Ia3ony+1uLRusifm0BMWBrgx+xfkPPqkXrX/xo9JQp97lS4/o8aNbZUCFCc2IM7wP2Qd4DBHYMhGTEYIiKqQ0oPRWQkMHiw92Pl5QGnTulmgVcp63855//IQEqPmFq9NgoHKG5fT0KvnD+4NhkRESlDypTou+8GamqkHa+4GJgzR9Mp1mWHy/D5cz8hZ00NPjnkuv7XkNhdSL1RofW/pPaIqZVwrHZ5BCPNPvSAwRARkdlImRJ98qT045WUaDLFunjrUeQ+a8v/yTt5Fc6jj+OxFpYTGHXFXqSODcPgWZ3RJEbB9b+krm+mZsKx2gFK3dmHBsNhMiIis8nKss1ikkvz5sAff3jfLzMTGD/e75cRNUI/+T+A7zlAaufz2Nto4B4bDpMREQULvV2Q5J7qLCUQ8vN1XfN/rgRwJQBP+T8B1v/xha9FB9WusaV1QruBMBgiIlKSHi9I3goVys3HquKa5v94Uzuw3bNH2nNq5wCplc/jy/AdcZiMiEgxSteU8VXtC/kvvwDz5rn2UMhN4rm65v+EOx5zzf9RoP6PFO4CWynczQ5TsrdQjSn8KuHUehkxGCIi1entguTuQh4dbfu3drJ0UhJw7pxtqrw/l4eoKNtz7TxMsfaW/9O2YSFSu9rW/5Kc/6NkgOEpsK2PVkGHmoUXFcacISIio7JagZde0s8ilp4u5PagZcECoG3biwFETo77vBYp3n/fduF3E5B4y//pfclPSO130r/8HyWHI+srReCJluvsBdGaYWphMEREJCdfh1KUviBJqSn0xhvOvRee8lrsM6LcsfeC2BOFL/At/+cq/85R6fwYb4nS7ihV00eKIFozTC0MhoiI5OLPUIrSFyRfZzzZuatTc+IEMG7cxefZ1ekFkVL/Z/QtYRjyoAz1f6QEe4EWfJQasD7+ONCxo/YzBr0lyPuY0G4GDIaIiIDA8018HUpR64KUkyNtP3cXfHeF9Nz0GImkJBy69RG8/XIUcifuwdazHQFcDPJc839kPGd/gz1fSA1Yb7hBHzk4ak/hDwZC56qqqsTcuXNFmzZtREREhEhJSRELFiwQVqtV8jFKS0sFAFFaWqpgS4nIsFavFiIpSQjbZcN2S0qybZcqL8/5+fXdLBbbzZfj+6O6WoiWLaW1KS/Pp+NWfbpO7L59oXjp8ufFZSGFzqcHq+hzyS7xzxvzxN5P/qfY6QkhhMjMlHZ+mZn+v0Z1te37YLF4/jyTk2376Ym773VysvLfOxmpdf3Wfc/Q4sWLsXz5cvz73/9Gp06dsHXrVkydOhWRkZF44IEHtG4eERmdXPkmvuT+qJVPkp8P/P679/1atpTUQ1V+pByfPbMbOWtq8N9DPfCHuLiIq2z5P75SIz/GqD0tRl0zrHYvrUozwHUfDG3evBmpqakYMWIEAKBNmzbIysrC1q1bNW4ZERmenPkmUi+2zz8PzJypzgVJaoB2++0e26Nq/o8/1MqPUXvxU7kYbc0wf2s5BUj3wVC/fv2wfPly7N+/H1deeSV++OEHfP3111i6dKnWTSMio5Mz30TqRVmtQAiQHqClpjr+b+36P7lbYtTN//GHmr02Ru1pMQp/JiDIRPfB0KOPPorS0lK0b98eISEhsFqtePrppzG+nsX+KisrUVlZ6bhfVlamRlOJyGik9pwUF9sK2dV3AdTjUIrEAK2qR298/dwO5LxdhpzdMtf/UYOavTZG62kxCn9qOclJ0YwkGWRlZYmkpCSRlZUldu3aJd566y0RFRUlVq1a5fE58+bNEwBcbkygJlJYdbUtETcz0/ZvoAmlch+vLqlJz3WTkOtLrtZb0urq1RcTtmu1qcZiETWAeL7l06K55ZRTcyNwVoyK3SLemLxJlPx4XJt2+0Pp7wspx8NvsVSl67fug6GkpCSxbNkyp21PPfWUaNeuncfnVFRUiNLSUsetqKiIwRCR0uSYkaXk8dzxNkvI39lgersou3kvf0OSuBmrHZtaWH4XU67YJNbM2SJOHzutbXvJfDzMClQrGNL9MNnZs2fRoEEDp20hISGoqanx+Jzw8HCEh4d7fJyIZCZ3BWC1Vtyub2irPt6Sq3UylHIx/ycKH5/8LxrhJOJxFEcRj3z0x+UNf8NDXTfoJ/+HzEvjati6X6h1ypQp+PLLL/Haa6+hU6dO2LFjB+6++2789a9/xeLFiyUdgwu1EilI7gVJtVjg1N0MlpYtpU1L19lil9UV1ch/5cda+T/Jjsdq5/+Mnp6M9jddBksDi4atDTJKLhQb7Oy/+zr5bWUAIgGuWl9eXo4nnngCa9aswfHjx5GQkIDx48fj73//O8LCwiQdg8EQkYLkXiFbqxW3617IiouBO+7w/rzMTKCeCR1qcK7/0xF/iOaOx+z1f0YPO49RD7dHbOeWGrZUZ+QMXpRcKNYs7D3CgCMgUisY0v0wWdOmTbF06VJOpSfSK7lXyNbLittxcdL206h7317/J/fLRlh/ootL/Z+Rl+9F6lgN6//onZzBi1rDusHO06xAFeg+GCIinZO7ArAWK267uzAmJgLR0cCpU8oW85NI1AjsXmOr/5Oz2QD1f/RMzuBFjYVizaRuLadmzYCRIxV/Wd0Pk8mBw2RECvIw1u/gb86QXMfzxtOFsXZCtae6QQr/xV87/yd3dwoKq1tdbIIc+T9mzHGROydNq2Fdk1Dr+u1Tz1BRURGSk5O970hE5iF3sUE1ixdK+as+Kgpo1Ei1JRhc83+6Ox5zzf+pZ/0vb4GOWXNc5F7lXi/DuhQYX+bhN27cWDz++OPi9Glj1aDgqvVEKpC72KAaxQulFl388ktF6wYdLjgiXrlto7ixxfciDBVOL+1X/R9vNZrshRh9rZ8UDKSucv/OO9KOJ/U7lJen5FkFLbWu3z4Nk3377bd48MEHUVRUhKeffhpTp05VLkqTEYfJiFQi97CL0sM4WVnAhAne95N5xpj7/J+LXPN/fDjn+ob9AOD994EHH1S3dIGeSB3WatkSWL7cey+Z2sO6JqPW9duvnKG33noLc+fORYsWLfD8889joM7HQRkMEZFbKuZ7KJ7/A0jLh4mOBk6c8H6sYM1x8Ra82PmSF+ZmSrjPxyC31Lp+N/C+i6tJkyZh//79GDVqFEaMGIGbb74ZBw4ckLttRETKsi9kavEQeFgsQHKy3zPGyo+U44P0zbgj5RvENC7H/83ujhd2DkBhdStE4BxGxnyP1yfl48gPJ/Bt+VV49NOB6DDycv8LIUrJh5ESCAHBm+Niz0nzxh7UzJplC6DqY58SnpjovD0piYGQQfg9tV4IgaFDh6K8vBwvvvgiPv30U0yfPh3z589H06ZN5WwjEZEyFEjW9q3+Ty+ZTuQCOQMYjZdHUJQ9eLnnnvqDQ1+SqetOCTfL7Lwg4VMwtHz5chQUFKCgoAB79+5FSEgIunTpgunTp6Nbt274z3/+g44dO2LNmjXo2bOnUm0mIvJOar6Rp0JvEmeMSa3/M3pyc1x3d2dl6//IEcCoXD9JM2lpwLlz0qqMSw0ydbImHfnOp5yh5ORk9O7d23Hr2bOny4KoCxcuRGZmJnbv3i17Y/3FnCEik/Fn2rgPydrVFdXIX7YTPy3/Gr8UhuCHmquQj/6oQYgj/2d035NInaHy+l9S82HqY7GYZ2iHNYJ0T9cJ1PU5duwYEhISYPU2xqoiBkNEJuJtNpWfF3p7/Z/ctVZYCn/F05iLZFwMtn5vEIMf+s9E5xfvRlyXmEDOIDDZ2cAtt/j33Kgo4PXXzREIAZwJZgCGDYaEENi0aRMGDBgg52EDwmCIKABGqlIsc3XhI9tLkPvMfuSss+f/hONmZONDjAUgnGeg6GXmkNUKxMYCJ0/6/twvvwRuuEH+NukZZ4LpmmGDIT1iMETkJ6NVKQ5w2MNb/Z92oQewxXIdIqt+h9uBLz30JEh9D2rTQ7u15O57npysWJVxkk6Xy3EQkQL02vNixJW4/VgawbX+z5UArgQA1/yfxkWw3PC75+P6upSDEnydUSb3EidGxJlgpsdgiEhLeu15MepK3BJnU51tGIlP0jcjd60Vnxzs5LL+1+CYH5F6YyVGzm6HuC611v/6zxZp7dCyRo+vM8oCXWdNjWBejdfgTDBzU3SxD53g2mSkS3peH8qo6y1VV9vW4HL3vgKiBhDHGsSKCJzxff2v1auFaNlS/++Ll/dAALbzeOedwNdZ87YGmhzUeA3SLbWu3wyGiLRgv2B5ulhZLLZFSWVeEFQyqYtZZmZq0776XAgya+oEA1ZYhBUWcTNWC0CIKxoWitk98sSml3aK6kov77OnwNXXz626WtEFX13aW7fNcgbaagTzev6DgVSh1vXbr+U4iChAUpZNsOeeaEHqUIvOqhRXV1Qj72AK/tV6AUpEnNNjh5GExyKeRa9hUdjz0f+wv6I1ntk6EP1ndK1/IdT6hgzd8ZR7k51tm+k2aJBtcdhBg2z3s7Olnp50Si8P4W0YFZC2jIXWr0F0AXOGiLTgR6Kvquxrdnmrv6KDKsXlR8rx+bO7kbOmdv5PdzTAY7gBX2JIs+/RsXckevzzL/hndz+CN2+Bq119q5xrkYweSFKwtxwdX4J5f/Nw1HgNogsYDBFpQe89L/Wt2WWn4ewj1/o/rut/jU5riKGz+6FJzLDAXkxqQPr88+4DGi2T0f1JCpaS1K9GMK/3PxgoqDAYItKCEXpe7EMtd9/tWsAvKkrVptSu/5O7pSUKznQCcHEY7IqGB5Ha5SBSpyiw/pfUgLTukJSdkXo4pPZgqRHM6/0PBgoqDIaItKDAaumKcVfJ+NQpxWsN2ev/5L5ThpwfXev/XHvJT0h1Wv+rjSLtCDhwNUoPhy89WGoE80b4g4GCBhOoiZRmtdqqAmdl2f61J3wqneQaKPvF0R2FEljLj5Tjw9mbMfGyrxHTuBz/N7s7lu4YgMLqVojAOYyM+R6vT8rHkR9OYHP5VZjz2UB0GHm5sguh2gNX4GKgaiclcDVKD4cvPViBvidSqPEaRHaKzlXTCU6tD3JqTVf2h5QaKXptv0q1hoq3HRWvjt8obmzxvQhDhdv6P9mPbPZc/0ct7j7L5GTv07u91f1xNx1fi++EP+UU/H1PfKHGa5BuqXX95tpkZGx6reAM+LZ6uh6X5MjKsk0B9yYzExg/3vt+F85RFB9B4QErsjYlIOe7uAv5Pxe55v/o6C9/fz8nXxYD1eo77e+6bsFSgZp0SbXrt6Khlk6wZyhI6bkgmy9FFfVaYVfGnqHqrPfFucgYp+f9hiRxM1YLC6yi9yW7xKJheWLPRwdEjbVG8VPTpOdFSg+Hlt9pf3qwiBTGniEZsWcoCFmttoJ1nnIctF6FW+pf2QsWAPPnS+s9UkPtv8BjYoDJk4EjR+pPYPXwHtvr/xS/sx4zjz8BQDglKdYAsAD489l/oXn6VIVOyA0texPr6+HQw3falx4sIhWwZ0hG7BkKQnpfO0tq/kVUlLTeIzW467mIjr7YFgk9FXXzfxqgWvyGJGHV0znqtTdRL99p5uiQjqh1/ebUejImvU9Xljoz6NQpz48JoV79GU/5Tfb2RUU5T7G/sNK5GHMzdq/e77H+z4SQd5Fs1UmNHS2LH0qhl+90IJWriQyKwRAZk96nK0upkVI3wPBE6YuflCChUSPgyy+B48dhjY5B/u5LkfOP08i5taj++j+lNcAdEtqgRtCq9+KHevpO+1O5msjAGAyRMem9IJuUoor33w/Mm+f9WMeO2QIWpf4ylxIkHD6Mja/9jDe2dr2w/ldzx8PhqMCQmF0YPawCox5qj7guV1187oYiaW1Q4wKvl54XT/T+nSYKYiy6SPrmqWChEQqyeSuqOHeu7f/XbX9dDz6o3OrmgOSL//IPovBOYT/8IZoj2nISky//GtmPbMHJY1Z8dKwX7nrresR1iXF+kv0C7+kcLRYgOVmdC7yeel5qs3/H338fuOuui71xtenlO00UrBTNSNIJJlAblJQp50ZI9qxvGrc9odfTdGY1EnwlJu5OCMkSs3vkiU0v7RTVlT4kPHs6R7WTlvU4ddxT0ro9cV2v32kilXBqvYw4td6AjF6w0Bfupnq7o8DU6uqKanz9yg/oOmc4IqtOoAFc/3MgAFS3iEfokd9gaejnyLq7c0xOtvV0qDlVW09Tx719x+fPB9q2NeZ3mkgmal2/GQyR/mhVb0XLoMpqBV56yTYk5k3dCsA+stf/yVljdeT/3IxsfAhbkOAUEMkZJOglaNVDYKaHmkJEBqDW9ZsJ1KQ/Wsz60XpZj5AQIDZW2r5+JPge2V6C3Gf2I/fLRvjq9y44jz6Ox6ItJ9HsshgUXLUQ13y3DDhafPGJF6bQy/Ie6GWGkh6mjut9ZhuRyTAYIv1Re9aPp+GK4mLbdrWGTmRM8BU1ArvX/ILc5UeQs9m1/o/r+l/9APQDrA/ro/dGaVoHZnqf2UZkMgyGSH/UnPWjp0J8AU6trq6oxtfLdyPnrVLk/JjiVP8HAHpf8iNGX3cSqdOT0GHk5bA0aON6EK2DBLPQ68w2IpNiMETaqC9/RM16K3oarpBSm6jO1GrX/J9ujsfCUYHBMbuQ6q7+D2mLNYWIdMUQdYaKi4txxx13IDo6Go0bN0a3bt2wbds2rZtlDJ7q9GgpO9uWPDpoEDBhgu3f2nV01KwhpLfhCm+1idLScGR7CZZP2ISbYgrQIjEMf3muj8f6Px97qv9D2jJCnSwiE9H9bLI//vgD3bt3x6BBg3DfffchJiYG//vf/9CmTRtcfvnlko5h2tlkWicFe2qT1Cnzasz6kbq6fIAzuHxmD2I3bIAQAgcbXoms/ESs3RJ3If/notr5P33u7ITQCHb4GoYeZrYR6Rin1l8wZ84cfPPNN8jPz/f7GKYMhnwJOtTiz3Ripadj29vkbbhC5SnO1nc/QNW9MxFResyxrQhJeAAvYA3S3OT/eKliTfqll5IDRDrEYOiCjh07YtiwYTh8+DA2btyIxMRETJs2DXfddZfH51RWVqKystJxv6ysDMnJyeYJhvRaw0SvvTA6KcRnz/858p/1mHHsCQDCaRy7BoAFwJ/P/gvN06cq3h4iIq2pFQzpPmfo119/xauvvoq2bdvi888/x7333ov7778fb731lsfnLFq0CJGRkY5bcnKyii3WAV+SgtWkt/wcOwl5Oko5sr0Er91+Mf/n1ud64eZjy1E3EAJsP1aLxYLmS+fpI/eLiChI6D65oKamBj179sTChQsBAN27d8dPP/2EV199FZMmTXL7nIyMDKSnpzvu23uGTEOvQYcepxPbhygqK4FVq2zbjh9XbLhC1Aj8lHMAOa8Uu63/MyHkXSRbdTK7jYjIJHQfDMXHx6Njx45O2zp06IDVq1d7fE54eDjCw8OVbpp+6THoAKRNJ05MtAUoWVnK50/Ul2AuY6BRu/5P7o9t8Gt1WwBtHY875f+U1wB3SDgoi/EREclG98FQ3759sW/fPqdt+/fvR+vWrTVqkQHotYaJtzo6QgDnzgGDB1/crtTsN4WrTtvzf3LXWvFJYUecklr/Z4OXxVrtWIyPiEg2uk+gLigowHXXXYcFCxZg3Lhx+P7773HXXXdhxYoVuP322yUdw9SzyQD1koKlzopx1yMTHQ2cPOm6rxLtVSjB/Mj2Enz07H7krLOv/3WxdzLachIjL9uL1FtCMXT2VWgS06T+tulsdhsRkRZUu34LA/joo49E586dRXh4uGjfvr1YsWKFT88vLS0VAERpaalCLdSp1auFSEoSwnZZtd2Sk23b1XitpCTPr1VdLURenhCZmUJ8+aUQiYnOz619s1hs7a6ulqeteXmeX6v2LS+v3sPUWGvEj9n7xT8G54lrmux2efrloQdFeo88sfHFnaLqXJX09q1ebTtni8X1fbBYlPn8iIh0SK3rt+57huRgyp4hOzVqmARa00jtKfdZWbbK195kZgLjxzttcs3/cR6uvbbJbqT2PRF4/R8W4yMiUu36rfucIQqQ0gtvyrHQqVqz3+yB4Z490va/kJdTfqQcXzy/GznZUvJ/OgfWRru0NNt7xmJ82mAhRCJTYTBEgZFjoVM1Zr+562nxxGJBdUw8/vWawNpxBRfyf/o4Hq6d/zPkwc64JK6X/+2qTzCtIG+k4EKPy9gQkaIYDFFg5OjVUXr2m6dhPDfEhf8Zd+wlrHn34tDd5aGHkNq1EKmTm+O6uzohNKKff20xIyMFFwrPMiQifdJ9BWrSOTl6dZRcwbu+YTxcCH5qKUIybsGHWIM0XNtkNxYO3YCfcg7gl8pWeHbrQFw/sysXQvWFPbio2yNnDy6ys7VplzvehnwB25Avq38TBR0mUFNg5JwKrkTSsMTk7CfxOL5Gf0S0bIpRN1ZdyP+J8e81yUava+R5ote184hMjAnUZAzeCikC0nt1lEgaljiMN3xkQ6S/fh0uibvE/9eSSqn8Gb3l5ciRT6YmvS5jQ0SKYzCkBr1dpORmX+jUXV6Ir706MiQN117/6/DXFXhVwnOumX09oEYgpFT+jJTjqv09NFpwoddlbIhIcRwmU5qRkkcDpWHQZ6//k/v2n8jZleKo/xOK8ziHxgiBFR4r/oSEAGfPAmFhyjYy0HpMgRwXUP97aLRhJ1b/JtIdta7fDIaUpNTFT2kG6clyrf8T5XjMXv/n3q6bMXLdLO8H83ZBDvQ9USp/Rspxo6KAU6fU/x4aMbjQYhkbIvKIy3HISJPlOKqrXZenUHKJCbn4uqyGyoq3HRXLJ2wUw1t+L8JQ4dTMKMtJMemyfLH64c2i/Gi57QmZmdKW3sjM9PyicrwnMi0B4vdxtfoeGnFpETWXsSGieql1/WbOkFKMljwK6LLGSu38n9zNLfD9mc4A4hyPe63/E2geiFzviVL5M4Hm2yj9PZQzn0wtrP5NZDoMhpRitORROZbVkEl1RTW+eW03ct6y5/+0BdDW8bjr+l+tPR8skIKOcr4nSiXnypXMq+T30IjBRTBV/yYirxgMKcVoM1M07sk6XXIanz/7o9f1v0amt0N8Nx/W/wpk6r+c74lSVba9HVcqpb+HDC6ISMcYDClF6SUm5KZBT9bRnceQ+//2IXddI3z1+1WorLX+V5TlFEam7EHqLaEYmh7g+l/+DtXI+Z7IWY/Jl+MKAURHu0+gtu+jp+8hEZEGGAwpRamLnyeBznZSoSfLnv+T+2oxcr615//EOh5XdP0vf4Zq5H5PlMqf8XZcQL3vIRGRAXFqvdKUWGJCymv4WkNGoWnQrvk/zvk9rvk/HqsBqU+pqeFaVKBW43tIRCQz1hmSkeZrkylZt0fOWkYy1Vhxzf9xrf9zMf8ntp4j6YC39+S994CWLY2RGGyQ+lFERHYMhmSkejCk1kVHiUJ+fvYguOb/RDgec83/UWHZCzl5ek9uuw3IyjJHdXEiIg0wGJKRqsGQmstvKLXcgYRgzn3+z0Wu+T8GT0+r+56cOAGMG2e86uJERAbCVeuNSO2ihUrNAPMwDVpq/Z/R05LQcZSX+j++qB2IxMTYth0/ru5QT+33xN4jp4OaTEREFDgGQ3I5fx649151L5AqzADzVv/nhpY/InXoOYx6yMf6P1K562mrTYthKSNWFyciIo8YDMkhOxu45x7b0IknSlwgFapl5Fv9n2vqP1gg+VOeetpq02KpEKNVFycionoxGAqUlAt2bXJeIGWqZSRqBPZ89D/kvHxY3vo/geRP1bcUhlPjNRiWMlp1cSIiqhcTqAPhbTaXO74mM0vhxwyw2vk/ubtS8D8P9X8u5v/4WP8n0Cn/UpPDa1PivXVHqfpDRETkhAnURuAtd6Q2JZc9kFhd2Z7/k5ttxcdK5f9YrbZA5q67Asuf8qcHTa1hKbWrixMRkaIYDAXC14uvkhdIDzPAju48ho+e2YecL2TI//HGW7KznZT8KX+GmNQcllJqaQ0iIlIdg6FASL34tmwJLF+uygVSsfwfb3zNnQLqDyZ9WY1dq8VG/VnvjIiIdIfBUCCkXLBbtrT1HISFKdYM1/yfKwBc4Xhcsfo/dlKTneuqL5isbyiqNq2HpTz0yBERkXEwGAqElNyR5csVCYRUyf+RypfcKUB6T46noajaOCxFREQBYjAUKBVzR1TN//GpYT7mTgkhvSen7lCUVhWoiYgoaDEYkoNCuSNS839GT7wUfe/pLF/+j698TVxesMC3IJFDUUREpCAGQ3KR6YLtLf+nV5PdSL3uBFKnK5T/4w977pTUobK2bb3vQ0REpBIGQzpwuuQ0vnhuN3JWV+OTwg44qWX+D+D7Ehr23KlbbpF2fFZmJiIiHWEwpBHX/J/ejsc0zf/xdwmNtDTggw+A226zBVPuaDUFnoiIqB5cjkMl7vN/LnLN/9EgTg10CQ3AFhCNG+e63ZdjEBERQb3rN4MhBdnzf3Lf/hM5P7iu/+Wa/+Pj+l9y8rbOmi/rbfmxVhoREVFdXJvMoHSX/yOVt1pBUpbQsGNlZiIiMhAGQzLwO//H10RlJUmtFSR1P06HJyIigzBcMLRo0SI89thjeOCBB7B06VL1XrhW4CJi47DnzwTkLD/qf/0ffxOVlSJ1hpeeZoLpKZgkIiLDMlQwVFBQgBUrVqBLly7qvnB2NsT9D8BSbAtcLACaIQlb8QK+x0AAPtb/8ZSoXFxs265FkrG3ddb0NhNMb8EkEREZVgOtGyDV6dOncfvtt+P1119H8+bN1XnNktPYMuafELfcAlHsnE+TiGKsxi34vP8/ULytBN+d7ozHvhiITqlX1J8IXd+ipvZts2Z5np6uFHutIODizC87rRdDrcseTNbNcbIHk9nZ2rSLiIgMyTDB0PTp0zFixAgMHjzY676VlZUoKytzukl1dOcxrLhjE0bEFCAm3oLEnJch4PpGNYCAxWLB0IMrkNC1pfQT8SVRWW32ddYSE523JyXpZ0q8XoNJIiIyLEMMk7377rvYvn07CgoKJO2/aNEiLFiwQNK+9a3/NQAbkAyZZljZyZ2oLDe9zwSTc9YbERERDBAMFRUV4YEHHsAXX3yBiIgISc/JyMhAenq6435ZWRmSk5Md96srqvHt6z8h599/XKj/4379r4lX7QCek/CCvgQuRkhU1vNMML0Hk0REZDi6D4a2bduG48ePo0ePHo5tVqsVmzZtwrJly1BZWYmQOr0W4eHhCA8PdzlW7hMFWPdx+IX6P10v7l+r/s/I9CuRcPWF+j8bIC0Y8iVwMVqist4YIZgkIiJD0X0F6vLychw6dMhp29SpU9G+fXs8+uij6NzZe+FCewVLoBSArYKlvf7P6LQQDJt9FS6Ju8T1ifaqzN4CFylVmWuzJwADzsflkhXeKfWZEBGR7rAC9QVNmzZ1CXiaNGmC6OhoSYFQbW1CfsPNXU8gdVI99X9qs8+wGjvWdpF1F7j4M8PKnqjsbmq4uyUrWE/nIqU+EyIiMi3dB0Ny2nkiGZGX+rgEhq+Biy/HlZKozHo6rpT6TIiIyJR0P0wmB1m62bTonZFjFflgxh4zIqKgxlXrZaTVqvUBkXMVeSIiIgNS6/ptmKKLpqPn4oxERERBxFQ5Q4aiVj0dDjUREZHJMRjSKzXq6TA5m4iIiMNkumUvzujNiRP+HZ+LnRIREQFgMKRfISHAcxLKX6en+74oKRc7JSIicmAwpGctW3rfx58kaiZnExEROTAY0jOlkqi52CkREZEDgyE9UyqJmoudEhEROTAY0jN7ErW94nRdFguQnOz7CvdKHZeIiMiAGAzpmX1RUsA1cAlkUVKljktERGRA5guGrFZgwwYgK8v2r95nTNkXJU1MdN6elBTY2mRKHZeIiMhgzLU22dtvo1lGhjGLDCpVKZoVqImISKe4UKuMHG8mAJe3kivAExER6RIXalULiwwSERGZGoMhgEUGiYiITIzBUG0sMkhERGQ6DIZqY5FBIiIi0wnVugG6YLHYZpWxyCAREZHpmK9niEUGiYiIqBZzBUNvv80ig0REROTEXMNko0cD48ezyCARERE5mCsYAmyBz8CBWreCiIiIdMJcw2REREREdTAYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIyNQZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITM18C7WqwWoF8vOBo0eB+Higf3/bArFERESkOwyG5JadDTzwAHD48MVtSUnACy8AaWnatYuIiIjc4jCZnLKzgbFjnQMhACgutm3PztamXUREROSR7oOhRYsW4ZprrkHTpk0RExODMWPGYN++fVo3y5XVausREsL1Mfu2WbNs+xEREZFu6D4Y2rhxI6ZPn44tW7Zg3bp1qK6uxtChQ3HmzBmtm+YsP9+1R6g2IYCiItt+REREpBu6zxn67LPPnO6vXLkSMTEx2LZtG66//nqNWuXG0aPy7kdERESq0H0wVFdpaSkAICoqyuM+lZWVqKysdNwvKytTvF2Ij5d3PyIiIlKF7ofJahNCID09Hf369UPnzp097rdo0SJERkY6bsnJyco3rn9/26wxi8X94xYLkJxs24+IiIh0w1DB0IwZM7Br1y5kZWXVu19GRgZKS0sdt6KiIuUbFxJimz4PuAZE9vtLl7LeEBERkc4YJhiaOXMmcnNzkZeXh6SkpHr3DQ8PR7NmzZxuqkhLAz78EEhMdN6elGTbzjpDREREuqP7nCEhBGbOnIk1a9Zgw4YNSElJ0bpJ9UtLA1JTWYGaiIjIIHQfDE2fPh2ZmZnIyclB06ZNUVJSAgCIjIxEo0aNNG6dByEhwMCBWreCiIiIJLAI4a5KoH5YPCQkr1y5ElOmTJF0jLKyMkRGRqK0tFS9ITMiIiIKiFrXb933DOk8ViMiIiKDM0wCNREREZESGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIyNQZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIyNQZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIyNQZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZkagyEiIiIyNQZDREREZGoMhoiIiMjUGAwRERGRqTEYIiIiIlNjMERERESmxmCIiIiITI3BEBEREZmaYYKhV155BSkpKYiIiECPHj2Qn5+vdZOIiIgoCBgiGHrvvfcwa9YszJ07Fzt27ED//v0xfPhw/Pbbb1o3jYiIiAzOIoQQWjfCm2uvvRZXX301Xn31Vce2Dh06YMyYMVi0aJHX55eVlSEyMhKlpaVo1qyZkk0lIiIimah1/dZ9z9D58+exbds2DB061Gn70KFD8e2332rUKiIiIgoWoVo3wJsTJ07AarUiNjbWaXtsbCxKSkrcPqeyshKVlZWO+6WlpQBsESYREREZg/26rfQglu6DITuLxeJ0Xwjhss1u0aJFWLBggcv25ORkRdpGREREyjl58iQiIyMVO77ug6EWLVogJCTEpRfo+PHjLr1FdhkZGUhPT3fc//PPP9G6dWv89ttvir6ZelNWVobk5GQUFRWZKleK583zNgOeN8/bDEpLS9GqVStERUUp+jq6D4bCwsLQo0cPrFu3DjfffLNj+7p165Camur2OeHh4QgPD3fZHhkZaaovkV2zZs143ibC8zYXnre5mPW8GzRQNsVZ98EQAKSnp2PixIno2bMn+vTpgxUrVuC3337Dvffeq3XTiIiIyOAMEQzdeuutOHnyJJ588kkcPXoUnTt3xn//+1+0bt1a66YRERGRwRkiGAKAadOmYdq0aX49Nzw8HPPmzXM7dBbMeN48bzPgefO8zYDnrex5G6LoIhEREZFSdF90kYiIiEhJDIaIiIjI1BgMERERkakxGCIiIiJTM2Qw9MorryAlJQURERHo0aMH8vPz691/48aN6NGjByIiInDZZZdh+fLlLvusXr0aHTt2RHh4ODp27Ig1a9Yo1Xy/+XLe2dnZGDJkCFq2bIlmzZqhT58++Pzzz532WbVqFSwWi8utoqJC6VPxiS/nvWHDBrfn9PPPPzvtF2yf95QpU9yed6dOnRz7GOHz3rRpE0aNGoWEhARYLBasXbvW63OC4fft63kHy+/b1/MOlt+3r+cdLL/vRYsW4ZprrkHTpk0RExODMWPGYN++fV6fp8Zv3HDB0HvvvYdZs2Zh7ty52LFjB/r374/hw4fjt99+c7t/YWEhbrrpJvTv3x87duzAY489hvvvvx+rV6927LN582bceuutmDhxIn744QdMnDgR48aNw3fffafWaXnl63lv2rQJQ4YMwX//+19s27YNgwYNwqhRo7Bjxw6n/Zo1a4ajR4863SIiItQ4JUl8PW+7ffv2OZ1T27ZtHY8F4+f9wgsvOJ1vUVERoqKi8Je//MVpP71/3mfOnEHXrl2xbNkySfsHy+/b1/MOlt+3r+dtZ/Tft6/nHSy/740bN2L69OnYsmUL1q1bh+rqagwdOhRnzpzx+BzVfuPCYHr16iXuvfdep23t27cXc+bMcbv/I488Itq3b++07Z577hG9e/d23B83bpy48cYbnfYZNmyYuO2222RqdeB8PW93OnbsKBYsWOC4v3LlShEZGSlXExXh63nn5eUJAOKPP/7weEwzfN5r1qwRFotFHDx40LHNCJ93bQDEmjVr6t0nWH7ftUk5b3eM+PuuTcp5B8vvuzZ/Pu9g+H0LIcTx48cFALFx40aP+6j1GzdUz9D58+exbds2DB061Gn70KFD8e2337p9zubNm132HzZsGLZu3Yqqqqp69/F0TLX5c9511dTUoLy83GWxu9OnT6N169ZISkrCyJEjXf6y1FIg5929e3fEx8fjhhtuQF5entNjZvi833zzTQwePNilSrueP29/BMPvWw5G/H0Hwsi/bzkEy++7tLQUAOpdhFWt37ihgqETJ07AarW6rFYfGxvrsqq9XUlJidv9q6urceLEiXr38XRMtflz3nU9++yzOHPmDMaNG+fY1r59e6xatQq5ubnIyspCREQE+vbti19++UXW9vvLn/OOj4/HihUrsHr1amRnZ6Ndu3a44YYbsGnTJsc+wf55Hz16FJ9++inuvPNOp+16/7z9EQy/bzkY8fftj2D4fQcqWH7fQgikp6ejX79+6Ny5s8f91PqNG2Y5jtosFovTfSGEyzZv+9fd7usxteBvG7OysjB//nzk5OQgJibGsb13797o3bu3437fvn1x9dVX46WXXsKLL74oX8MD5Mt5t2vXDu3atXPc79OnD4qKivDMM8/g+uuv9+uYWvG3jatWrcKll16KMWPGOG03yuftq2D5ffvL6L9vXwTT79tfwfL7njFjBnbt2oWvv/7a675q/MYN1TPUokULhISEuER7x48fd4kK7eLi4tzuHxoaiujo6Hr38XRMtflz3nbvvfce/va3v+H999/H4MGD6923QYMGuOaaa3Tzl0Qg511b7969nc4pmD9vIQT+9a9/YeLEiQgLC6t3X7193v4Iht93IIz8+5aL0X7fgQiW3/fMmTORm5uLvLw8JCUl1buvWr9xQwVDYWFh6NGjB9atW+e0fd26dbjuuuvcPqdPnz4u+3/xxRfo2bMnGjZsWO8+no6pNn/OG7D9xThlyhRkZmZixIgRXl9HCIGdO3ciPj4+4DbLwd/zrmvHjh1O5xSsnzdgm61x4MAB/O1vf/P6Onr7vP0RDL9vfxn99y0Xo/2+A2H037cQAjNmzEB2djbWr1+PlJQUr89R7TcuOdVaJ959913RsGFD8eabb4o9e/aIWbNmiSZNmjiy6ufMmSMmTpzo2P/XX38VjRs3Fg8++KDYs2ePePPNN0XDhg3Fhx9+6Njnm2++ESEhIeKf//yn2Lt3r/jnP/8pQkNDxZYtW1Q/P098Pe/MzEwRGhoqXn75ZXH06FHH7c8//3TsM3/+fPHZZ5+J//3vf2LHjh1i6tSpIjQ0VHz33Xeqn58nvp73888/L9asWSP2798vdu/eLebMmSMAiNWrVzv2CcbP2+6OO+4Q1157rdtjGuHzLi8vFzt27BA7duwQAMRzzz0nduzYIQ4dOiSECN7ft6/nHSy/b1/PO1h+376et53Rf9/33XefiIyMFBs2bHD63p49e9axj1a/ccMFQ0II8fLLL4vWrVuLsLAwcfXVVztNy5s8ebIYMGCA0/4bNmwQ3bt3F2FhYaJNmzbi1VdfdTnmBx98INq1aycaNmwo2rdv7/Tj0gtfznvAgAECgMtt8uTJjn1mzZolWrVqJcLCwkTLli3F0KFDxbfffqviGUnjy3kvXrxYXH755SIiIkI0b95c9OvXT3zyyScuxwy2z1sIIf7880/RqFEjsWLFCrfHM8LnbZ867el7G6y/b1/PO1h+376ed7D8vv35ngfD79vdOQMQK1eudOyj1W/ccqGBRERERKZkqJwhIiIiIrkxGCIiIiJTYzBEREREpsZgiIiIiEyNwRARERGZGoMhIiIiMjUGQ0RERGRqDIaIiIjI1BgMERERkakxGCIiIiJTYzBERIaUlZWFiIgIFBcXO7bdeeed6NKlC0pLSzVsGREZDdcmIyJDEkKgW7du6N+/P5YtW4YFCxbgjTfewJYtW5CYmKh184jIQEK1bgARkT8sFguefvppjB07FgkJCXjhhReQn5/PQIiIfMaeISIytKuvvho//fQTvvjiCwwYMEDr5hCRATFniIgM6/PPP8fPP/8Mq9WK2NhYrZtDRAbFniEiMqTt27dj4MCBePnll/Huu++icePG+OCDD7RuFhEZEHOGiMhwDh48iBEjRmDOnDmYOHEiOnbsiGuuuQbbtm1Djx49tG4eERkMe4aIyFBOnTqFvn374vrrr8drr73m2J6amorKykp89tlnGraOiIyIwRARERGZGhOoiYiIyNQYDBEREZGpMRgiIiIiU2MwRERERKbGYIiIiIhMjcEQERERmRqDISIiIjI1BkNERERkagyGiIiIyNQYDBEREZGpMRgiIiIiU2MwRERERKb2/wEjQjuKqrbTWAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {
      "filenames": {
       "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_5_1.png"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "# Importing various packages\n",
    "from math import exp, sqrt\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "n = 100\n",
    "x = 2*np.random.rand(n,1)\n",
    "y = 4+3*x+np.random.randn(n,1)\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x]\n",
    "XT_X = X.T @ X\n",
    "theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(theta_linreg)\n",
    "# Hessian matrix\n",
    "H = (2.0/n)* XT_X\n",
    "EigValues, EigVectors = np.linalg.eig(H)\n",
    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
    "\n",
    "theta = np.random.randn(2,1)\n",
    "eta = 1.0/np.max(EigValues)\n",
    "Niterations = 1000\n",
    "\n",
    "\n",
    "for iter in range(Niterations):\n",
    "    gradients = 2.0/n*X.T @ ((X @ theta)-y)\n",
    "    theta -= eta*gradients\n",
    "print(\"theta from own gd\")\n",
    "print(theta)\n",
    "\n",
    "xnew = np.array([[0],[2]])\n",
    "Xnew = np.c_[np.ones((2,1)), xnew]\n",
    "ypredict = Xnew.dot(theta)\n",
    "ypredict2 = Xnew.dot(theta_linreg)\n",
    "\n",
    "n_epochs = 50\n",
    "M = 5   #size of each minibatch\n",
    "m = int(n/M) #number of minibatches\n",
    "t0, t1 = 5, 50\n",
    "\n",
    "def learning_schedule(t):\n",
    "    return t0/(t+t1)\n",
    "\n",
    "theta = np.random.randn(2,1)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "# Can you figure out a better way of setting up the contributions to each batch?\n",
    "    for i in range(m):\n",
    "        random_index = M*np.random.randint(m)\n",
    "        xi = X[random_index:random_index+M]\n",
    "        yi = y[random_index:random_index+M]\n",
    "        gradients = (2.0/M)* xi.T @ ((xi @ theta)-yi)\n",
    "        eta = learning_schedule(epoch*m+i)\n",
    "        theta = theta - eta*gradients\n",
    "print(\"theta from own sdg\")\n",
    "print(theta)\n",
    "\n",
    "plt.plot(xnew, ypredict, \"r-\")\n",
    "plt.plot(xnew, ypredict2, \"b-\")\n",
    "plt.plot(x, y ,'ro')\n",
    "plt.axis([0,2.0,0, 15.0])\n",
    "plt.xlabel(r'$x$')\n",
    "plt.ylabel(r'$y$')\n",
    "plt.title(r'Random numbers ')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8e7110b",
   "metadata": {
    "editable": true
   },
   "source": [
    "In the above code, we have use replacement in setting up the\n",
    "mini-batches. The discussion\n",
    "[here](https://sebastianraschka.com/faq/docs/sgd-methods.html) may be\n",
    "useful."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74a7859a",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Momentum based GD\n",
    "\n",
    "The stochastic gradient descent (SGD) is almost always used with a\n",
    "*momentum* or inertia term that serves as a memory of the direction we\n",
    "are moving in parameter space.  This is typically implemented as\n",
    "follows"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "619df8e8",
   "metadata": {
    "editable": true
   },
   "source": [
    "$$\n",
    "\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c759cfc4",
   "metadata": {
    "editable": true
   },
   "source": [
    "<!-- Equation labels as ordinary links -->\n",
    "<div id=\"_auto1\"></div>\n",
    "\n",
    "$$\n",
    "\\begin{equation} \n",
    "\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t},\n",
    "\\label{_auto1} \\tag{1}\n",
    "\\end{equation}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a4571f9",
   "metadata": {
    "editable": true
   },
   "source": [
    "where we have introduced a momentum parameter $\\gamma$, with\n",
    "$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n",
    "indicate the gradient is to be taken over a different mini-batch at\n",
    "each step. We call this algorithm gradient descent with momentum\n",
    "(GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a\n",
    "running average of recently encountered gradients and\n",
    "$(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory\n",
    "used in the averaging procedure. Consistent with this, when\n",
    "$\\gamma=0$, this just reduces down to ordinary SGD as discussed\n",
    "earlier. An equivalent way of writing the updates is"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bebfdd9",
   "metadata": {
    "editable": true
   },
   "source": [
    "$$\n",
    "\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9e635e8",
   "metadata": {
    "editable": true
   },
   "source": [
    "where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb544f8b",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Algorithms and codes for Adagrad, RMSprop and Adam\n",
    "\n",
    "The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n",
    "\n",
    "The codes which implement these algorithms are discussed after our presentation of automatic differentiation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d088ad85",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Practical tips\n",
    "\n",
    "* **Randomize the data when making mini-batches**. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented.\n",
    "\n",
    "* **Transform your inputs**. Learning becomes difficult when our landscape has a mixture of steep and flat directions. One simple trick for minimizing these situations is to standardize the data by subtracting the mean and normalizing the variance of input variables. Whenever possible, also decorrelate the inputs. To understand why this is helpful, consider the case of linear regression. It is easy to show that for the squared error cost function, the Hessian of the cost function is just the correlation matrix between the inputs. Thus, by standardizing the inputs, we are ensuring that the landscape looks homogeneous in all directions in parameter space. Since most deep networks can be viewed as linear transformations followed by a non-linearity at each layer, we expect this intuition to hold beyond the linear case.\n",
    "\n",
    "* **Monitor the out-of-sample performance.** Always monitor the performance of your model on a validation set (a small portion of the training data that is held out of the training process to serve as a proxy for the test set. If the validation error starts increasing, then the model is beginning to overfit. Terminate the learning process. This *early stopping* significantly improves performance in many settings.\n",
    "\n",
    "* **Adaptive optimization methods don't always have good generalization.** Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications.\n",
    "\n",
    "Geron's text, see chapter 11, has several interesting discussions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "937063d3",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Using Automatic differentation with OLS\n",
    "\n",
    "We conclude the part on optmization by showing how we can make codes\n",
    "for linear regression and logistic regression using **autograd**. The\n",
    "first example shows results with ordinary leats squares."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e08215a6",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[3.81651921]\n",
      " [3.10399758]]\n",
      "Eigenvalues of Hessian Matrix:[0.30971881 4.52950417]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theta from own gd\n",
      "[[3.81651921]\n",
      " [3.10399758]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABX90lEQVR4nO3deXwTZf4H8E96t1AKpfQuUJCjtNwgILdyiBzBiigqKh6rgigCrrC6Cq7IiqKgeKz+WFhlCx4UUkFRlJZDwAUBKTdIgXKUcrZQ6JU+vz9i0qZNmqPJzCTzeb9eeUFmJpNnkpnON8/xfTRCCAEiIiIilfKRuwBEREREcmIwRERERKrGYIiIiIhUjcEQERERqRqDISIiIlI1BkNERESkagyGiIiISNUYDBEREZGqMRgiIiIiVWMwRKRAS5cuhUajMT38/PwQExOD+++/H0ePHpWtXLNmzYJGo5Ht/T3BiRMnoNFo8M4778hdFCKyk5/cBSAi65YsWYK2bduiuLgYv/zyC+bMmYPMzEwcOnQIjRo1krt4RERegcEQkYKlpKSgW7duAIABAwZAr9fjtddew+rVqzFhwgSZS0dyKisrM9UaElHdsJmMyIMYA6Pz58+blhUXF2PatGno1KkTwsLCEB4ejl69ekGn09V4vUajwbPPPosvvvgCSUlJCAkJQceOHbFmzZoa265duxadOnVCYGAgEhMTrTb7FBcXY+bMmUhMTERAQADi4uIwadIkXL161Wy75s2bY8SIEVizZg06d+6M4OBgJCUlmd576dKlSEpKQr169XDrrbdi586dNj8PY3NiZmYmnnnmGURERKBx48ZITU3F2bNnaxz7rFmzauyjefPmePTRR2vsc8OGDXjyySfRuHFjNGjQAA8//DCKioqQl5eHsWPHomHDhoiJicH06dNRVlZWY78VFRWYM2cOmjZtiqCgIHTr1g0///xzje2OHj2KBx54AJGRkQgMDERSUhI+/PBDs22ysrKg0WjwxRdfYNq0aYiLi0NgYCCOHTuGGzduYPr06UhMTERQUBDCw8PRrVs3LF++3ObnR0QG/ElB5EFycnIAAK1btzYtKykpweXLlzF9+nTExcWhtLQUP/30E1JTU7FkyRI8/PDDZvtYu3YtduzYgddffx3169fHvHnzcPfdd+Pw4cNo0aIFAODnn3+GVqtFr169sGLFCuj1esybN88sCAMAIQRGjx6Nn3/+GTNnzkTfvn2xd+9evPbaa9i2bRu2bduGwMBA0/a///47Zs6ciZdffhlhYWGYPXs2UlNTMXPmTPz888948803odFo8NJLL2HEiBHIyclBcHCwzc/liSeewPDhw5GWlobc3Fy8+OKLeOihh7BhwwanP+snnngCqampWLFiBXbv3o2//e1vKC8vx+HDh5Gamoq//OUv+Omnn/DWW28hNjYWU6dONXv9okWL0KxZMyxYsAAVFRWYN28ehg0bho0bN6JXr14AgAMHDuC2225D06ZNMX/+fERHR+OHH37Ac889h4sXL+K1114z2+fMmTPRq1cvfPLJJ/Dx8UFkZCSmTp2KL774Am+88QY6d+6MoqIi7Nu3D5cuXXL62IlURxCR4ixZskQAENu3bxdlZWXi2rVrYt26dSI6Olr069dPlJWVWX1teXm5KCsrE48//rjo3Lmz2ToAIioqShQWFpqW5eXlCR8fHzF37lzTsh49eojY2Fhx8+ZN07LCwkIRHh4uqv7ZWLdunQAg5s2bZ/Y+X375pQAgPv30U9OyZs2aieDgYHH69GnTsj179ggAIiYmRhQVFZmWr169WgAQGRkZdn1OEydONFs+b948AUCcO3fO7Nhfe+21Gvto1qyZeOSRR2rsc/LkyWbbjR49WgAQ7777rtnyTp06iS5dupie5+TkCABWP79BgwaZlg0dOlTEx8eLgoICs30+++yzIigoSFy+fFkIIURmZqYAIPr161ej/CkpKWL06NE1lhOR/dhMRqRgPXv2hL+/P0JDQ3HnnXeiUaNG0Ol0NfqJfP311+jduzfq168PPz8/+Pv7Y/HixTh48GCNfQ4cOBChoaGm51FRUYiMjMTJkycBAEVFRdixYwdSU1MRFBRk2i40NBQjR44025ex5qVqMxMA3HvvvahXr16NZqFOnTohLi7O9DwpKQmAoT9USEhIjeXGMtkyatQos+cdOnRw6PWWjBgxwuy5sUzDhw+vsdzS+1j7/DZt2gS9Xo/i4mL8/PPPuPvuuxESEoLy8nLT46677kJxcTG2b99uts977rmnxvvceuut+P777zFjxgxkZWXh5s2bTh8zkVoxGCJSsM8//xw7duzAhg0b8NRTT+HgwYMYN26c2Tbp6ekYO3Ys4uLisGzZMmzbtg07duzAY489huLi4hr7bNy4cY1lgYGBppvolStXUFFRgejo6BrbVV926dIl+Pn5oUmTJmbLNRoNoqOjazTVhIeHmz0PCAiodbml8ltS/ZiMTXN1CQwcKaulclr7/EpLS3H9+nVcunQJ5eXl+OCDD+Dv72/2uOuuuwAAFy9eNHt9TExMjX2+//77eOmll7B69WoMHDgQ4eHhGD16tKwpGIg8DfsMESlYUlKSqdP0wIEDodfr8X//93/45ptvMGbMGADAsmXLkJiYiC+//NIsB1BJSYlT79moUSNoNBrk5eXVWFd9WePGjVFeXo4LFy6YBURCCOTl5aF79+5OlcEdAgMDLX4m7upbY+3zCwgIQP369eHv7w9fX1+MHz8ekyZNsriPxMREs+eWcjzVq1cPs2fPxuzZs3H+/HlTLdHIkSNx6NAh1xwMkZdjzRCRB5k3bx4aNWqEV199FRUVFQAMN8iAgACzG2VeXp7F0WT2MI7mSk9PN6vxuHbtGr799luzbe+44w4AhoCsqpUrV6KoqMi0XgmaN2+OvXv3mi3bsGEDrl+/7pb3s/b59e3bF76+vggJCcHAgQOxe/dudOjQAd26davxsFSLV5uoqCg8+uijGDduHA4fPowbN264+rCIvBJrhog8SKNGjTBz5kz89a9/RVpaGh566CGMGDEC6enpmDhxIsaMGYPc3Fz84x//QExMjNNNJf/4xz9w5513YvDgwZg2bRr0ej3eeust1KtXD5cvXzZtN3jwYAwdOhQvvfQSCgsL0bt3b9Noss6dO2P8+PGuOvQ6Gz9+PP7+97/j1VdfRf/+/XHgwAEsWrQIYWFhbnk/X19fDB48GFOnTkVFRQXeeustFBYWYvbs2aZtFi5ciD59+qBv37545pln0Lx5c1y7dg3Hjh3Dt99+a9douB49emDEiBHo0KEDGjVqhIMHD+KLL75Ar169zPphEZF1DIaIPMzkyZOxaNEivP766xg3bhwmTJiA/Px8fPLJJ/j3v/+NFi1aYMaMGTh9+rTZjdcRgwcPxurVq/HKK6/gvvvuQ3R0NCZOnIibN2+a7VOj0WD16tWYNWsWlixZgjlz5iAiIgLjx4/Hm2++aTasXm4vvvgiCgsLsXTpUrzzzju49dZb8dVXX0Gr1brl/Z599lkUFxfjueeeQ35+PpKTk7F27Vr07t3btE27du2wa9cu/OMf/8Arr7yC/Px8NGzYEK1atTL1G7Ll9ttvR0ZGBt577z3cuHEDcXFxePjhh/Hyyy+75biIvJFGCCHkLgQRERGRXNhniIiIiFSNwRARERGpGoMhIiIiUjUGQ0RERKRqDIaIiIhI1RgMERERkaqpIs9QRUUFzp49i9DQUIvp7ImIiEh5hBC4du0aYmNj4ePjvvobVQRDZ8+eRUJCgtzFICIiIifk5uYiPj7ebftXRTAUGhoKwPBhNmjQQObSEBERkT0KCwuRkJBguo+7iyqCIWPTWIMGDRgMEREReRh3d3FhB2oiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNdmDoU2bNmHkyJGIjY2FRqPB6tWrrW771FNPQaPRYMGCBZKVj4iIiLyb7MFQUVEROnbsiEWLFtW63erVq/Hrr78iNjZWopIRERGRGvjJXYBhw4Zh2LBhtW5z5swZPPvss/jhhx8wfPhwiUpGREREaiB7zZAtFRUVGD9+PF588UUkJyfLXRwiIiLyMrLXDNny1ltvwc/PD88995zdrykpKUFJSYnpeWFhoTuKRkRERF5A0TVDv/32GxYuXIilS5dCo9HY/bq5c+ciLCzM9EhISHBjKYmIiMiTKToY2rx5M/Lz89G0aVP4+fnBz88PJ0+exLRp09C8eXOrr5s5cyYKCgpMj9zcXOkKTURERB5F0c1k48ePx6BBg8yWDR06FOPHj8eECROsvi4wMBCBgYHuLh4RERF5AdmDoevXr+PYsWOm5zk5OdizZw/Cw8PRtGlTNG7c2Gx7f39/REdHo02bNlIXlYiIiLyQ7MHQzp07MXDgQNPzqVOnAgAeeeQRLF26VKZSERERkVrIHgwNGDAAQgi7tz9x4oT7CkNERESqo+gO1ERERETuxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqskeDG3atAkjR45EbGwsNBoNVq9ebVpXVlaGl156Ce3bt0e9evUQGxuLhx9+GGfPnpWvwERERORVZA+GioqK0LFjRyxatKjGuhs3bmDXrl34+9//jl27diE9PR1HjhzBqFGjZCgpEREReSONEELIXQgjjUaDVatWYfTo0Va32bFjB2699VacPHkSTZs2tWu/hYWFCAsLQ0FBARo0aOCi0hIREZE7SXX/9nPbnt2koKAAGo0GDRs2tLpNSUkJSkpKTM8LCwslKBkRERF5ItmbyRxRXFyMGTNm4IEHHqg1Qpw7dy7CwsJMj4SEBAlLSURERJ7EY4KhsrIy3H///aioqMBHH31U67YzZ85EQUGB6ZGbmytRKYmIiMjTeEQzWVlZGcaOHYucnBxs2LDBZrthYGAgAgMDJSodEREReTLFB0PGQOjo0aPIzMxE48aN5S4SEREReRHZg6Hr16/j2LFjpuc5OTnYs2cPwsPDERsbizFjxmDXrl1Ys2YN9Ho98vLyAADh4eEICAiQq9hERETkJWQfWp+VlYWBAwfWWP7II49g1qxZSExMtPi6zMxMDBgwwK734NB6IiIiz6OaofUDBgxAbfGYgtIgERERkRfymNFkRERERO7AYIiIiIhUjcEQERERqRqDISIiIlI1BkNERESkagyGiIiISNUYDBEREZGqyZ5niIiIiLyQXg9s3gycOwfExAB9+wK+vnKXyiIGQ0RERORa6enA888Dp09XLouPBxYuBFJT5SuXFWwmIyIiItdJTwfGjDEPhADgzBnD8vR0ecpVCwZDRERE5Bp6vaFGyNJUWsZlU6YYtlMQBkNERETkGps316wRqkoIIDfXsJ2CMBgiIiIi1zh3zrXbSYQdqImIiKTkQaOsHBYT49rtJMKaISIiIqmkpwPNmwMDBwIPPGD4t3lzRXYqdkrfvoZRYxqN5fUaDZCQYNhOQRgMERERScEDR1k5zNfXMHweqBkQGZ8vWKC4mjAGQ0RERO7moaOsnJKaCnzzDRAXZ748Pt6wXIF5hthniIiIyN0cGWU1YIBkxXKb1FRAq/WYvlEMhoiIiNzNQ0dZ1Ymvr8cEdmwmIyIicjcPHWWlFqwZIiIicjfjKKszZyz3G9JoDOsVNsrKIR6cMoA1Q0RERO7moaOs7ObhKQMYDBEREUnBA0dZ2eXrr4F77vHolAEaISzV13mXwsJChIWFoaCgAA0aNJC7OEREpGYe3JxUwzffAPffbz0lgLH5LyfHqWOU6v7NPkNERETuYi3w8ZBRVrVKTwfuvbf2bTwkZQCDISIiIndITzckWqzafBQfb+g75KlNYkbGJJL2UnjKAPYZIiIicjVvn3rDVhLJ6hSeMoDBEBERkSupYeoNR2p6FDgxa3UMhoiIiFzJkak3PJUjNT0ekDKAwRAREZErqWHqDWMSyeo5k6ry9QW++soj+kcxGCIiInIlNUy9UVsSSaMVK2yPNjPS64GsLGD5csO/EjchMhgiIiKqq6o3c72+9loTjcYj+tHYZC2JZEICsHKloaO4PRSQvZpD64mIiOrC0hD6xo0NfYM0GvOO1N4w9UZVqamAVut8EknjqLvqnc2No+4+/9z1ZbaAGaiJiIicZe1mbgyCGjcGLl2qXJ6QYAiEPKAfjdvp9YYaIGudzTUaFMbGIuzMGWagJiIiUiRbQ+g1GiA4GPjpJyA/3/On3nCWtSzc9oy6O3NGkiIyGCIiInKGPTfz06cNN/5x46Qrl5LUloW7pES+clXDDtRERETOUMMQ+rqwlYX76FF5ymUBgyEiIiJnqGEIvbPsycL92We2R91VH6nmJrIHQ5s2bcLIkSMRGxsLjUaD1atXm60XQmDWrFmIjY1FcHAwBgwYgP3798tTWCIiIiNbiQe9ZQi9M+xtQnzyScPz6p+h8fk//+me8lUjezBUVFSEjh07YtGiRRbXz5s3D++++y4WLVqEHTt2IDo6GoMHD8a1a9ckLikREVEVtSUelHMIvcwJDAHY3zTYqpXlXEXx8Yblo0a5vmwWyN6BetiwYRg2bJjFdUIILFiwAC+//DJS/xyG+J///AdRUVFIS0vDU089JWVRiYiIzBkTD1rqJCzHEPraOixLWRZHmhAHDLCYq6hCaLDz0x1uLaaRovIMaTQarFq1CqNHjwYAHD9+HC1btsSuXbvQuXNn03ZarRYNGzbEf/7zH4v7KSkpQUmVXuqFhYVISEhgniEiInIPa8PHpVRbziPAELRJFRAZcwidOWO535BGYwjScnLMPqfiq8XYsDAbuhU38e2R1jhXEQLA/XkCZa8Zqk1eXh4AICoqymx5VFQUTp48afV1c+fOxezZs91aNiIiIhNfX0MNh9SMQdiZM8CUKbXnPJoyxVADI0WQZmxCHDPGZhbuy39cwdq390O3xhc/nEnBdXQ3bVofp3Hd/aWVv8+QPTTV2mKFEDWWVTVz5kwUFBSYHrm5ue4uIhERkbSqzun10EPAxYvWtxUCyM01BE5SsTZ3WXw8zr/+CRZ80RgDG+1G5C2hePhffbDyTC9cRyjifM7hmeRNWPfGThzPDZSkqIquGYqOjgZgqCGKqdL+mJ+fX6O2qKrAwEAEBkrzARIREUnOWpOYLVLnPPpz7jKxcRNyvt6JzC3++ODoUPz+9ySzzdoHHYG221lon4xE14eSoPEx3PMLCwslKaaig6HExERER0dj/fr1pj5DpaWl2LhxI9566y2ZS0dERCSD2nL42CJhzqOSwhJkfZAN3fIbyDjYFmcqBprW+aIcfRtmQzugEKOeT0SLAa0BtJasbNXJHgxdv34dx44dMz3PycnBnj17EB4ejqZNm2LKlCl488030apVK7Rq1QpvvvkmQkJC8MADD8hYaiIiIpnYyuFjibHDsptzHl3JuYrv3t4P3bc+WHc6GdfQzbSuHq7jzrhsaEfoMfzFZIS37FzLnqQlezC0c+dODBxYGS1OnToVAPDII49g6dKl+Otf/4qbN29i4sSJuHLlCnr06IEff/wRoaGhchWZiIhIPo42dbk559HJX05D9+4f0GWGYtOV9ihHb9O6aJ/zGNX6MLT3B+P259sjqGEvl7+/KyhqaL27FBYWIizM/UPziIiI3C4ry9Bp2l4JCS7NeSQqBHYvPwTdp+eh+18Mfi9uY7Y+OfAoRnU5A+3jEej+SDv4+Dk/Vkuq+7fsNUNERETkAOM0ILXl8ImIAN57zzCSywU5j0qvl2Ljomzo0q4j48AtyNUnATB0gvaBHn3CsqHtfxWjnkvELXe0AtDK8EJjNmw58y/ZgcEQERGRJ7Enh88nn9S5JqjgVAG+f2c/dDrgu1PJKERX07oQFGFoTDa0w8sxfHoSItp0qrkDpWTDtgObyYiIqO6UkIFZbSwFG3VsEsv99Swy5h+F7uf6yLrcHmUIMK2L8snHyFaHoB0bhDumtEdweHDtZXNBNmyp7t8MhoiIyDp7ghxnawAYQNVdHT9DUSHw+9dHoPvkHDJ+jcKum+b5f9oG/AFt51xoH2uMHo8l29f/xzgVh7URb1am4rCEwZALMRgiInKCPUGOszUAHtSE4m3KbpRh04fZ0C27hoz9LXFSH29ap0EFbgvdB23fy9A+1wythyY6/gb2dvDOzLQ5hQk7UBMRkXysBTlnzhiWf/ONYZ4ra8n/apsPy559KyUg8pLaq8LThVg3fx90qwS+O5WMq6KLaV0wbmBIdDa0w0ox4sUkNEnqULc3s3fov9TZsGvBYIiIiMzVluG4apATFlZ78r+q82EZawDs3bc7JxS1N8Dx8NqrMzvPGfr/rA9B5qX2KMVtpnVNNBcw8pZD0N4bgEEvtEdIRA/XvbG9Wa4lzIZtC4MhIiIyZyvDsTHIycqyb39VawDs3XfVAMqV7A1wPKn26k+iQmDfqqPQfXwWum2R2HmjHYDKgKO1fw60nU5COyEcPR9Phm+Am7JR2zP0X4Js2I5gMEREROZc3XxRtQZAziYUewMcJdRe2am8uBybP8pGxrJC6LITkVNeOceXBhXoWX8/tH0uQTu5Kdre1QKAE32AHGXP0H83ZcN2FoMhIiIyZ2/zxYABwNKljtUAyNWE4kiAI3ftlQ3Xzl7DD/P3QbdKj7UnknFFVM7xFYSbGBSZDe2dJRj5YltEpbSXvHwADEHlN99YroVzYTZsV2EwRESkdtX70Nx2m33NHAMGOF4DIFcTiiMBjgI7AJ/bcx4Zbx+Gbn0wfr7QAaWonOOrseYSRrQ4CO09fhgypR3qHb5hKNtFPaC30B/KkU7hznQgN76mpMQQLANAfr5z7yUVoQIFBQUCgCgoKJC7KEREyrJypRDx8UIYwgHDIz5eiBdfFEKjMTyqrjMuW7my9n0kJJhvU/097d23q6Slmb+XtUdamhCZmfZtm5np+nL+qUJfIfatPirmDM4Ut9bLrvHWt/jniGldM8WmD/aI8pJyw4usfZe2vqvq2zizbV1eU8vrCr74QpL7N4MhIiK1MgYl1e+0xqDkxRftD3LKyw3BgTGYKC+3/d6OBFB15UiAU15uKJulz8b4+SQk2D5GB5XdLBMb398jpnbNFC39TtR42x71ssWbQzLFft1RUaGvMH+xre9y5Ur7tnFkf9U58xobrysAJLl/M+kiEZEa2Zsl+NgxYOtW9+TZkTKHj/F4bTXPGbMiGztbA5ab/1w0mqwovwg/zs+GbmU51hxPwiXR2LQuEMW4o0k2tENuYuT0NojpFFX7sdX2XcbFGY7jzBnr2xiPH3A8g7SzWadtvK4QQBjApItEROQG9vah2brVfZ2EfX2l64Ds6AgnN3YAPr/vAr59+xB06wLxU357FKOnaV245jKGJx6ANtUXQ6e1R/3o7rZ3aM93Wdt64zbGPlOA4x3Ine10but1EmEwRESkRgrsJOx2jgY4qamVo8vqWHt16Lvj0H1wCrotjbH9ejIEKjsHJ/qdgrZ9DrQPh6HP0ynwC+rj2M5d+R05sq+q2zp7Pink/GIwRESkRh6YJdglbAU4lprunKi90pfqse3/9kO35DIyfm+GI2UtALQwre8WcgDaXvnQPhOLlLtbQePT1PljcuV35Mi+qm7r7PmkkPOLfYaIiNTI0T40alDH6TduXLyB9e9mQ/dNKdYca4sLoolpXQBKcHvEXowadBOjprVCXDcXBgH2fJfGPkNnz9r+vgHHzw1nzycbr5Oqz5CP2/ZMRETKZexDA1T2mTFSaJZgtzJ2mK7ef8WYnTo93eLL8vdfwL8nbIY25ldENAFGz+2BJUf74pIIx3CswaKIWcgc8yEunLiB7y90xzPL+7k2EALs+y4XLgTef7/2bYzftzPnhrPnkz2vk4Jbx6opBIfWExFZIfUQdyUyDqW3Nty+2lD6w+uOi3l3ZYreob8LDfRmmzbzzRX/1+x1cTMs0vFcO3Vlz3fpyPftzLnh7Plk5XVS5RliMxkRUV1JOUTcHTy9/HWVlQUMHGhzs0+T3sN7f4zEodKWZsu7BB+Etud5aJ+OQQeffdCMvbdmk4+Lh+RbZc93KVUGakfPJwuvKywqkuT+zWCIiKgu6tjPhBRg+XLggQdsbjYOaViBcfBHKQaEZ0N7x3WMmtYKCT1iDRs4m2uHrJLq/s3RZEREzrJ3FnRSNjtHNCVFXsLy+7Zi2PRkhDXtWnMDe3PtZGUZgiG11sQpEIMhIiJnODILOm90inXs55PIeE+PhzRRiBD58EHN71MAQFw8Xj35TO3fpb05c8aOBS5frnzOmkTZcTQZEZEzHMm4qzZ6vaH2Y/lyw796vdwlMqkor8Cvi/fhb7dlITnoGFoNaoZpa+/A0+Ijw3rUHNGk0WigeX+h7aDW3pw5VQMhwOaINXI/1gwRETlDjRmc7aHAPlTFV4uxYWE2dCtu4tsjrXGuIsW0zg9l6N8oGwNvb4yLXf6FyI9fd376jb59Ddtby7VjDWsSZcdgiIjIGWrN4FwbBfWhuvzHFax9ez90a3yx7kx7FKFyjq9QFGJYwj5oRwoMm56MRold/lzTH3jpMedH1tU2/5kt1ubuIklwNBkRkTPkyOCs5CHwChhJdTzrFHQLjkOX1RBbClKgr/J7P87nHEYlHYV2XAgGTG6PwAaBbikDAMu1Y40bA5cu2X5tWhowbpxj76fk86KOOJqMiEjJHJ0Fva7c1fzkqhups7OW10FFeQV+++8h6D7Lh25nHPaVtAJQOcdX+6Aj0HY7C+2Tkej6UBI0Pm6opbP0+Vma/0yvBwYNsr0/R2sSFdgs6ZHcmtJRIZiBmojcRooMzitXGrIgW8qMrNE4/16Wyu5spuS0NOsZnKs+0tKcK+ufiguKxff/2CGebrdRxPqcNdu1L8rEgIa7xHujs8Txjafq9D52ceTzM2a5tvQ9Wshybff7u+O8UBCp7t9sJiMi7yZFE4I738NdzU/W+vc4mynZzizOyMx0uGboSs5VfPf2fui+9cG608koQj30xWbE4ByuIgwNYhtg5CgN7preDuEtGzm0b6c58/kZXwNYrkl05DNXQLOkFCS7f7s11FII1gwRqZQraz7kkplpX41LZqb9+3RwLi6H9umimo8TW3LFwtQscXuj34QfSk27uRsrxRnEyPud1uXzc1VNojvOCwWS6v7NPENE5J2cnIVccdwxhN8dOZKcnbXc+JYVArv+exCv9c9Cp+DDaN4nHs+n98eGK11QDn8kBx7FsjavYyXGIAbVjlXq77Qun19qKnDihKGGLC3N8G9OjuP9e5jawaXYgZqIvI83ZYd2xxB+d91IU1MNTT2WOvRayNVTer0UGxdlQ5d2HRkHbkGuPglAEgDAB3r0CcvG6H6XMLbfecRd2Q988AFgIUO05N9pXT8/X9+6dyJnageXYjBERN5HhpFNbmMrkZ+xb0jfvvbv0503Uksjqar0oSo4VWDo/6MDvs9NRiEq5/gKQRGGxmRDO7wcw6cnIWL/ceAvfwG+tWNIupTfqRICEXecFyrGYIiIvI83NSG4Ywi/u2+k1Wo+cn89C907R6HbUB9ZlzugHLeZ1kX55GNkq0PQjg3CHVPaIzi8p2FFejpwzz2Ov7cU36kSAhGpUzt4OfYZIiLvo4Rf7q5kbH6KizNfHh/vXFbnOvbvsUVUCOz58jBmD8xCl5CDaNozFpO/6Y+fLndFOfyRFPAHZvTMwrbP9uFsSQQ+O9QPI16/FcHhwYYdGJs5nSHFd+rmz89urj4vVIxD64nI+8iRHVoKrh7CbylhX0KC/XNxVVF2owybPsyGbtk1ZOxviZP6eNM6DSrQu0E2tH2vQPt8c7Qa3Lz2ndk7TL8qOb5TF35+dcIM1HWm+GCovLwcs2bNwn//+1/k5eUhJiYGjz76KF555RX4+NhXscVgiEiFXJnTxZvV4UZaeLoQ6+bvg26VwHenknFVNDStC8YNDInOhnZYKUa8mIQmSRH2l2n5cuCBB+zfXs7v1IsDESXgdBx/euutt/DJJ5/gP//5D5KTk7Fz505MmDABYWFheN7ZalQi8n4OjmxSLQdHNp3ecQ4Z848g46d62HCpA8qq9P9pormAkbccgvbeAAx6oT1CIno4VyZHm7rk/E5dMTKMZKf4mqERI0YgKioKixcvNi275557EBISgi+++MKufbBmiEjFlPzLXcll+5OoEMhOPwrdx2eh2x6J3260M1vf2j8H2k4noZ0Qjp6PJ8M3wAXlt5Vd2Sg8HPjqK0MworDPjVyDNUN/6tOnDz755BMcOXIErVu3xu+//44tW7ZgwYIFcheNiDyBUn+5K3iCzfLicmz+KBu6LwqRsS8ROeWtAbQGYOj/0yt0H7R9LkM7uSnaDGsBING1BTB2ULY1muyzz4A77nDte5MqKT4Yeumll1BQUIC2bdvC19cXer0ec+bMwbhx46y+pqSkBCUlJabnhYWFUhSViMg+1ua1MmZSttX3xQ01StfOXsMP8/dBt0qPtSeScUV0Nq0Lwk0MjtoL7Z2lGDG9LaJSOtTpveySmgqsXGnIM3SpWp6hxo2BTz+VPWgk76H4YOjLL7/EsmXLkJaWhuTkZOzZswdTpkxBbGwsHnnkEYuvmTt3LmbPni1xSYmI7GArOzYAPPkkEBZmufnHhTVK5/acR8bbh6FbH4yfL3RAKXqZ1kVoLmJEy4PQjgnA4BdSUC/Syf4/dWFM4JiVZXgAhs+EzWLkYorvM5SQkIAZM2Zg0qRJpmVvvPEGli1bhkOHDll8jaWaoYSEBPYZIiL5OTJsvHqQU8eZ5kWFwIFv/4Duw9PQbY3A/4pSzNbf4n8C2g4noH20EW77S4pr+v8Q1QH7DP3pxo0bNYbQ+/r6oqKiwuprAgMDERgY6O6iERE5zpEMyVWbzbRap+ZbKy8ux9bP9kP3nyvQ/Z6IP8pvAXCLaX2Pevug7X0R2knxSBrREhqf5k4fGpGnUnwwNHLkSMyZMwdNmzZFcnIydu/ejXfffRePPfaY3EUjInKcI8PGqwY5YWF2z7dW1K47fngnG7qV5Vibk4RLoqNps0AU444m2dAOuYmR09sgplOK9X0SqYTig6EPPvgAf//73zFx4kTk5+cjNjYWTz31FF599VW5i0bkfTxgqLfHszWvVXXGIMfYZ8aGedrNeLWwJ0rQ07QsXHMZwxMPQJvqi6HT2qN+dHcnC0/knRQfDIWGhmLBggUcSk/kbgoe6u1Vaptg0wW+K+yLEgShhd9JaDvkYNT4hujzdAr8gvq49H3oT/wB4RU4USsRVXbMrd4MY+yzkp4uT7m8lbUJNmszYABEXDwENBZXV0CDPE00htyhR3b6URwraYp3fxuAAVM6wS9I8b97PVN6uiE55MCBhulDBg40PFf69aLXG2oaly83/KvXy10i2Sl+NJkrMAM1US1sZftV4qSm3vJr3HhTGjsWuHzZ4iZCo8HN+pGYHLUCFcdysBiPAwB8UPmn2/A/DTQrOd+ay1k71+o4sk82HlYDLNn9Wzjg1KlTjmyuGAUFBQKAKCgokLsoRPYrLxciM1OItDTDv+Xl7nmfzEwhDH/Sa39kZrrn/R21cqUQ8fHmZYuPNyz3VCtXCqHRGB5VjksPCD004m6sNC1+CJ+Li75NzI8/IcHx45fq/PJk1s61r76qubzqQ6MxfCdK+0yN55ml8mo0iryGpLp/OxQMhYSEiFdeeUVcv37dXeVxCwZD5HGkvOGnpdkXDKWluf69HeWBf8ztdfrlj0RBkHmQcxIJ4m6sFM18c8VzHbPEz+/sEqVFpXUPZLwxoHS12s41e64XJf2AEMJwjnhgACfV/duhZrKtW7fihRdeQG5uLubMmYMJEya4q8LKpdhMRh5F6up3e5MAZmbKO8eXJzbn1aKivALbF++H7t+XoNvTFIdLW8AHevTFZsTgHBoEliC+VwJGPROPDmNaQ+Njua+Qwzy1eUdK9k4Ua0taGlDL1FGS8pTrvBqp7t9O9Rn6/PPP8fLLLyMiIgLvvfceBijog7OEwRB5DDlu+Mb3tDbUWylBhpx/zF3UR+nm5Zv46b1s6L4qwbdH2yJfNDGt80cpBjbeC+2gIoya1hrx3R3IR2QvLwso3caRLOG1UVJgsXy5oZO3LUoK4CDd/dup0WQPP/wwjhw5gpEjR2L48OG4++67cezYMVeXjUh9Nm+2O7GeyxiHegOVtQNGxucLFsh/c7Q3c7MjGZ7tUccRQxcPX8LSJ7bg7tjtiGhcgVFv3IrFR/oiXzRBGAowrtkvWPHcVlw4eRM/XOyGiSv6uycQAuQ5vzxRXc8hjQZISDAEzUphb7JPR5KCehGnx1sKITBkyBBcu3YN77//Pr7//ntMmjQJs2bNQmhoqCvLSOQanjACSa4bvnGot6VRJgsWKKPZRI4/5k7OLn/s55PQLciBblMj/FKYggpU5vhJ8D0Dbbtj0D4Uin4TUxBQv7frymuLXOeXp3HkHKqeK0pJPyCqspXs01grqKQATkqOdDD6+OOPxWOPPSbat28v/Pz8RGBgoOjevbuYPHmyWLx4sbj99ttFfHy82LFjhxu6NzmPHahJ0g6jdencKvfILiWPMDJ2ALXWgdXVHUAd6HCqL9OLbZ9lixk9M0W7wKM1Nu0UfFC81j9T7Eo7KCr0Fa4pnzPkPr88hb3n2tdf1zxHnBnZJxUroxaVPABBkaPJ4uPjxZgxY8Q777wjtmzZIoqLi2tsM2fOHJGcnOyyAroCgyGVk3IEUl2DLqlv+I6SO1iS8o+5nYHD2wkLRLRPntliP5SKOxr9Jt6/J0uc2JLrujLVldLPLyWx91yT+5pwlKW/UQoO4BQZDNkjLy9P+Pj4uHq3dcJgSMWkHE7qqqBLqb/elDIcW6o/5namHLgfaQIQIhQFYmzCL+K/E7eIKyeuurYsrqTU80uJPCxwsJsHBXCKHFpvZ7MbNm3ahP79+7tyt3XC0WQqJtUIJFeP0rGUJTYhQb7+O+4Yjl2XPlxS9P+y89xZ0Pw9tPtLX/SflILABoGuLYO7KO38UjJP6GvoxRQ9tN7TMBhSMamGk7oj6FLKH2F3DMdW8JQAFeUV+O2/h5Dx6TlM2voQInHebOoLIwENkBAPjacOQ1fK+UVUC6nu35y9j7ybVCOQ3DFKx9dXGTlKHBmObU95nRyh5U4lhSXIfD8buuU3kHGoFc5WtAPQDvvxIb7BGFRAYx4QaTSG6VKVNmLIEa48vxhYkYdjMETeTarhpN6cw+PMGfu3y8qq/Yao1xtqhCx9F0IYvo8pUwCt1u030ys5V/Hd2/uh+9YH604n4xq6mdbVxzXcGbcP2pFRKGq/FKFzX1ZuygG5KbiWj8hebCYj72esiQDMb8KunH7AU7I4Oyo9HXj6aeDCBdvbRkQAFy9WPrd0Q5R5SoATW04j470/oMsMxcYrHaCv8nswxicPo9ocgfb+YNw+pYN5/x/WfFjGqT3IzdhMRuQqUiQUNGZxHjPGc5Kw2WLtRmdN1UAIsNzsJXHSP1EhsCvtEHSfnoduRwz2FrcBEG9anxx4FNquZ6B9ogm6jU+Cj1+05R0ppclSSRRUy0dUV6wZIvWQ4te9t4zScdVEldVrxCSoGSq9XoqsD7KhS7uOjIOtcFofa1rnAz36hmVDO+AqRj2XiJa3N3PqPRRFrlorD534kzwLa4aIXE2KX/epqYZfwp7epGKr07RRWBhQUGB9ffXO1W7qw3X1ZAG+f2c/dDrg+9xkFKKraV09XMfQ2H3QDi/H8BfboXGrTg7tW9Hk7K/DqT3IizAYInI1b2hSsfcG9uCDwEcf2b8/FzYnntp2BhnvHoNuQ31kXe6ActxmWhflk49RrQ9Be18w7pjSHkENe9a+M0/sEyT3qDxvHjRAqsNgiJzjiTcPsp+9N7BWrRzfn5N9uESFwO9fH4Huk3PQ/RqF3TeTAMSZ1icF/AFtl1xoH4/ArY+2g49fP/vK5omjoZTQX4cTf5IXYZ8hcpwn3jzIMfaOjjt2DGjZ0rlRdHYE1GU3yrBxUTZ0y64h40BLnNIbOj/7QI/+yEK/4B1I6hiELq+NRKs7Wzp+nJ46Gkop/XWkGKlJqibZ/dutk30oBOcmcyEpJz0ledk7h5WL57oqyC0QK577RYxrtkWE4arZLoNRJP7ZcK64HtKk7nOkSTlvnavZOW+aSEtzf1m8df4uUgSPnZtMiVgz5CL2TMsQEQG89x4QF8emM29g7+i4Oo6iO73jHDLmH4Hup3rIvNQBZQgwrYvUXMDIVoegHRuIIbf8gcAJD7qmJkcptSvOUFrZ9XpDmbKyDM8HDDA8eP1THXFuMhdiMOQi9v4BNmLTmXewt3+YA/3IRIVA9teHsOetH3B4fxl+Ke2OzeiLChi2bxNwHNpOp6B9rDF6TGgH3wBf18+RJtW8de6gtCSfbDonN+HQelIeR4fIyjjXFLmQvaPjbGxXXlyOzR9lQ/dFIYr3HsbLFf/Aw6i8eZ7XRGFrp4loN+chtBnWAkAL8x24eo40Tx4NJUeST2vBrtyj2ohcwEfuApAHcfSmYPzjOGWK4Q8pqc61s9fwzbRtGN9iCyJDruH2aZ1xas8lfFTxNOJgHthEIR9375mFNjf3WN6Zq/PaGEdDGYOH6jQaQ1OfUkdDGUflxcWZL4+Pd30Akp5uqIkaONBQmzZwoOG5cVSgtVFtAK9/8ghsJiP72aqar40r+y5wWL+ind2Vh2/nH4FufTB+vtABpaic4ysS53HQtz0a6S/AYghSW/OOO/rJeMNoKHdfD7WNuLP374AS+12RR2AzGSlPbVXztrgqCy37JiiOqBDYrzsG3UdnkLEtAv8rSgFQOcfXLf4noO1wAtpHG+G2NhfhO6SWSV9ra+pyR14bKeatcxVrQY87k3zaymdkL2ahJoVjMESOsXbzsMUV/S7YN0ExyovL8cu/9kH3+VXo9ibieHkrAJUJGHvU2wdt74vQTopH0oiW0Pg0N6xYvty+N7B083RXPxlPmEJFrh8B9k7LYosS+10RVcFmMnKO8VfqmTOGPgHVZyw3ctWoFlePJCKHXc+7jh/f3QfdynKszUnCJdHYtC4QxRgUuRfaocUYOb0tojtEWt6JK5q6vGUyXHvJmRjS3hF31vC6pDpiMxkpW9Wq+eBgy/0ujM+feKLu7+fqkURkl7y9+fj2nUPQ/RCEn/I7oASVc3yFay5jROIBaO/xw5CpKagffavtHbqiqcsTanJcRe5pNxyp0ZFqVBuRG3A0GdWdtVEtRq+9ZqjVSU93/j04Q7YkRIXAwTV/4J93ZqFXaDZiO0bgL1/0w9r8W1GCILTwO4kXumQha8EenL/RAP/5ow9S5/VE/ej69r2BsakLqDmKy5GbpzEYHzfOu5P7OfIjwB3sHXH31VfSjGojchPWDJFrGH+tz5ljCH6qq2u/Hk/OCaNw+lI9tn66Dxn/uQLd781wtKwlgMp5vrrX2w9trwvQToxDsvYWaHya1e0NPanTstzk/hFgbz+t1FTDQw21deSV2GeIXMed/XqUlnHXw924eAM/zs+G7ptSrPkjCRdFhGldAEpwe8ReaAffxKjprRHbJbqWPdUBUyTYppRpN9TWT4sUg9NxuFCdP0z+0baPu/9we0NOGFdy8LzM338B3759CLp1AVh/vgOKEWxa10hzBcOb74f2bl8MnZaC0NhQKY7A+7j6b4WSfgTw7yDJgB2olYJ5bezn7ip9a80rERHAgw8C4eGGP9hq+ANt53l5+Pvj0H1wCrot4dh2LQUClR2Tm/vlQptyHNrxDdDn6RT4h/SR8gi8jzv+Vsgx7UZtZeHgBPJSrBkCHJ9zR601EbZIVaVv/L50OmDZMvNh/d4SqNb2K7yW81IAOPyX+ViyuzN0e5ricKn5/F5dQw5A2zMf2mdi0T61FTQ+VjrGkmPc/beCzVSkUmwmc6FaP0xrv+beew944QXmtXGElFX63hyo1lbDoNXW2i+rAhqcRjwSkYMK+MIfpRjYeC+0g4owalprxHdXUQdzqZp1pMqBxWYqUiHJ+vwKD3D69Gnx4IMPivDwcBEcHCw6duwodu7caffrCwoKBABRUFBgvmLlSiE0GiEMt9TKh6Vl1h6Zma49WE9n/Eyrf4bGZStX1v09ysuFiI+3/p1oNEIkJBi28zS1nZMajRCzZ9t1Xr4e9b5Y8dwv4urJq3IfkTxWrqx5jsTHu+b8qy4zk38riNzE6v3bxRSfZ+jKlSvo3bs3/P398f333+PAgQOYP38+GjZsWLcdc84d95BiJm25c6+4i41zUgjg5hvz7drV39+LwH0Lb0NY0zAXF9IDGGsNq58jxvQOdcl3ZYncw9+JqM4U34H6rbfeQkJCApYsWWJa1rx587rvmHPuuI+7MwR7683HxjmpgUBwWaF9+1LreSlHxmbmwCLyeIqvGcrIyEC3bt1w7733IjIyEp07d8Znn31W62tKSkpQWFho9qihrjdKY+ZVR2bIVhN3Zgj21puPnefkTf9QCNjICKzW89LeWsMPPjDMu5WVZQig6sLeLM1q/U6IPIDig6Hjx4/j448/RqtWrfDDDz/g6aefxnPPPYfPP//c6mvmzp2LsLAw0yMhIaHmRo7OuWPpOefckYeX3XwuHb2Mz/+yBa9OvmLX9sGvTDccOs/Lmuz9kfPCC4YJSAcOrPtUMa6aYoSI5OPWHkku4O/vL3r16mW2bPLkyaJnz55WX1NcXCwKCgpMj9zc3JodsIydcK11ljZ2wv3qq5odMRMS3NMR05uVlxs6kKalGf6ta+dmKTpqu9EfmSfFu9pM0T9st/BFmQCE8EG5OIV4oUctHfibNBGipMRyB2Gel/Z3ZnbHOcPvhMjlpOpArfg+QzExMWjXrp3ZsqSkJKxcudLqawIDAxEYGFj7jjnnjnTckYzOw+a3qiivwG//PQTdZ/nQ7YzDvpJWAJqa1ncIOgxt93Mo7fwiNB9MAVDtnDS6cAFo2dLw2Z04wfOy+nDz224znAPW0jtY4qq+RO7uK0dEbqP4PEMPPPAAcnNzsbnKyKAXXngBv/76K7Zu3WrXPhzOM8RkZq7j7nxACs69UlJYgg0L9iLjy5vIONQKZysqm2Z9UY5+DbOhHViIUVNaILFflaZcS+dkVZ6US8md34+1IHvcOOCddwzPHf3z5u45vojIIcwz9Kf//e9/ws/PT8yZM0ccPXpU/Pe//xUhISFi2bJldu/DZjWbq5twyMCb8wFZcfn4FfHF01vEmLitoj4KzQ63PgrFmLit4ount4hLxy7XvqOSEiEiIjz7s3Nnrh9b+ZhefLH2c8/aIy2t7mUjIpeRqplM8TVDALBmzRrMnDkTR48eRWJiIqZOnYonn3zS7tdz1nqZKGXGbTc7seU0dO/+AV1mA2y62h76KhkrYn3OYVTbo9COC8HA59ojsIGN5lsjT//s3FkjaG/G52PHgK1bDbVS588bOk3botTPk0ilOFFrFSNGjMCIESPkLgY5ykvzAYkKgV1ph6D79Dx0O2Kwt7gNgHjT+pTAo9B2OwPtk5Ho+mBb+Pg5McTfkz87d+f6sXf4/NatlYGNXg/Mn297qhgPGYFIRK7lEcEQeSgvygdUer0UWR9kQ5d2HRkHW+G0PglAEgDAB3r0DcuGdsBVjHouES1vbwWgVd3e0JM/O0cyhDtTC+NMoKik2d+JSHEYDJH7GPMBeeiv8asnC/D9O/uh0wHf5yajEF1N6+rhOobG7oN2eDmGv9gOjVt1cu2be/Jn5+5aLWcDRQ8bgUhE0mEwRO7j6l/jEowcO7XtDHTzj0G3IRQbr7RHOW4zrYv2OY+RrQ9De18w7pjSHkENe7r0vc14ck2Gu2u16hIocvg7EVngER2o64odqGshxdB0V6QvcEeuIhj6/+z58jB0/8qD7n/R2HOzrdn6doHHoO1yGtrHI9D9kXbw8ZM4absnpn4wdnC2Fazk5Dh/rhk7aAOWA0VPSDtARDZJdf9mMKRmbgowLKpL0OXikUllN8qwcVE2dMuuIeNAS5zSV3Z+9oEevRvsg7bfFYx6rjlaDW5u937dRsG5lKySIljxxECRiBzCYMiFGAxZ8PXXwNixNZcr7Ze1vcOobdQyFJ4uxPdv74NOJ/DdyRQUIMy0Lhg3MDRmL7R3lWP4tLZokhTh4oNQKSmCFU8MFInIbgyGXEiRwZCcf8S/+Qa4/37rs3W7ohnDVeqQb+f0jnPImH8Eup/qIfNSB5QhwLQuUnMBI1sdgnZsIAa90B7B4cGuLTcZMFghojpgniFvJmXzlKX3vvfe2rep69BnV3JgZJKoEMhOPwrdx2eh2x6J3260A1DZSbdNwHFoO52C9rHG6DGhHXwDZBiJpbbgwNdX/nOIiMgGBkNSs9b/5cwZw3J3Nk8Zk+HZSwkJ/ewccbTo7+cx/+HTOFHeGkBrAIAGFegVug/aPpehndwUbYa1ANDCfWU1shbwuCoIVltARUTkZmwmk5KL+r84zd4mJyMlTE1gY2RSBTQ4jXgkIgcV8EUQbmJw1F5o7yzFiOltEZXSRNry2po8tK6dwOWsVSQikhj7DLmQYoIhueebWr4ceOAB+7ZNSFBGnyEASE+HuGcMAAFNlcUVfz57DP+G5paW0I4JwOAXUlAvsp4sxbRa62eLvUGwO+f7IiJSIPYZ8kZyzzflSJI7mRP6iQqB/bpj0H10BrptbRCPb7AQzyMBlTUiBf4RyHt0BhYvGg/fAJmDttrm47LFnj5a7p7vi4hIxRgMSUnu+aZsZe4FDDfS5ctlqWEoLy7HL//aB93nV6Hbm4jj5ZVzfO1EEs7Xa4FJ7bLQvz8QO6wjGvXvh0ZKufHbmo/LHrUFwe6e74uISMUYDElJ7vmmapviwWjFispkeRK4nncdP767D7qV5ViT0w6XRSfTukAUY1DkXmiHFmPk9LaI7tAJQCcre5KZK2rzaguC5a5VJCLyYgyGpKSE+aasTVYpYebevL35yJh3CBnrg/BTfgeUoHKOr3DNZYxIPADtPX4YMjUF9aNvdXt5XKIutXn2BMFy1yoSEXkxdqCWgxKmEZBweLaoEDi45g/oPjwN3S8R+LUoxWx9C7+T0HbIgfbhhuj9VAr8gjwwRrc1H5eRtSDYVudnKeb7IiJSGI4mcyHFBUOA1+eK0ZfqsfXTfdAtvQLd3uY4VtbcbP2t9fZBe9tFaCfFo93IltD4aCzvyJPYmo9r+nRDfyxng2BOTkpEKsNgyIUUGQx5oRsXb+DH+dnQfVOKNX8k4aKonOMrACW4o8leaAffxMhprRHbJVrGkrqRrVq/ugbB7qxV9PIAnYg8D4MhF2Iw5D7n913AmncOQbcuAOvPd0AxKuf4aqS5guHN90N7ty+GTktBaGyojCWVkLuDCnfsn8kciUiBGAy5kMcFQwr/hX7ou+PIWHQKui3h2HYtBQI+pnXN/XKhTTkO7fgG6PN0CvxD/GUsKdmFyRyJSKEYDLmQRwVDCvyFri/VY/vi/dAtuQzdnmY4UpZotr5ryAFoe+ZD+0ws2qe28o7+P2oh9xQxRES1YAZqNZJzEtdqbl6+ifXz90L3dSnWHGuLfNHBtM4fpbi98V6MGlSEUdNaI757OwDtJCkXuRiTORIRMRhSDAVMt3Dh4EWsefsgdN8H4Me89riJHqZ1YSjA8Gb7oL1bgzunpaBBfDe3lIEkxmSOREQMhhRDpl/oR9efgG7hCeg2N8LWwhRUoDLxX1Pf09Am/wHtQ6HoN6k9/EN6u+x97aLwvlNegckciYgYDCmGRL/QK8or8L+lB6BbfBG6XQk4WNoSQHPT+s7BB6HtcR7ap2PQ8d7W0PjE1+n9nKbAvlNeSe4pYoiIFIDBkFK48Rd68dVi/LwgG7ovb+LbI22QV1GZAdoPZRgQvhfa269j1NRb0LRXEoAkh9/DpRTUd8rrKWGKGCIimXE0mVKUlgIhIYamIWt8fYEbN4CAAJu7u3T0Mta+fQC6tX744WwKilDftK4BCjAsYT+0WmDY9GQ0bBbmiiNwDY5ukocSpoghIqqGo8nUZuvW2gMhwLB+61arfYb+2HASuoU50G1siC0F7VGBPqZ18b5nMSrpKLQP1MeAye0RUP82FxbehTi6SR6pqYbO+eyjRUQqxGBIKeztC3TmDJCVBZw7h4rIKOw8EQHdvy9D91sc9pe0AtDMtGnHoMMY1f0ctH+JQpcH2kLjE+uWorsURzfJx9eXASYRqRKDIaWwsy+QeH4KNJcuAgB8AMQgHgexEPsxAL4oR/9Ge6EdeA2jXmiJ5n3aAGjjvjK7A0c3ERGRxNhnSCmMfWWsjOoRVf71qbK8AhpoILB56By0/3giGiU2dH9Z3cnG58A+Q0RE6iHV/dvH9iYkCeOoHqByFM+fKmB4Xj0QAgAfCGg0GvQ78AkaNfWCiVBr+Rw4uomIiNyBwZBCiAqBndfbYnmbWcgTUWbrLiACGtTyZVXtVOwNUlMNw+fj4syXx8dzWD0REbkc+wzJqPR6KTIX7oVu+Q1kHGyFMxXtALwKH7yM/sjCgOD/IalLMPr11wBvTrG9Q2/qVMzRTUREJBEGQxK7erIA383bB923Gnyfm4JrqJzjqx6u4864bGhH6HHXtM5o3OoOw4qsLOBNO3Ze107FSpv+gqObiIhIAgyGJHBq2xno5h+DbkMoNl5pj3JUzvEV7XMeo1ofhvb+YNz+fHsENexVcwdSTJnA6S+IiEilGAy5gagQ2PPlYej+lQfd/6Kx52ZbAJX9X9oFHoO2y2loH49A90fawccvyvrOAPdPmcDpL4iISMU4tN5Fym6UYeOibOiWXUPGgZY4pa+c4NQHevRusA/aflegnZKIW+5oVsueauGOKRM4/QURESkUp+OwYu7cufjb3/6G559/HgsWLJC1LAWnCrBu/n7odALfnUxBAbqY1oWgCENisqG9qxzDp7VFk6SOdX9DV3cq1uuBDz7g9BdERKRqHhUM7dixA59++ik6dOggWxlyfz2LjPlHofu5PrIut0cZKuf4itRcwMhWh6AdG4hBL7RHcHhP1xfAVZ2KLdUy1cabRqoRERFV4THB0PXr1/Hggw/is88+wxtvvCHZ+4oKgb3fHIHuk3PQbY/CrptJACrn+GoTcBzaTqegfawxekxoB9+AOnRiloq1PkK14fQXRETkpTwmGJo0aRKGDx+OQYMG2QyGSkpKUFJSYnpeWFjo0HuV3SjD5o/3QfdFITL2t8CJ8so5vjSowG2h+6Dtexmjnm2KNsNaAGjh6OHIR6831AjZGwi5YqQaERGRgnlEMLRixQrs2rULO3bssGv7uXPnYvbs2Q69x7Wz17DunX3QrarA2pPJuCo6m9YF4wYGR2dDe2cpRkxvi8hk+Zrp6mzzZvubxtw1/YXS8hkREZGqKT4Yys3NxfPPP48ff/wRQUFBdr1m5syZmDp1qul5YWEhEhISamx3dlceMt45At36YGy42AGlqMzxE6G5iJG3HIR2TAAGT22PkIgedT8YJXCk7098fN1GqlnCfEZERKQwih9av3r1atx9993wrVJzoNfrodFo4OPjg5KSErN1lhiH5l29chW5mfnQfXQGum1NsKMo2Wy7Vv450HY8Ce2EcPR6Ihm+AV5YW5GVBQwcaHu7994DJk92bY2Ntb5Kxhoo5jMiIqIqpBpar/hg6Nq1azh58qTZsgkTJqBt27Z46aWXkJKSYnMfxg+zuW82Tugrt9egAj3q74e29yVon01A27taQOOjqWVPXsCYV8hWNmtX5xViPiMiInIQ8wz9KTQ0tEbAU69ePTRu3NiuQKiqE/qmCEQxBkfuxaihxRg5vS2iO7R3ZXGVz93ZrK2x1VeJ+YyIiEgmPnIXQErLnt+BS+f1+Pb8rXjy836I7hApd5HkkZpqaJKKizNfHh/vvqYqe/sqMZ8RERFJTPHNZK4gVTWbiaeMlpKynPb2VcrMZM0QEREBYDOZ5/Kk0VKuymZtj759DZ+Drb5KzGdEREQSU1UzmdsZR0tV7xtjnP09PV2ecimBsa8SUNk3yaiufZX0ekPN0/Llhn/1+joUlIiI1IbBkKvUltnZuGzKFHXfqN3RVyk93TBKbeBA4IEHDP82b67uwJOIiBzCPkOuwj4x9rOnr5I92zBvERGRV2OfIU/D0VL2s9VXyZ5+V7Zq4jQaQ02cVqvMzutERKQYbCZzFXtndefs77Wzt9+VI3mLiIiIasFgyFWMo6Wqdw420miAhASOlqqNI/2uWBNHREQuwmDIVdw5WsrI20dNOVLbw5o4IiJyEQZDruTOzM5qGDXlSG0Pa+KIiMhFGAy5WmoqcOKEYdRYWprh35ycugdCashf5EhtjxQ1cUREpAocWi83W0PI1TTbu/FYbWWprnqslkaeJSQYAiEOqyci8mhS3b9ZMyQne5q+1DRqypnaHnfUxBERkaowGJKLvU1fahs15Uy/K2PeonHjDP96eg0ZERFJikkXpWRsEjtzBnjhBfsSBqpx1FRqquHYbWWgJiIicgEGQ1Kx1LfFmqpNX2qd7d1WlmoiIiIXYTOZFKw1idly7pw8o6a8PZ8RERFRFQyG3K22rMq2GJu+3Jm/qDo15DMiIiKqgkPr3c3e2eyrsjZc3p6Z3OuCs8ATEZGCcNZ6Kbg7uAAcH+VVW9OXO/vRcBZ4IiJSKfU2k0nVHOToKC93NH3ZQ035jIiIiKpQZzAk5fQWtubQAoAmTYBly+RNGKi2fEZERER/Ul8wZKs5CDA0B7lqBJWt0WAaDfDJJ8CDD8qbMFCN+YyIiIigxmBIjuYgKUeDOYuzwBMRkUqprwO1XM1B1rIqA4YRZ3JnWjbWYI0ZYwh8qtaccRZ4IiLyYuoLhuRsDqo+GsxSVur4eENQIkdtkbEGy1KZOAs8ERF5KfXlGapXzzBqzNb0FtVz/LiaknP6SJFygIiIyAap8gypLxhq0KAyEAEsNwe5OxDR6w0BmbW+S1IFZERERAomVTCkvg7UgPwdmpnTh4iISDHU12fIyFqHZilqYpjTh4iISDHUGwwB7p3eojbM6UNERKQY6mwmkxtz+hARESkGgyE52MpKDTCnDxERkUQYDMlF7k7cREREBEDtfYbkJmcnbiIiIgLAYEh+cnXiJiIiIgBsJiMiIiKVYzBEREREqsZgiIiIiFRN8cHQ3Llz0b17d4SGhiIyMhKjR4/G4cOH5S4WEREReQnFB0MbN27EpEmTsH37dqxfvx7l5eUYMmQIioqK5C4aEREReQGPm7X+woULiIyMxMaNG9GvXz+7XiPVrLdERETkOlLdvz1uaH1BQQEAIDw83Oo2JSUlKCkpMT0vLCx0e7mIiIjIMym+mawqIQSmTp2KPn36ICUlxep2c+fORVhYmOmRkJAgYSmJiIjIk3hUM9mkSZOwdu1abNmyBfHx8Va3s1QzlJCQwGYyIiIiD8JmsmomT56MjIwMbNq0qdZACAACAwMRGBgoUcmIiIjIkyk+GBJCYPLkyVi1ahWysrKQmJgod5GIiIjIiyg+GJo0aRLS0tKg0+kQGhqKvLw8AEBYWBiCg4NlLh0RERF5OsX3GdJoNBaXL1myBI8++qhd++DQeiIiIs/DPkN/UnisRkRERB7Oo4bWExEREbkagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlXzmGDoo48+QmJiIoKCgtC1a1ds3rxZ7iIRERGRF/CIYOjLL7/ElClT8PLLL2P37t3o27cvhg0bhlOnTsldNCIiIvJwGiGEkLsQtvTo0QNdunTBxx9/bFqWlJSE0aNHY+7cuTZfX1hYiLCwMBQUFKBBgwbuLCoRERG5iFT3b8XXDJWWluK3337DkCFDzJYPGTIEW7dulalURERE5C385C6ALRcvXoRer0dUVJTZ8qioKOTl5Vl8TUlJCUpKSkzPCwoKABgiTCIiIvIMxvu2uxuxFB8MGWk0GrPnQogay4zmzp2L2bNn11iekJDglrIRERGR+1y6dAlhYWFu27/ig6GIiAj4+vrWqAXKz8+vUVtkNHPmTEydOtX0/OrVq2jWrBlOnTrl1g9TaQoLC5GQkIDc3FxV9ZXicfO41YDHzeNWg4KCAjRt2hTh4eFufR/FB0MBAQHo2rUr1q9fj7vvvtu0fP369dBqtRZfExgYiMDAwBrLw8LCVHUSGTVo0IDHrSI8bnXhcauLWo/bx8e9XZwVHwwBwNSpUzF+/Hh069YNvXr1wqeffopTp07h6aeflrtoRERE5OE8Ihi67777cOnSJbz++us4d+4cUlJS8N1336FZs2ZyF42IiIg8nEcEQwAwceJETJw40anXBgYG4rXXXrPYdObNeNw8bjXgcfO41YDH7d7j9oiki0RERETuoviki0RERETuxGCIiIiIVI3BEBEREakagyEiIiJSNY8Mhj766CMkJiYiKCgIXbt2xebNm2vdfuPGjejatSuCgoLQokULfPLJJzW2WblyJdq1a4fAwEC0a9cOq1atclfxnebIcaenp2Pw4MFo0qQJGjRogF69euGHH34w22bp0qXQaDQ1HsXFxe4+FIc4ctxZWVkWj+nQoUNm23nb9/3oo49aPO7k5GTTNp7wfW/atAkjR45EbGwsNBoNVq9ebfM13nB9O3rc3nJ9O3rc3nJ9O3rc3nJ9z507F927d0doaCgiIyMxevRoHD582ObrpLjGPS4Y+vLLLzFlyhS8/PLL2L17N/r27Ythw4bh1KlTFrfPycnBXXfdhb59+2L37t3429/+hueeew4rV640bbNt2zbcd999GD9+PH7//XeMHz8eY8eOxa+//irVYdnk6HFv2rQJgwcPxnfffYfffvsNAwcOxMiRI7F7926z7Ro0aIBz586ZPYKCgqQ4JLs4etxGhw8fNjumVq1amdZ54/e9cOFCs+PNzc1FeHg47r33XrPtlP59FxUVoWPHjli0aJFd23vL9e3ocXvL9e3ocRt5+vXt6HF7y/W9ceNGTJo0Cdu3b8f69etRXl6OIUOGoKioyOprJLvGhYe59dZbxdNPP222rG3btmLGjBkWt//rX/8q2rZta7bsqaeeEj179jQ9Hzt2rLjzzjvNthk6dKi4//77XVTqunP0uC1p166dmD17tun5kiVLRFhYmKuK6BaOHndmZqYAIK5cuWJ1n2r4vletWiU0Go04ceKEaZknfN9VARCrVq2qdRtvub6rsue4LfHE67sqe47bW67vqpz5vr3h+hZCiPz8fAFAbNy40eo2Ul3jHlUzVFpait9++w1DhgwxWz5kyBBs3brV4mu2bdtWY/uhQ4di586dKCsrq3Uba/uUmjPHXV1FRQWuXbtWY7K769evo1mzZoiPj8eIESNq/LKUU12Ou3PnzoiJicEdd9yBzMxMs3Vq+L4XL16MQYMG1cjSruTv2xnecH27gide33Xhyde3K3jL9V1QUAAAtU7CKtU17lHB0MWLF6HX62vMVh8VFVVjVnujvLw8i9uXl5fj4sWLtW5jbZ9Sc+a4q5s/fz6KioowduxY07K2bdti6dKlyMjIwPLlyxEUFITevXvj6NGjLi2/s5w57piYGHz66adYuXIl0tPT0aZNG9xxxx3YtGmTaRtv/77PnTuH77//Hk888YTZcqV/387whuvbFTzx+naGN1zfdeUt17cQAlOnTkWfPn2QkpJidTuprnGPmY6jKo1GY/ZcCFFjma3tqy93dJ9ycLaMy5cvx6xZs6DT6RAZGWla3rNnT/Ts2dP0vHfv3ujSpQs++OADvP/++64reB05ctxt2rRBmzZtTM979eqF3NxcvPPOO+jXr59T+5SLs2VcunQpGjZsiNGjR5st95Tv21Hecn07y9Ovb0d40/XtLG+5vp999lns3bsXW7ZssbmtFNe4R9UMRUREwNfXt0a0l5+fXyMqNIqOjra4vZ+fHxo3blzrNtb2KTVnjtvoyy+/xOOPP46vvvoKgwYNqnVbHx8fdO/eXTG/JOpy3FX17NnT7Ji8+fsWQuDf//43xo8fj4CAgFq3Vdr37QxvuL7rwpOvb1fxtOu7Lrzl+p48eTIyMjKQmZmJ+Pj4WreV6hr3qGAoICAAXbt2xfr1682Wr1+/HrfddpvF1/Tq1avG9j/++CO6desGf3//Wrextk+pOXPcgOEX46OPPoq0tDQMHz7c5vsIIbBnzx7ExMTUucyu4OxxV7d7926zY/LW7xswjNY4duwYHn/8cZvvo7Tv2xnecH07y9Ovb1fxtOu7Ljz9+hZC4Nlnn0V6ejo2bNiAxMREm6+R7Bq3u6u1QqxYsUL4+/uLxYsXiwMHDogpU6aIevXqmXrVz5gxQ4wfP960/fHjx0VISIh44YUXxIEDB8TixYuFv7+/+Oabb0zb/PLLL8LX11f885//FAcPHhT//Oc/hZ+fn9i+fbvkx2eNo8edlpYm/Pz8xIcffijOnTtnely9etW0zaxZs8S6devEH3/8IXbv3i0mTJgg/Pz8xK+//ir58Vnj6HG/9957YtWqVeLIkSNi3759YsaMGQKAWLlypWkbb/y+jR566CHRo0cPi/v0hO/72rVrYvfu3WL37t0CgHj33XfF7t27xcmTJ4UQ3nt9O3rc3nJ9O3rc3nJ9O3rcRp5+fT/zzDMiLCxMZGVlmZ23N27cMG0j1zXuccGQEEJ8+OGHolmzZiIgIEB06dLFbFjeI488Ivr372+2fVZWlujcubMICAgQzZs3Fx9//HGNfX799deiTZs2wt/fX7Rt29bs4lIKR467f//+AkCNxyOPPGLaZsqUKaJp06YiICBANGnSRAwZMkRs3bpVwiOyjyPH/dZbb4mWLVuKoKAg0ahRI9GnTx+xdu3aGvv0tu9bCCGuXr0qgoODxaeffmpxf57wfRuHTls7b731+nb0uL3l+nb0uL3l+nbmPPeG69vSMQMQS5YsMW0j1zWu+bOARERERKrkUX2GiIiIiFyNwRARERGpGoMhIiIiUjUGQ0RERKRqDIaIiIhI1RgMERERkaoxGCIiIiJVYzBEREREqsZgiIiIiFSNwRARERGpGoMhIvJIy5cvR1BQEM6cOWNa9sQTT6BDhw4oKCiQsWRE5Gk4NxkReSQhBDp16oS+ffti0aJFmD17Nv7v//4P27dvR1xcnNzFIyIP4id3AYiInKHRaDBnzhyMGTMGsbGxWLhwITZv3sxAiIgcxpohIvJoXbp0wf79+/Hjjz+if//+cheHiDwQ+wwRkcf64YcfcOjQIej1ekRFRcldHCLyUKwZIiKPtGvXLgwYMAAffvghVqxYgZCQEHz99ddyF4uIPBD7DBGRxzlx4gSGDx+OGTNmYPz48WjXrh26d++O3377DV27dpW7eETkYVgzREQe5fLly+jduzf69euHf/3rX6blWq0WJSUlWLdunYylIyJPxGCIiIiIVI0dqImIiEjVGAwRERGRqjEYIiIiIlVjMERERESqxmCIiIiIVI3BEBEREakagyEiIiJSNQZDREREpGoMhoiIiEjVGAwRERGRqjEYIiIiIlVjMERERESq9v8x1BIQwAc54QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {
      "filenames": {
       "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_16_2.png"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Using Autograd to calculate gradients for OLS\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "def CostOLS(beta):\n",
    "    return (1.0/n)*np.sum((y-X @ beta)**2)\n",
    "\n",
    "n = 100\n",
    "x = 2*np.random.rand(n,1)\n",
    "y = 4+3*x+np.random.randn(n,1)\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x]\n",
    "XT_X = X.T @ X\n",
    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(theta_linreg)\n",
    "# Hessian matrix\n",
    "H = (2.0/n)* XT_X\n",
    "EigValues, EigVectors = np.linalg.eig(H)\n",
    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
    "\n",
    "theta = np.random.randn(2,1)\n",
    "eta = 1.0/np.max(EigValues)\n",
    "Niterations = 1000\n",
    "# define the gradient\n",
    "training_gradient = grad(CostOLS)\n",
    "\n",
    "for iter in range(Niterations):\n",
    "    gradients = training_gradient(theta)\n",
    "    theta -= eta*gradients\n",
    "print(\"theta from own gd\")\n",
    "print(theta)\n",
    "\n",
    "xnew = np.array([[0],[2]])\n",
    "Xnew = np.c_[np.ones((2,1)), xnew]\n",
    "ypredict = Xnew.dot(theta)\n",
    "ypredict2 = Xnew.dot(theta_linreg)\n",
    "\n",
    "plt.plot(xnew, ypredict, \"r-\")\n",
    "plt.plot(xnew, ypredict2, \"b-\")\n",
    "plt.plot(x, y ,'ro')\n",
    "plt.axis([0,2.0,0, 15.0])\n",
    "plt.xlabel(r'$x$')\n",
    "plt.ylabel(r'$y$')\n",
    "plt.title(r'Random numbers ')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623ad923",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Same code but now with momentum gradient descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b7fe53c1",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[4.]\n",
      " [3.]]\n",
      "Eigenvalues of Hessian Matrix:[0.28641189 4.39287528]\n",
      "0 [-15.65565751] [-18.44347438]\n",
      "1 [-0.02618169] [0.02215597]\n",
      "2 [-0.02447466] [0.02071142]\n",
      "3 [-0.02287894] [0.01936105]\n",
      "4 [-0.02138725] [0.01809873]\n",
      "5 [-0.01999282] [0.01691871]\n",
      "6 [-0.0186893] [0.01581562]\n",
      "7 [-0.01747077] [0.01478446]\n",
      "8 [-0.01633169] [0.01382052]\n",
      "9 [-0.01526688] [0.01291943]\n",
      "10 [-0.01427149] [0.0120771]\n",
      "11 [-0.013341] [0.01128968]\n",
      "12 [-0.01247118] [0.0105536]\n",
      "13 [-0.01165807] [0.00986552]\n",
      "14 [-0.01089797] [0.00922229]\n",
      "15 [-0.01018743] [0.00862101]\n",
      "16 [-0.00952322] [0.00805892]\n",
      "17 [-0.00890232] [0.00753349]\n",
      "18 [-0.00832189] [0.00704231]\n",
      "19 [-0.00777931] [0.00658316]\n",
      "20 [-0.00727211] [0.00615394]\n",
      "21 [-0.00679797] [0.00575271]\n",
      "22 [-0.00635475] [0.00537764]\n",
      "23 [-0.00594042] [0.00502702]\n",
      "24 [-0.00555311] [0.00469926]\n",
      "25 [-0.00519105] [0.00439287]\n",
      "26 [-0.0048526] [0.00410646]\n",
      "27 [-0.00453622] [0.00383872]\n",
      "28 [-0.00424046] [0.00358844]\n",
      "29 [-0.00396398] [0.00335448]\n",
      "theta from own gd\n",
      "[[3.98706221]\n",
      " [3.01094846]]\n",
      "0 [-0.00370554] [0.00313577]\n",
      "1 [-0.00346394] [0.00293132]\n",
      "2 [-0.00316561] [0.00267887]\n",
      "3 [-0.00286972] [0.00242847]\n",
      "4 [-0.00259385] [0.00219502]\n",
      "5 [-0.00234197] [0.00198187]\n",
      "6 [-0.00211371] [0.00178871]\n",
      "7 [-0.00190742] [0.00161414]\n",
      "8 [-0.00172117] [0.00145652]\n",
      "9 [-0.00155308] [0.00131428]\n",
      "10 [-0.00140139] [0.00118591]\n",
      "11 [-0.00126452] [0.00107008]\n",
      "12 [-0.00114101] [0.00096557]\n",
      "13 [-0.00102956] [0.00087126]\n",
      "14 [-0.000929] [0.00078616]\n",
      "15 [-0.00083826] [0.00070937]\n",
      "16 [-0.00075639] [0.00064009]\n",
      "17 [-0.00068251] [0.00057757]\n",
      "18 [-0.00061585] [0.00052115]\n",
      "19 [-0.0005557] [0.00047025]\n",
      "20 [-0.00050142] [0.00042432]\n",
      "21 [-0.00045244] [0.00038288]\n",
      "22 [-0.00040825] [0.00034548]\n",
      "23 [-0.00036838] [0.00031174]\n",
      "24 [-0.0003324] [0.00028129]\n",
      "25 [-0.00029993] [0.00025381]\n",
      "26 [-0.00027064] [0.00022902]\n",
      "27 [-0.0002442] [0.00020665]\n",
      "28 [-0.00022035] [0.00018647]\n",
      "29 [-0.00019883] [0.00016826]\n",
      "theta from own gd wth momentum\n",
      "[[3.9993736 ]\n",
      " [3.00053008]]\n"
     ]
    }
   ],
   "source": [
    "# Using Autograd to calculate gradients for OLS\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "def CostOLS(beta):\n",
    "    return (1.0/n)*np.sum((y-X @ beta)**2)\n",
    "\n",
    "n = 100\n",
    "x = 2*np.random.rand(n,1)\n",
    "y = 4+3*x#+np.random.randn(n,1)\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x]\n",
    "XT_X = X.T @ X\n",
    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(theta_linreg)\n",
    "# Hessian matrix\n",
    "H = (2.0/n)* XT_X\n",
    "EigValues, EigVectors = np.linalg.eig(H)\n",
    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
    "\n",
    "theta = np.random.randn(2,1)\n",
    "eta = 1.0/np.max(EigValues)\n",
    "Niterations = 30\n",
    "\n",
    "# define the gradient\n",
    "training_gradient = grad(CostOLS)\n",
    "\n",
    "for iter in range(Niterations):\n",
    "    gradients = training_gradient(theta)\n",
    "    theta -= eta*gradients\n",
    "    print(iter,gradients[0],gradients[1])\n",
    "print(\"theta from own gd\")\n",
    "print(theta)\n",
    "\n",
    "# Now improve with momentum gradient descent\n",
    "change = 0.0\n",
    "delta_momentum = 0.3\n",
    "for iter in range(Niterations):\n",
    "    # calculate gradient\n",
    "    gradients = training_gradient(theta)\n",
    "    # calculate update\n",
    "    new_change = eta*gradients+delta_momentum*change\n",
    "    # take a step\n",
    "    theta -= new_change\n",
    "    # save the change\n",
    "    change = new_change\n",
    "    print(iter,gradients[0],gradients[1])\n",
    "print(\"theta from own gd wth momentum\")\n",
    "print(theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "749000b3",
   "metadata": {
    "editable": true
   },
   "source": [
    "## But noen of these can compete with Newton's method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e7b4abe1",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[3.92351924]\n",
      " [3.025027  ]]\n",
      "Eigenvalues of Hessian Matrix:[0.33104875 3.95055425]\n",
      "0 [-11.75315452] [-11.48461009]\n",
      "1 [3.84783351e-15] [-1.7484672e-15]\n",
      "2 [-4.8897518e-16] [-4.48356153e-16]\n",
      "3 [4.61653285e-16] [3.79106945e-16]\n",
      "4 [-4.8897518e-16] [-4.48356153e-16]\n",
      "beta from own Newton code\n",
      "[[3.92351924]\n",
      " [3.025027  ]]\n"
     ]
    }
   ],
   "source": [
    "# Using Newton's method\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "def CostOLS(beta):\n",
    "    return (1.0/n)*np.sum((y-X @ beta)**2)\n",
    "\n",
    "n = 100\n",
    "x = 2*np.random.rand(n,1)\n",
    "y = 4+3*x+np.random.randn(n,1)\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x]\n",
    "XT_X = X.T @ X\n",
    "beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(beta_linreg)\n",
    "# Hessian matrix\n",
    "H = (2.0/n)* XT_X\n",
    "# Note that here the Hessian does not depend on the parameters beta\n",
    "invH = np.linalg.pinv(H)\n",
    "EigValues, EigVectors = np.linalg.eig(H)\n",
    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
    "\n",
    "beta = np.random.randn(2,1)\n",
    "Niterations = 5\n",
    "\n",
    "# define the gradient\n",
    "training_gradient = grad(CostOLS)\n",
    "\n",
    "for iter in range(Niterations):\n",
    "    gradients = training_gradient(beta)\n",
    "    beta -= invH @ gradients\n",
    "    print(iter,gradients[0],gradients[1])\n",
    "print(\"beta from own Newton code\")\n",
    "print(beta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d47c2b1c",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Including Stochastic Gradient Descent with Autograd\n",
    "In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e2b04ba9",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[4.08928088]\n",
      " [2.79578306]]\n",
      "Eigenvalues of Hessian Matrix:[0.31588332 4.45642521]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theta from own gd\n",
      "[[4.08928088]\n",
      " [2.79578306]]\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABLq0lEQVR4nO3de3xU1b3///fkyj3cE0ICwVblYhWviBiFKqRWKRgRFY1o1WMVLxRthaoFWhHlpxZbtB5OLbRS8EZEqqcilQSwiEWB+vWG4AGMyM1bAiiBDOv3xzhDJjOTuWTPzJ7Zr+fjkUeYPXv2rD3DzP5krc/6LJcxxggAAMChMpLdAAAAgGQiGAIAAI5GMAQAAByNYAgAADgawRAAAHA0giEAAOBoBEMAAMDRCIYAAICjEQwBAABHIxgCbGj+/PlyuVy+n6ysLPXo0UOXX365Nm/enLR2TZs2TS6XK2nPnwq2bdsml8ulhx56KNlNARChrGQ3AEBo8+bNU9++fXXw4EH961//0owZM1RVVaUPP/xQnTp1SnbzACAtEAwBNnbCCSfotNNOkyQNHTpUbrdbU6dO1ZIlS3TttdcmuXVIpsOHD/t6DQG0DMNkQArxBka7d+/2bTt48KDuuOMODRw4UHl5eercubMGDx6sF198MeDxLpdLt9xyi5566in169dPbdq00UknnaSXXnopYN+XX35ZAwcOVG5urvr06RNy2OfgwYOaMmWK+vTpo5ycHPXs2VMTJkzQ119/7bdfSUmJLrroIr300ks6+eST1bp1a/Xr18/33PPnz1e/fv3Utm1bnXHGGXrrrbfCvh7e4cSqqirddNNN6tq1q7p06aLy8nJ99tlnAec+bdq0gGOUlJTommuuCTjmihUrdMMNN6hLly7q0KGDrr76ah04cEC7du3S2LFj1bFjR/Xo0UN33nmnDh8+HHDcI0eOaMaMGerVq5datWql0047Ta+99lrAfps3b9a4cePUvXt35ebmql+/fnrsscf89qmurpbL5dJTTz2lO+64Qz179lRubq62bNmib775Rnfeeaf69OmjVq1aqXPnzjrttNO0aNGisK8fAA/+pABSyNatWyVJxx13nG9bfX29vvzyS915553q2bOnDh06pH/+858qLy/XvHnzdPXVV/sd4+WXX9a6dev0m9/8Ru3atdOsWbN08cUXa9OmTTrmmGMkSa+99ppGjRqlwYMH6+mnn5bb7dasWbP8gjBJMsZo9OjReu211zRlyhSVlpbqnXfe0dSpU/XGG2/ojTfeUG5urm////znP5oyZYruvvtu5eXlafr06SovL9eUKVP02muv6f7775fL5dJdd92liy66SFu3blXr1q3Dvi7XX3+9LrzwQi1cuFA1NTX6xS9+oauuukorVqyI+bW+/vrrVV5erqefflobNmzQr371KzU0NGjTpk0qLy/Xf/3Xf+mf//ynHnzwQRUWFmrSpEl+j58zZ4569+6t2bNn68iRI5o1a5YuuOACrVy5UoMHD5Ykvf/++zrrrLPUq1cvPfzwwyooKNCyZct022236fPPP9fUqVP9jjllyhQNHjxYTzzxhDIyMtS9e3dNmjRJTz31lO677z6dfPLJOnDggN5991198cUXMZ874DgGgO3MmzfPSDJr1641hw8fNvv27TOvvPKKKSgoMOecc445fPhwyMc2NDSYw4cPm+uuu86cfPLJfvdJMvn5+aaurs63bdeuXSYjI8PMnDnTt23QoEGmsLDQfPvtt75tdXV1pnPnzqbx18Yrr7xiJJlZs2b5Pc8zzzxjJJm5c+f6tvXu3du0bt3afPrpp75tGzduNJJMjx49zIEDB3zblyxZYiSZpUuXRvQ63XzzzX7bZ82aZSSZnTt3+p371KlTA47Ru3dvM378+IBj3nrrrX77jR492kgyjzzyiN/2gQMHmlNOOcV3e+vWrUZSyNfv/PPP920rKyszRUVFpra21u+Yt9xyi2nVqpX58ssvjTHGVFVVGUnmnHPOCWj/CSecYEaPHh2wHUDkGCYDbOzMM89Udna22rdvrx/96Efq1KmTXnzxxYA8keeee05DhgxRu3btlJWVpezsbD355JP64IMPAo45bNgwtW/f3nc7Pz9f3bt31/bt2yVJBw4c0Lp161ReXq5WrVr59mvfvr1Gjhzpdyxvz0vjYSZJuvTSS9W2bduAYaGBAweqZ8+evtv9+vWT5MmHatOmTcB2b5vC+clPfuJ3+8QTT4zq8cFcdNFFfre9bbrwwgsDtgd7nlCv36pVq+R2u3Xw4EG99tpruvjii9WmTRs1NDT4fn784x/r4MGDWrt2rd8xL7nkkoDnOeOMM/SPf/xDkydPVnV1tb799tuYzxlwKoIhwMb++te/at26dVqxYoVuvPFGffDBB7riiiv89qmsrNTYsWPVs2dPLViwQG+88YbWrVunn/70pzp48GDAMbt06RKwLTc313cR/eqrr3TkyBEVFBQE7Nd02xdffKGsrCx169bNb7vL5VJBQUHAUE3nzp39bufk5DS7PVj7g2l6Tt6huZYEBtG0NVg7Q71+hw4d0v79+/XFF1+ooaFBf/jDH5Sdne338+Mf/1iS9Pnnn/s9vkePHgHH/P3vf6+77rpLS5Ys0bBhw9S5c2eNHj06qSUYgFRDzhBgY/369fMlTQ8bNkxut1t/+tOf9Pzzz2vMmDGSpAULFqhPnz565pln/GoA1dfXx/ScnTp1ksvl0q5duwLua7qtS5cuamho0N69e/0CImOMdu3apdNPPz2mNsRDbm5u0NckXrk1oV6/nJwctWvXTtnZ2crMzFRFRYUmTJgQ9Bh9+vTxux2sxlPbtm01ffp0TZ8+Xbt37/b1Eo0cOVIffvihNScDpDl6hoAUMmvWLHXq1Em//vWvdeTIEUmeC2ROTo7fhXLXrl1BZ5NFwjubq7Ky0q/HY9++ffr73//ut+95550nyROQNbZ48WIdOHDAd78dlJSU6J133vHbtmLFCu3fvz8uzxfq9SstLVVmZqbatGmjYcOGacOGDTrxxBN12mmnBfwE68VrTn5+vq655hpdccUV2rRpk7755hurTwtIS/QMASmkU6dOmjJlin75y19q4cKFuuqqq3TRRRepsrJSN998s8aMGaOamhr99re/VY8ePWIeKvntb3+rH/3oRxo+fLjuuOMOud1uPfjgg2rbtq2+/PJL337Dhw9XWVmZ7rrrLtXV1WnIkCG+2WQnn3yyKioqrDr1FquoqNC9996rX//61zr33HP1/vvva86cOcrLy4vL82VmZmr48OGaNGmSjhw5ogcffFB1dXWaPn26b59HH31UZ599tkpLS3XTTTeppKRE+/bt05YtW/T3v/89otlwgwYN0kUXXaQTTzxRnTp10gcffKCnnnpKgwcP9svDAhAawRCQYm699VbNmTNHv/nNb3TFFVfo2muv1Z49e/TEE0/oz3/+s4455hhNnjxZn376qd+FNxrDhw/XkiVLdM899+iyyy5TQUGBbr75Zn377bd+x3S5XFqyZImmTZumefPmacaMGeratasqKip0//33+02rT7Zf/OIXqqur0/z58/XQQw/pjDPO0LPPPqtRo0bF5fluueUWHTx4ULfddpv27NmjAQMG6OWXX9aQIUN8+/Tv31/r16/Xb3/7W91zzz3as2ePOnbsqGOPPdaXNxTOD3/4Qy1dulS/+93v9M0336hnz566+uqrdffdd8flvIB05DLGmGQ3AgAAIFnIGQIAAI5GMAQAAByNYAgAADgawRAAAHA0giEAAOBoBEMAAMDRHFFn6MiRI/rss8/Uvn37oOXsAQCA/RhjtG/fPhUWFiojI379N44Ihj777DMVFxcnuxkAACAGNTU1KioqitvxHREMtW/fXpLnxezQoUOSWwMAACJRV1en4uJi33U8XhwRDHmHxjp06EAwBABAiol3igsJ1AAAwNEIhgAAgKMRDAEAAEcjGAIAAI5GMAQAAByNYAgAADgawRAAAHA0giEAAOBoBEMAAMDRCIYAAICjEQwBAABHIxgCAACORjAEAAAcjWAIAAA4GsEQAABwNIIhAADgaARDAADA0ZIeDK1atUojR45UYWGhXC6XlixZEnLfG2+8US6XS7Nnz05Y+wAAQHpLejB04MABnXTSSZozZ06z+y1ZskRvvvmmCgsLE9QyAADgBFnJbsAFF1ygCy64oNl9duzYoVtuuUXLli3ThRdemKCWAQAAJ0h6z1A4R44cUUVFhX7xi19owIAByW4OAABIM0nvGQrnwQcfVFZWlm677baIH1NfX6/6+nrf7bq6ung0DQAApAFb9wy9/fbbevTRRzV//ny5XK6IHzdz5kzl5eX5foqLi+PYSgAAkMpsHQytXr1ae/bsUa9evZSVlaWsrCxt375dd9xxh0pKSkI+bsqUKaqtrfX91NTUJK7RAAAgpdh6mKyiokLnn3++37aysjJVVFTo2muvDfm43Nxc5ebmxrt5AAAgDSQ9GNq/f7+2bNniu71161Zt3LhRnTt3Vq9evdSlSxe//bOzs1VQUKDjjz8+0U0FAABpKOnB0FtvvaVhw4b5bk+aNEmSNH78eM2fPz9JrQIAAE6R9GBo6NChMsZEvP+2bdvi1xgAAOA4tk6gBgAAiDeCIQAA4GgEQwAAwNEIhgAAgKMRDAEAAEcjGAIAAI5GMAQAAByNYAgAADgawRAAAHA0giEAAOBoBEMAAMDRCIYAAICjEQwBAABHIxgCAACORjAEAAAcjWAIAAA4GsEQAABwNIIhAADgaARDAADA0QiGAACAoxEMAQAARyMYAgAAjkYwBAAAHI1gCAAAOBrBEAAAcDSCIQAA4GgEQwAAwNEIhgAAgKMRDAEAAEcjGAIAAI5GMAQAAByNYAgAADgawRAAAHA0giEAAOBoBEMAAMDRCIYAAICjEQwBAABHIxgCAACORjAEAAAcLenB0KpVqzRy5EgVFhbK5XJpyZIlvvsOHz6su+66Sz/4wQ/Utm1bFRYW6uqrr9Znn32WvAYDAABruN1SdbW0aJHnt9udlGYkPRg6cOCATjrpJM2ZMyfgvm+++Ubr16/Xvffeq/Xr16uyslIfffSRfvKTnyShpQAAwDKVlVJJiTRsmDRunOd3SYlne4K5jDEm4c8agsvl0gsvvKDRo0eH3GfdunU644wztH37dvXq1Sui49bV1SkvL0+1tbXq0KGDRa0FAAAxqayUxoyRmoYgLpfn9/PPS+XlCbt+J71nKFq1tbVyuVzq2LFjyH3q6+tVV1fn9wMAAGzA7ZZuvz0wEJKObps4MaFDZikVDB08eFCTJ0/WuHHjmo0QZ86cqby8PN9PcXFxAlsJAABCWr1a+vTT0PcbI9XUePZLkJQJhg4fPqzLL79cR44c0eOPP97svlOmTFFtba3vp6amJkGtBAAAzdq509r9LJCVsGdqgcOHD2vs2LHaunWrVqxYEXbcMDc3V7m5uQlqHQAAiFiPHtbuZwHb9wx5A6HNmzfrn//8p7p06ZLsJgEAgFiVlkpFRUeTpZtyuaTiYs9+CZL0nqH9+/dry5Ytvttbt27Vxo0b1blzZxUWFmrMmDFav369XnrpJbndbu3atUuS1LlzZ+Xk5CSr2QAAIBaZmdKjj3pmk7lc/onU3gBp9mzPfgmS9Kn11dXVGjZsWMD28ePHa9q0aerTp0/Qx1VVVWno0KERPQdT6wEAsJnKSs+sssbJ1MXFnkCovFxS4q7fSQ+GEoFgCAAAG3K7PbPGdu705AiVlvr1CCXq+p30YTIAAOBQmZlShKM88WT7BGoAAIB4IhgCAACORjAEAAAcjWAIAAA4GgnUAAAgNmFmg6UKgiEAABC9YHWCioo8BRW/qxOUKhgmAwAA0ams9FSQbrr6/I4dnu2VlclpV4wIhgAAQOTcbk+PULCazd5tEyd69ksRBEMAACByq1cH9gg1ZoxUU+PZL0UQDAEAgMjt3GntfjZAMAQAACLXo4e1+9kAwRAAAIhcaaln1pjLFfx+l8uz+nxpaWLb1QIEQwAAIHKZmZ7p81JgQOS9PXt2StUbIhgCAADRKS+Xnn9e6tnTf3tRkWd7itUZougiAACIXnm5NGoUFagBAEALWb2kRSKXyMjMlIYOjc+xE4hgCACAZLF6SYs0WiIjkcgZAgAgGaxe0iLNlshIJJcxweppp5e6ujrl5eWptrZWHTp0SHZzAABO53ZLJSWhKzm7XJ4ena1bIxvisvp4NpGo6zc9QwAAJJrVS1qk4RIZiUQwBABAolm9pEUaLpGRSARDAAAkmtVLWqThEhmJRDAEAECiWb2kRRoukZFIBEMAACSa1UtapOESGYlEMAQAQDJYvaRFmi2RkUhMrQcAIJlSuQJ1nCXq+k0FagAAksnqJS3SZImMRGKYDAAAOBrBEAAAcDSCIQAA4GgEQwAAwNEIhgAAgKMRDAEAAEcjGAIAAI5GMAQAAByNYAgAADgawRAAAHC0pAdDq1at0siRI1VYWCiXy6UlS5b43W+M0bRp01RYWKjWrVtr6NCheu+995LTWAAAkHaSHgwdOHBAJ510kubMmRP0/lmzZumRRx7RnDlztG7dOhUUFGj48OHat29fglsKAADSUdIXar3gggt0wQUXBL3PGKPZs2fr7rvvVnl5uSTpL3/5i/Lz87Vw4ULdeOONiWwqAABIQ0nvGWrO1q1btWvXLo0YMcK3LTc3V+eee67WrFkT8nH19fWqq6vz+wEAADFyu6XqamnRIs9vtzvZLbKUrYOhXbt2SZLy8/P9tufn5/vuC2bmzJnKy8vz/RQXF8e1nQAApK3KSqmkRBo2TBo3zvO7pMSzPU3YOhjycrlcfreNMQHbGpsyZYpqa2t9PzU1NfFuIgAA6aeyUhozRvr0U//tO3Z4tqdJQGTrYKigoECSAnqB9uzZE9Bb1Fhubq46dOjg9wMAAKLgdku33y4ZE3ifd9vEiWkxZGbrYKhPnz4qKCjQ8uXLfdsOHTqklStX6qyzzkpiywAASHOrVwf2CDVmjFRT49kvxSV9Ntn+/fu1ZcsW3+2tW7dq48aN6ty5s3r16qWJEyfq/vvv17HHHqtjjz1W999/v9q0aaNx48YlsdUAAKS5nTut3a85brcnqNq5U+rRQyotlTIzW37cCCU9GHrrrbc0bNgw3+1JkyZJksaPH6/58+frl7/8pb799lvdfPPN+uqrrzRo0CC9+uqrat++fbKaDABAbJJ80Y9Kjx7W7hdKZaVnOK5xL1RRkfToo9L557fs2BFyGRNsMDC91NXVKS8vT7W1teQPAQCSo7mL/ne19Gzl+eelyy8PnRPkcnnav3Vr7AGdN0G7aSjy3SSpur/+VXkVFXG/fhMMAQAQb2Eu+nr+eesDopb0QoVqb2MuV8va7XZ7puiHyEsykj5WNx2rvXG/fts6gRoAgJSXjFlZLakN1Fx7vTIzpWeeaVkAFyZB2yWpu/bGfvwoEAwBABBPiZ6V1dLaQOHaK3kCpm7dYm7iof2H9N6f3oj58VYjGAIAIJ4SPSurpb1QcWrvlte267GxK/WTgjfVpX29JvxtcFSPj6ekzyYDACBlRZKXk6hZWVJ0vVBDh7asHWH22/fZPq2Y876WLa3Xsk299X8NvSX1VobcKtVq9de7OiKXXDIKuaZERoZ05Ehk7WkBgiEAAGIR6eyw0lLP9h07gvfYeGdllZa2vE1W9OrE2N4jDUe08ZlNWvbX3XrlzY5aUztADRrkuz9bhzS5ze91p3uWOtRHmAuUgEBIIhgCACAyjXuBNm+Wpk0LDBa8eTmNZ1llZnoCpDFjPIFE48d4Z5PNnm1NvSErenWiaO/ud/fq1TmbtGyZS69uP057TT9J/Xy7fz97m8r6blfZqNY6v2SzWt/wy+YTs5OEqfUAAIQTrBeoOZ07S88+6xmK8gY5wY5RXOwJLKyaVu+drh6uVyeS2kBB2mt6Fund82/X3z48TcveKdDGb/v6PaSd9um8gvdVNrReZTeW6JihvfzbFenr9506SXkSdYasQDAEAIhZJDV3Qmk6bJaICtTe9krBe3WiqQ3kdmvH7Of03nPvq/qjAj321TjVqaPfLqe0/kBlA3er7LKOGnxdf+W0ywk8TnW1Z3p/NFwu1RUWKm/HjrhfvxkmAwCgscYBS/fu0m23xT604x02e/ZZqWvXo0HQ2LHxW4ajvNwT8ATLZ4qgFyow8flyv/vzM/ZoRMlHKitzafiE49R9gP/QWFDRzpTzBm4PPCBVVET32BjQMwQAgFe0w2GRysz0n86eiGU4IuyFCp74nO27P1uHNKTjeyobVKuy8QU66dLjlJEVZWWeaHuGvhs+rDv//IRcvwmGAACQWjYcFq14LsMRgcDEZ/8Cio0Tn4dN6K92Be1a9oThcpkkTxHH3/1O6tnTF7gl6vpNMAQAQIwJvi1ixUKnETq0/5D+Nfc9LXu2NrrEZyvFkMuUqOs3OUMAAESyBIXVIimAGEoEQ2BbXtuuZf+9TctWtVLV7v7ar5P97g9MfB6kuGphLlM8EQwBABDLUhjeGjxTp0p/+IP05ZeJee4QxR6/nfqAXv2/7wdUfPaKKfHZauXl0qhR8Z9RFyWCIQAAYlkKo3GPxoknBh8Csvq5Q+Q1Hfl0h3JvqNBf9LxekKeHJXjic/fo2hYPmZnR94TFGTlDAABEUqywZ09p/nxpz57gPRrBemyaziJresxocobcbrl79lLG7s+CruV1RC7tUoFmnrBAI0a3sybxOclIoLYQwRAAICwrihU2zeXZu1e67LKYj9k48fnzDZ/oT4fGhz+Pqirb9bzEigRqAAASyYoE32BDQJmZUR0zVOLz5VoU2XnEkv/kcARDAAB4WZXg27SH6OOPpTVrgh4zsOJz8MTnihM+l5ZG8Nyx5D85HMNkAJDOErEWFvyFmO3lrTgdc8VnKxdhTREMkwEAWibMRRlxEGK2l9mxQ7rkEj3abYbu//wG7TX+09oDKz6frACZmZ73bsyYo9P6vbw5SLNnp00glEj0DAFAOgq1tESSl4FIa2GqWB+RS5+qSH20VW30TWQVn4P17L34YmCQ+91aXun2ntIzBACIjdvtuVgG+1vXGE9ANHGiJzeGXgTL7Jj9nHo2U8U6Q0a9VKO3J8xT/weuDl/xubmevW3bGP60EMEQAKSbcEtLtGQZCPg0TXw+o8FENN9r4JC2Uruc5ncK1bO3Y4dnOz17liIYAoB4SGbicqRTq6OZgk0idojE56O9O330UWQHCjfbi569hCMYAgCrJTtxOdKp1ZHul+zzSaLd7+7Vq3M2adkyl17dflzzic8/O0M6syj8bK/S0uaflJ69hCMYAgAr2WF4o7TUc9Ft6UVZssf5JFDjis/L3inQxm/7Surmu7+d9jVJfC6RVHL0AFbM9opHzx6axWwyALBKmNlECa0DY9XSEnY5nzgKrPjc3u/+U1p/oLKBu1V2WUcNvq6/ciLJ92nJbK/qamnYsPD7pdGyG6GwNpmFCIYAB7BDTovdLmJclIMKXvH5KG/F57Iyl4ZPOE7dB3QLcaRmtOT/owOLK4bC1HoAiJRdclrsNrzR0qUl7HY+MQqX+By84nP3lj1psDXKmgoVMFFcMeEIhgCkNjvltFiduGyFSC7KoUTazt27PRd2G12co0p8DlXxOZ7CBfBWLBqLiDFMBiB12S2nJd2GN8KdT2NJnl0WPPH5qMDE5xAVnxMhmurgdhj+TSJyhixEMASkKTvmtFiRuGwnoc6nqSScn+WJz4lgtwDe5sgZAoBw7JjTkojhjUT2FoQ6n6YSUAwweOLz0eTnwMRn/6ExW6CGkC0RDAFIXXbM0ZFanrjcnGQki3vP5w9/kH7+89D7WXwhT0ric7zZMYAHwRCAFGZlcUGrtSRxOZRkJotnZkr5+ZHt24ILue0Tn1vKrgG8wxEMAUhdTpiC7B0S27HD0yuTzPWq4nAhb3HF51Rj5wDewWwfDDU0NGjatGn629/+pl27dqlHjx665pprdM899ygjIyPZzQOQbOk8BTnYkFgoicg1CXchlzyB2OefN3uYwMRn/96dwMTnQSGO1Ay7zsJyQgCfiozN3XfffaZLly7mpZdeMlu3bjXPPfecadeunZk9e3bEx6itrTWSTG1tbRxbCiCpGhqMqaoyZuFCz++GhmS3qGUWLzbG5TLGc7mM/OeWW+J7/pG0y+Xy7Peduh11ZsmUteamASvNMVnbAnbPz9htKo5ZbRbc9LrZ/e4ea9pYVOT/JEVFfm1KumBtLC62VxttIFHXb9tPrb/ooouUn5+vJ5980rftkksuUZs2bfTUU09FdAym1gNIKeGmX0cinknVzz0nXXGFp51BGJdLhzvl65FTF+gf/+7yXeJztu/+4InPFvX0R1PDJ9ns2ntlI0yt/87ZZ5+tJ554Qh999JGOO+44/ec//9Hrr7+u2bNnJ7tpABAf4aZfRyKeSdXduoUMhCTJZYxyvtylV5ZnapUGSkpQ4rPb7RlWTGZeVTTikWSPmNg+GLrrrrtUW1urvn37KjMzU263WzNmzNAVV1wR8jH19fWqr6/33a6rq0tEUwHAGlZMq47nxT/C9o3MW63LLshIXOIzNXwQI9sHQ88884wWLFighQsXasCAAdq4caMmTpyowsJCjR8/PuhjZs6cqenTpye4pbAtuqKRaqyaVm3xxd+b+Lzjn1/p/gj2v2NJqTT0nBY/b8So4YMY2T5nqLi4WJMnT9aECRN82+677z4tWLBAH374YdDHBOsZKi4uJmfIieyymjkQjWjWBIvEwoWeHJ8oBa/4LGXIrW0qUU/tUIZstAabHZdnQYuQM/Sdb775JmAKfWZmpo4cORLyMbm5ucrNzY1302B3dlrNHIhGc9OvYxFhT1M0FZ8P9/uFXI9OlGSj6eHU8EGMbB8MjRw5UjNmzFCvXr00YMAAbdiwQY888oh++tOfJrtpsLNUS6QEmgpVP6lLF8/vL74If4wILv6xV3weKpUWWVffyYrhbGr4IEa2Hybbt2+f7r33Xr3wwgvas2ePCgsLdcUVV+jXv/61cnIiW4GYqfUORHc50kWwIEE6um3zZmnaNM+2YBf/Jj2gwSs+HxVY8blX+PZVV3t+JM/naejQ6AIOq4ezgx2vuDj1i3A6UKKu37YPhqxAMORAixZJ48aF3y/GXArAVsJc/AMrPrf3e3hgxefI/tAM+dzRBDLxqgvExIm0QDBkIYIhB6JnCE7T6OL/TXaelr/VUcteavBLfPbKz9ijESUfqazMpeETjlP3Ad1CHDSMlgYy4YpLxpqInSqBUKq0M4kIhixEMORA4WbjJGu2C5IjzS86wROf41zx2YpAJh5/tKTKDNJUaWeS2XI2WU1NjYqLi+PVFsA6JFLCK00vOrEnPlvEigKHVtcFSpUZpKnSTgeJKhjq27evJk2apMmTJ6tt27bxahNgjXRezRyRSaOLTvDE56PDW4GJzyWKa8VnKwKZSItLRrJfqswgtWs707z3NJyohsnWrFmjn//856qpqdGMGTN07bXXxrNtlmGYzOEc/iFvEbu+dpG0K175KAkU18TnlrJiiMvK4exUyRO0Yztt3HuasOt3LEvd/+UvfzFFRUVm4MCBpqqqKpZDJFRtba2RZGpra5PdFCB1LF5sTFGRMZ7LlOenqMizPRXaVVXlv0+oHxt9h9XtqDNLpqw1Nw1YaY7J2hbQ1PyM3abimNVmwU2vm93v7kluYxsaPK+7yxX8dXW5jCku9uzXnMWLPfs2PY53W6T/3xYujOz9Xriw5efeEnZrp/f1D/b+RfP6x0mirt8xBUPGGPPNN9+Ye++917Rp08aMHj3abN682cp2WYpgCIiSXb8go2mX3S46QbgPu83bC94394+oMufkbTBZOuTXtGzVm6Ed15uZZVVm/cIPjPuwO2ltDcqqQCZYgFtcHN3/s0QEvw0NnscvXOj5HS7QS1Y7I+UNaEO1IdKANo5sHwwdOHDArF692kycONFkZGSY3NxcM2nSJFNXV2dl+yxBMAREIdIvyPr6ll8Y4tEubzvsdNFpZNf/22P+euNqc2XJ66aba09Ac76fvdVM+EG1WXrPm2bfzn0JbVtMrAhkjGl5oGFVT1UoVvWUxrud0bDpZ6QxWwZDf/zjH81Pf/pT84Mf/MBkZWWZ3Nxcc/rpp5tbb73VPPnkk+aHP/yhKSoqMuvWrYtXe2NCMAREIdIvyK5dW35hiEe7vF/c9fXGZGY2v29mpme/OKrfV29WPLze3DWoygxs/UFAE9qpzowqWGsev3yl+bhqe1zbEjdW9JhYwaqeqlDHDRa4xHLceLUzWinQe2rLYKioqMiMGTPGPPTQQ+b11183Bw8eDNhnxowZZsCAAZY10AoEQ0AUIv2CtOrCYHW7vF/cSfyrd/M/t5k5l1abkflrTTvVBTzlKa3fN1MGV5nq2RtM/b74BmOOY1VPlVe8hpKsbmcs6Bnysbzo4u7du1VYWCi3223lYVuE2WRAFCKd7RJMPGdoRTsLJ4FLsuz7bJ9WzHlfy5bWx7fiMyJj5SzISP/f3XOPdN550T1XsmdrpkBxWlsWXYxE9+7dtWLFCqsPCyBRSks9X4ChviCbY0z4QnvxalfTFdqtrGHTRPCKz4N89wdUfC7/njLeaPBc9PYeltw2KVGQrjIzrfv/F2k9pfvu8/xEMyXdynbGguK0Pi2sxx7I5XLp3HPPtfqwABLF+wUpHf1CjFakF5BoNNeuYF/c3uAp1Dm4XJ7FTL3BUxi7392rp372uq7q8y8V5HyhU6/qp1+9OlSrageqQdn6fvY2TfjBSi2959/6cuchVX1+oiZPlk7+938ro7inp3dh3DjP75IST20X2F+0wbK3oGeqvL/e4rQ9e/pvLypKqaKkLcXaZACCC1aIrVs3ae/e8I+tqvIEGfEYAgizQnvAvmPGeP4d7K/eZr7sg1d8Piqw4nOv5tvYWEtXZEfihBtKCsYGw0tRS/aQXQgs1GohgiEgRk2/IM86S/re98IPVT3yiPTzn1tf0dbbnh07pF27pC+/lDIyPEMNQ4cG//KOIniypOJzqCVAmkrFC6ZThQqqw0l2tes0QDBkIYIhwELhelvuvFN66KHAi0ZLe0PC9bY0F2yF+KvX8sTncEuABMMFMzWE+/8XjAXJ+U6XsgnUANJccwvgPvywNGlS8L+ejfEERD/7mfTtt54chUi74iPpbWlu8dXvElV9ic8/Xh1Z4vOlxykjq3v49nmFW8k9mHjkV8F65eWexVNXr5Zee82TLB1ODMn5SA56hoB4suk4vCWCndvq1dFNy49k6Cya3pYgQ0+7392rV+ds0rJlLr26/TjtNf69O9/P3qayvttVNqq1hk3or3YF7SJvf1ORTudvjJ6h1JMCU9LTBT1DQKqz8UrQlgg2LTjaXo7GvTnev7qbBo7R9LZ8N7V/w61/1jPrj22U+Hw0AApMfC6RVBJdu0OJpiegaSkApA6mpKcdeoaAeAg1rJPus4hiKdjockmdO0utWnmCIy9v4FhfH3VvyxVaqKd1NFcjosRnK0Qz88jlSt//B4mS7J7XaGY2IiYkUFuIYMgiyf7iSRXhhnXSuQs9lmnIoXgDx2nTpKlTo3pouatS7fp0S07F50hmHmVmenKrZs1KXLvSjV16XvlejCuCIQsRDFnALl88kv2/fKJdNsJKdnhtYp2GHIzLJfXsKWOMtOMzudT88Yykw10KlLXjE2XkZrfsuVuCOkPx5dSeVwdK1PXb8grUSEPeL56mX+zJqLRaWenpebBzNd9I82asnkVkl9cmVEXbWBgjffqpHtxxpYykI2qmIrbLJZfLpZy5jyU3EJI8r8HHH3uKVAbjvYhPnOgJYOHhdnv+mFi0yPM72GvjdnsCzVAzFiVeV0SNYAjNs9MXj52CssaafoF3j3Aqdo8ekX35R8Jur015ubRtm6f3a8ECqWvXFh3uPxqoCj2lLzOaGe6yavkAq96TNWuar9bdeB03RB7Mh0uo53VFDJhNhuZF88UTz+nB4YIyl8sTlI0alfwEyqIiqUsXT3Xk5qbd7t0bmFsUy9CjXV+bxrPNWrcOOnRmpOb6enzumnBA/R+4WjmtLz86DOgNOvfsiXxIMNwwopXDwcnqIUxFoYa9gtWO4nVFPBgHqK2tNZJMbW1tspuSehYuNMbzFdX8z8KF8W1HVVVk7aiqim87Glu82BiXK7ANjbc1vd/l8vz84hehH+tyeY4dKTu+NsEsXmzcPQr92rRdRWavuhi3grwW3tejuNiYhgZLnt8UFfkfv6jo6Gvd3PsZ7XtijHXvS0ODZ5+FCz2/rXgt7KShIfB9ae7/QKr8f4clEnX9ZpgMzYu0bkq8K63a7a/BSHpjunSRCgv97ysqkp591jMEY9XQo91em0aONBzR+r99oJll1Tr3p8eo9c6PNVRVukILNVRVOk4f6fdtJsslI9O0j8jKei3hhhG9FbWtHA4uLfW8364QfV8ul2cadnN1huySBxZP0Q57WfG6Ak0QDKF5dvnisUtQ5hXJF/gXX0h/+Ysnb2bhQs/vrVs9+TNW5jzY7LXZ/e5ePfWz13VVn3+pIOcLnXpVP/3q1aFaVTtQh9RKO7JL1OUHhbrjnjb6fKdbvzlwp1yLF8tV1CTh2socoHCBzs03W5+H4i3MJwV+fiIJ9OyWBxYv0QbzLX1dgSDIGULz7FJp1RuUhSt/n6i/BiP9At+zJ3ChRqt7cpL82hzaf0j/mvuelj1bG3vF58brPgXL52lJyYBIAtfmEp0bi6Z3ze32FJO8/Xbpb3/zf46iouYL89k1DyweYgnmm1sfj4KHiEVcB+FsgpwhCwTLtygujj6PoqVt8OZvWJHT0RLR5i00zvv43e+sz3lI8Guz+Z/bzJxLq83I/LWmneoCmn5K6/fNlMFVpnr2BlO/r75lTxYu1yecSPPerHxPgrW5a1djJk6MLO/HSXkx3pyhYPla4fLG0j2fCgm7fhMMIXJ2+OKxQ1BmTHRf4MHanJkZecJopOL42tTtqDNLpqw1Nw1YaY7J2hbQ5PyM3abimNVmwU2vm93v7mnx8/lYkdQcaWDRtWtsF+R4tNkuExcSxU5/6MBWEnX9pgK1ZI+qvYicXd6vUJWWG1fBlYJPGQ6lpRV0LXptjjQc0cZnNmnZX3frlTc7ak3tADXoaCHDbB3SkI7v6Udn1qrsmh468ZJjlZFlcQqiVcuaRLrC+MMPS5dd5tkW6v0M955Y1eZkVjFPFtb5QhAJW0EirqGWTTQbWba0Cx6pyaperuZ6Y8JNGQ7WQ5SMXq7v7Pp/e8xfb1xtrix53XRz7Qlo6vezt5oJP6g2S+950+zbuS/+DbJyqCjSnoeW9q5ZOZ0+1qGjVGaH3mfYSqJ6hpydQB1NoS+kDysL6zWX+Ftd3XziruTpSfjd76T8/IT3clmS+BxPViaaR5pwGy6RO1FtjnXigl16TWPVuFAnkEDODYbiPVsj1b+U0lU8AuBQX+CRXhjz8wNnnMXJlte2a9l/b9OyVa1Utbu/9utkv/tPaf2BygbuVtllHTX4uv7KaTcoIe0KyuqSAZEGOi25IFvZ5mhnTNlpMWUgxTg3ZyieY/J8KdmTVfkckbJB3se+z/ZpxZz3tWxpvZZt6q3/a+jtd39+xh6NKPlIZWUuDZ9wnLoPaGbtr0SLNNfHqvfLCvFocyR/WLGKe2z4o9X2yBmyUNAxx3jN1rC6pD+sk+jpyknI+3Afdpu3F7xv7h9RZc7J22CydMjvKbNVb4Z2XG8e+FGV2fD0h8Z92G3Zc8fF4sWhXzu7fp4SPTMq2uUs4EG+aEogZyje4lG110mF0lJRopetSFDByt3v7tWrczZp2TKXXt1+nPaafpL6+e7/fvY2lfXdrrJRrTVsQn+1Kzg59MHsqEsXTzXvxjp3lubOtaY6tdU9A4kuCGiXxZRTCfmiaMK5wVA8qvbypWRvyVi2Ig4XRtsnPlsl1AVLkr780prjx2s4u6WJ2NGw8dp0tsQfrQgmrv1OFvn000/NlVdeaTp37mxat25tTjrpJPPWW29F/PiQ3WxWd2c7rVBaqknmdOUWThlOaMVnO4j30E86DWc7qVq1FXi9UgrDZN/56quvNGTIEA0bNkz/+Mc/1L17d3388cfq2LFjyw9u9V/tNlswE00kc521KGcoBU98Ppr8HJj47D80lvLi2cuabj0Ddlu3z+7oSUMQtg+GHnzwQRUXF2vevHm+bSUlJdY9gZXd2Xwp2Z9NF3gMXvH56LT24BWfuyelrQkRzwtWug1n22Ux5VTBH60IwvbB0NKlS1VWVqZLL71UK1euVM+ePXXzzTfrhhtuCPmY+vp61dfX+27X1dU1/yRWFfriSyk1JDKfoxlpn/jcEvG8YKVjz4BNg3xb4o9WBBPXQTgL5ObmmtzcXDNlyhSzfv1688QTT5hWrVqZv/zlLyEfM3XqVCMp4CdhC7XaZTFR2Er9vnqz4uH15q5BVWZg6w8CUhTaqc6MKlhrHr98pfm4anuym5tc8czvSuecEZaziAwLw6YMFmr9Tk5Ojk477TStWbPGt+22227TunXr9MYbbwR9TLCeoeLi4vgXbWqMYl5QsIrP7f3uD6z4nJOkltpQJAvhxrqYbaoVc4T1WBg2JSSq6KLth8l69Oih/v37+23r16+fFi9eHPIxubm5ys3NjXfTmscaO44Ul8RnpwbW8Rr6YTgbkm2Gy2EPtg+GhgwZok2bNvlt++ijj9S7d+8QjwAS4LsA5cinO7TlgwZVrivSP/7dxfrEZ6cv7RKvCxY5NpD4oxU+th8mW7dunc466yxNnz5dY8eO1b///W/dcMMNmjt3rq688sqIjpGwtU3gCF/Pnqfse+5S2wN7fdtqVKTb9aheUHmQxOd2sT0R603Fn1N73YAUkajrt+2DIUl66aWXNGXKFG3evFl9+vTRpEmTmp1N1hTBEFqiccXnwxve0f93aKIko4xG+xyR5JK0e/oTKvj1jS1/0kQvKgsANkQwZCGCIQex6C/9YInPGXJrm0rUU5/6BUI+LQlQmrbb7ZbOPz/84+K44j0AJBsJ1Dgq2V35yX7+SLUgvyaSxOfb8p9R8c44FOsL1u7OnSN7bCrVwgEAmyIYsjurE2ijDWxSJYE3ylWoY6r4/FxXaVwEbYkmQAnV7kgXIqVKLgC0GMNkdmZ1Am20gY0dEngjCd4izK/Z/fd/69U/bmlU8bmb327HZX2sm4v/rrNP2qf+Faep9agR/s9VXS0NGxa+zZEOXYVrd3PIGQLgAOQMWSglgyGrE2ijDWzskMAbafAWYZAyVFVaqaG+2+20T+cVvK+yofW6uN8HKvif+5p/LquL9UUaXAV7HonZZE6QKkPUQJwk6vodNA8UNhDNYpLhhFulW/Ks0u12x+f5Y+EN3pq2wTvsVVl5dFuEw1I9tFOntP5AUwZXq3r2Rn2xL1dLdg7STZd+roJpN4V/Lm+xPuloQOIVS7G+SIfTmuYPFRURCDlBZaUn+B42TBo3zvO7pMT//z4ASxAM2ZWVi0nGEtgkczHLKIK3fZ/t0+svfRXRYR//c2u9/U0/3b9mqM69faBn6YtoA0Vvsb6ePf33jSVAiTTf59lnPUNvCxd6fm/dSiCU7qL5YwBAi5FAHalEd1dbuWp3LIFNPFcNDyfC4O32vHl6/MB4HdGN2qaZ6qkdylDo4atOV4+M+bn8ZohZVRU50tWzhw5laMRJwgXoLpcnQB81iv8XgEXoGYpEMrqrvRfKpsMxXi6XZ1HB0tLwx4olsLHy+aMVYfC250BbNShbx2TX6JVe/yWXjEy0w1ex9oB5y/hfcUXswYrVw25ID8keogYciGAonGR1V1t5oYwlsEnihfpwx64R7Xf5sD36uOoTbT5Uohu23yvX4sVyRTt8lcweMMnaYTekh2QOUQMOxWyy5th1RlVxcfSLSXqDOin4Kt2hLrxWPX8YjSs+r9x9vN7VD0IOexmXS65Qr3u0w5lWzxCLFbOG4GV1CQcghTG13kIxv5h2+VKy6kIZa2AThwt18IrPR13rmqcnzU8ledb88onHtPJYA0UgHuwSoAM2wHIcdmCX7mpvfkpLxZr4a8HzR1/xebxcS/OC1xmyuFfKN1SViOcCwvEOUY8Z4wl8ggXo5JIBlqJnqDl26RlKRW63vvzr3/Xe3zZq1btd9Ifdl2q3Cvx2+X72NpX13a6yUa01bEJ/tStoF/Q4CRs+YqgKdpKgIWrAzhgms1CLc4boro7Iof2H9K+57+nTJ/6u4Zv/qALt8t1XoyJN1gM6UPB9lQ2tV9mNJTpmaK8kthZIAQTocDiCIQu16MUkn6RZjROfq3b313At1/MaI8n4TVU0ckkuyeXw1wsAEDmCIQu1+MWku9qnucTnDLlVo2L10E4FncRPTxoAIAokUNuJVRWHU1A0ic+XnP6JCn/bTDJ5sGrOAAAkGcFQpKya0WVn3+Un1P57k958/ZCeeuckLfukn/aafpL6+XYLTHw+2XPHokWRPQ/F4gAANkIwBB3af0ibbvm9ihfNUsdDe5UnaYSkfirSAT2q5Rqu8wreb5T4XCKpJPBAya7mDABADMgZcqjGic/td2/RU6pQYNKzJLnUsOBpZV85NvxBmX0HALAQOUOwVPDE597KkFvbgsz+kr6r/OySsqfcKV1+SWTFGSkWBwBIMQRDaSrSxOcbjl2p4nURrpAdSc4U1ZwBACmGYMhuWlBkbfe7e/XqnE1atsylV7cfF1ni86IPpXERHDyapGcHz74DAKQegiE7CVbPqKjIM/QUpEfFW/F52bO1WvZOgTZ+21dSN9/97bQvfOJzvJKenTD7DgCQFkigtgtvpeumb0eTStdNKz7vV3u/3U9p/YHKBu5W2WUdNfi6/sppl9P885L0DACwKRKoncTt9vQIBQtGjJGR9NVlN2mQGagt7mMkHa36nJ+xRyNKPlJZmUvDJxyn7gP8h8bCIukZAOBwBEN2sHq1/9BYEy5JnRv2qKc+0XYV+So+l13TQydecqwysrq37PlJegYAOBjBkA3U/nuT8iLY7+HRq3X8H087WvHZSiQ9AwAcimAoCZomPud9e7yqI3jcqbeXSgXt4tcwkp4BAA5EMJQggYnPR3t3MnSsvlae8lTb/GrvpaUJay8AAE5BMBQnoSo+ezVOfL7ouI+U9/Pa0AczhiRmAADihGDIIpFWfA5IfHa7pZLLmz94ly6efB4naEHRSQAAYkEw1AIxVXxuKsxMMknSF19EvhxGKouy6CQAAFYgGIqCJRWfm4p0mYtolsNIRaGKTu7Y4dn+XdFJAACsRjAURnOJz1Kwis+DQhwphHgth5FKwhSdlMslTZzoGSpkyAwAYDGCoSaiSXyOqeJzU6WlnqGgcMthJGMmWaLyd8INFRoj1dQ4Y6gQAJBwjg+GYk58topdl8NIZP4OQ4UAgCRyZDBkSeKzley2HEai83cYKgQAJFHKrVo/c+ZM/epXv9Ltt9+u2bNnR/QY76q3E099SdXvf++7xOejAhOfe8Wh5RGww7Ry7yr2oYat4rGKvfc5ww0VWvmcAADbY9X6INatW6e5c+fqxBNPjOnxs98uleR5MVuc+BwPdlgOIxn5O3YdKgQAOEJGshsQqf379+vKK6/U//zP/6hTp04xHeOykje04KZ/afe7e/X2N/10/5qhOvf2gcppl2Nxa1NYsvJ3vEOFPXv6by8qYlo9ACCuUqZnaMKECbrwwgt1/vnn67777mt23/r6etXX1/tu19XVSZLm/mdwXLvZ0kIy83fKyz3T55M9VAgAcJSUCIaefvpprV+/XuvWrYto/5kzZ2r69OmxP2G0uTt2yPWxSrKn+tthqBAA4Ci2HyarqanR7bffrgULFqhVq1YRPWbKlCmqra31/dTU1ET+hJWVnmTeYcOkceM8v0tKPNut2N/uvPk70tF8HS/ydwAAacj2s8mWLFmiiy++WJmNLr5ut1sul0sZGRmqr6/3uy+YiLPRQ00p9wYBTXNXmtvfmKNVkxv3FKVKL1KwOkPFxcmZ6g8AcKREzSazfTC0b98+bd++3W/btddeq759++quu+7SCSecEPYYEb2Y0U4pD7d/Y95ihVJqLUSaKoEbACAtMbX+O+3btw8IeNq2basuXbpEFAhFLNop5ZGsNu+1Y4d0ySWh77PrQqTk7wAAHMD2OUMJE+2U8mimljfX+ea9b+JET08MAABIKNv3DAVTXV0d2wOfe0763veCD/dEO6XcyqnlLEQKAEDSOKtn6PrrQ8/28k4pbzqDysvl8iQQe6eUh9s/FixECgBAwjkrGPLy5uk0DoiinVLe3P6xYiFSAAASzpnBUKg8nWiXhAi1f7Sa9joBAICEsf3Ueiv4pubJu0xrI1VVgXk6sVagfvFFT+9RU40XHw21EKkdZ5MBAJBETK1PFG+eTktq6ninoA8d6nlcsFpC3iAp1H0EQgAAJAU9Q1VV0pdfWlsMsbnAikKGAABEhArUFgoaDHkrSj/8sHTZZaFrAT37rHTppQlqKQAA8EpUMOTMBGpvns4jj0iTJjVfFPGKKzz5PAAAIC05Mxjyzg7r2jX8khput6dnKFVXoQcAAM1yVjD0pz95coS2bvXkAkVT5JDlMgAASEvOCoYuvdQz48ubsBxNkUPvchkAACCtOHtqvXdJjUhXn2e5DHgxKxAA0oazeoaaarykRiRYLgOSJ3+spMSzzt24caHXuwMApARnB0OSJ3fo2Web/6ue5TLgVVnpWdeuaW9isPXuAAApgWBI8uQSPf106PuN8ax4D2dzuz3FOYOVYgi13h0AwPYIhrzGjJEWL/bkEAUzdWp6DYW43VJ1tbRokec3F/DwVq9uPr/MGBLtASAFEQw1Vl4ubdsmTZ8e/P50GQoh5yU2kSbQk2gPACmFYCiY//mf4NvTYSiEnJfYRZpAT6I9AKQUgqGm0nkohJyXlvGWYvAu59IUifYAkJIIhppK56GQdA70EqFxKYamAZH39uzZ1BsCgBRDMNRUOg+FpHOglyjl5Z517Xr29N/uXe+uvDw57QIAxMzZFaiD8Q6F7NgRfDjJ5fLcn4pDIekc6CVSebk0ahQVqAEgTRAMNeUdChkzxhP4NA6IUn0oJJ0DvUTLzPSscwcASHkMkwWTrkMh5LwAABDAZUywLoL0UldXp7y8PNXW1qpDhw6RPzBdF+OsrPTMKmucTF1c7AmEUjXQAwCknZiv31EiGHKqdA30AABpI1HXb3KGnIqcFwAAJJEzBAAAHI5gCAAAOBrBEAAAcDSCIQAA4GgEQwAAwNEIhgAAgKMRDAEAAEcjGAIAAI5GMAQAAByNYAgAADgawRAAAHA02wdDM2fO1Omnn6727dure/fuGj16tDZt2pTsZgEAgDRh+2Bo5cqVmjBhgtauXavly5eroaFBI0aM0IEDB5LdNAAAkAZcxhiT7EZEY+/everevbtWrlypc845J6LH1NXVKS8vT7W1terQoUOcWwgAAKyQqOt3VtyOHCe1tbWSpM6dO4fcp76+XvX19b7bdXV1cW8XAABITbYfJmvMGKNJkybp7LPP1gknnBByv5kzZyovL8/3U1xcnMBWAgCAVJJSw2QTJkzQyy+/rNdff11FRUUh9wvWM1RcXMwwGQAAKYRhsiZuvfVWLV26VKtWrWo2EJKk3Nxc5ebmJqhlAAAgldk+GDLG6NZbb9ULL7yg6upq9enTJ9lNAgAAacT2wdCECRO0cOFCvfjii2rfvr127dolScrLy1Pr1q2T3DoAAJDqbJ8z5HK5gm6fN2+errnmmoiOwdR6AABSDzlD37F5rAYAAFJcSk2tBwAAsBrBEAAAcDSCIQAA4GgEQwAAwNEIhgAAgKMRDAEAAEcjGAIAAI5GMAQAAByNYAgAADgawRAAAHA0giEAAOBoBEMAAMDRCIYAAICjEQwBAABHIxgCAACORjAEAAAcjWAIAAA4GsEQAABwNIIhAADgaARDAADA0QiGAACAoxEMAQAARyMYAgAAjkYwBAAAHI1gCAAAOBrBEAAAcDSCIQAA4GgEQwAAwNEIhgAAgKMRDAEAAEcjGAIAAI5GMAQAAByNYAgAADgawRAAAHA0giEAAOBoBEMAAMDRCIYAAICjEQwBAABHS5lg6PHHH1efPn3UqlUrnXrqqVq9enWymwQAANJASgRDzzzzjCZOnKi7775bGzZsUGlpqS644AJ98sknyW4aAABIcS5jjEl2I8IZNGiQTjnlFP3xj3/0bevXr59Gjx6tmTNnhn18XV2d8vLyVFtbqw4dOsSzqQAAwCKJun7bvmfo0KFDevvttzVixAi/7SNGjNCaNWuS1CoAAJAuspLdgHA+//xzud1u5efn+23Pz8/Xrl27gj6mvr5e9fX1vtu1tbWSPBEmAABIDd7rdrwHsWwfDHm5XC6/28aYgG1eM2fO1PTp0wO2FxcXx6VtAAAgfr744gvl5eXF7fi2D4a6du2qzMzMgF6gPXv2BPQWeU2ZMkWTJk3y3f7666/Vu3dvffLJJ3F9Me2mrq5OxcXFqqmpcVSuFOfNeTsB5815O0Ftba169eqlzp07x/V5bB8M5eTk6NRTT9Xy5ct18cUX+7YvX75co0aNCvqY3Nxc5ebmBmzPy8tz1H8irw4dOnDeDsJ5Owvn7SxOPe+MjPimONs+GJKkSZMmqaKiQqeddpoGDx6suXPn6pNPPtHPfvazZDcNAACkuJQIhi677DJ98cUX+s1vfqOdO3fqhBNO0P/+7/+qd+/eyW4aAABIcSkRDEnSzTffrJtvvjmmx+bm5mrq1KlBh87SGefNeTsB5815OwHnHd/zTomiiwAAAPFi+6KLAAAA8UQwBAAAHI1gCAAAOBrBEAAAcLSUDIYef/xx9enTR61atdKpp56q1atXN7v/ypUrdeqpp6pVq1Y65phj9MQTTwTss3jxYvXv31+5ubnq37+/XnjhhXg1P2bRnHdlZaWGDx+ubt26qUOHDho8eLCWLVvmt8/8+fPlcrkCfg4ePBjvU4lKNOddXV0d9Jw+/PBDv/3S7f2+5pprgp73gAEDfPukwvu9atUqjRw5UoWFhXK5XFqyZEnYx6TD5zva806Xz3e0550un+9ozztdPt8zZ87U6aefrvbt26t79+4aPXq0Nm3aFPZxifiMp1ww9Mwzz2jixIm6++67tWHDBpWWluqCCy7QJ598EnT/rVu36sc//rFKS0u1YcMG/epXv9Jtt92mxYsX+/Z54403dNlll6miokL/+c9/VFFRobFjx+rNN99M1GmFFe15r1q1SsOHD9f//u//6u2339awYcM0cuRIbdiwwW+/Dh06aOfOnX4/rVq1SsQpRSTa8/batGmT3zkde+yxvvvS8f1+9NFH/c63pqZGnTt31qWXXuq3n93f7wMHDuikk07SnDlzIto/XT7f0Z53uny+oz1vr1T/fEd73uny+V65cqUmTJigtWvXavny5WpoaNCIESN04MCBkI9J2GfcpJgzzjjD/OxnP/Pb1rdvXzN58uSg+//yl780ffv29dt24403mjPPPNN3e+zYseZHP/qR3z5lZWXm8ssvt6jVLRfteQfTv39/M336dN/tefPmmby8PKuaGBfRnndVVZWRZL766quQx3TC+/3CCy8Yl8tltm3b5tuWCu93Y5LMCy+80Ow+6fL5biyS8w4mFT/fjUVy3uny+W4slvc7HT7fxhizZ88eI8msXLky5D6J+oynVM/QoUOH9Pbbb2vEiBF+20eMGKE1a9YEfcwbb7wRsH9ZWZneeustHT58uNl9Qh0z0WI576aOHDmiffv2BSx2t3//fvXu3VtFRUW66KKLAv6yTKaWnPfJJ5+sHj166LzzzlNVVZXffU54v5988kmdf/75AVXa7fx+xyIdPt9WSMXPd0uk8ufbCuny+a6trZWkZhdhTdRnPKWCoc8//1xutztgtfr8/PyAVe29du3aFXT/hoYGff75583uE+qYiRbLeTf18MMP68CBAxo7dqxvW9++fTV//nwtXbpUixYtUqtWrTRkyBBt3rzZ0vbHKpbz7tGjh+bOnavFixersrJSxx9/vM477zytWrXKt0+6v987d+7UP/7xD11//fV+2+3+fsciHT7fVkjFz3cs0uHz3VLp8vk2xmjSpEk6++yzdcIJJ4TcL1Gf8ZRZjqMxl8vld9sYE7At3P5Nt0d7zGSItY2LFi3StGnT9OKLL6p79+6+7WeeeabOPPNM3+0hQ4bolFNO0R/+8Af9/ve/t67hLRTNeR9//PE6/vjjfbcHDx6smpoaPfTQQzrnnHNiOmayxNrG+fPnq2PHjho9erTf9lR5v6OVLp/vWKX65zsa6fT5jlW6fL5vueUWvfPOO3r99dfD7puIz3hK9Qx17dpVmZmZAdHenj17AqJCr4KCgqD7Z2VlqUuXLs3uE+qYiRbLeXs988wzuu666/Tss8/q/PPPb3bfjIwMnX766bb5S6Il593YmWee6XdO6fx+G2P05z//WRUVFcrJyWl2X7u937FIh893S6Ty59sqqfb5bol0+XzfeuutWrp0qaqqqlRUVNTsvon6jKdUMJSTk6NTTz1Vy5cv99u+fPlynXXWWUEfM3jw4ID9X331VZ122mnKzs5udp9Qx0y0WM5b8vzFeM0112jhwoW68MILwz6PMUYbN25Ujx49WtxmK8R63k1t2LDB75zS9f2WPLM1tmzZouuuuy7s89jt/Y5FOny+Y5Xqn2+rpNrnuyVS/fNtjNEtt9yiyspKrVixQn369An7mIR9xiNOtbaJp59+2mRnZ5snn3zSvP/++2bixImmbdu2vqz6yZMnm4qKCt/+//d//2fatGljfv7zn5v333/fPPnkkyY7O9s8//zzvn3+9a9/mczMTPPAAw+YDz74wDzwwAMmKyvLrF27NuHnF0q0571w4UKTlZVlHnvsMbNz507fz9dff+3bZ9q0aeaVV14xH3/8sdmwYYO59tprTVZWlnnzzTcTfn6hRHvev/vd78wLL7xgPvroI/Puu++ayZMnG0lm8eLFvn3S8f32uuqqq8ygQYOCHjMV3u99+/aZDRs2mA0bNhhJ5pFHHjEbNmww27dvN8ak7+c72vNOl893tOedLp/vaM/bK9U/3zfddJPJy8sz1dXVfv9vv/nmG98+yfqMp1wwZIwxjz32mOndu7fJyckxp5xyit+0vPHjx5tzzz3Xb//q6mpz8sknm5ycHFNSUmL++Mc/BhzzueeeM8cff7zJzs42ffv29ftw2UU0533uuecaSQE/48eP9+0zceJE06tXL5OTk2O6detmRowYYdasWZPAM4pMNOf94IMPmu9973umVatWplOnTubss882L7/8csAx0+39NsaYr7/+2rRu3drMnTs36PFS4f32Tp0O9f82XT/f0Z53uny+oz3vdPl8x/L/PB0+38HOWZKZN2+eb59kfcZd3zUQAADAkVIqZwgAAMBqBEMAAMDRCIYAAICjEQwBAABHIxgCAACORjAEAAAcjWAIAAA4GsEQAABwNIIhAADgaARDAADA0QiGAKSkRYsWqVWrVtqxY4dv2/XXX68TTzxRtbW1SWwZgFTD2mQAUpIxRgMHDlRpaanmzJmj6dOn609/+pPWrl2rnj17Jrt5AFJIVrIbAACxcLlcmjFjhsaMGaPCwkI9+uijWr16NYEQgKjRMwQgpZ1yyil677339Oqrr+rcc89NdnMApCByhgCkrGXLlunDDz+U2+1Wfn5+spsDIEXRMwQgJa1fv15Dhw7VY489pqefflpt2rTRc889l+xmAUhB5AwBSDnbtm3ThRdeqMmTJ6uiokL9+/fX6aefrrffflunnnpqspsHIMXQMwQgpXz55ZcaMmSIzjnnHP33f/+3b/uoUaNUX1+vV155JYmtA5CKCIYAAICjkUANAAAcjWAIAAA4GsEQAABwNIIhAADgaARDAADA0QiGAACAoxEMAQAARyMYAgAAjkYwBAAAHI1gCAAAOBrBEAAAcDSCIQAA4Gj/P890SB5pKtw/AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {
      "filenames": {
       "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_22_2.png"
      }
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theta from own sdg\n",
      "[[4.10188623]\n",
      " [2.87242312]]\n"
     ]
    }
   ],
   "source": [
    "# Using Autograd to calculate gradients using SGD\n",
    "# OLS example\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "# Note change from previous example\n",
    "def CostOLS(y,X,theta):\n",
    "    return np.sum((y-X @ theta)**2)\n",
    "\n",
    "n = 100\n",
    "x = 2*np.random.rand(n,1)\n",
    "y = 4+3*x+np.random.randn(n,1)\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x]\n",
    "XT_X = X.T @ X\n",
    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(theta_linreg)\n",
    "# Hessian matrix\n",
    "H = (2.0/n)* XT_X\n",
    "EigValues, EigVectors = np.linalg.eig(H)\n",
    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
    "\n",
    "theta = np.random.randn(2,1)\n",
    "eta = 1.0/np.max(EigValues)\n",
    "Niterations = 1000\n",
    "\n",
    "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
    "training_gradient = grad(CostOLS,2)\n",
    "\n",
    "for iter in range(Niterations):\n",
    "    gradients = (1.0/n)*training_gradient(y, X, theta)\n",
    "    theta -= eta*gradients\n",
    "print(\"theta from own gd\")\n",
    "print(theta)\n",
    "\n",
    "xnew = np.array([[0],[2]])\n",
    "Xnew = np.c_[np.ones((2,1)), xnew]\n",
    "ypredict = Xnew.dot(theta)\n",
    "ypredict2 = Xnew.dot(theta_linreg)\n",
    "\n",
    "plt.plot(xnew, ypredict, \"r-\")\n",
    "plt.plot(xnew, ypredict2, \"b-\")\n",
    "plt.plot(x, y ,'ro')\n",
    "plt.axis([0,2.0,0, 15.0])\n",
    "plt.xlabel(r'$x$')\n",
    "plt.ylabel(r'$y$')\n",
    "plt.title(r'Random numbers ')\n",
    "plt.show()\n",
    "\n",
    "n_epochs = 50\n",
    "M = 5   #size of each minibatch\n",
    "m = int(n/M) #number of minibatches\n",
    "t0, t1 = 5, 50\n",
    "def learning_schedule(t):\n",
    "    return t0/(t+t1)\n",
    "\n",
    "theta = np.random.randn(2,1)\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "# Can you figure out a better way of setting up the contributions to each batch?\n",
    "    for i in range(m):\n",
    "        random_index = M*np.random.randint(m)\n",
    "        xi = X[random_index:random_index+M]\n",
    "        yi = y[random_index:random_index+M]\n",
    "        gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
    "        eta = learning_schedule(epoch*m+i)\n",
    "        theta = theta - eta*gradients\n",
    "print(\"theta from own sdg\")\n",
    "print(theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6d7455a",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Same code but now with momentum gradient descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "40a68e4f",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[3.84094234]\n",
      " [3.01300561]]\n",
      "Eigenvalues of Hessian Matrix:[0.26776828 4.97262227]\n",
      "theta from own gd\n",
      "[[3.83774539]\n",
      " [3.01544605]]\n",
      "theta from own sdg with momentum\n",
      "[[3.8186717 ]\n",
      " [3.06511966]]\n"
     ]
    }
   ],
   "source": [
    "# Using Autograd to calculate gradients using SGD\n",
    "# OLS example\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "# Note change from previous example\n",
    "def CostOLS(y,X,theta):\n",
    "    return np.sum((y-X @ theta)**2)\n",
    "\n",
    "n = 100\n",
    "x = 2*np.random.rand(n,1)\n",
    "y = 4+3*x+np.random.randn(n,1)\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x]\n",
    "XT_X = X.T @ X\n",
    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(theta_linreg)\n",
    "# Hessian matrix\n",
    "H = (2.0/n)* XT_X\n",
    "EigValues, EigVectors = np.linalg.eig(H)\n",
    "print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
    "\n",
    "theta = np.random.randn(2,1)\n",
    "eta = 1.0/np.max(EigValues)\n",
    "Niterations = 100\n",
    "\n",
    "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
    "training_gradient = grad(CostOLS,2)\n",
    "\n",
    "for iter in range(Niterations):\n",
    "    gradients = (1.0/n)*training_gradient(y, X, theta)\n",
    "    theta -= eta*gradients\n",
    "print(\"theta from own gd\")\n",
    "print(theta)\n",
    "\n",
    "\n",
    "n_epochs = 50\n",
    "M = 5   #size of each minibatch\n",
    "m = int(n/M) #number of minibatches\n",
    "t0, t1 = 5, 50\n",
    "def learning_schedule(t):\n",
    "    return t0/(t+t1)\n",
    "\n",
    "theta = np.random.randn(2,1)\n",
    "\n",
    "change = 0.0\n",
    "delta_momentum = 0.3\n",
    "\n",
    "for epoch in range(n_epochs):\n",
    "    for i in range(m):\n",
    "        random_index = M*np.random.randint(m)\n",
    "        xi = X[random_index:random_index+M]\n",
    "        yi = y[random_index:random_index+M]\n",
    "        gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
    "        eta = learning_schedule(epoch*m+i)\n",
    "        # calculate update\n",
    "        new_change = eta*gradients+delta_momentum*change\n",
    "        # take a step\n",
    "        theta -= new_change\n",
    "        # save the change\n",
    "        change = new_change\n",
    "print(\"theta from own sdg with momentum\")\n",
    "print(theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1f7f38d",
   "metadata": {
    "editable": true
   },
   "source": [
    "## AdaGrad algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
    "\n",
    "<!-- dom:FIGURE: [figures/adagrad.png, width=600 frac=0.8] -->\n",
    "<!-- begin figure -->\n",
    "\n",
    "<img src=\"figures/adagrad.png\" width=\"600\"><p style=\"font-size: 0.9em\"><i>Figure 1: </i></p>\n",
    "<!-- end figure -->"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b85cc96c",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Similar (second order function now) problem but now with AdaGrad"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3ced9e13",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[2.]\n",
      " [3.]\n",
      " [4.]]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theta from own AdaGrad\n",
      "[[2.00019432]\n",
      " [2.99888222]\n",
      " [4.00105497]]\n"
     ]
    }
   ],
   "source": [
    "# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n",
    "# OLS example\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "# Note change from previous example\n",
    "def CostOLS(y,X,theta):\n",
    "    return np.sum((y-X @ theta)**2)\n",
    "\n",
    "n = 1000\n",
    "x = np.random.rand(n,1)\n",
    "y = 2.0+3*x +4*x*x\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x, x*x]\n",
    "XT_X = X.T @ X\n",
    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(theta_linreg)\n",
    "\n",
    "\n",
    "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
    "training_gradient = grad(CostOLS,2)\n",
    "# Define parameters for Stochastic Gradient Descent\n",
    "n_epochs = 50\n",
    "M = 5   #size of each minibatch\n",
    "m = int(n/M) #number of minibatches\n",
    "# Guess for unknown parameters theta\n",
    "theta = np.random.randn(3,1)\n",
    "\n",
    "# Value for learning rate\n",
    "eta = 0.01\n",
    "# Including AdaGrad parameter to avoid possible division by zero\n",
    "delta  = 1e-8\n",
    "for epoch in range(n_epochs):\n",
    "    Giter = 0.0\n",
    "    for i in range(m):\n",
    "        random_index = M*np.random.randint(m)\n",
    "        xi = X[random_index:random_index+M]\n",
    "        yi = y[random_index:random_index+M]\n",
    "        gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
    "        Giter += gradients*gradients\n",
    "        update = gradients*eta/(delta+np.sqrt(Giter))\n",
    "        theta -= update\n",
    "print(\"theta from own AdaGrad\")\n",
    "print(theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed1fa9e4",
   "metadata": {
    "editable": true
   },
   "source": [
    "Running this code we note an almost perfect agreement with the results from matrix inversion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "787b2b5e",
   "metadata": {
    "editable": true
   },
   "source": [
    "## RMSProp algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
    "\n",
    "<!-- dom:FIGURE: [figures/rmsprop.png, width=600 frac=0.8] -->\n",
    "<!-- begin figure -->\n",
    "\n",
    "<img src=\"figures/rmsprop.png\" width=\"600\"><p style=\"font-size: 0.9em\"><i>Figure 1: </i></p>\n",
    "<!-- end figure -->"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac61b6e0",
   "metadata": {
    "editable": true
   },
   "source": [
    "## RMSprop for adaptive learning rate with Stochastic Gradient Descent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "5febac4f",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[2.]\n",
      " [3.]\n",
      " [4.]]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theta from own RMSprop\n",
      "[[1.9996357 ]\n",
      " [3.00788598]\n",
      " [3.99589367]]\n"
     ]
    }
   ],
   "source": [
    "# Using Autograd to calculate gradients using RMSprop  and Stochastic Gradient descent\n",
    "# OLS example\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "# Note change from previous example\n",
    "def CostOLS(y,X,theta):\n",
    "    return np.sum((y-X @ theta)**2)\n",
    "\n",
    "n = 1000\n",
    "x = np.random.rand(n,1)\n",
    "y = 2.0+3*x +4*x*x# +np.random.randn(n,1)\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x, x*x]\n",
    "XT_X = X.T @ X\n",
    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(theta_linreg)\n",
    "\n",
    "\n",
    "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
    "training_gradient = grad(CostOLS,2)\n",
    "# Define parameters for Stochastic Gradient Descent\n",
    "n_epochs = 50\n",
    "M = 5   #size of each minibatch\n",
    "m = int(n/M) #number of minibatches\n",
    "# Guess for unknown parameters theta\n",
    "theta = np.random.randn(3,1)\n",
    "\n",
    "# Value for learning rate\n",
    "eta = 0.01\n",
    "# Value for parameter rho\n",
    "rho = 0.99\n",
    "# Including AdaGrad parameter to avoid possible division by zero\n",
    "delta  = 1e-8\n",
    "for epoch in range(n_epochs):\n",
    "    Giter = 0.0\n",
    "    for i in range(m):\n",
    "        random_index = M*np.random.randint(m)\n",
    "        xi = X[random_index:random_index+M]\n",
    "        yi = y[random_index:random_index+M]\n",
    "        gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
    "\t# Accumulated gradient\n",
    "\t# Scaling with rho the new and the previous results\n",
    "        Giter = (rho*Giter+(1-rho)*gradients*gradients)\n",
    "\t# Taking the diagonal only and inverting\n",
    "        update = gradients*eta/(delta+np.sqrt(Giter))\n",
    "\t# Hadamard product\n",
    "        theta -= update\n",
    "print(\"theta from own RMSprop\")\n",
    "print(theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee8a6301",
   "metadata": {
    "editable": true
   },
   "source": [
    "## ADAM algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
    "\n",
    "<!-- dom:FIGURE: [figures/adam.png, width=600 frac=0.8] -->\n",
    "<!-- begin figure -->\n",
    "\n",
    "<img src=\"figures/adam.png\" width=\"600\"><p style=\"font-size: 0.9em\"><i>Figure 1: </i></p>\n",
    "<!-- end figure -->"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2d0f16b",
   "metadata": {
    "editable": true
   },
   "source": [
    "## And finally [ADAM](https://arxiv.org/pdf/1412.6980.pdf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "59349e6f",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Own inversion\n",
      "[[2.]\n",
      " [3.]\n",
      " [4.]]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "theta from own ADAM\n",
      "[[2.0001042 ]\n",
      " [2.99949077]\n",
      " [4.00048049]]\n"
     ]
    }
   ],
   "source": [
    "# Using Autograd to calculate gradients using RMSprop  and Stochastic Gradient descent\n",
    "# OLS example\n",
    "from random import random, seed\n",
    "import numpy as np\n",
    "import autograd.numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from autograd import grad\n",
    "\n",
    "# Note change from previous example\n",
    "def CostOLS(y,X,theta):\n",
    "    return np.sum((y-X @ theta)**2)\n",
    "\n",
    "n = 1000\n",
    "x = np.random.rand(n,1)\n",
    "y = 2.0+3*x +4*x*x# +np.random.randn(n,1)\n",
    "\n",
    "X = np.c_[np.ones((n,1)), x, x*x]\n",
    "XT_X = X.T @ X\n",
    "theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
    "print(\"Own inversion\")\n",
    "print(theta_linreg)\n",
    "\n",
    "\n",
    "# Note that we request the derivative wrt third argument (theta, 2 here)\n",
    "training_gradient = grad(CostOLS,2)\n",
    "# Define parameters for Stochastic Gradient Descent\n",
    "n_epochs = 50\n",
    "M = 5   #size of each minibatch\n",
    "m = int(n/M) #number of minibatches\n",
    "# Guess for unknown parameters theta\n",
    "theta = np.random.randn(3,1)\n",
    "\n",
    "# Value for learning rate\n",
    "eta = 0.01\n",
    "# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980\n",
    "beta1 = 0.9\n",
    "beta2 = 0.999\n",
    "# Including AdaGrad parameter to avoid possible division by zero\n",
    "delta  = 1e-7\n",
    "iter = 0\n",
    "for epoch in range(n_epochs):\n",
    "    first_moment = 0.0\n",
    "    second_moment = 0.0\n",
    "    iter += 1\n",
    "    for i in range(m):\n",
    "        random_index = M*np.random.randint(m)\n",
    "        xi = X[random_index:random_index+M]\n",
    "        yi = y[random_index:random_index+M]\n",
    "        gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
    "        # Computing moments first\n",
    "        first_moment = beta1*first_moment + (1-beta1)*gradients\n",
    "        second_moment = beta2*second_moment+(1-beta2)*gradients*gradients\n",
    "        first_term = first_moment/(1.0-beta1**iter)\n",
    "        second_term = second_moment/(1.0-beta2**iter)\n",
    "\t# Scaling with rho the new and the previous results\n",
    "        update = eta*first_term/(np.sqrt(second_term)+delta)\n",
    "        theta -= update\n",
    "print(\"theta from own ADAM\")\n",
    "print(theta)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d490115",
   "metadata": {
    "editable": true
   },
   "source": [
    "## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n",
    "\n",
    "Presently, instead of using **autograd**, we recommend using [JAX](https://jax.readthedocs.io/en/latest/)\n",
    "\n",
    "**JAX** is Autograd and [XLA (Accelerated Linear Algebra))](https://www.tensorflow.org/xla),\n",
    "brought together for high-performance numerical computing and machine learning research.\n",
    "It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bc7785a",
   "metadata": {
    "editable": true
   },
   "source": [
    "### Getting started with Jax, note the way we import numpy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f1090d5e",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import jax\n",
    "import jax.numpy as jnp\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from jax import grad as jax_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac6bf976",
   "metadata": {
    "editable": true
   },
   "source": [
    "### A warm-up example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ed55121d",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/jax/_src/numpy/lax_numpy.py:173: UserWarning: Explicitly requested dtype float64 requested in asarray is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.\n",
      "  return asarray(x, dtype=self.dtype)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1268ba940>]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABQCUlEQVR4nO3deXxU1eH+8c+dmaxkhZCFECDsS9hBFNkURami4K6tim1tUyOVUmu/WlutbeXXRUstYLRWXLHugDu4sCj7JoussgWSAAnJTNZJMnN/fwxEEQQCydyZyfN+veYFTG7Iw4iZh3POPccwTdNERERExE9sVgcQERGR5kXlQ0RERPxK5UNERET8SuVDRERE/ErlQ0RERPxK5UNERET8SuVDRERE/ErlQ0RERPzKYXWA7/J6veTn5xMbG4thGFbHERERkTNgmiZlZWW0adMGm+3UYxsBVz7y8/PJyMiwOoaIiIichby8PNq2bXvKawKufMTGxgK+8HFxcRanERERkTPhcrnIyMiofx8/lYArH8emWuLi4lQ+REREgsyZLJnQglMRERHxK5UPERER8SuVDxEREfErlQ8RERHxK5UPERER8SuVDxEREfErlQ8RERHxK5UPERER8auA22RMREREmojXA3uXQvlBiEmB9kPBZvd7jAaNfEydOpXBgwcTGxtLcnIy48ePZ9u2bcddM3HiRAzDOO5x/vnnN2poEREROXMz188k9+PJMC0Lnr8S3vyJ78dpWeR+PJmZ62f6NU+DyseiRYvIyclh+fLlLFiwgLq6OsaMGUNFRcVx111++eUUFBTUP95///1GDS0iIiJnzla0nRkHPiHXVn7c87m2CmYc+ARb0Xa/5mnQtMuHH3543K9nzZpFcnIya9asYcSIEfXPR0REkJqa2jgJRURE5Ox5PWRv+Ahs5cxITAAgu9RFbkIcMxLjySlxkl08Hy72+G0K5pzWfDidTgBatmx53PMLFy4kOTmZhIQERo4cyV/+8heSk5NP+nu43W7cbnf9r10u17lE+v6slbW8unofOw+V87fr+jbJ1xAREQk4e5eCK5/so7+ckZhAbkI8HsMgp6SU7FIX4PRdlzncL5HO+m4X0zSZMmUKw4YNIysrq/75sWPH8vLLL/Ppp5/y2GOPsWrVKi6++OLjCsa3TZ06lfj4+PpHRkbG2UY6pRqPl//3wVZeW72fvCOVTfI1REREAk75wfqfjqisAsBjGISZ5tHiceJ1Te2sy8fdd9/Nhg0beOWVV457/sYbb+SKK64gKyuLcePG8cEHH7B9+3bee++9k/4+999/P06ns/6Rl5d3tpFOqXVsBOd3bAXAexsLmuRriIiIBJyYlPqf/qWVb6bCME1qDYPchLiTXtfUzmraZdKkScybN4/FixfTtm3bU16blpZG+/bt2bFjx0k/HhERQURExNnEaLAr+qSx9Oti3t2QT/bITn75miIiIpZqPxTi2jDDVsGGSN/77YyDh9kcEX50DYhBtjfGd52fNGjkwzRN7r77bt566y0+/fRTMjMzT/s5xcXF5OXlkZaWdtYhG8vYrDTsNoNNB1zsKao4/SeIiIgEO5ud3D6XkZsYD0DrujouqKomu9RFTomTGYnx5PYZ49f9PhpUPnJycnjppZeYPXs2sbGxFBYWUlhYSFWVbw6pvLyce++9l2XLlrFnzx4WLlzIuHHjSEpKYsKECU3yB2iIli3CGdrJN/Xy7oZ8i9OIiIj4hzepKxm1BgBXllfWT3tke2PISR+NN6mrX/MYpmmaZ3yxYZz0+VmzZjFx4kSqqqoYP34869ato7S0lLS0NC666CL+9Kc/nfFCUpfLRXx8PE6nk7i4uNN/QgO9tiqP+97cQPfUWD6cPOL0nyAiIhLkiot3cMk7E6gzDP4adwM/6Nyj0Xc4bcj7d4PWfJyup0RFRfHRRx815Lf0uzG9UnjgbYOthWXsPFRO5+QYqyOJiIg0qVcW/pU6w6BjNVx8ywMQ5v8t1b+t2R0slxAdzvAuSYCmXkREpHn4pGQlAN1sfYm0uHhAMywfAFf2aQPAuxsKTjuaIyIiEsy2bn+PnWEmDtNkRN9fWh0HaKbl49JeKYTbbew8VM62g2VWxxEREWky/1vlOzSuR2Uklw4YaHEan2ZZPuIiwxjRtTUA723QhmMiIhKaamsr+cy9B4AOkRcR4bB+ygWaafkAGNfXt++Ipl5ERCRUfb76SY7YbSR4vFx8Xo7Vceo12/IxukcKEQ4bu4sq2JzfNIfZiYiIWOnNbW8B0Lm8NSO6t7M4zTeabfmIiXBwcXffSbvvaupFRERCjLN0D0vxnT6fnnA94Y7AecsPnCQWOHbXyztf5mvqRUREQsp7y/9BrWHQwQ1jzr/B6jjHadbl4+LuybQIt3OgtIq1+0qtjiMiItJo5uZ/DkDrii71R4sEimZdPqLC7Vza03eE8DtfasMxEREJDbt2f8JXdg8O06Rd+k9x2APr7T6w0ljgqn6+qZf3Nhbg8WrqRUREgt/ba2YA0K0igivOO9/iNCdq9uVjWOfWxEeFcbjMzYpdxVbHEREROSeeuhrec20HIKJ6MAPbJVqc6ETNvnyEO2z8oHcqAPM09SIiIkFu+bqnOWw3iPd46dr1Z9hsJz+R3krNvnwAjOvrm3r5YFMhNXVei9OIiIicvTnbXgcgs6wlVw/oZHGak1P5AIZktiI5NgJnVS1Ldhy2Oo6IiMhZKXMd4LNa3xICb91l9GoTZ3Gik1P5AOw2gyv6+LZb110vIiISrD5a8Rhum0FGjcmArOswjMCbcgGVj3rHpl7mf3WQqhqPxWlEREQabs7+hQC0cnbg6v7p1oY5BZWPo/pnJJDRMorKGg+fbD1odRwREZEG2bt3CV/aarGZJtWOm+icHGt1pO+l8nGUYRiM+9Z26yIiIsFk3trpAHSvDOPifhdYnObUVD6+5djUy2fbDuOqrrU4jYiIyJnxeup4p2QzAKazP1ceXccYqFQ+vqV7aixdkmOoqfPy4aZCq+OIiIickVVfPkuB3SDG48WMu4mMltFWRzollY9vMQyDq49utz5vvaZeREQkOMzd8j8AOpcncGX/rhanOT2Vj++4up9vdfDSr4s45Kq2OI2IiMipVZQXsqDmEADFzlH1W0cEMpWP78hoGc2Adgl4TXhnQ4HVcURERE5p/orHqbYZtKkxSWozjqSYCKsjnZbKx0mMP3pv9Nz1ByxOIiIicmrz8j4FINnZngkD2lqc5syofJzED3qnYbcZbNjvZNfhcqvjiIiInNT+/ctZbbgxTJM95Vdwac9UqyOdEZWPk0iKiWBY5yQA5mrhqYiIBKh3Vj8BQLeqMPp1u4CYCIfFic6Mysf3GN//6F0vX+ZjmqbFaURERI7n9dQx98hGAGylWVx9dK+qYKDy8T0u7ZlKZJiN3UUVbNjvtDqOiIjIcdZufIkDdoj2etlTO54RXVtbHemMqXx8j5gIR/3cmaZeREQk0Mzb8hIAXcriGd27K+GO4HlLD56kFjg2hPXOhnw8Xk29iIhIYKisLOKjat9O3CWlwxjfL3BPsD0ZlY9TGNG1NQnRYRwuc7Ps62Kr44iIiADwyYppVNoMUmu9HHFcyqD2iVZHahCVj1MId9j4QW/fTnFztOeHiIgEiHn75gOQ5szg6v4Z2GyGxYkaRuXjNI4NZX24qZCqGo/FaUREpLkrLFjHCrMSgJ3OsfVnkgUTlY/TGNQ+kfSEKMrddXy85aDVcUREpJl7Z9U/MQ2DrpV24lr2p0danNWRGkzl4zRsNqN+z4856zT1IiIi1jG9XuYWrQMg3NWTCf2Da6HpMSofZ+DYf9xF2w9TXO62OI2IiDRXX371P/baIcrrZYPzyvqT2IONyscZ6JwcS+/0eOq8Ju/qpFsREbHIvE0vANC1PJZ+HTuSGh9pcaKzo/Jxho6NfrytqRcREbFAdVUJH1btB8BZOjTo9vb4NpWPMzSubxvsNoP1eaU66VZERPxu4aonKLMZJNd62VlzEZdnBccJtiej8nGGWsdGMLyL76TbOdpuXURE/GzO7vcBSHe14ZKebYmNDLM40dlT+WiAY1Mvc9Yd0Em3IiLiN4cObmKZWQHArtLLmdA/+Pb2+DaVjwa4tGcK0eF29h2pZO2+EqvjiIhIM/HuysfxGgadqwyqwvowvEvwnGB7MiofDRAd7qifY9PCUxER8QfT62Xe4dUARDl7Mq5vG8Lswf32HdzpLXBs6uXdDQXU1HktTiMiIqHuq21v87XdJMJrstF5BeODdGOxb1P5aKChnZJIiYugtLKWz7YdsjqOiIiEuDkb/gtAt4poWrfMoG/beIsTnTuVjway24z6HeXeWrvf4jQiIhLKatxlfFC5D4Dy0vOZ0D8dwwiuE2xPRuXjLFw7oC0An249RElFjcVpREQkVC1a9W+cNoOkOi8by0czYUDwT7mAysdZ6ZYaS682cdR6TN7ZoD0/RESkaczb9S4AGa5UzuuYTNvEaIsTNQ6Vj7N0bPTjzTWaehERkcZXVLSVJV4XAHtLL61/3wkFKh9n6ap+bXDYDL7c72TnoTKr44iISIh5f+XjeAyDTtUGh7z9GNs7zepIjUbl4ywlxUQwqptvk5e31mrPDxERaVzzDq4AoIWzK5dnpRIT4bA4UeNR+TgH1xwdAnt73QE8Xm23LiIijWPrtnlss3kJM002ll7JNSGy0PQYlY9zMLpHMnGRDgqc1SzfVWx1HBERCRFzNzwDQPfyKKJjMhjaKcniRI1L5eMcRDjsjOvrO9xHC09FRKQx1NZW8n75LgCqnIMZ3z8duy349/b4NpWPc3TtQN/UywebCqlw11mcRkREgt3nq2dyxGaQWOdlQ9mlXBtiUy6g8nHO+mckkJnUgqpaDx9sKrQ6joiIBLm5O98GoENZMr3atqZLSqzFiRpfg8rH1KlTGTx4MLGxsSQnJzN+/Hi2bdt23DWmafLwww/Tpk0boqKiGDVqFJs3b27U0IHEMIz6VqqpFxERORelJbtZ5HECsL90dEjt7fFtDSofixYtIicnh+XLl7NgwQLq6uoYM2YMFRUV9df87W9/4/HHH2f69OmsWrWK1NRULr30UsrKQncvjAkD2mIYsGxXMXlHKq2OIyIiQer9Ff+gzjDo4Ia82gH16wpDTYPKx4cffsjEiRPp1asXffv2ZdasWezbt481a9YAvlGPadOm8bvf/Y5rrrmGrKwsnn/+eSorK5k9e3aT/AECQXpCFEM7tQK054eIiJy9uQVfABDv7Mzo7im0bBFucaKmcU5rPpxO39BQy5YtAdi9ezeFhYWMGTOm/pqIiAhGjhzJ0qVLT/p7uN1uXC7XcY9gdP3ADADeWJuHV3t+iIhIA+3Y+SFf2Tw4TJNNpVdy/aDQnHKBcygfpmkyZcoUhg0bRlZWFgCFhb4FlykpKcddm5KSUv+x75o6dSrx8fH1j4yMjLONZKnLeqUSG+Eg70gVK3YfsTqOiIgEmXnrnwKge0UEjqj2jOza2uJETeesy8fdd9/Nhg0beOWVV074mGEcfz+yaZonPHfM/fffj9PprH/k5eWdbSRLRYXbubKvb9/9N7TwVEREGqCutpp3XTsAqHEO4poB6TjsoXtD6ln9ySZNmsS8efP47LPPaNv2m2Gh1NRUgBNGOQ4dOnTCaMgxERERxMXFHfcIVtcd3fPj/Y0FlGvPDxEROUPL1j1Nkd0g3uNlg2tM/ftJqGpQ+TBNk7vvvpu33nqLTz/9lMzMzOM+npmZSWpqKgsWLKh/rqamhkWLFjF06NDGSRzABrRLpOPRPT/e31BgdRwREQkSc7e/AUBHVxI926bQNQT39vi2BpWPnJwcXnrpJWbPnk1sbCyFhYUUFhZSVVUF+KZbJk+ezKOPPsrbb7/Npk2bmDhxItHR0dxyyy1N8gcIJIZh1O94qqkXERE5E07nPj6r860VLCi9mOsGBefax4ZoUPl48skncTqdjBo1irS0tPrHq6++Wn/Nfffdx+TJk7nrrrsYNGgQBw4cYP78+cTGhnaLO+baAW2xGbByzxH2FFWc/hNERKRZ+2j5Y9QYBu3cJnvrBnFVn9Dc2+PbGjztcrLHxIkT668xDIOHH36YgoICqqurWbRoUf3dMM1Banwkw7v4Vihr9ENERE5nbv4SABKdHRnTM4346DCLEzW90F1Ka6FjC4XeXLsfj/b8EBGR77F7z0I22GqxmyZbSsdxfTOYcgGVjyZxac8U4iIdFDirWfp1kdVxREQkQM1bMx2AbpXhhLXoyLDOSRYn8g+VjyYQGWbn6n6+w+ZeXRWc+5aIiEjT8tTV8I5zKwDe0v5cMyAdu+3ke2KFGpWPJnLD0aGz+ZsPUlJRY3EaEREJNCvWP8NBu0Gsx8uXZWPr3zeaA5WPJpKVHkePtDhqPF7mrtdhcyIicrx5214HoFNZS/p3SKdDUguLE/mPykcTMQyDG48eCvTq6v2YphaeioiIT3lZAZ/UHAbgUOkIbhzcfEY9QOWjSY3vn064w8aWAhebDgTnab0iItL45q94jGqbQdsak3zvMMZmpVkdya9UPppQQnQ4l/XynXfz6up9FqcREZFAMXf/ZwAkOdszrl9bosLtFifyL5WPJnbj0QVEc9fnU13rsTiNiIhYLS/vC9YaNdhMk62lV9S/TzQnKh9NbGinVrRNjKKsuo4PNumwORGR5m7e6mN7e4TRKimLPm3jLU7kfyofTcxmM7h+oK/VvrZK262LiDRnXk8d80o2AWA4+3DDoAwMo3ns7fFtKh9+cN2gthgGLNtVzN5iHTYnItJcrdnwPPl2aOH18lXFOCb0T7c6kiVUPvwgPSGqfsvc11dr9ENEpLmas2U2AF3K4hnRsyOJLcItTmQNlQ8/OXYP9+tr8qjzeC1OIyIi/lZZfogF7oMAFJWOaFY7mn6XyoefXNozhZYtwjnocrNo+2Gr44iIiJ99vHIaVTaDNjUmzrBLms0hciej8uEnEQ471xyd23tlpfb8EBFpbubumw9AsqsdNw5u32wOkTsZlQ8/uuk83xDbp1sPUeistjiNiIj4S37+alYabgzT5OvSsdwwuK3VkSyl8uFHnZNjOa9DS7wmvL46z+o4IiLiJ/NWTQOga5WDHp3OJy0+ytpAFlP58LNjox+vrs7D69VhcyIioc70eplX/CUADmcWN53XzuJE1lP58LMf9E4jLtLB/pIqPt9ZZHUcERFpYus2vkSeHaK9XvK913BRt9ZWR7KcyoefRYbZ6zeV0cJTEZHQN++rlwDoUhbHVQN74bDrrVevgAVuHuIbclvw1UEOl7ktTiMiIk2lqvIIH1XnA+B0Xli/51Nzp/Jhge6pcfTLSKDOa/LmWu14KiISqj5dOY1ym0FKrZeEtAlktIy2OlJAUPmwyM3HFp6uysM0tfBURCQUzd37IQBtXOncPKSjxWkCh8qHRa7s04YW4XZ2F1WwbFex1XFERKSRFRauZ7lZCcDh6qu4pEeKxYkCh8qHRVpEOLi6fuGp9vwQEQk1766ahmkYdK2yMaLfaMIdess9Rq+EhW45eq/3h5sKKCrXwlMRkVBher3MPbwGgHBnT27W3h7HUfmwUFZ6PH0zEqj1mLyxRgtPRURCxcYtr7PHDpFek6iWt9EhqYXVkQKKyofFfni0Dc9esU87noqIhIg5G58DoGt5C64/v5+lWQKRyofFruybRmykg31HKvnia+14KiIS7NzVTj6s8m0iaVYN45KeWmj6XSofFosOd3DtAN/phi8v146nIiLBbuGqJyiz2Whd52VAnzsI046mJ9ArEgBuObbj6ZaDHHRVW5xGRETOxVtfvwtAW1caN2lvj5NS+QgAXVNiGdwhEY/X5LVVuu1WRCRYHT60mRVUABAfcyNtE7Wj6cmofASIHw5pD/gOm/No4amISFCat+JxPIZB52qD8cOusjpOwFL5CBCXZ6WSGB1GvrOahdsOWR1HREQayPR6mXdoFQCtKnsxqluyxYkCl8pHgIgMs3PdQN/C09krtPBURCTYbNk+j10Ok3CvSe/ud2G3GVZHClgqHwHklqNTL59uO0TekUqL04iISEO8tuYpALpXRHPThYMtThPYVD4CSGZSC4Z1TsI0fWs/REQkONS6K/jY7fu+nR4xmpS4SIsTBTaVjwDzo/N9ox+vrsrDXeexOI2IiJyJBSv+hdNuo1Wdl6tGTLI6TsBT+Qgwl/RIJi0+kuKKGj7YWGh1HBEROQNv75gLQNeqNlzYNc3iNIFP5SPAOOy2+tNuX1y+1+I0IiJyOsXFO1ht9+3tMTDzdgxDC01PR+UjAN14XgYOm8GavSVszndaHUdERE7G64HdS3j93V9SZxh0rIabR99odaqgoPIRgJJjI7k8KxWAl3Tei4hIQJm5fia5H0+GaVnw/JV86t4FwLUVFcz+/DfMXD/T2oBBQOUjQN12QQcA5qw7gLOq1towIiJSz1a0nRkHPiHXVs62sDC2RITjME2O2D3MOPAJtqLtVkcMeCofAWpwh0S6pcRSVevhrbX7rY4jIiIAXg/ZGz4ip6SUGYkJ/DkpEYD2tbX8NyGOnBIn2Rvm+6Zk5HupfAQowzD40QW+225fXL4X09R5LyIiltu7FFz5ZJe6+EVJKesjfft5fB0eTk5JKdmlTnAd8F0n30vlI4BN6J9OTISDXYcrWPZ1sdVxRESk/GD9TzvW1tX/PMw0yS51nfQ6OZHKRwCLiXBwzYB0AJ5ftsfaMCIiAjEpAJjAX1smAGAzTWoNg9yEuBOuk5NT+Qhwtx7d8XTBVwfZX6LzXkRELNV+KMS14cGklhQ5HDi8XhbtO1C/BiQ3IR7i0n3XyfdS+QhwXVJiubBzK7ymbrsVEbGczU5un8uYFxsDwE1l5SR4vWSXusgpcTIjMZ7cPmPAZrc4aGBT+QgCtx+97fZ/q/ZRXasV1CIiVsqr821/4DBNbnOW1T+f7Y0hJ3003qSuVkULGg6rA8jpje6RQtvEKPaXVDFvfT43DM6wOpKISLNVdGAtOKBfZSypP5oD5Yd8azzaDyVbIx5nRCMfQcBuM+rXfjy3dI9uuxURsciu3YtYZveNdgzPvBsjcwT0vg4yh2uqpQFUPoLEjYMziAyz8VWBi9V7S6yOIyLSLOUufgTTMMiqDOMmneNy1lQ+gkRCdDjj+/luu31u6R5rw4iINEOHD23mY9O3f0dW/E1Eh2vlwtlS+Qgitw/tAMCHmwopdFZbG0ZEpJl55tMHqTUMOlcZ3HrZ3VbHCWoNLh+LFy9m3LhxtGnTBsMwmDNnznEfnzhxIoZhHPc4//zzGytvs9YjLY7zMlvi8Zq8vGKv1XFERJqNMtcB5lb5DozrZLuUdq2iLU4U3BpcPioqKujbty/Tp0//3msuv/xyCgoK6h/vv//+OYWUb0w8Ovoxe8U+3HW67VZExB9mf/Y7Kmw22taYjBt5r9Vxgl6DJ6zGjh3L2LFjT3lNREQEqampZx1Kvt+YnimkxUdS4KzmnS8LuG5gW6sjiYiENHe1k1eKV4PdIKNyICO66v3tXDXJmo+FCxeSnJxM165dufPOOzl06ND3Xut2u3G5XMc95Ps57DZuO7rp2Kwvduu2WxGRJjZv8R8pthu0rvMyfND/YRiG1ZGCXqOXj7Fjx/Lyyy/z6aef8thjj7Fq1Souvvhi3G73Sa+fOnUq8fHx9Y+MDG2gdTo3n+e77XZzvouVu49YHUdEJGR56mp4Nm8BAO2d3bh2cBeLE4WGRi8fN954I1dccQVZWVmMGzeODz74gO3bt/Pee++d9Pr7778fp9NZ/8jLy2vsSCEnITqcawf4plue/WK3xWlERELXJ8v+xn4HxHq8dOtyr26vbSRNfqttWloa7du3Z8eOHSf9eEREBHFxccc95PTuuLADAPO/Osi+Yp12KyLS2Eyvl2d2vAFAF2c6tw3vb3Gi0NHk5aO4uJi8vDzS0tKa+ks1K52TYxnZtTWmCc8v22N1HBGRkLNi3X/YYvcQ4TVp2WoSbRKirI4UMhpcPsrLy1m/fj3r168HYPfu3axfv559+/ZRXl7Ovffey7Jly9izZw8LFy5k3LhxJCUlMWHChMbO3uwdG/14dVUeZdW11oYREQkxz2x8BoAerkR+NHKYxWlCS4PLx+rVq+nfvz/9+/uGn6ZMmUL//v35wx/+gN1uZ+PGjVx99dV07dqV22+/na5du7Js2TJiY2MbPXxzN6JLazq1bkG5u4431uy3Oo6ISMj4auscVhjV2E0TbBMZ0C7B6kghpcErZ0aNGnXK2zs/+uijcwokZ85mM7jjwkwenLOJ55bu4bYLOmC36RYwEZFz9ezqxwHoVd6C64ZfrttrG5nOdgly1wxIJz4qjL3FlXy29fv3UxERkTOTl/cFC+p82xhUVYznB721ZrGxqXwEuehwBzef1w6AZz7fZXEaEZHgN+uLP+E1DHpWOBh93vWE2fVW2dj0ioaA24e2x2EzWL7rCJsOOK2OIyIStIqKtjK32reGrqbkEm45+o87aVwqHyEgLT6KK/r4hgWfWaLRDxGRszV78e+pMQw6VRv07PEjEluEWx0pJKl8hIg7h3cE4N0NBRQ4qyxOIyISfMrLCnjFuQUAR/EQfjqis8WJQpfKR4jISo/n/I4tqfOaPPfFHqvjiIgEnTcW/Z5ym0F6jUli+k/ITGphdaSQpfIRQo6NfsxeuY9yd53FaUREgkeNu4wXDi0HIPFIb+4c0dXiRKFN5SOEXNQtmY6tW1BWXcerq3RAn4jImXp3yR85bDdIqvPijv4xgzskWh0ppKl8hBCbzeCnw3yjH89+vps6j9fiRCIigc/rqWPWvvkApJd04o4RvbSpWBNT+Qgx1wxIp1WLcA6UVvHh5kKr44iIBLzPlv+DPXaTGI+XgrqJjM1KtTpSyFP5CDGRYXZ+dH57AP6zZPcpt8IXEWnuTK+XZ7a/CkAXZxtuvrAfDm0q1uT0CoegWy9oT7jDxpd5pazaU2J1HBGRgLX6y1lsstUR4TXZ7bqFGwdnWB2pWVD5CEFJMRFcO6AtAE8v/triNCIigeu/G54GoIcrgbGDzyc2MsziRM2DykeIunN4JoYBH285xI6DZVbHEREJONu2v8sXVGIzTfaVXMuPL8y0OlKzofIRojq2juGynr5FU08v1pbrIiLf9eyqfwCQVR7Neb0uJjU+0uJEzYfKRwj7+Ujfbbdz1h+g0FltcRoRkcCxf/9yPqwtAuBQ0RX8bERHixM1LyofIax/u0TOy2xJrcdk1he7rY4jIhIwnv/iEbyGQY9KB+0zf0CXlFirIzUrKh8hLvvo6MfLK/bhqq61OI2IiPWKi7bzduU+AKqKLuLnIztZnKj5UfkIcaO6JtM1JYZydx2zV+yzOo6IiOVmL/k9bptBx2pwtJzAoPbaSt3fVD5CnM1m8LMRvlb/7Oe7cdd5LE4kImKdyvJD/K90MwBhxUPIHtVFW6lbQOWjGbiqbxtS4yI5VOZm7rp8q+OIiFjmjcUP4rIZtKkxKQm7kUt7pFgdqVlS+WgGwh02fjLMd/967uKv8Xi15bqIND+17gpeKFwKQKsjWdw5shs2m0Y9rKDy0UzcPKQdcZEOdh2uYL4OnBORZui9zx/hoN2gVZ2XPbU/ZHz/dKsjNVsqH81ETISDiUM7ADBj4U4dOCcizYrXU8esvR8AkFGSyR0jehEZZrc4VfOl8tGMTLwwk6gwO5sOuFiyo8jqOCIifrNo5TR22U1aeL18XfFDbh7SzupIzZrKRzPSskU4N5/n+x9uxmc7LU4jIuIfptfLf7fOBqBraSrXDx1ITITD4lTNm8pHM3PniEzC7AYrdh9hzd4jVscREWlyaze+yJe2WsK9JrtcN9dPQYt1VD6ambT4KCYcXWQ187OvLU4jItL0nv0yF4CeZfFcNnAoiS3CLU4kKh/NUPbIThgGfLL1EFsKXFbHERFpMtt3fsBisxybabLvyLXcOVwHyAUClY9mqGPrGH6QlQbAkws1+iEioWvWir8B0Ks8igt6X0JqfKTFiQRUPpqtX4zybbn+7oZ89hZXWJxGRKTx5eev5oPawwAUF4/VAXIBROWjmcpKj2dk19Z4TY1+iEhoeuHzP+IxDLpV2unSZTyZSS2sjiRHqXw0Y5Mu7gzAm2v3c6C0yuI0IiKNp+TI17xZsRuAmuKR3H30+50EBpWPZmxQh5Zc0LEVtR6TXI1+iEgIeWXRg1TbDDq4ISXjh3RPjbM6knyLykczN2m0718Dr67O46Cr2uI0IiLnrrKyiNmlGwGIKh7MPZd0sziRfJfKRzN3QcdWDGqfSE2dl6cW7bI6jojIOXt70e9x2gzSak1iUn9KVnq81ZHkO1Q+mjnDMJg0ugsAs1fupajcbXEiEZGzV1tbyXMFSwBIOtKTSaN7WJxITkblQxjRJYm+beOprvXynyUa/RCR4PXh53+h0G6QWOclvGU2/dslWh1JTkLlQ3yjHxf7Rj9eXLaXkooaixOJiDSc6fXy7O53AGhf0oG7RvezNpB8L5UPAWB0j2R6psVRWePh2S92Wx1HRKTBlqz6FzvtJtFeL7T4OedltrQ6knwPlQ8Bjo1++O58ee6LPTgray1OJCLSMP/56kUAupWmkH3pUIvTyKmofEi9y3ql0j01ljJ3Hf/9XGs/RCR4rN/4MutttYSZJkb4TxjaqZXVkeQUVD6kns1mcM/RO1+e/WIPpZVa+yEiweE/a6cD0NMVx88uuwzDMCxOJKei8iHHOTb6Ue6u45klWvshIoHv668XsJhyDNMk0v4jjXoEAZUPOY7NZjD5kq4AzPpit+58EZGA959ljwLQqyKSH192g0Y9goDKh5xgTM8UeqTFUVHj0b4fIhLQCgvW8VHdYQDiuZYLNOoRFFQ+5AS+0Q/f2o/nl+7hiEY/RCRA/XfRH6gzDLpW2rn98p9p1CNIqHzISY3pmUKvNhr9EJHA5Szdw9wq39q0NuYYjXoEEZUPOSnD+Gbtx/NL91CsM19EJMA898n9VNkMOrjhljG/sTqONIDKh3yvS3ok0zs9nsoaD08t1uiHiASOqsojvOHcAEDH2qFc0Lm1xYmkIVQ+5HsZhsGvLvWt/Xhh2R4OuaotTiQi4vPyxw9QareRUuvl5ssftjqONJDKh5zSRd2SGdAugepaL9M/22l1HBER6mqrefXw5wBk1fbj/E5pFieShlL5kFMyDIN7L+sGwCsr95F3pNLiRCLSbHk9sHsJb71xO4UOgwSPl9vGTLU6lZwFlQ85raGdkriwcytqPSZPfLLD6jgi0szMXD+T3I8nw7QszOev5LWy9QDc6Kpi5e7Hmbl+pqX5pOFUPuSM/HqMb/TjzbX72XW43OI0ItKc2Iq2M+PAJ+TayvkiKpJtEeFEe714jDpmHPgEW9F2qyNKA6l8yBkZ0C6R0d2T8Zrwz481+iEifuL1kL3hI3JKSpmRmMCfWiUC0LWmhmcS4skpcZK9Yb5vSkaCRoPLx+LFixk3bhxt2rTBMAzmzJlz3MdN0+Thhx+mTZs2REVFMWrUKDZv3txYecVCU8b49v1458t8thS4LE4jIs3C3qXgyie71MU1rnLyw8LANFkfGUlOSSnZpU5wHfBdJ0GjweWjoqKCvn37Mn369JN+/G9/+xuPP/4406dPZ9WqVaSmpnLppZdSVlZ2zmHFWr3axHNFH9+q8sfma5hTRPyg/CAAHmBLRLjvOcMgzDTJLnWdcJ0EhwaXj7Fjx/LnP/+Za6655oSPmabJtGnT+N3vfsc111xDVlYWzz//PJWVlcyePbtRAou1fnVJV2wGfLzlIGv2llgdR0RCXUwKAHNiWtSXjzDTpNYwyE2IO+E6CQ6NuuZj9+7dFBYWMmbMmPrnIiIiGDlyJEuXnnxIzO1243K5jntI4OqcHMN1A9sC8NcPt2KapsWJRCSktR+KMy6N/3d0rceFlVWs3ZNXvwYkNyEe4tKh/VCLg0pDNGr5KCwsBCAl5fgGmpKSUv+x75o6dSrx8fH1j4yMjMaMJE1g8iVdCXfYWLn7CAu3HbY6joiEMpudOxNiqLbZSKzz8O+Dvu852aUuckqczEiMJ7fPGLDZLQ4qDdEkd7t890hj0zS/95jj+++/H6fTWf/Iy8trikjSiNokRDFxaAfAN/rh9Wr0Q0SaxradC9hq8x3t8Mcj1YR962PZ3hhy0kfjTepqTTg5a47G/M1SU1MB3whIWto3290eOnTohNGQYyIiIoiIiGjMGOIHd43qxCsr97G1sIy5Xx5gQv+2VkcSkRBjer08uuh+TIdB78oIhkxaDQUrfYtLY1Kg/VCyNeIRlBp15CMzM5PU1FQWLFhQ/1xNTQ2LFi1i6FDNx4WShOhwfjGqE+C788Vdp3vsRaRxzV/6D9Y63ISZJqM7/Y7oyAjIHA69r/P9qOIRtBpcPsrLy1m/fj3r168HfItM169fz759+zAMg8mTJ/Poo4/y9ttvs2nTJiZOnEh0dDS33HJLY2cXi90xNJOUuAj2l1Qxe8U+q+OISAipcZfxz20vADCwLI3bRl9tcSJpTA0uH6tXr6Z///70798fgClTptC/f3/+8Ic/AHDfffcxefJk7rrrLgYNGsSBAweYP38+sbGxjZtcLBcVbuee0b651n9/upOy6lqLE4lIqPjv+zkccBi0qvPyg/MeI8yuDblDiWEG2L2SLpeL+Ph4nE4ncXFxp/8EsVSdx8uYfy5mV1EFv7y4M1OOngEjInK2Dh3cxJXv30SVzWC4azDTc/6LzXbymxYkcDTk/VtVUs6Jw27jN5f5Csd/luzmoKva4kQiEuz+Pv9uqmwGnaoNbhrzqIpHCFL5kHN2eVYqA9snUlXr4XFtuy4i52D9ptl86C0GoB0/YkS3VIsTSVNQ+ZBzZhgGD/ygBwCvrcnToXMicla8njoeXfk3APq5Ysm5+hcWJ5KmovIhjWJg+0Su6J2GacLUD7ZaHUdEgtCcz+5ni91DC6+X9kn/R7dU3agQqlQ+pNHcd3k3wuwGi7cfZvF2bbsuImeuzHWAf+37AIBuRzrzyyvGnOYzJJipfEijad+qBbee3wGAR9/fgkfbrovIGXryo7s4YjdIrzHp2+sRkuMirY4kTUjlQxrVpIs7ExfpYGthGW+u3W91HBEJArv3LOSViq8BSCodzc8v6mVxImlqKh/SqBJbhDPp4i4APDZ/G5U1dRYnEpFA99dFv6XOMOhZEcaVI39Ni4hGPXZMApDKhzS624a2J6NlFAddbnIX7bI6jogEsMUr/skXVOIwTQz3T7h+UIbVkcQPVD6k0UU47Nw/1nfr7VOLvuZAaZXFiUQkENW4y/jr5mcB6F2STM6VN2DXhmLNgsqHNImxWamcl9kSd52Xv+rWWxE5iZfm38M+OyTWeYlO+A3Du7S2OpL4icqHNAnDMPjDlT0xDJj3ZT5r9h6xOpKIBJDDhzbzVNFKANoW9eO+K4dbnEj8SeVDmkxWejw3DPTN3z7yzld4deutiBw1bcEvqbQZdKw26NHjt3RsHWN1JPEjlQ9pUvde1o2YCAdf7nfy9roDVscRkQCwYfOrzKs7BICjZDy/vLSHxYnE31Q+pEm1jo0g56LOAPzto61UuHXrrUhz5vXUMXXl/wOgr6sFE0b9hPioMItTib+pfEiT+/GwDrRrGc1Bl5snF35tdRwRsdA7ix5kk62OaK8Xd10ONw3WrbXNkcqHNLkIh73+1Nunl+xib3GFxYlExAoV5YVM2/MuAF2LM5k8biwOu96GmiP9Vxe/uKxXCsO7JFFT5+VP735ldRwRscBTH95Fkd0grcZLfPJ9XNg5yepIYhGVD/ELwzB4aFwvHDaDj7cc4tOtB62OJCJ+tHfvEl4s3w5AQtFI7r9qsMWJxEoqH+I3nZNj+PGwTAD++M5XVNd6LE4kIv7yt4X3UWcY9KhwMGzQPaQnRFkdSSyk8iF+NeniziTHRrC3uJL/fr7b6jgi4gdLVj7BYspxmCaeion8bFRnqyOJxVQ+xK9iI8PqF59O/3Qn+Tr3RSSk1bor+Oum/wDQu7Q1d115E5FhdotTidVUPsTvru7XhsEdEqmq9fCX97dYHUdEmtDLC+5h79HzW6LifsPoHilWR5IAoPIhfmcYBn+8KgubAe9tKODzHUVWRxKRJlBUtJWnDi8DIKO4D/dfNdLiRBIoVD7EEj3bxHHr+e0B+P3cTVp8KhKCps2/m3Kbjcxq6Nf7QToktbA6kgQIlQ+xzK8v60br2Ah2F1Xw1KJdVscRkUa0ecubzKspBCCmbAKTLulucSIJJCofYpm4yDB+f2VPAGYs3MmeIu18KhIKTK+Xvyz7M6Zh0McVzZ1X3q1FpnIclQ+x1Lg+aQzr7Nv59PdzN2GaptWRROQcvbPo92y01xHl9dI69tdc1D3Z6kgSYFQ+xFKGYfDI1b0It9tYsqOI9zYWWB1JRM5BRXkh/9w9F4CeJR347YRxFieSQKTyIZbr2DqGX4zqBMAj73xFWXWtxYlE5Gw9+f7R81tqTS7s/yhp8drJVE6k8iEB4RejOtGhVTSHytw8Nn+71XFE5Czs2/c5syt9//92qLiYO0b0tjiRBCqVDwkIkWF2Hrk6C4Dnl+1h3b4SixOJSEM9+sm91BoGPSod/Oyqh3DY9RYjJ6e/GRIwRnRtzYT+6Zgm3P/WRmo9XqsjicgZWrR8Ol/YKnCYJr0SJjEos5XVkSSAqXxIQHnwih4kRoextbCMpxdr7w+RYFBbW8nfNz0FQP+y1kyZcKvFiSTQqXxIQGkVE8Efxvn2/vjXJzvYdbjc4kQiclJeD+xeAhvf4L+v3cLeMEjweLl26D+JjQyzOp0EOIfVAUS+a3y/dN5el8/i7Ye5/62NvHLn+dhshtWxRASYuX4mtqLtZG/4CFz5HLHZeKFtG7DbyKhLYK99KdDP6pgS4DTyIQHHMAz+Mj6LqDA7K3Yf4dXVeVZHEpGjbEXbmXHgE3JtvlHJJxITKLPbSKqrY2OEC1uR7laT01P5kICU0TKaX4/pCsCj72/hkKva4kQigtdD9oaPyCkpZUZiAn9qlchbsb7D4oocDnJKnGRvmO+bkhE5BZUPCVh3XJhJ37bxlFXX8cDb2npdxHJ7l4Irn+xSF3eVlPJaXCym4ZsSzSkpJbvUCa4DvutETkHlQwKW3Wbw1+v6EGY3+HjLQeZ9mW91JJHmrfxg/U+jvd/8YyDMNMkudZ30OpGTUfmQgNY9NY5fXtwFgIfmbeZQmaZfRCwTkwLA5vAwHm+ZAIDdNKk1DHIT4k64TuT7qHxIwMse1YlebeIorazlQU2/iFin/VDK49L4eWoyXsOgY00N6/bk1a8ByU2Ih7h0aD/U6qQS4FQ+JOCF2W384/q+OGwG8786yDsbdPKtiCVsdn4UH4XTbifG4+GFgkMYQHapi5wSJzMS48ntMwZsdquTSoBT+ZCg0CMtjknHpl/mbuJwmdviRCLNz/8++i1fO+owTJPHit3Ee785AiHbG0NO+mi8SV0tTCjBQpuMSdC466JOfLi5kC0FLn4/ZxNP/mgAhqHNx0T8Ye/eL/hn/rtgszG8sgPn/2oO5C3zLS6NSYH2Q8nWiIecIY18SNDwTb/0wWEz+HBzIW+vO2B1JJFmodZdwa8/votKm42uVTZ+de2L2BwOyBwOva/z/ajiIQ2g8iFBpVebeO4ZfWz6ZTMHSqssTiQS+v7+9i1sc3iJ9XgZ23EqnVMSrY4kQU7lQ4LOL0Z1on+7BMrcddz72pd4vbr7RaSpLF75b15x+06YPs99CT++ZKzFiSQUqHxI0HHYbTx+Qz+iwuws21XMrKV7rI4kEpKKirby4KZcAAa6Evm/W/6qQx6lUah8SFDKTGrB767oAcBfP9zK9oNlFicSCS1eTx3/985tlNhttHObXH3BU6TGR1odS0KEyocErR8Oaceobq2pqfPyq1fXU1PnPf0nicgZmfX+z1lhqyLCa9IrfBITBvWwOpKEEJUPCVqGYfC3a/uQEB3G5nwX//xYR3mLNIZNX73O9OIVAPQtyeKBG+6wOJGEGpUPCWrJcZFMndAbgNxFX7N0Z5HFiUSCW0V5Ib9Z9gh1hkFWeQS3X/EvEqLDrY4lIUblQ4Le2N5p3DQ4A9OEX722niMVNVZHEglaf3z7JvY7ILnWS5/0vzGimw6Jk8an8iEh4Q/jetKpdQsOutzc98YGHT4nchbmfvoAH3iLsZsmmZXX8+srR1kdSUKUyoeEhOhwB0/c3J9wu42PtxzkpeV7rY4kElT27l3Co3vnAtDvSAYP3Hwv4Q69RUjTaPS/WQ8//DCGYRz3SE1NbewvI3KCXm3i+e3Y7gD8+b0tbCvU7bciZ6LWXcG9n0yq3z597LDpdGwdY3UsCWFNUmt79epFQUFB/WPjxo1N8WVETvDjCzswqltr3HVeJr2ylqoaj9WRRALe43NvYavdQ5zHS4eoB7hhcEerI0mIa5Ly4XA4SE1NrX+0bt26Kb6MyAkMw+Af1/clKSaC7QfLeWjeJqsjiQS0xSv+xUtVvu3Tu5RcxB9uuEanRUuTa5LysWPHDtq0aUNmZiY33XQTu3bt+t5r3W43LpfruIfIuUiKieCJm/phM+C11ft5c81+qyOJBKSiw1t4cPPTAPQvTWDyDX8mPirM4lTSHDR6+RgyZAgvvPACH330Ef/5z38oLCxk6NChFBcXn/T6qVOnEh8fX//IyMho7EjSDA3tnMQ9o7sC8OCcTezQ9usix/F66rjvXd/26e3dMLLPDPplJFgdS5oJw2ziexIrKiro1KkT9913H1OmTDnh4263G7fbXf9rl8tFRkYGTqeTuLi4powmIc7jNbn92ZV8vrOIzskxzLv7QqLDHVbHEgkIT82dyPTSNUR4TUZ4J/HYj3+m6RY5Jy6Xi/j4+DN6/27y+6hatGhB79692bFjx0k/HhERQVxc3HEPkcZgtxlMu6kfybER7DxUzoNvb9L+HyLAhs2vkVuyGoB+zr48dMuPVTzEr5q8fLjdbrZs2UJaWlpTfymREyTFRPDvm/tjM+CtdQd4dVWe1ZFELFVeVsBvlvu2T+9dHsk9107XOg/xu0YvH/feey+LFi1i9+7drFixguuuuw6Xy8Xtt9/e2F9K5IwM6diKX4/pBsAf5m5mfV6ptYFELGJ6vTz45o3kOwySa72M6fkEvTMSrY4lzVCjl4/9+/dz8803061bN6655hrCw8NZvnw57du3b+wvJXLGfjGyE5f2TKHG4+UXL62hqNx9+k8SCTGvzP8tnxgl2E2TIfbbuH3U+VZHkmaqyRecNlRDFqyINERZdS1XT/+CXUUVnN+xJS/9ZAgOu7aPlhDm9cDepVB+kB1lBfxwWy5VNhvDXJn8M/ttIsPsVieUENKQ928t/ZdmIzYyjKduHcj4GV+wfNcR/vrhVn53RU+rY4k0upnrZ2Ir2k72ho/AlU8N8Ls2qVRFhJNca5IxYJSKh1hK/+yTZqVLSiz/uL4vAP9Zspt3vsy3OJFI47MVbWfGgU/ItZUDMK1lAlsiwon0ejkUZtCyap/FCaW5U/mQZmds7zR+MaoTAPe9sYHN+U6LE4k0Iq+H7A0fkVNSyozEBO5PasmL8b4h8GqbjZwSJ9kb5vumZEQsovIhzdK9Y7oxvEsSVbUe7nx+NYfLtABVQsTepeDKJ7vUxW1OF+/GfnM6bU5JKdmlTnAd8F0nYhGVD2mW7DaD6TcPoGNSC/Kd1fz8xdVU1+pfghICyg8CcMRmY3FUVP3TYaZJdqnrhOtErKDyIc1WfHQYz9w+iLhIB2v3lfLAWxu1A6oEv5gUKg2DnNTW7An3bR4WZprUGga5CXHHXSdiFZUPadY6to5h5g8HYrcZvLXuAE8t/v4TmEWCQW1qX+5JSWZTRAQAtzhdrN2TV78GJDchHuLSof1Qi5NKc6byIc3esC5JPDTOd8vtXz/cyoKvNBwtwcnrqeNXr/yA5VG+4nGdq4z7j5QCkF3qIqfEyYzEeHL7jAGbbrUV66h8iAC3XdCBH53fDtOEX76yjg37S62OJNIgptfLQ6+MY5HdiWGaDPGk8VBt1HHXZHtjyEkfjTepq0UpRXy0yZjIUQ+N68W+I1Us3n6YHz+3irfvupCMltFWxxI5I0+8dStzPPsBuKxmGH/96ZOAt36HU2JSoP1QsjXiIQFAIx8iR4XZbcz84QB6psVRVF7DxFkrKa2ssTqWyGm99P5knqnYAMDI8u786Y4Z2GyGb2olczj0vs73o4qHBAiVD5FviYlwMOuOwaTFR/L14Qp+9uIa3HW6BVcC14dLpvL3Qx8DcL4rlUcmvqyt0yXgqXyIfEdKXCSz7hhMbISDlbuPcO/rG/B6dQuuBJ4V657ldztfxmsYDCiL5YEb36Jli3CrY4mclsqHyEl0T40j99aBOGwG73yZz5/e+0p7gEhA2bbjPSavf4wam0FWRTh3X/EWmcmxVscSOSMqHyLf48LOSfWH0M36Yg/TP91pcSIRn7z9y/j5kvsot9noUmXj1qGzGZyZanUskTOm8iFyCuP7p9fvAfLYgu28uHyvxYmkuSsu2s7PPrqTYruNDLfJdb2e4gf9ulkdS6RBVD5ETuOOCzP55cWdAfjD3E2882W+xYmkuaooL+Rnc69nv8MgudbLVe3+wS3Dz7c6lkiDqXyInIFfXdq1fhOyKa+tZ9H2w1ZHkmamxl3GXa9eyXaHl3iPlzEJD5B9+eVWxxI5KyofImfAMAz+eFUWV/ZJo9Zj8rMXVrP06yKrY0kz4amrYcrsy1nrcBPl9XKx46fcd+0tVscSOWsqHyJnyG4zePyGflzcPRl3nZefPLealbuPWB1LQo3XA7uXwMY3YPcSzLpafj97HItsLhymyUWeCTz0o8kYhmF1UpGzZpgBdv+gy+UiPj4ep9NJXFzc6T9BxM+qaz3c+cJqluwookW4nRd/OoQB7RKtjiVBbub6mdiKtpO94SNwfbOu6MmWScyMjwbT5HL3CB796XTC7Pp3owSehrx/62+wSANFhtn5z22DGNqpFRU1Hm5/dqUOopNzZivazowDn5BrK69/7vXYFr7iAfSqieUvP1HxkNCgv8UiZyEyzM4ztw/ivA4tKauu49b/rmTjfqfVsSRYeT1kb/iInJJSZiQmkJsQx8fRUfypVUsABlZV80q5k3BbQA1Ui5w1lQ+RsxQd7uDZOwYzoF0CzqpabnlmOWv2llgdS4LR3qXgyie71FVfQH6VnIRpGPSsdjOr8BCG64DvOpEQoPIhcg5iIhw8/+PzvjUCsoJlXxdbHUuCTfnB+p+2r60D0wTDwDBNXi44iHGS60SCmcqHyDmKjQzjuR8PZljnJCprPEyctVL7gEjDxKRgArPiY7kvOam+eJiGwTMJccddJxIKVD5EGkF0uINnbh9Ufxvunc+vZsFX+leqnBlP+iD+ktSax1v67prqU13Nuj1531oDEg9x6dB+qMVJRRqHyodII4kMs5P7o4GMzUqlxuMl+6U1vLY6z+pYEuCqKo+Q/eIoXo2NAuDCyipeLjiEHY6uAXEyIzGe3D5jwGa3NqxII1H5EGlE4Q4b/765P9cOaIvHa3LfGxuYuXAnAbadjgSI4uId3Db7IpY7KrGZJgNrW/FkVdhx12R7Y8hJH403qatFKUUan8PqACKhxmG38Y/r+9A6NoLcRV/ztw+3ccjl5g9X9sRm066U4rNrzxLu+uQXHAgziPN4uS7ux0y+dgqG6fXd1VJ+0LfGo/1QsjXiISFG5UOkCRiGwf+N7U7r2Aj+9O5XPLd0D0Xlbh67oS8RDr2RNHervnyZKWsepdRhI6XWy63tHuL2MTf4PmjYIXO4tQFFmpjKh0gT+smwTFrHRvDr19bz7oYCDrnc5N46kJYtwq2OJhaZ99lfeGTPK7jtNjLdcGefpxh3/jCrY4n4ldZ8iDSxq/q2YdbE84iNcLByzxHGz/iCnYfKrI4lTe07B8Th9fDk2z/jwb2v4LYZ9KoM4/6Rb6p4SLOkg+VE/GTHwTJ+/Pwq8o5UERvpYOYPBzC8S2urY0kjO9kBcV7gsVZJvBDnO6flvPIEHrxuLpkpLS1MKtK4dLCcSADqkhLLnLsuZHCHRMqq65g4axUvLtujO2FCzHcPiHMb8NvWreqLR3d3FI/f8bGKhzRrWvMh4ketYiJ46adDuP+tjby19gC/n7uZDfud/Gl8FpFhWoga9I4eEIetnBmJCbgNg3UREayJigTgkvIKHnd7MCL1rVeaN418iPhZhMPOY9f35f/GdsdmwOtr9nNd7lLyjlRaHU3O1bcOiPuh08UzCfH1xePqsnL+ebhYB8SJoPIhYgnDMMge2YkXfzKEli3C2XTAxbjpn7Nw2yGro8m5KD+ICbwe24K3YmPqn3aYJn8uOnLcdSLNmcqHiIUu7JzEO5OG0bdtPKWVtdzx3Coen7+NOo/X6mhyFgrr3PwyOYlHklpRZfN9ew0zTeoMg1wdECdST+VDxGLpCVG8ln0BN5/XDtOEJz7dyY1PL9c0TJCZ8+lfuWHdoyxsEY3t6CLiu0pKWasD4kROoPIhEgAiHHamXtObJ27uT2yEgzV7S/jBE0t458t8q6PJaVRWFvObWZfw+7yXKLHbSKjz4jUMckqc/KLUBeiAOJHv0pJrkQByVd829M9I4J7/rWPtvlImvbKORdsP84dxPYmLDDv9byBNx+s54cyVlRvf4uFVfyQvzHdmz7DKFNr1GUNiVT7ZxR8BzvpPz/bGgA6IEwG0yZhIQKrzeHnikx1M/2wnXhPS4iN59JreXNQt2epozc7JNg3zAM8kJjEzPgqvYdCqzstVsbdwz3UPYD92eOBJyopGPCSUNeT9WyMfIgHIYbcxZUw3hndtzW9e/5I9xZXcMWsV1w1sy++v6El8tEZB/OXYpmHYyskGDjjsPNC6FWsjfbfQtq0xeGjY/zi/R+/vfKIOiBP5Phr5EAlwVTUe/jF/G89+sRvThOTYCB65OovLeqVgGIbV8UKb1wPTssg9umnY6IpKlkdFUnH0TpbRFZU8Xh2BbfJGjWpIs6ft1UVCSFS4nd9f2ZPXf34BHZNacKjMTfZLa7jjuVXsKaqwOl5oO7pp2LDKatJqa/mkRXR98bjV6WLaoSJs2jRMpMFUPkSCxKAOLXn/nuHkXNSJMLvBwm2HGfPPxTw+fxvVtR6r4wWvk5w+e8yevcu4v3Urbk5PpSAsDI4OFIeZJvcdKf3m99CmYSINomkXkSC063A5D83bzJIdRQC0TYziN5d1Y1yfNthsmoo5EydbSApAXBue6DGKJYc2s9tbjPvoSEc3dw3bIsIJM01qDYOcklKyj95Ky+3van2HNHuadhEJcR1bx/DCj8/jyR8OIC0+kv0lVdzzv/VcNeNzPj9aSOTUvnv6LPjuYvllZA3/OfQ5WynBbbPRv9rN9a4ytkWEk6NNw0Qahe52EQlShmEwtncaI7u15tnPd5O7aBebDrj40X9XMLxLEvdd1p3ebeOtjhkYvnvba8aQ406fBehX7eb/Wrei2OH7tphRW8foFuOITK0mt/Azckqc9SMdvh8NZiTGQ/posrXYVKRBNO0iEiKKy91M/2wnLy3fS63H97/1yK6tufvizgzu0NLidNb43qmV6CRyw2vwYLDfYefdbx0CF+718ssSJze7ygi//V1mOjd+z/RMOrl9xuBN6spd/e7y459KJDA15P1b5UMkxOwrrmTax9uZ+2U+Hq/vf+/zMlty90WdGd4lqVndnpv78WRmHPjk+PUZwPSEOJ5KTCCxro4SxzcDwDbTZOG+AyR6jx7sd+1/ofd1vp9r0zCRU1L5EBH2FVfy5KKveWNNXv1ISJfkGG67oD0TBrQlJiIEZ12/XRBatIa3f06uvYIZiQnklJRyk6uce5NbsSIqqv5Twrxeam02LSQVOUcqHyJSr8BZxdOLd/Hqqjwqa3y3kcZEOLhuYFtuGdKOrimxFic8d987vQLkJsTxWXQUX0VE+G6VPTryk1JXR6eaWpZGR9UXjtyEuKNFxek7i0Wbh4mcMW2vLiL10uKjeGhcL351aVfeWrOfF5btZVdRBc8t3cNzS/eQlR7H+H7pXNWvDcmxkVbHPSvf3QIdIM9h5+Gklqz81igHhoFhmjx6uJh9YQ6ePDoiooWkIv6lkQ+RZsbrNfni6yJeWLaXz7Yeou7ouhCbAcO6tOayXimM7p5CanyQFJHvbIE+sqISl93Gushv8ju8XupsNhymSd3RqRUvBjZMsmvCoLL4m99PC0lFzkpATLvMnDmTv//97xQUFNCrVy+mTZvG8OGnnztV+RDxnyMVNby3IZ+31h1g3b7S4z6WlR7H6O4pjOrWmqz0eMLsgbctUE2dl92rPqTbRzcDcGdKa5ZHfzPSkVFbS9ca37bo3zu18sv1kLdCC0lFzpHl5ePVV1/l1ltvZebMmVx44YU89dRTPPPMM3z11Ve0a9fulJ+r8iFijT1FFby3sYCPtxxkfV4p3/7OEB1uZ0C7RM7LbMngDi3plR5HXKT/T9Z1Vtay8YCTlXuOsGr3EdbllTDG8zlPhE8HfCfOjm3bBtMwCDNNflbqrF9s+u27XXIT4pmRGE9O+miyL5nm9z+HSCiyvHwMGTKEAQMG8OSTT9Y/16NHD8aPH8/UqVNP+bkqHyLWKyp38+nWQ3yy5SDLdx3BWVV7wjVtE6PonhpHj7RYOifH0DYxivSEaFrHRmA/hy3ePV6Tg65qDpRWcaCkiu0Hy9haWMbWAhf5zuoTrr80ejv/8T4MUD+qcezOlcFV1ZxXXX1c8QA0tSLSBCwtHzU1NURHR/P6668zYcKE+ufvuece1q9fz6JFi075+SofIoHF6zXZcaiclbuLWbH7CGv3lpy0BBwTZjdIjY8kMTqcuMgwYiMdxEWGERVu59tbjJgmVNbUUVZdh6u6FldVHUcqaih0VdfvT3IybROjGNQ+kfMyW3FeZiKdWkVh/Ks3ubYK32jG902vXD0TKos0tSLSRCy926WoqAiPx0NKSspxz6ekpFBYWHjC9W63G7fbXf9rl8t1wjUiYh2bzaBbaizdUmO59YIOgG/6Y2uhiy0FLrYUlLG7uIIDJVUUuqqp9ZjkHaki70jVWX9Nh80gLSGS9IQoMpNi6JkWS/e0OLqlxp50uie3z2VHNxM7xRbonUaddR4RaVxNdqvtd3dRNE3zpDsrTp06lT/+8Y9NFUNEmkB8dBhDOrZiSMdWxz1f5/FysMxNQWkVzqpaXNW1vpGNqlqqaj0n/D7R4Q7iIh3EHh0hSYgOo01CFMmxkQ2auvEmdSUHyC7+CHDWP5/tjYH00XiTup7tH1VEmoDl0y4nG/nIyMjQtIuINJy2QBexjKXTLuHh4QwcOJAFCxYcVz4WLFjA1VdffcL1ERERRERENHYMEWmObHZthy4SBJpk2mXKlCnceuutDBo0iAsuuICnn36affv2kZ2dffpPFhERkZDWJOXjxhtvpLi4mEceeYSCggKysrJ4//33ad++fVN8OREREQki2l5dREREzllD3r8Db79kERERCWkqHyIiIuJXKh8iIiLiVyofIiIi4lcqHyIiIuJXKh8iIiLiVyofIiIi4lcqHyIiIuJXKh8iIiLiV02yvfq5OLbhqsvlsjiJiIiInKlj79tnsnF6wJWPsrIyADIyMixOIiIiIg1VVlZGfHz8Ka8JuLNdvF4v+fn5xMbGYhiG1XEs53K5yMjIIC8vT2fdNCG9zv6h19l/9Fr7h17nb5imSVlZGW3atMFmO/WqjoAb+bDZbLRt29bqGAEnLi6u2f/F9ge9zv6h19l/9Fr7h15nn9ONeByjBaciIiLiVyofIiIi4lcqHwEuIiKChx56iIiICKujhDS9zv6h19l/9Fr7h17nsxNwC05FREQktGnkQ0RERPxK5UNERET8SuVDRERE/ErlQ0RERPxK5SMIud1u+vXrh2EYrF+/3uo4IWfPnj385Cc/ITMzk6ioKDp16sRDDz1ETU2N1dGC3syZM8nMzCQyMpKBAweyZMkSqyOFlKlTpzJ48GBiY2NJTk5m/PjxbNu2zepYIW/q1KkYhsHkyZOtjhI0VD6C0H333UebNm2sjhGytm7ditfr5amnnmLz5s3885//JDc3lwceeMDqaEHt1VdfZfLkyfzud79j3bp1DB8+nLFjx7Jv3z6ro4WMRYsWkZOTw/Lly1mwYAF1dXWMGTOGiooKq6OFrFWrVvH000/Tp08fq6MEFd1qG2Q++OADpkyZwptvvkmvXr1Yt24d/fr1szpWyPv73//Ok08+ya5du6yOErSGDBnCgAEDePLJJ+uf69GjB+PHj2fq1KkWJgtdhw8fJjk5mUWLFjFixAir44Sc8vJyBgwYwMyZM/nzn/9Mv379mDZtmtWxgoJGPoLIwYMHufPOO3nxxReJjo62Ok6z4nQ6admypdUxglZNTQ1r1qxhzJgxxz0/ZswYli5dalGq0Od0OgH0d7eJ5OTkcMUVV3DJJZdYHSXoBNzBcnJypmkyceJEsrOzGTRoEHv27LE6UrPx9ddf8+9//5vHHnvM6ihBq6ioCI/HQ0pKynHPp6SkUFhYaFGq0GaaJlOmTGHYsGFkZWVZHSfk/O9//2Pt2rWsWrXK6ihBSSMfFnv44YcxDOOUj9WrV/Pvf/8bl8vF/fffb3XkoHWmr/W35efnc/nll3P99dfz05/+1KLkocMwjON+bZrmCc9J47j77rvZsGEDr7zyitVRQk5eXh733HMPL730EpGRkVbHCUpa82GxoqIiioqKTnlNhw4duOmmm3jnnXeO+0bt8Xiw2+388Ic/5Pnnn2/qqEHvTF/rY99M8vPzueiiixgyZAjPPfccNpu6+tmqqakhOjqa119/nQkTJtQ/f88997B+/XoWLVpkYbrQM2nSJObMmcPixYvJzMy0Ok7ImTNnDhMmTMBut9c/5/F4MAwDm82G2+0+7mNyIpWPILFv3z5cLlf9r/Pz87nssst44403GDJkCG3btrUwXeg5cOAAF110EQMHDuSll17SN5JGMGTIEAYOHMjMmTPrn+vZsydXX321Fpw2EtM0mTRpEm+//TYLFy6kS5cuVkcKSWVlZezdu/e45+644w66d+/Ob3/7W01znQGt+QgS7dq1O+7XMTExAHTq1EnFo5Hl5+czatQo2rVrxz/+8Q8OHz5c/7HU1FQLkwW3KVOmcOuttzJo0CAuuOACnn76afbt20d2drbV0UJGTk4Os2fPZu7cucTGxtavp4mPjycqKsridKEjNjb2hILRokULWrVqpeJxhlQ+RL5j/vz57Ny5k507d55Q7DRQePZuvPFGiouLeeSRRygoKCArK4v333+f9u3bWx0tZBy7jXnUqFHHPT9r1iwmTpzo/0Ai30PTLiIiIuJXWkEnIiIifqXyISIiIn6l8iEiIiJ+pfIhIiIifqXyISIiIn6l8iEiIiJ+pfIhIiIifqXyISIiIn6l8iEiIiJ+pfIhIiIifqXyISIiIn6l8iEiIiJ+9f8Bor1Qczd146QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {
      "filenames": {
       "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_39_2.png"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def function(x):\n",
    "    return x**2\n",
    "\n",
    "def analytical_gradient(x):\n",
    "    return 2*x\n",
    "\n",
    "def gradient_descent(starting_point, learning_rate, num_iterations, solver=\"analytical\"):\n",
    "    x = starting_point\n",
    "    trajectory_x = [x]\n",
    "    trajectory_y = [function(x)]\n",
    "\n",
    "    if solver == \"analytical\":\n",
    "        grad = analytical_gradient    \n",
    "    elif solver == \"jax\":\n",
    "        grad = jax_grad(function)\n",
    "        x = jnp.float64(x)\n",
    "        learning_rate = jnp.float64(learning_rate)\n",
    "\n",
    "    for _ in range(num_iterations):\n",
    "        \n",
    "        x = x - learning_rate * grad(x)\n",
    "        trajectory_x.append(x)\n",
    "        trajectory_y.append(function(x))\n",
    "\n",
    "    return trajectory_x, trajectory_y\n",
    "\n",
    "x = np.linspace(-5, 5, 100)\n",
    "plt.plot(x, function(x), label=\"f(x)\")\n",
    "\n",
    "descent_x, descent_y = gradient_descent(5, 0.1, 10, solver=\"analytical\")\n",
    "jax_descend_x, jax_descend_y = gradient_descent(5, 0.1, 10, solver=\"jax\")\n",
    "\n",
    "plt.plot(descent_x, descent_y, label=\"Gradient descent\", marker=\"o\")\n",
    "plt.plot(jax_descend_x, jax_descend_y, label=\"JAX\", marker=\"x\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a994430",
   "metadata": {
    "editable": true
   },
   "source": [
    "### A more advanced example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "863db98d",
   "metadata": {
    "collapsed": false,
    "editable": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/jax/_src/numpy/lax_numpy.py:173: UserWarning: Explicitly requested dtype float64 requested in asarray is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.\n",
      "  return asarray(x, dtype=self.dtype)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x1162c32b0>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGdCAYAAAAvwBgXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB8SklEQVR4nO29eZhcZZn3/z21dFUvVb2mO0t3VhAStkgCCQjIJsuIijqMDMgMDvq+/AQGZGZwAiqo+ObngAsygjLziigKuKKOiGFU0BEiISxhSyBJJ+mkt/Ra3dVd+3n/qHrOOd2pqj7L85zzVNf9ua5cF+l0dz0k1efc53t/7++tqKqqgiAIgiAIwgN8Xh+AIAiCIIjqhQoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8I+D1AcqRy+XQ29uLSCQCRVG8Pg5BEARBECZQVRUTExNYvHgxfL7ymofUhUhvby+6urq8PgZBEARBEDbo6elBZ2dn2c+RuhCJRCIA8v8j0WjU49MQBEEQBGGGWCyGrq4u7T5eDqkLEdaOiUajVIgQBEEQRIVhxlZBZlWCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCkIQ9hyfxrWf2IJHOen0U15B6+y5BEARBVBNf3fIWfv1qH9oaQvjrdZ1eH8cVSBEpQzqbQyab8/oYBEEQRJVweCIJAOgZmfL4JO5BhUgJ4skMTtv8e3zkga1IZagYIQiCIMQTS6QBAIOFgqQaoEKkBPuG4xiaTGL7/lF88w+7vT4OQRAEUQVMJDIAgMFYwuOTuAcVIiVIGlSQb/5hN97si3l4GoIgCKIamCBFhGAk03ohksmpuOUnO8gvQhAEQQhDVVVMJvOKyAApIkQikx+dWtJUi2g4gFcPjeM//6fb41MRBEEQ85V4Koucmv/vockksuw38xwqRErAFJGFjWF85pI1AICvPvUW9hye9PJYBEEQxDwlNp3W/junAsOT1dGeoUKkBMmCIhIK+HDZuk6ceXQbUpkc/vWnO5CrkiqVIAiCcA9mVGUMxKgQqWqYWTUU8EFRFGz+0Amor/Fj275R/NerfR6fjiAIgphvMKMqY3CiOnwirhUimzdvhqIouOmmm9x6SUckC/G64aAfANDZXIdLTlwMAOg+HPfsXARBEMT8hBQRgWzbtg0PPPAATjzxRDdejgtGRYRRF8oXJczIShAEQRC8iM1SRKplckZ4ITI5OYkrr7wS//Ef/4Hm5mbRL8cNvRDxax9j6kg1LSMiCIIg3GG2IlItWSLCC5HrrrsO733ve3H++eeLfimusGIjFNT/isKFoiRJke8EQRAEZ1ghoij531dLuqrQ7buPPvooXnzxRWzbts3U5yeTSSSTegUYi3mXZsqKDaaCAHpRQooIQRAEwRtmVu1qrsOBkSlSRJzS09ODG2+8EQ8//DDC4bCpr9m8eTMaGxu1X11dXaKONyfMrGr0iIQL/02KCEEQBMEbpoisWlAPgDwijtm+fTsGBwexbt06BAIBBAIBPPPMM/jGN76BQCCAbPZIVWHTpk0YHx/XfvX09Ig63pwk0keaVUMFdSRJighBEATBGaaIHNXeAKB60lWFtWbOO+88vPrqqzM+9rGPfQzHHnssPv3pT8Pv9x/xNaFQCKFQSNSRLMECzYytmbDWmiFFhCAIguALU0SWt9XDp+jpqu1Rc12FSkVYIRKJRHD88cfP+Fh9fT1aW1uP+LiMFBvf1c2qpIgQBEEQfGGFSFNtDdoaQhicSGIgNv8LEUpWLUGx8d0QKSIEQRCEIFiOSCQcQHs03x2ohnRVoVMzs3n66afdfDlHlB/fJUWEIAiC4AtTRCLhADoiYbyGmNB01b7xabzRG8OCSAgndjYJe525IEWkBKSIEARBEG4yoSkiQa0dI3JyZuveYVzz0Au467e7hL2GGagQKYG2fdegiLCihHJECIIgCJ6oqorJZF4RiYYDaI+w1ow4RSRVeOCu8XtbClAhUoJi47tsgsarHJHx6TRe7x335LUJgiAIccRTWbBJ3Ug4iI6CIiIyXVUrRALelgKuekQqiWLju6wocVsRSaSzePDP+3D/07sRS2Tww49vwOlHtbl6BoIgCEIcrC0T8CkIB32uKCJJKkTkJjmHIqKqKhS2EEAQmWwOP95+EF//77dmGJb2DU/h9KOEvjRBEAThIkajqqIomiIi0iOSyubvc0GPWzNUiJSg+PZd34w/N6olvBmNp/CRB57DWwOTAIAlTbVoCAWwa2ACk8n0HF9NEARBVBJGoyoAdBTGd1m6qt/H/8E3ncn3grxWRMgjUgLWfgkXMasCumIiihcPjOKtgUnU1/jx2UvW4Pf//G6cuqIFADA5a1U0QRAEUdnEDIoIALQ2hGakq4ogVVi1QmZVCVFVtagiEvQrYEWp6CwR5p4+qasJ15yxAqGAHw2FN+hEkgoRgiCI+cTErELE71PQ1pBXRURliaSKJIh7ARUiRWB9M2Dm+K6iKIYRXrGKyFQqX+jU1ejds4ZQ/r9JESEIgphfzG7NABCerirL1AwVIkUwFhmzK0Vt8Z1gRSReUD3qQ7oiwyrlSVJECIIg5hWzFREA6Igww6qo1owcZlUqRIrA2i6KcmTvTJuc8VIRoUKEIAhiXsEUkegMRUTs5EyKzKryYhzdnT2iq2WJiFZEUgVFpEZXRFghMkGtGYIgiHlFbPpIRUR0lghTRMisKiHFjKoM1xSRZEERCRkUEWrNEARBzEuKKSKi01VThQdqUkQkpNjoLsOtdNViikgklH+DklmVIAiCD8OTSSke7op5RIQrImRWlZdyikiooIiIbs2QIkIQBCGWiUQaZ9/1NM7/yjM4NDbt8VlYIXKkIiLMI0KtGXnRNu8WqRLdas2U84hMJjPIse1IBEEQhC0OjExhIplBfyyBf3hwG2IJ71KrY9r4rmFqZla6Km8oWVViWJFRLMLdLbNqsakZ4xuUFSoEQRCEPcam9MJj18AEPvnwi0hnvdmuXqw1IzpdNUmKiLyYUUREB5oVyxEJBXwIFKJdqT1DEAThjJF4CgDQ2VyLuho//mf3ED7z89egqu4rzsUCzUSnq5JHRGI0j0gRs2q48A8mOuK9mCKiKIruEyHDKkEQhCNGp/KFyAlLGnHv374TPgV47IUe3Pf0HlfPoaqq9nAZDc/cRatNzghIV6WpGYlhEzHFzapsakZ0oNmRighgyBIhRYQgCMIRo/G8CtFcX4PzVnfg8+8/DgBw12934c+7h1w7RzyVBbOAGBURQJ+cEaGIpLP5F6VkVQlhikix8d1woTgRrYjEC1Mz9TUzq2PaN0MQBMEHpog01+Vv/ledthzvO2kxAOCPbx927RysLRPwKUfcd0Smq9LSO4nRk1VLKyIip2ayORXTadaamXkG2jdDEATBB+YRaa6r0T521IIGAO4mWBuNqrPTvEVmiWjju1SIyEdZs6q2fVecIjJt+N71IVJECIIgRMAUkZZ6vRBhD3uxafdGeYsZVRki01U1syq1ZuQjUWZ8V8sRyYhTRKYKaodPObIYaii8UckjQhAE4Qy9NXNkIeKmIhIrMrrLYFkiQhSRwn0sSIqIfJRTRHSzqjhFJJ7S/SGzZTpSRAiCIPhgNKsyorWFhz0Xw82KZYgw2iN5RaSfsyKiqiolq8pMsoyBRzerilNEWIZIXehIRUb3iHiXAMg4PJHEuV95Gvf899teH4UgCMIys82qgKE146pHpHRrpqlOTGHEJmYA8ohIiTa+WyxZ1QVFZCpVfGIGmBnz7jXPvHUYew/H8X//Z6+Q+GGCIAhRJNJZ7Vo7QxEJy6WIaAMSmRzXoLWUIUGWpmYkpJwiEnLBrMri24spIlqOiAStmX1DcQD5J4c3emMen4YgCMI8LN494FMQMQwF6IWI+4pItIgiwu45qjpTxXBKyqDqU46IhGjju0XNqnp1Kgpt824xRUSi8d3u4bj238/ucS/8hyAIwilsdLeprmaGF4+pElOprGt7Z8oqIoYHYp75VawQ8fsU+H3KHJ8tFipEilDWrOqiIlJfU8QjIpFZdf+MQmTYw5MQBEFYQx/dnalCNBiKAbeus+YLEX6FUVoSoypAhUhRyo/vio94n9LMqvIqIqqqYt/QlPb757tHZkh9BEEQMsMKkSbD6C6Qb1PUFq79brVnyplVFUXRzKQ8C5GkJAvvACpEimJm+67QqRnNrCqvR2RoMoXJZAaKknd1T6ezeOXgmKdnIgiCMMtooTXTMqsQAYBoLZucccewWi5HBNDvRUmOSrwsm3cBKkSKUt6syv8NMRu28K6YR0SWiHfWllncWIt3HdUGAHh2N7VnCIKoDEaKZIgwmDLhViGit2aOVEQA3RLA8wFYlgwRgAqRojD/h1fJqtrCu6JTM/k36mQyw3WUyyrdhYmZFW31OH1VKwAyrBIEUTkUyxBhuJ2uqrdm5lBEeBYipIjIjRlFJJXNCcvOKKeIMI9INqcK9anMxb6CIrKstQ7vWpVXRF46MIbplNitxARBEDwotmeGwcZo3do3wwqeaKlCJMhfiSezquTohUhpRST/eWJuuuU8InVBP9ik2YSH6ar7hvNG1RVt9VjWWofFjWGksjls3z/q2ZkIgiDMMlrIEWku4hFxUxFRVbWsWRUQ1JohRURuklqyamlFJP95YhSJclMzPp+ChhrvR3hZmNny1nooioLTCqoItWcIgqgEmFm1ub5Ya8a9ULN4KgsmrrvZmqGpGclJZEqP7wb8PgQK4S8J4YpI8Tel1yO8+dHdQiHSVgcABp8IGVYJgpAfFmhWTBFxc2qGqSF+n6KNDc9GL0Q4Ts1Qa0ZeVFXVJKtS+fuaYVWUIlIm4h3wfgPv4ckk4qksfArQ1ZIvRE4rFCI7Do655jQnCIKwy9hUmULExX0zxjCz2dvWGSEB9xx2nwuSIiIfRumrdCFSCDUTpIiwiPe5FJEJjxSR/QV/yOKmWq13ubipFiva6pFTgef3jnhyLoIgCDMk0llNeS4+vuueR2SuiRlATGuGzKoSY6w4i7VmAGPMuxhFRFt6V8SsCniviBhHd42cRu0ZgiAqALbwzu9Tik6qRF3MEdHCzELFjaqAoNbMHMq/m3h/Aslg/9A+BZoXZDYiRqmMaIpIEbMq4H2oGfOHLGutm/FxyhMhCKISMGaIFGuHuKuIlE9VBWhqpuowju6W7NcxRURAqJmqqmWX3gEGRcTj1szy1pmKyMaV+UJkZ/8EhieTrp+LIAjCDKNljKqAu1MzrDUTrS2jiGgPv5SsWhUkyozuMvTFd/wVkWQmp41yFRvfBfR0Va/2zZRqzbQ1hHDswggAYNs+yhMhCEJORsoYVQHD1IwLgWbmFBH+rZmkZlYt/sDtJlSIzIL944SLhJkxwgJkMkbcoHKUGuXSx3fdn05RVdWQqlp/xJ+z4mQglnD1XARBEGbRwsyKZIgAHikiJcLMADGtGd2sWvpe5xZUiMxC27xbRhEJCVREpgpO7tqgH/4SHpWIh2bVwxNJTBVGd5e21B3x5ywuebggfRIEQciGtnm3yMQMoKsTqWxOyHXeiFeKCHlEJIb14Mo5iYUqIswfUiJDBPA20IxFuy9pri36Bm5tCAEAeUQIgpAWZlZtKtGaaagJaKs0RE/OmCpERHhEqBCRF5YNUmp0FxA7NcM27xZbeMdgZlUvPCLGaPditDUUFJFJUkQIgpATTREpUYj4fIpr19m59swAYqdmZBjfLX23q1KsKCJiWjPlM0QAbxWR7uHyhUhrfUERiZMiQhDVSiqTw7Z9I/jz7iGc1NWEC49b6PWRZjBS8Ig01ZW++UfDQUwkMsILkZhHrRnmEQn6vTerUiEyi3KbdxlsakaMWbV8hgige0TiHhQi+1kh0laiECFFhCCqkqlUBr94uRd/2DmIP+8e0pJLw0EfXrvjQgQkGBNlsHj3Uh4RQC8MRE/O6K0ZM4FmHJfeSTS+S4XILJjKES5rVq1iRWQo7xFZ0XakURUAWsmsShBVyRd+9QYe3daj/b6tIYTx6RQS6RwOjExh5YIGD083E23hXZlCJOrS5IypiHeBu2Zqyjx0u4X3pZBkmFJEBFSnjLk27wLeeURUVdUUkWKju4BuVh2fTmtvdIIg5j/b9+ezgz66cSl+df0ZeP7W8/COjnyu0O7BSS+PdgRzBZoBxnRVdxSRYlHzDJqaqTK08d0y/zhCFZFk+c27gL6TIJnJuXqzHzSM7nY1F1dEmmqDYFPHzJlOEMT8JpnJYm/ByH7dOUfhhM5G+HwKjmrPqyC7D8tTiCQz+sK7UmZVQE86FTk1o6qqpmy73ZrRtu9K4BGhQmQWmlm1XGsmwHJEvFFEjKO9bvpE2MRMZ3NdySra51PQUjCsDtEIL0FUBXsG48jmVETDASyMhrWPH1Vox8ikiLCFdz6lfDvEjX0zU6kssoUobbd3zTCzqgxTM96fQDISmiJSzqzK3hTeKCIBv09LXXXTJ7JvDqMqg0Z4CaK62NkfAwAcuzA6Y0cXU0T2SFSIjBjaMr4SoZGAO4UI+95+n1IySRswREbwbM1kqTUjLZWgiAC6YdVNnwgzqi5vLd6WYbDJmREyrBJEVbCrfwIAcOyiyIyPa4XI4ThUVXX9XMXQNu+WMaoCullV5NSM0ahaaskqYGjNiDCrUsS7fJgb3xWoiJiYmgEMMe8uKiL758gQYVBrhiCqi52FQuSYhTMLkWWt9fD7FEwmM+iXZP/UaLywZ6ZMhgigezZiAh/2zGSIAGIDzUgRkRAz47thzazqTY4I4M3iOxbvvrzE6C6DRngJorrQFJFZhUhNwIdlBQV1z2Dc9XMVY3SOzbsMN6Zm2IPnXAq40O27ZFaVDzOKiN6a8U4R8WKEl428tUfCZT9P94iQIkIQ852xqZSmdrBxXSO6YXXC1XOVYq6Fdwx9akbcNVa3ApS/3ocMIZq8Wlxp8ojIi5nxXaaIiBidNe0R8aA1EzMRvAMYF9+RIkIQ8x3Wlulsri06girbCO/IHAvvGG4oItpuszmKAfZgrKpAOsunEEnR1Iy8JE0sAmJtG69yRABDa8YlRSSdzWGqUCRFy8y7A9SaIYhqolRbhrFKshFeNr7bUl/+OhZ1YSAgYVYRMdyPeLVnyKwqMbpHpFxrpuAREaCITJlURNw2qxoLnrkVEVaIUGuGIOY7pYyqDE0RkcQjwqb55lJE9Ij3tLCJn6RpRcRYiPC575BZVWKsKCJJAYpInJmXTCoibnlEWFumrsY/5/IqbQMvtWYIYt7DMkSOWRgt+uerCoXI0GQS41PumetLwcyq5VJVAX1qJqfqLXPemFVEFEXRCgYehUgupyJTCFIjs6qEmDEPCVVEClMzdXN6RPI/JG4pIrFptg+hvJwJ6IrIVCqrmW8Jgph/5HIq3iooIqtLKCINoQAWNeYN7rsPe29YNZsjEg76ECgEnonyiWgKvAlVQs8ScV4UMX8IQIqIlJgxDzFFJJtTkcnynetmbxCzgWZueUTYD2K0du6FzQ2hgPbmJlWEIOYvh8amEU9lUeP3lU1c1tsz3vtEzOaIKIqiT85Mi7nOMnWjnBWAwTNLhAoRyTGjiBjfNDxVkWmD/FcrWaCZPjEztyKiKAra6ildlSDmO8wfsqq9AcEyLVtZDKupTE67Zs41vguIn5xh6oaZyRWei++ME581c7Ta3UDoCTZv3oxTTjkFkUgE7e3tuPTSS7Fr1y6RL+kYMx4R4z8cz8kZ5g+p8fvmrFK1HBHXCpG5V1UbaSHDKkHMe3b2sR0zxdsyDFkUkbFCW8anmGszi943Y2Y4ghHi6E3UJ2Z8ZaPl3UJoIfLMM8/guuuuw9atW/HUU08hk8ngggsuQDwuh3u6GJqLucwbw+fjaxxiaGFmcxhVAWNrxh3zF9u3YEYRAXTD6hC1Zghi3rJzoPzoLkOWLBFjhki5hXcMbd+MKEVEa82YUUQ4tmYkSlUFAHOPtzZ58sknZ/z+wQcfRHt7O7Zv346zzjpL5EvbRmvNmBinSmVyfBWRpLnRXcD9QDNNETHhEQEMI7xUiBDEvGXXHKO7DFaIHBydRiKdNaUAiMCsP4TBFBFR6aoJrTVjxiPC7+FXplRVwGWPyPj4OACgpaXFzZc1TS6nmk6b0xbfcdw3EzcZ7w7oPyCum1VNKiJtWroqtWYIYj6SSGfRPZRXt48tMbrLaK2vQVNdEKoK7D3snSJuds8MI2LIEhEBG981p4jw2zeTlChDBHCxEFFVFTfffDPOOOMMHH/88UU/J5lMIhaLzfjlJkYn8Vxz3Vq6KsclRNro7hwL7wBdEYmnssjmxK/XZq5x860ZMqsSxHxm9+AksjkVjbVBdERDZT9XURR954yH7Rl2PZprdJehKSLCpmYKiogpjwi/h99UtSoi119/PXbs2IFHHnmk5Ods3rwZjY2N2q+uri63jgdgpvHUbPa/CLNqvQlFpMFgGo27kNURszC+C+iO9KEKLUSyORVv9MaEJSoSRKVjjHY3Y3iUwbA6piki5h6ooi4pIl5NzcgwMQO4VIjccMMN+OUvf4k//OEP6OzsLPl5mzZtwvj4uParp6fHjeNpsH9gv0+ZMz00HBRhVjUXZgbkCyH2JnKjPTNhYXwXqPzWzM0/ehl/9Y0/4eYfveKK4kQQlcYuk0ZVBitE9nhYiIwwj4hFRUTY1IyJ4QgGz9YM84iUG7l2E6FmVVVVccMNN+DnP/85nn76aaxYsaLs54dCIYRC5SU+kZg1qgJAmDmYuZpVzcW7MxrCAYzEU64YVvVk1flvVt26dxi/eLkXAPDzlw5BAXDXZSfBb8JlTxDVwpt95aPdZ7NKAkXEbLw7Q/jUjOYRcTnQzERMhZsIPcV1112Hhx9+GD/84Q8RiUTQ39+P/v5+TE9Pi3xZ21iqTrUNvN4oIoAhS8QFRURvzZj0iDBFJJ6sqPZGNqfiC796AwCwblkz/D4FP3vpEP7lJ6SMEIQRsxMzDOYR6R6Kc02ktsJ4IYag0eR1jLWiRSsiplozWo4Ix9ZMNRQi999/P8bHx3H22Wdj0aJF2q/HHntM5MvaxpYiwtGsasUjArg7wjthMdCMmVXTWdW10DUe/OiFHrzRF0M0HMADV63DvX/7znwx8uIh3PKTHVSMEASA0XgKgxP5tqvZQmRJUy1qg36ksjn0jHrzMMra2A0mr2Oip2asKSL8WjOymVWFt2YqiaSN6pSrImJhagZwb9+MqqqWx3fDQT/qa/yIp7IYnkyZ/jovGZ9O4+7f5pN/bzr/HWhtCOGvTlgEVQX+8dGX8NMXD6Im4MPmD53g8UkJwlveLrRXOptrtQeiufD5FKxcUI/Xe2PYPTiJFWV204hiUmt/my1E3JmacTvQLJmRyyMixykkQY93n7s6lUER0ffNiE1XjaeyYEKA2dYMYGjPVIhh9d7fvY3heAqrFtTjqtOWaR9/74mLcM/la6EowCPPH0DPyJSHpyQI7+kdyysaXc11lr7O650z7Bprtnhyb2rGG7NqVU3NVAp67r8ZRYSN73qviIj2iLB496BfsWRuYobVSoh533N4Et99dh8A4LOXrDniSeGSExfjtJWtAIBfv9rn9vEIQioOFQqRRU1hS1/X2VwLABiIJbifyQxsIMBsIcIUkXgqK8TXYu2eQx6RqsCKIsJuyF7liADueURihraMlQVJbN9MJSy++9Kv30Qmp+LcY9tx9jHtRT/nvScuAgD8egcVIkR10zeeL0SWNNVa+jrdxO7Nw8mk5UJEV4B5X2cz2RwyBak5bOqew39qhgoRCdFT7kyYVYP83hQMy1MzLnlEmOISMWnwYrQVFJERyRWRXf0T+P3OQQR8Cm577+qSn3fRcQvhU4BXD41j/7C8ixsJQjS9Y3lFY1GjxUJES1x2/+Ekk81pCrZZj0hNwKepFbyVZ+O9w8w9h6tZtZrGdysNO/06vkvvrOWIRNxSRKatje4ytCwRydNVt+8fBQBsWNmi9bCL0doQwumr2gBQe4aobphHZLHF1oyX+UJsqShg/hoL6KoI7ywRYyFiThHhmKwqWaCZHKeQBBZOVjGKCMsRcbE1Y4WWQmtmSHKz6o6DYwCAEzub5vxcas8QhLEQsaaIsNUPXjycTBZa33mvm5VCRMzkDHuIrfH74DMRlihk1wwVIvKRtCBXaUvvOCoiUylrikhDoTCQvTUje7rqKwfzW6FP6myc83MvPG4h/D4Fr/fGsG+I2jNE9RFPZhArXBMWNVpTRNjqh9F4CjmXM3msGlUZoiZn2L3DbHtERGuGPCISwgoRK3G7PKdmmHRYb1ERca01Y1ERqQSz6nQqi7cKOzPMKCIt9TU4fRVNzxDVCzOqRsIB07unGM2FaPVMThUWm14KqxkiDFH7ZrQHXxP3G0DQ0jsqROTDSoWqL73jo4hkcyqm06w1Y9Ij4rJZ1ezmXQbrB49I7BF5o28c2ZyKtoaQ6ae7S6g9Q1QxhwpGVasTM0D+xseuW263Z7RUVZuKCO/CycroLkBTM1WDpUAzjv06AFoRApiv2N0e37X69GMsRGSNRn+lR2/LmB1NvmDNQgR8Ct7oi2HvYe8WeBGEF/SxDBGLbRmGvpnb3UIkblMREbVvJmFhpQhgyBHhGfFOHhH5sBTxzqZmOCkiU4UfEp9i/o3JvCTiWzPW9swwmAybU4GxKTlVEStGVUZzfQ3edVR+euYJas8QVYZdoyqjxaMRXvutGTEekaSFJauAoTXD4eE3LdmuGTlOIQkJCwuIeCsi8ZTuDzH7ZF5b8JJMczTMFsOuIhL0+9BUl/8aWUd4dxSMqid2zW1UNcKmZ/6L2jNEldE7nm/N2C1EWJaI24nLTBGJWC1EQqKmZszfbwBBrRlSROTDytQMb0WE/ZDUWZhvry28gVOZnNDWR0zziFhfXKdfdOQzrI5Pp7G3MPlykgVFBAAuXLMQQb+Cnf0Tnu3NIAgvsJshwvDKO6Y97Fm4xgL6dW+C804vKwq88fN4tGaS5BGRF1s5IpwUkamUtYkZYKapVaQqMqFNzVhf1tzqUT/YDK8dyqshnc21mlxslsa6IM4otGd+Q+0ZooroG7eXqspg03RuFyLM4yHN1IxVRSSoT8043WxPZlWJ0cZ3TZlVOSsiKeuKiLGSnk6JK0RiWo6IdUWkTeLJmVcK/hCragjj/DUdAIBn9wxzOhFByI2qqroiYrMQafFIJbWbI6Ilq05znprJ2JuaUVUgnXVYiFCyqrwkLCgieo4IL7OqtVRVAFAURWvPiC1EWMS7dUVES1KUsDWzozAxc6KJILNibFiRzxN58cAotzFugpCZkXgKyUwOigJ0NIZsfQ/PWjN2p2YEKSJ6XIQ1syrgvD1DZlWJsbR9l6NMBljfvMtg7RlRrZlEOqvJePY8IoWYdwkVETYxc1JXk62vX7WgHm0NISQzOc30ShDzGbbsrq0hZCkm3YgWdOhyu9bq5l2GvmtGVGvGmkcEcG5Y1ZbekSIiH9Yi3nWZjMlcTpjSzKrWfkjYOVg8PG/YU4CiAA0W1BqGHvMulyJyeCKJ3vEEFAU4fok9RURRFGxY0QIA2ErtGaIK6B13NroLeLdvhj3sWS9EClMzvAPNMtYUEUVRNAWDVyFCioiEJNPm57p5VqeAcXxXLkWE/fA1hAKmFjPNRlazKlNDjlrQYPnCZGTDynwh8pfuER7HIgip0f0h9iZmAP3hZHTK3X0zkzbNqkwJTmVyXFuwWqCZSUUEMGaJODsHFSISY0URqfH7wOI+ePhEmKJhxSMCALU1Yj0iWry7DaMqoI/vymZWZYvurASZFYP5RLbvH9X6rgQxX+lzmCEC5AMBgfxai3HOBtBy6IFm1h72jA8qPNdpaIFmFlpcvLJEyKwqMdpct4kKVVEUrkl32sI7iz8kmllVlCIyzcLM7KkGXq79LofuD7HXlmEc3d6A5rogptNZ8okQ855DDuPdgfzNr7HW/aBDdo21qoD6fXpLhOd11mqgGcBv8R0pIhKjvTFMVqhalggHuc6pIjIlSBHRJ2bsKSKNdfrCKLfXfpdCVVU9UdWhIuLzKTh1BWvPkE+EmN+wPTN2Ft4ZafVgms7u+C6gP/DxmpI0fi+zgWaAYUjCaWuGds3IixVFBNALlgQPRcSmR0TED4gRvTVjTxFhTz6qyn/8zS4HR6cxEk8h6FewelHE8fdj7Zm/7CWfCDG/YVMzixwWIi0ut2xVVcWkTbMqYPDipfi1X7XcKkuKCKfWjAUbghvIcQoJyOZULSTG9Fw3x22IdqdmRHtEYlqqqj1FJBTwaz/EY9NytGdYkNmxC6O2RxCNMMPqC/tGkCGfCDFPSWdzGJxgHhH7rRlAzxJxa6x/KpUFS1mwalYF9Ac+ntOJCW04woZZ1UEhksnmwMRp8ohIRsrwD2v2jSFGEbFYiATlbs0AQFPha8em3DOmlUNvyzjzhzCOXRhFNBxAPJXF670xLt+TIGRjIJZATgWCfgVt9fbCzBgtLObdpWk61pZRlJmrMcxSK2A6kXkLrTwM8dg3Y0xlJY+IZBhbG2b7ZkwR4To1Y9Gsyn6oRLdm7JpVAb2IcdMhX47Xe/kWIn7yiRBVgHHHjJ1RfiNavlDcHY+INjFjYbu5EREJ1kmLEe8AEOKw48z40E2FiGQwqSvgUxAwWYiEOa5l1iLeLfQLARcUEYetGQBoKhhWxyQpRPYNTQEAjmpv4PY9ySdCzHd6OUzMMNyeprM7McMQoYh4NTWTzOb/HxQlf7+TASpEClhdyQzwVUQSFsLUjNQWWjmixnd5KCJNtfmLzviU9x6RdDaHvkI6ZFdzHbfvy3wiz+8bQVaS6SCC4AkzqjqdmAH0oEO3WjN2M0QYIh74EnbuORxaM9rort9nSx0SARUiBexVp/w8InYc1ABQWyiGhJlVeXhE6uTxiPSOTSOn5n+gF0Sc9bmNrFkURUMogIlEBm/2kU+EmH9oiohDoypgGN91uTXjVBHh2QJPOrjnOFFEjIWILMhzEo+xo4iEOU7N2JkpB8RIhkZi086SVQE9S0SG1syBkXxbZmlLHdengYDfh/XLmwFQ3DsxP+njsGeG4fb4rt3Nu4w6AdOJthQRLUfEfiHCzKqy+EMAKkQ0tHh3C9VpOMhPEUnYVURYa0ZYxLuzZFVAzxKRwaxqLER4o/tEyLBKzD8OFVozixt5tGb0QsSNoMNJh4WItlxUQKCZPY8Ih9YMFSLyoY9SuduvA/Jz3cxXYMVBDRh6l8KW3hUUEUfju/mLjgytmZ6Rgj9ERCFi8InIkiJLELzgqYg01+WvCTnVHaWUKSIRSRQRVVUND79W7jkcWjMFsyoVIhKitUY8UEQShjeV1YAtbXxXgCKSzanak4TdZFVA94iMSxBo1lNQREQUIicsaURt0I+xqTTeHpzk/v0JD9m2DVi0CFiwQP+1aFH+41XAVCqjPUjw8IgE/T7tuuBGzLvT1gzv8d1UNqcFrLmdI5Ikj4i8WNm8y2Cf69TAZPx6qx4RXTLkH59u3DQZcTK+K1GgmcjWTNDvw9quJgDAC/vJJ1LxbNsGtLbm5xxPPRVT/f0IDQ3hgqEhhIaGMNXfD3zwg16f0hXYxEwkFHDkFzPi5gjvhNNChPN0ovHh1VqOiHOPCGvNyJKqClAhomHPrMonRyRp6NlZDQoSsQOBwSZmwkGfIxlPRrNqV4tzebkYzLC6fd+okO9PuEB3N3DqqcCGDZgaGUEIgOLzo2XJaviWnYQ/LTsJvuVr0br8nehJeH1Yd+A5McNg6azDLozw6gvv5BjfZfcbRbGmTPBozchoVrWvt88z7IzvhjltQrQ7MQMYJUP+ikhMM6o6ewIymlVVVfVsdn18Oq0ZZnlmiBhZtyxfiLywnwqRiiMeB771LeDzn0dqYgKrAByqqUXDiRdgwSkfQCDafsSXnK0A1zzxJm46/2jLm7MrCZ7+EIY+OeNGa8ZpoBm/zChgpifRyvVwvppV5+9PjkVsBZpxSla1M0/OqDOM7/K+yeuju87eJk0FY1oqk0MindNGjt2G+UPaGmpsS7RzcfKyZihKXnkZnEigPcLvCZIQSHc38P73I/faa/gsgC/XNSGy7hIsOfkS+MP5BN7s1Diy8VFozX2fHzVtS/HAH/fiiVf78KUPnoB3v2OBd/8PAmETM4s4TMwwWhrca804nZqpDfJuzdgLsOSRrMrMqrJs3gWoENHQPSLWFRHHHhEbOwe0MxRu6jk1b4DisU2WwSPMDADqa/wI+BRkcirGplOorRHTFpkLVoh0ClJDgHzeyjEdEezsn8D2faO4+IRFwl6L4EA8Djz4IHDnnUgNDGABgMwx78Lii2+EL5R/n6SHDyL2/M8w+fofgOzM9uLv3uzHZx9/HQdHp/H333keH1i7GHe87zg0F5725wt9hdbMEq6tmUIh4mprxlmgGb/WTOHh0+L1mueuGTKrSoiuSlhXRBIOx3f11oyNrZCGipp3loge7+6sEFEURYp01Z5RcUZVI8wnQu2ZCuDXv0buhhvwmYEB1PsD8J9/LRZcugm+UB2S/bsx+LMvoff/fhKTO7YcUYQMDAzg3GM7sOVTZ+GaM1bApwC/eLkX//v72+ddzH/vONszI6I1I78iwnu5qD6lae0WzLM1Q2ZVCdFT7ux4RPiYVe0oIkG/D0F/vh3DO11VX3jnXDhrlGByRuTEjJH1y/J5IlSISEw8Dvz7vwO33ILPAfhy00K0ffRuRNddAgAYf+5H6P/ezZh++zlAzSEIYHrVKqgNDVAVBerAANrb856R+lAAn71kDX72yXehvsaP5/eN4D/+tNe7/zcB6Jt3+SkibN/MkAvju85bM0wR4ePF0zyJVhURLq0Z+cyq8pzEYzTzkI2VzE4VEWZ2tfqmZIQFbeDl1ZoB5EhXPVAIMxNdiDDD6uuHxoUl3hIOiMeB227LKyH79+Peleux6Op7EFp4FLJT4xj48e0Y++P3ADWHOgC3LliARDyO8O7dwMQEkMsB7UcaV9d2NeH29x0HAPjKll14o3d+7BxSVRX9rBDhaFZtdVER0QLNbD5U1XIONEvabMdz3TVDhYh8aG8MG+EyjgPNbBRBRkTsQQD4bN5lMMOql6FmmkdE0Oguo7O5Fh3REDI5Fa8cHBP6WoRFWBFyzz14N4Cvn3A+mj78WfhC9UgcfAN9370Rib3bEQRw69KlmNiyBV/q74evzlzxetn6TrxnTQfSWRU3PfYS1yVpXhFLZLSHnIVR/oqIG2ZVNjXjONCMc46I1XZ8iMN+MypEJMZOMaDniDhURGwUQUZ4/5Aw9NaMc0XE61CzbE7FoVF3FBFFUbT2zHZqz8jD4CDwoQ8hd889OAvAjo2Xoe2vboLi82Py1acw8MgmZCeGEFUUTD35JL7U3Q3fe94D+MxfExRFweYPnYC2hhq8NTCJu3+7S9z/j0swNaSpLsh14o15REanUkI9NclMFqls/vreYHPEmj3spbMq0lkOu8WcekScmFVZxDt5ROTDVqAZM6tyUkTsjO8C4hbfaa0ZHh4Rj0PNBmIJpLI5BHwKV8NdKbQ8kX2UsCoFg4PAJZcgs2ULlkPBG+d9As3v/nsAwPjWH2P4iXuAXBZRnw/DU1MIXHihpQLESFtDCF/+8IkAgP/8n248u3uI2/+GF7AMEZ5qCAA0F64JqgqMTYlTRZgaAgD1NgPNjNdmHipX0uaSU2rNzHPsjO+GJAg0A4Dawjl4KyITHBbeMbxefMeMqkuaa+G3mF5rBy1hdf8oLcDzGlaEbNuGVl8A0+/7J0TXfwAAMPK7BzD2zEMAgK5gEMPxOAJh5zfc81Z34G9PXQoA+Kcfv6J5FCqRfgFGVQAI+H1aMSKyPcP+7kMBHwI2VYB88Fj+v3k88Nm95vOYmtGSVUkRkQ87u2a0pXecIt6tLNwzwttIxdAVER5m1byqEvNIEelxaWKGsXpRFLVBP2KJDHYfpgV4nmEsQvxBhD54K+rXnA01m8HhX96FiRd+CQA4o60N+yYnuRQhjM+8dzU6m2vRN57AT188yO37ug2bmFkoQElscSFLZNKhURXIt9zqOLbAbSsimkckB1W194CTJEVEXpiqYeUfhxUtqUzO0VOvnrJnVxHhm/rHEGFWHfPIrCpy624xZizAo70z3mAsQoIhhP/6c6g76lTk0kkM/uyLmHrzGQDAGUuW4Jm+Pvhq+IaQ1YcC+MSZKwEA3312X8UqY6IUEcBoWBU3wut08y6DZ6iZ3Ws+U+xVVVc2rEKtGYlhBiQr/zjGatZJz86ug5rBO/WPoZlVeYzvehxopi27E5iqOhs92Ix8Iq5jLEJqalF72edRu/ydyKWmMfjjO5DYux31AG7buBHP7NsHX0BMyPSH13WiIRTA3sNx/E+FekX6Y0wREVCIuDDCq2WIONwFVFvDTxGxk1uV/3z9/mS3PcOMuxRoJiEpG4WI8U2RclCI2J0pZzDJkOeooKqqiDGPyDyYmnErzMwIM6zS5IzLGIqQtnAD6j5yJ8JdxyOXjGPgsc8i2fMqoj4fxrZswZ1//rOwIgTIR4r/9bpOAMBDz+4T9joiEauI5AuRIRdaM3bj3Rn6glEOrRkbSd7A7ELE3j0nTYqIvLBCImShSgz4FDDfoxPzkPOpGb6pf0C+6mcjdXxzRDzyiLg0umuELcDbPzyFwxPi0yMJzCxC6hpRd/n/QWjxMchOxzDwyK1I9e7MT8bE4whYHM21y9+dtgwA8PtdgzgwPCX89XjTp8W78y9EWurzrRmRG3j11oyz0WOe04l2IxsURdEKCLuFCHvotnKvE408J/EYO30zRVG4jFMlbIwOG9HNqs7n2xls867fp2gz9E5gyaqTyQyXOXwrTKeyWiHgZiHCFuABwHZqz4jHWIRE2lB/xZdR07ES2clRDPxwE1IDe/QihKMpdS5WLmjAWe9YAFUFvvfcPtdelwfxZEZTRkWYVd1pzeSvrw0OlV02nTjFozXjIMRSzxKx2ZohRURe7C4C4pF0l3SqiGhubn6KCJuYiYQDUBTn467GLBK3J2fYsrtIOKB5VdxCzxOh9oxQDEXIgqZFaLjy3xBs7URmfBD9P/w00kP7PSlCGB87fTkA4LEXeipqlJf5QyKhgOPWRjHcaM3om3cdKiKsBc5TEbFxzXf68EuFiMTYXQTEI+bdqUeEZ++SMcFxdBfIZwawFo/boWZuj+4aOXVFPmH1eQo2EwNbXrdhQ74IaVuG+iu/jEBjO9LDB9H/g1uQGe31tAgBgHe/YwGWt9ZhIpHBz1865MkZ7NA/Ls6oCrizgTfOyaxaV8NvOtHu0jvA+eK7JJlV5SWVsT6+C/BJutPDbRzmiHA0q7LWTLSW31NQk0eTM14YVRkbVrQCAF47NK6pTAQnjMvr9u1D6+Jj0HDFZgQaWpAa2Iv+H/5rPrLd4yIEAHw+BVedthxAvj1jNwPCbfoEFyJtbHxX4AbeCU7juzyXi9qNeDd+jd3WDJlVJUabmrHamuGQdKeH2zhTRHiO72qtmRC/VgZLV3V78d0BlzNEjCxsDGNFWz1yKsW9c2XW8rqvrjoVjZd/Cf7aKJKHdmLgkU3ITY1JUYQwLlvfiboaP94amMRze4a9Po4p+gUaVQFdERmbTiMjyDsW5zQ1U8fxgc9OkjfDcWvG5r1OJPKcxGNSNpJVgZlJd3bRwm1sKiLsB4Tn+G6MY5gZgxlW3Z6ccTvMbDYbV+bbM5Vy85GeeBy45RatCHn5pAux4EO3wRcMY2rPNgw8dhtyybhURQiQb3N+6OQlAIDvb93v8WnMITJVFQCaC9N0qiquZasVIg6vZfpQgHOPj5MQS6etGfKISEommwMLPbRsVmXVqQOPiBMHNQCEBQSa8frhNeJVqFnPiPuju0Y2rsy3Z7buJUXEMWyD7n334SwAr77rCrRedEN+g+6OLTj80y9CTSelK0IYl5+S3z/zh12D3FcyiEBkhgiQn8pjRnZRDyiTvJJVOUa8O2nHO1Xh7T50i0Sek3iIMSrXrlnVWWvGmUeE5w4EBi8504gXoWaqqhpSVcVv3S0GK0Re7x33LEdlXmDcoOsL4M2LbkDTGVcAAMaefRTDv/kGoOa4Lq/jzXGLo1jSVItEOoc/V0DSqmiPCGBY/yDousC27zqemuH4wOekHc92ktl9+KVkVUkxpqLaL0ScKyJOA834Ts3weYowwsyqbt6MhyZTmE5noSj5zbte0BENYyX5RJzR3Q1s3JjPCKmNIvWRLyJy0oVQc1kM//abGP/TwwDELK/jiaIoOH91OwDgqTcGPD7N3Gjx7lGRhQi7LojxjnGLeOeYYO3kmu/0nkNmVUlJZvNvLEXJp6Vage/UjM2IdwFTM2IUEfbk455Zlakhi6Jh24oTDzYUVBHyiViEjeeedhpS3d1obFuG+r/7KsJLT0AuOYXBn34Rky//BgDQFYkIWV7Hm/esWQgA+N3OAS29WEYS6aw2ViuqNQPo3jFRigi31gzHB76kg2u+UxU+aWOdiWjkOYmHGMPMrIZ3OR2lUlXV9kpoRlhAjkg8xb8Q8cKsemgs7w/pdHHZXTGYYXVrNxUipmF+kBtuwGcGBtC66hS0fvQuBJsWIj3ah/7v/xMSe18AAHSFQtg7NCR0bwwvTl3RgkgogKHJFF7uGfP6OCUZKKgh4aBP+9kVgejrAnuocmq85zmd6OSa7+ThV1VV3axKrRm5YB4RO9n7jsNlDF/ndHw3mclxe8Jiscg8WzOaWdXFQkQbP2zyVqrXfSIx8omYYXAQuPjivB8ECu497W/Q8uHPwheqQ2L/DvR/72akh3vyG3TXrMG+WAwByZUQRk3Ah7OPlb8906cZVWu5pCuXQmS+UC6naoWD02sZr+nEbE7VfBq2ChHt4df6PceJH1Ik8pzEQ5yMMzltzRjfTLbNqobeJ68RXl6xyEaYWXXcRbOqZrYT2OM2A/OJqCqwrZt8IiWJx4GvfAU47jhkXnwRbaF6JD78WTSf9XdQFB8mXv4NBn70OeQSE6gB8ht0X31V+nbMbN6zpgMA8NQb/R6fpDT9Lv3s6PlC/K8LccOoLa/tu04VEWNLxe3WTMqQ1UKKiGQ4K0Qc9usKX+dTgKDf3lOH8c3Ma4R3UohZVQ8vcgsmL4t0/Ztl4yo2xkvtmaKwVsw//zNuGxpCtGMVGq6+B3VHnYpcOomhJ+7ByG+/CeQyqAEw4eIGXd6cfcwCBHwK9hyOY+/hSa+PU5Q+waO7DF0R4e8dY/4Qv09xPK7KK8Ha+PDpdmsm7WAwQyTynMRDUll78e6AM5kMMGSIBPy25U+fT+Hq6Ab4GbyMGKdm3Iq4dutiagbWnnmOCpGZMEPqKacUWjHAN066EAs+ehcCzA/y8L8g/upTAIAuvx/xeBw1dd76fpwQDQe198N/vylne8atIl4zq4pQRLSJGfvXVwYvsyrbth70K/BbHI4A+Cgifp+91xYFFSLQK0s7c9VOWzMJhwvvGDxn3AFd0owIMKtmc6pW6IimX3AypBU2FhbgvdEXc7U9JTUsJfWGG/CZAwfQVFOH6fffkg8pC9Rg6u2/oO+hm5Ae3AugMJ47NYVABRchDL09I2ch0ic43p0h0qzKvG4RDss764J8lt4ZHz7t4OThV0ajKuBSIXLfffdhxYoVCIfDWLduHf70pz+58bKmYQYeO/84jlszDjNEGDxT/wDDkwTHQiQc9Gt/X26EmmWyOQxO5JdpyaCItEfDWLkg7xOhbbzQskFy992HdwO4a9E70Pixb6B+9VlQsxmM/uFBHP7ZnVCTcQQB3LZxY0WM55rl/EIhsn3/qNClb3Zxq4hnLVsRxbl+HXPudQvX5K9d0+msI0XX6bZ1Jw+/SQkzRAAXCpHHHnsMN910E2677Ta89NJLOPPMM3HxxRfjwIEDol/aNHw8Is4UEW79Sy6jZVmtOONZiADuhpoNTaaQzanw+xRty6fX6HHvVdyeYYbUtWuReu01NEPBjlM/hIVX/huCTQuRGetH/w8/jdjzPwWgIgJg8sknceef/1wR47lmWdJUi+MWR5FTgd/vHPT6OEfgukdEiCLC74GKPeypKp8AS9uKiJPWjAP1XyTCT/PVr34V11xzDT7+8Y9j9erV+PrXv46uri7cf//9ol/aNI4KEYdxu/ryI16KiPOWBzOqAvneKk/0UDPxhQiTljsiIWn6oadVe7CZ0ZAai6Guvhnhv/k8ms/5Byj+AOI7/4Te796IVO8uAMAZ0SjGpqdRc+GFFWlKnYvzV8vZnklnczhcUGlEe0T01Q8p5DgHvLFrGY88pFrDNdrJA58WYGlbEbH/8JvOyrdnBhBciKRSKWzfvh0XXHDBjI9fcMEFePbZZ4/4/GQyiVgsNuOXGzCzqicpd9rCOz57EKZTzldps90MtUE/ApwrZz1LRHy6ar8LezKssqEQbPZmf8zVhFkp6O4GTj1VM6R+beV6LPqHf0ftipORSycw/OS/Y+gXX4aajKMOhVbM8DB8ksa184D5RP709hDX7dlOGZxIQlXz7eqWOrGtsGihEMmpwCSHzbZGmNfNabw7AAT8Pq19P+Xg30oLM7PtEbH/8JuSMFUVEFyIDA0NIZvNoqOjY8bHOzo60N9/5Pz85s2b0djYqP3q6uoSeTwNJ3IVL7Oq49aMNuPOQRER4A9huJmuagxkkoX2SBhHtTdAVfM3n6rAGNO+fz/q/UFMnve/0H7ZHfDXNSI12I3+hz6FyVeeBAB0BYMY37Jl3rViisGW4E2ns1KpZCwIsKMxBJ9gNTEc9Gt+Cd4+kUnOW8R5tMB1FdypImK/NVOVZtXZY1OqqhYdpdq0aRPGx8e1Xz09PW4cDykuZlVn47tOWzO8Uv8AY7w7/90sbm7g7ZcoQ8TIeYWlZ7KObXJlcBB4//u1mPbG1k60/t1XEV3/fgBAbNvj6CukpPoA3Lp6dX5pXYXmg1hFURSceXQbALnGurUiPupOES8q1Iz3zqxaDus09N1iTj0i9qdmggE5WtUMoY8bbW1t8Pv9R6gfg4ODR6gkABAKhRAKuW8qdOYRcbZrRnNQc1NEnBciIhURN82qMmWIGHnP6g58+5m9+MPOQaSzOemMY9zo7gbOOw+Z7m6sBDB64nuw4Pz/DV8wjGx8FENPfB2JvdsBABEAQ088MW+9IOXYuLIVj27rkcrA7HZbs6kuiP5YgvsDSlxbVcHnoYrHglF9z4z7UzNaa0aya47Q09TU1GDdunV46qmnZnz8qaeewumnny7ypS3BI+I95VAR4eYR4aCIiEhVZWjpqi74I5i8LJsi8s6lzWitr0EskcHz8zHufXAQuOoq4KSTkOruRgPLBrn4RviCYUx3v4jeB2/QihDNkHrxxVVXhAD6JNVrh8YRS8iRL+N2Ea+HmvG9LkxwvpaFOcQkJJ0qIkEOrZlq8ogAwM0334z//M//xHe+8x28+eab+NSnPoUDBw7g2muvFf3SpvF0fDfNVxHhMb6rbasU6BFxZ2pGTkXE71O09oxs0xKOYcvqHn4YZ05MoHHhUWidlQ0y+KPbkYuPIQjg1g0b5r0hdS4WNoaxoq0eOYn2EHmhiAD8rwvcWzOaR8S+F09vxzv0iDgJNLNZBIlCuBPsIx/5CIaHh/GFL3wBfX19OP744/HEE09g2bJlol/aNFrEux2PiIPqNP91nALNeCoi88CsmsuphohqecyqjPNXd+BHLxzEU28M4Pb3rRG63dQ1DK2YVgC5tRej/bz/BSUQRGasH4d/dZc2lhtVFAz/5jdV4wWZi40rW9A9FMfWvcM4b/WRbWu3cStVlSHquqD73fhcy/i0ZpxFNszH1owrlvRPfvKT+OQnP+nGS9lCS1Z1sn3XZo6ILtPJE2im91Ur1yMyHE8hnVWhKEB7RI4wMyNnHr0A4aAPh8am8WbfBNYsjnp9JPvE48CDDwJ33onUwAAigRAiF16HhuPPBQBMvfUchp74OtRkHH4AG+vr8fTQEAJVrILMZuPKVjzyfA+27pVNEXHJrFonxqzK+6EqzMGLpweaeTg1I5lZVa6yyCOcjDQ5bc1wU0TYD4j0UzPuBJqxC+mChpCUZtDaGj/OOGoBgApvzxh3xQwMINq8GK1X3Y2G48+Fmsti9A/fweGffwlqMo4aAFNf/jL+Z2yMipBZbFiR94m83jvuipG7HNmcigGXVyM01orZwMu7NVPHdXzX3vWVfV0yk7McNZ+WVBGR6zQe4SR/nxUiqWzOViqg05lyhja+y0ER4W3wMtLkUqAZG92VzR9i5AK29OzNIzN1KgKWklrYFfOVlevQ/vdfQ037CmQnRzHw6G2IPf8zAEAn25h7yy3APM8GsYPRJ/KCx3uIhiaTrq9GEOUR4W285+HFY/cbuwMKzA6gqrqab/W1q86sWgnwiHgH9P6bFZzOlDN4SIYM3k8RRliyaiKdE5okKevEjJFzjm2HogCvHYppPfmKYXAQuOQSLSX1lVMuRfuHPwdfqB6JntfR991/RLLntfyyutWrsX+ebMwVycZC6q7XwWbM5O3magRNKRXUmuGl7vLw4iUctuONX2e1PVO1UzOVACsg7CWrGt4UNnwiTmfKGXU1fFZUA2ILkYaaANi1LSZQgpYxVXU2CyIhnLy0GQDw35XUnilszc1s24ZWfwBTF9+IlnM/DsXnx8Qrv8XAo7chGx9FDQrL6l57bd5szBWJthCx29tCRE9Vda+IZ60ZntcEVVURLzyYNYSCXL4nj7ymhMN2vLGtYtUSoJtV5ZqaoUIEQNpBlRjwKdqN1Y55SF+AxGnpneSBZj6fYsgMEFeIMI9IR1ReRQTQd41sqZRCZHAQuOgipLq7EamNovYjX0LDie+Bmsti5L8fwMiT9wK5DKKKkm/FVGFAmV1YIfJ6b8xTn0jPSL4Q6Wx2T8ES0ZpJZnLI5tgWcU6KSNB5gnXSYTteURTb3kRZk1XpCgG9SgzZUETybwr741ROHdSM2pr813NRRDiPvM1GDzVzQxGpjEJk695hacKsisL2xWzYgMxbb6GpaRFarvoKwl3HIZeYxOBPPo+J7b8EkN8VM0ytGMt0RMNY2VYP1eM8kZ7RKQDA0hb31EQRgWbsgQrgs/QO4NSaYR4RB+14PUvE2jnSDu51IpHrNB7htG/mJEvE6Uw5ozbIrzWjrc7mtChqNqIc8kZk3TMzm1ULGrCyrR7prIo/vnXY6+MUJx4HbrstPxmzbx9aF70DrR+9C8HmRUiP9aPv+/+MRPeLqAdw25o1+V0xNBVjiw2sPeNh3PuBkXwh0uWBIsLTO8auY3U1fm6L+1gh4qQ141QRAQwbeG0qIuQRkRAn23cBvTpN2PCI8Fp6xzNHZJLliHB6ipiNPjkjRgFQVdX1QCYnaO2Z1yVsz7Ai5J57CpMx69F4+f+Bv74Jyf7d6P/+PyMzchA1AMa2bMGdr75KfhAHaIZVCQqRpS3uFSINoYBmjOXVlhLRYuaxXFTziPBQRKgQmT9oBh7bLmYHrZkMn0AzY+Kf1dny2Yg0qwKGFEVBrZnx6bRW4MnuEQGAi09YBAB48vV+jMTF7+Axzawi5KUT3oP2D38Wvpowpvdux8Ajm5CbGkMNgIl4nFJSOXBaQRF5oy8m7OejHLmcioOj+SK+y8VCRFEU7umqIq5jPMyqWoilE0XEZmsmSTki8uK4NeMg6S7JSRFhX5/NqbbGiBnZnKq1d3gZvGbTUp9/Yh4WdNNl/pCW+hrHf69ucFJnI05Y0ohUJodHnj/g9XHyzCpCdpz6YbT91Y1QfH5Mvvo7DP70C1BT07oplfwgXGiPhrFyQd4n8rwHeSKDE0mkMjn4fYrramIT5z1ULA8pyrHFXMumEznkiDi5Ntl9+NXNqnLd+uU6jUc4zd/XPSJ2xnf5BpoBQCJlvxCJG5Y5iZiaAYDWQiEyEk8K+f5aPHUFqCFA/mnw709fDgD4wdb9yDgoJLlgKELOAvDamR9F8zkfAwCMb/0xhp/4GpDLkilVEBs99ImwtsySploEXH5qbqzj6x1j5u9oLZ/RXcAwncghR8RRa8bmPYeSVSXGuSJif9+MPjXj7Mk96PchUOixTqXtb4ZkBq+gX3HcLipFayGtcXhSrCJSCf4QxiUnLkJLfQ16xxPeRr7PKEIUvHHe/0Lj6ZcDAEaf/i7GnnkIAHBGWxuZUgUhQyHipj+E0cR5rJ9lkkQ4KiI8I965tGYo0Gz+4GTXDOCwNcNJEQH4GFbjBoOXqI2wolszlZCqOptw0I+/PbULAPDdZ/d5cwgW2X7PPThL8WHnxf+I6Pr3AwCGt9yH2F9+AgA4Y8kSPNPXR6ZUQWxYkTesvtkXmzGC6gY9bGLGxdFdhrb4jntrhp8iEuagiCS5mFXtPfymtNFhuW79cp3GI9KOzar2ZLJsTtV2BTh5UzJ4yIaa01zQxAwAtDWwQkRMa6YSFREA+OjGZfD7FPylewRv9sXcfXFjZLsvgF3v/xctqGzov76KyZeeAFAoQvbtg4/2xQijIxpGZ3Mtcirw0oFRV19bL0TcV0R4m1WFtGYcPuypqsplv5jtqRmH9zpRyHUaj+DWmrH4pjCOgDmR6Rh8FBEWiSzuRtNSn2/NjAhqzegZIvLGuxdjUWMtLjpuIQDgITdVEVaEbNuGVn8QiQ/eivpjz4SaTePwL/5/xF//PQAqQtxk/bJ89P8L+9wtRLxszfAONYtN5x+qIjzHdwsPe6lszpaXK51VwXajOgo003JE7LVmZNtILtdpPCLpVBFhxiGro1SGwkU6RUTQxAygt2biqayQxXeVqogA0Eyrj798SGjgm4axCAmGEP7rz6HuqFORSycx+NMvYvqt5wBQEeI265bn2zPb97usiIy6H2bG4B3zPpEUp4gA9q6zxsKBj0fE7q4ZuW79cp3GA1RV5egRsaeI1Ph9XJL/eKT+adsqOfZVZxMNBxD05/9/RfhEBsYrI1W1GKcsb8bqRVEk0jk8tq1H7IsZi5CaWtRe9nnULn8ncqlpDP74DiS6XwRARYgXMEXkpQOjrk1RJdJZDMTy7VJPzKp1nFszBUUkWsvvfRsK+MCsc3YKEWPopROfhh6iSWbVeQHzaABOChFnrRlexiEeqX9xzmuzi6EoiqaK8G7PTCTSmCj8P1TK+K4RRVFw9enLAADf37pfW9rFHUMR0hZuQN1H7kS463jkEpMYeOyzSPa8mo9s37iRihAPeEdHBJFQAPFUFjv7J1x5zYMFNSQSCmhFgZs01fLdQTVR8IhEOG3eBfI/n04WjBqv+U6GAZzmiFAhIhlpw9OG24Fm7E3kdPMug0fqnxtmVQBoLfhEhjgbVgcK/pBoOCAsB0U0H1i7BE11QRwcncZPtgtQRYxFSKQVdVd+GaHFxyA7NY6BR29Dqncnoj5fPrL9z3+mIsQD/D4F79R8Iu4Em7Gtu10tdcIm5soR5W5WZYoI36KqzsHiOx5hZoDe1rGsiFBrRk5SGQ6FiOYRsaeI8BjdBfik/sUF7GcoRmuDGEVE94dUllHVSDjox7XvXgUA+OJ/valNMjjGuEF32zYsaO1Ew0fvQk3bMmQmhjDwyCakBvYg6vNhmCLbPUczrLrkEzng4eguYPSI8DKrMo8I32tZmIMi4vSaH7apiPBK8uZN1V9lWIXo9yna0iWr2G/N8J3pri28uZ2YVVkhwjMEqBit9WJGePsq2B9i5BNnrsT6Zc2YTGbwTz96xXmLZvYG3cXHoOHKf0Mg2o70cA/6v/8vSA8d0IsQCirzHFaIuGVY9XJiBtADzWKJjOP3u6qqWo5IhLPfzUlrJqntFnO60sO6IqKqquH15br1y3UaD3BqVAWctGZYdcy3NeNEEZlwSRFhI7y8zar9FTwxY8TvU/DVv1mL+ho/nt83ggf+uNf+NxscBN7//lkbdL8Ef20Uyd6d6P/Bp5GdOExFiGSsXdoEv09B33gCh8amhb+e14VIo6GFEnPYnklmctpDJs9dM4DD1oymSDi79doZkJgxOkyKiFwktblqJ8Yhu1MzfGUyrTXDQRGp9NZMpSsiALC0tQ63v+84AMBXn9qF13vHrX+T7m5g40Zkfv97LFd8ePVdV6D9rz8HXzCM6T0vYODR25CbjlERIiF1NQGsWRQF4I5PhLUAOz0qRAJ+n5b54TTmnRUyisLf7xZ24MVLcHr4ZF9vJTJixugwKSJyoaeqcgiXsegR4S2T8TCr6oFmYivmVkEx71q8ewVOzBTjsvWduGBNB9JZFZ967GXzUuzgIPC3fwusWYNUdzci9U1I/s0X0XTGFVAUHyZefhKDP/si1HSSihCJWedSe0ZVVa0Q8UoRAfgZVplRNRIKcIlGMOJEEeHVjtfNqubvOcYHZSpEJINH9r7t1gxnRYTH+K5bUzOi9s3MJ0UEyI8Lbv7QCWhrqMFbA5P4zOOvlf/3LSgg6OpC5tFHcUYigejSE9B69TdQu/wk5FIJDP3qboz89t/1DbpUhEjL+uXuJKyOxFOIp7JQlPzmXa/gZVgVEe/OcJJgzasdr5tVrSgi+r3Oi6moclT9XB6P7H3bZlWOC+8AIKwFmtlflKXliIg2q2obeMWYVSt5amY2rQ0hfPnDJ+Kah17AT7YfxPb9o/jSB4/H6avagMWLgb6+GZ+fArAKwGDbUkRO/SA6jjsXis+P1OH9OPyLzcgMHwQAdNXWYt/YGC2vk5j1y/IJqzv78wvwRK1e6BnNK4kdkbCnExW8Qs1EGVUBoDZovwXOa9s6+zeyoojwzq3iiXwnchm+ZlW7gWacFBEt4t1+EqOWrCraI8ICzTgqIvFkRruALW6aX0/4563uwLevWof2SAjdQ3Fc8R9/wb/8+BWMjk7ihwAUw69o53FIfvhzWHTNfWg44T1QfH5M7ngK/d+7GZnhg/ADeFdjI/ZSESI9CxvdWYDntVGVwSvUTBvdFfBAVVuTv97b8oiwaz43s6oFRYQVQZIZVQFSRPQlQAEHZlW7u2Y4OagZumRoXxGZdNmsOpXKYjqVnbHDwS59BX9IJBwQ8iTkNRcetxCnrWrFvz25Ew9vPYAfbz+IJ//xexgZG8JiRYECAP4gApFWAICq5jD11nOI/eWnSPW9BSD/5DH15S+j5uabAQoqqwjWL2vGwdFpvLBvFGcevUDIa3i5dddII6d9M0JbM0H7LXAt0Mzhw2fIhiIi6+guQIUIl6Q5pmikbLZmeCkiTpfeqapqiHgX+9ZoCAVQ4/chlc1hOJ5EZ43zC+ChsXxbZvE8asvMJhoO4s5LT8Cla5dg089exduDkwg2L5rxOWomhcnXfofY8z9HZrRX+3gEwNDEBGoaGlw+NeGEdctb8PjLvUINqz0eh5kxGjmZVfXWjAhFxH5wJLdAMxs5IrynNHlChQiH7H2n47tOZTqGExMVOw+bMxetiLB9M/2xBEbiKXRy2PbZV8hamG9tmWKsX96CJ248E2/2xXDKhtMAqICa/8fLjPYhl9D3k9QC+NTSpfjirl3wkSm14pi9AC8gIJ5bntZMQRGZdmhW1Voz4hQRe60ZPh4RO75EmRUR+U7kMnoh4mR812GgGW9FxGYhwtZmK4ruNxEJa88Mc8oS6S0UIos8dP27SdDvw4mdTXjwrs8i1bsLqb63kOp7a0YR8q9LlmByyxZ8qbubipAKxY0FeNIUIsysyqs1I0ARcTKdmOQ0oGBHEUlymBAVhXwnchmerRnru2b4KiJO5tsBPUOkvob/7H0xeI/w9hYmZrwcP/SCK664AuoPfgAVyP/q6IC6cSPUJ5/E5gMH4KOdMRWN6AV46WxOK+K9LkQamVmVU2tGpEfEznQir/aIURFRVXNx+LwW7omg6q9OeqCZF8mqfBURJ4l/gDFV1Z03ahvnEV5NEZknGSKWuOKKfGtGVYH+fuC554ALL6QCZJ6wYUV+jPcv3fwLkb6xBHJq/jq2IBLi/v2twC1HRGRrxlHEO5/2iFFRMXvfofFdieE5vpvK5pCzsKyJd4XKFJFkxto5GG5NzDBaOI/wsgyRxVWmiBDzH1aIPN89YvoJ2CwHDBMzXgdd6WZV+5N/gGCzqqOld3yu+cavN6vE660ZUkSkI8nDrGp4U7BWjxl4V6jGEVg71bpbEzMM5hEZ4uARUVVVU0Tm89QMUZ2c2NmEcNCH4XgKuwcnuX5vWfwhgDHQLOWo4BI5vuss4p2PRyTgU8C652a9iUlOry0C+U7kMjynZgBrPhHeEe/GFo+dHxK3wswYeqiZ89bMSDyFZCYHRQE6Gr2VlwmCNzUBH05emveJbOXcnukZLSgizd4X8CzQLJ1VHe3MihUUFRGtmbCTQoRTZIOiKJbTVUkRkRimYAQdtGbsVKfGz+VVofp8iva97MiG7rdm8gUDj9YMa8u0NYSk/EEjCKdsWJEPqvvL3mGu3/eAJGFmQP5ayB4KnRhWJwqKiHStGY4hluwBOGFREeE1HMET+U7kMmkOioiiKLbmunnNlBtxEmpWya2ZQ1qGiPdPdQQhgg0rdcMqT5+IDFt3GYqi6FkiNg2rmWwO8UKRILQ1YydHhGOIZdji1nca35UYpoiEHIYE2ckS4b30DgDqHKT+TbLxXZemZnjum9HCzKpxYoaoCtZ2NaEm4MPhiSS6h+Jcvmcup6L7cP57LWut5/I9neI0XZUZVQGxishUOmu5IGTtpjoOKy201ozJe47uT5FPMaZChIMiAhhkMg89IvnvZX8hk66IuLOnhW3gnU5nHW0MBvQMEVJEiPlKOOjH2q4mAPzGePcOxTGRzCAc9GHVAjkKEaehZqwQqQ36HbXcS8GGAlTVemTDlPaw57xA0mIjSBGpfPgVIjZaMwIid5kiYif1bzLBChF3Kub6Gr/29+40XbWqM0SIqmEjyxPh5BPZcXAMAHD84kYh0fF2cBpqpk/MiGkx1xoeHK0qz/HCAxcPRSRkcfkemVUlJsnBrArYW8ssQipzsgdhMuWuWVVRFLRxSldlhUi1paoS1cWGlQXDKiefyI6D4wDy48Gy0ORwA29MM6qKUXYDfp+WO2XFi6eq+iQQT0XEtFlVgBWAF/KdyGV4mFUBo0fEnCKiqqpeoXJ8YzhJ/Yu7PDUDAC0NfEZ42dRMteyZIaqTk5c2I+hX0DeeQM/ItOPv90pBETmpq9Hx9+KF08V3+uiuuOuYNp1ocddLthA0ydMjYrY1I2I4ghdVX4jw2DUDWN83k8rm2LJUMVMzNjwXbk/NAPoIr5PWTCabw0CMeUSoNUPMX2pr/Jp6sbXbWXsmnc3hjd4YALkUEc2s6lARETExw7Cz6dyoUrMWuhPCNhURGt+VEN5mVdMpdwblhKdUxn5AbLVmCkYqNwsRHq2ZgYkkcioQ9Ctoq6cwM2J+o+2d2evMsLqrfwLJTA7RcADLW70f3WXoKqm9a4Ie7y6uENGmEy0oIuxBLxz0wc9hqWjI6vhumsyq0pLi5CS2uviO+UMUxbkaY4QVEexNb4XJZP5JwtXWDIcRXuYPWdgYdmVrMEF4ie4TcaaIsLbMiZ1Nnu+YMdLq8OFEX3gnsjVj/YFP84dwUEMA64qINhxB47vywSNZFbA+NWOsTnleBNjcfCxhpzXjviLCRniHHGzgpR0zRDWxblkz/D4FB0entSA/O+zoYUZVefwhgH5NsLuV243WjJ1QM21ihtNUouWId1JE5IVba4aZVU2PUokJl2FyJPthtIIe8e5excwj1Kx3jDJEiOqhIRTA8UvyxYOTMV6jIiITmiJi0zcmcvMuQ0+wNv/Ap2WIcFJE7NoByKwqIfzMqlZbM4UwM85vCjY7P2FREUlnc1pR5q4i4uyiAwB94yzenYyqRHWw0aFPZDqVxduFLb4yTcwAuiIykcxYikNg6K0ZN8yq5nOjeGaIAHYi3ml8V1q8CjRLCFpAxBSRCYuKiNFTUqkekUXUmiGqBLZ3xu7kzOu948jmVCyIhLAwKlcBHw0HEPTn29V2rgvuKiJWPCJ84xGsKiI0visxXk/N8FZENI/ItDVFhP3whgI+IbHIpWhl47vxpO2AJtaaoTAzolo4ZXkLgn4F+4en8NbAhOWvf6UQZHZSZ6NURlUgH3TY6mCs35XxXRsxCcyDx1sRMe0REZDkzQv5TuQy3FozmkfEmiLCWyZjcuRE0qIiknI/QwTQWzOJdM7WyDEA9BZaM4uoNUNUCZFwEGcdvQAA8OsdfZa/foek/hAGU0rtmNi1QsSN1owdRYTX1EyQ7Teb+wyiAjR5Id+JXIZbsqrl1owYmYyNrFn1iHiRqgrknw5YhW5Hhp1KZbQoaDKrEtXEe09cBAD49at9ltVEPdpdLn8Iw4l3jF37RI7v2slr0hQRTsMAVu45xgBN2r4rIZoi4nprRrRHJGPp4uRFmBlQ2DfjYISXtWUaQgGhT0AEIRvnr+lAjd+H3YOTeGtg0vTXjU+n0T0UByCvIsKuCcMWVz+oqqoXIiLHd4PWx3d5KyIhC4qIsVih1oxk5HIq0tn8zdqzqRnO1SmbmsnmVIvVujetGcCZYZUmZohqJRoO4qx3sPZMr+mve7WghnS11Go/e7JhN9RsKpXV9rmIfDCpY8GRlnJEmEeEl1nV/PZdZhngHaDJC/lO5CJMDQF45IhYXUAkxjhUG/Rr8cFW2jOTCfczRBhOZFiamCGqmUsK7Zn/stCekTU/xIgeambtmsD8IQGfInRMNVIoRCYtTCdOcc5pCltYtGq838hmTgaqvBBJGwoR58mqNqdmOCsiiqIY0lXN/5BMeuQRAXRFxE6kM4WZEdXMeavbURPwYe/hOHb2m5ueYUbVtTIXIlqombXWjLZ5tzYo9IbbULjGTlpYpcFbEdGnZsy3ZmQc3QWqvBBJGSpJbtt3LeaIiKjaozayRLxszdi96ADGeHdqzRDVRyQcxNnvsDY984qk0e5GNJXU4sMJu+aJzBAB9OukFdVZzxHhZVY1r4jIPLoLVHshou2ZURwvS7PsEdHeGPwrVDv7ZiY9Gt8FdBnWnkeEFBGiurEyPTMYS6A/loBPgRYTLyNOWzOijeu2FJGkKEXETCEiRoHnRXUXIhk+GSKAjV0zmlmV/z+BHmpmXRGpvNYMZYgQ1c15qzsQCvjQPRTHG32xsp/LgsyOam/w5GfdLLpZ1VrQod6aEfv/pnlELBQi+tQM74j3ue85ojyJvJDzVC7BCpEgh38cpmykzG7fZbn/AhSRqGGE1yzMrOqFItLWoF90rKCqqhZmRqmqRLXSEArgnGPaAczdnvnBX/YDyG/wlRm7QYdaaybkkiJiISZBzxHhHfFuXhGRMcwMqPZChFOqKmB/fFfEGyNipxBhmyE9UUQKrRmLMuzoVFr7e1xIHhGiijHTnvnDzkE8veswgn4FnzhzpZvHs0xdTUCLUbfSnokl3FFE2ANbJqeavuaLUkRS2Zw2slyKpMR7ZoBqL0Q4paoChtaMxUAzET07O1Mzcc6jZVZgMuxQPGVJhmVtmbaGkLQ/YAThBuce245w0If9w1N4vffI9kwqk8MXf/0GAOBj71qBlQsa3D6iZZgqMmRBKXXLI2IMJTP7wKdNzXBWRIC5lXiZN+8CVIgA4FSIBKzmiIhZegfoiYKWpmZS4jdWlqI9GoJPyf97HLYwOaNNzJA/hKhy6kMBnHtsvj1zxy9fP8K78L3n9mHv4TjaGmpw/blHeXFEy9gxrDKPSERwIeLzKZoqYsYnks7mtPsNb0UEmHuElxQRifG2NSMm4h2wt29GyxHh5Oi2QijgR1dLHQBg96D5qOqeUTa6S/4Qgrjh3KMRCQfwwv5RfOzB5zWVc2gyiXt+9zYA4F8uPKZiViG02Rjr1zfvir+OaYWIieus0efCa2rG71MQ9OenPRNzKPE0visxfBWR/PdIZXPIzdGvA8QGzNiZmtGTVb1x0h9VkIr3WChE3ioEOL2jQ36ZmSBEs3pRFN+/ZgMioQC27RvFxx7chngyg69seQsTiQyOXxLFX6/r8vqYprGTJaIvvBNfbDHDqplN58wfEvQrXO43DLNKfNWO7+7btw/XXHMNVqxYgdraWqxatQq33347UinrI5qiSPNURAz/wMbo+FKIVUSsm1W9DDQD8uOEgDVFZGd/vhd+zMKokDMRRKWxtqsJ3/94vhh5ft8IPvLAc3h02wEAwO3vO05b/1AJMBO7tdaMO4FmgDVFhHeGCIN5PuZSRKp2fHfnzp3I5XL49re/jddffx1f+9rX8K1vfQu33nqrqJe0TFKAIgKY84loyapCFBFrhUgqk9OMVE113si2q1ghcthcIZLLqdrG0WMWRoSdiyAqDWMx8tqhGFQ1v5PmlOUtXh/NEnbG+vXWjPjrWMRCqBnviRmGVUVE1kJEWNl40UUX4aKLLtJ+v3LlSuzatQv3338/7r77blEvawmerZmAT4FPAXIq68eV/0HQpTKBgWYmzapjU/knDp/ijqRZDKuKyIGRKUynswgFfFjeWifyaARRcaztasL3rjkVf/ed56EA2PRXq70+kmXsLMNkD1+uKiImChHeGSIMpqjPaVbVckTkbM24qsOPj4+jpaV0VZ5MJpFM6tVvLFY+JdApPM2qiqIgFPBjOp01uQ1RnEdEn5oxp4iMTuULlqa6GsdR93ZhhchALIlYIj1nQcQWfB3d0YCAhGutCcJr3rm0Gc/+67lIZnJoK0ygVBKthdbMkBWz6rQ747uAtX0zohQRpqgn5rjn6Aq8nNdK1061Z88e3Hvvvbj22mtLfs7mzZvR2Nio/erqEmus4pmsCljLEkkKXHpnlAznCroB9B0vzR61ZYD8haM9kr/wmDGsav6QDvKHEEQpIuFgRRYhgK6ImN1BlczoD4FutGas7JvhvXmXYXa1iDa+K6kiYvkueMcdd0BRlLK/XnjhhRlf09vbi4suugiXXXYZPv7xj5f83ps2bcL4+Lj2q6enx/r/kQWYWTXE6Yma9d+8XkJklCXNGKlGC60ZtvPFK6y0Z3YVFJHVi8gfQhDzEaaIjMRTpiYRjcqEG6b7iJXxXUGBkWYVEdnHdy3/a11//fW4/PLLy37O8uXLtf/u7e3FOeecg9NOOw0PPPBA2a8LhUIIhdyr3nl6RACDcWiON0U2p2ptIRFvjFDAj1DAh2Qmh1gijcY5lA5WiDTVeV+IPLtn2JRhlRUiZFQliPkJezDK5FTEEuk5r0/axEwo4Mp0kAyKSNikRyQhuSJi+W+lra0NbW1tpj730KFDOOecc7Bu3To8+OCD8Pnkqsb4FyLmWjPGPxc11x0JB5GcTJrqX44WpM8WCQoRYO7WzHQqi33DcQBUiBDEfKUm4EM0HEAskcHQZGrOQkTLEHGhLQMADSHzXjxRiojZh1/ZFRFhp+rt7cXZZ5+Nrq4u3H333Th8+DD6+/vR398v6iUtk+RoVgWMHpE53hSG1o2oN0bUwuTMSDz/Oc1et2YWmGvNvD04gZyaf2JaUKH9b4Ig5qZNi3mf27DKrnVuranQFZG5r7GiFZE5PSLVOr67ZcsW7N69G7t370ZnZ+eMP7Oy2Ewk3M2qJme6WfhMwKcIm/iIWJicYeO7XppVAT1L5MDIFBLpbEm1iE3MHLswAkWpnIAmgiCs0dpQg71DcVOGVbZnxq0IgoiF8V3hOSJzKiJVumvm6quvhqqqRX/JAs9kVcB8a0ZbeCewX6fvmzGhiLBCxGNFpD0SQiQUQE6F1nopBvlDCKI6aDFs5p6LCa8UESvJqpxNtOY9IrR9V1rEeUS8X8lsZd/MqDa+620hoiiKnrBapj2zy6CIEAQxf2m10ZpxzyNiJdBMUI5I4WHWdKBZtSkilUCKc9/MrEwmMsyMYWXfDAs0a6n3fiunblgtrYjQjhmCqA70DbxmFBHWmnFJEbEQaBYvtGa454hYfPgVsduMB3KeyiW0ZFXegWYmZTKRb4qIthnS/NSM14oIYMgSKTHCOzSZxNBkCopCW3cJYr6jKSIm9s3oC+9c8ogUrrHJTE57qC3FVMGsyn1qxqQiotkBSBGRj1Qm71cJcveImFzJLPBNoS++K9+aSWdzWrEiRSEyx+QMa8ssa6nj/nRBEIRcWNk3E9PGd925LtQb/B7xOR742J97poi48PDrBDlP5RI8d80AVloz4t8U2vjudPkfkFHjwjuXeqvlYIrI3sOTRePpd5JRlSCqBmZWHbZgVnVraibo92k+v7l8IqIUEeseETlv+XKeyiVShb6Z24Fm+gIi8YrIXDkio3F94Z0baYRz0dVSh5pCKuyh0ekj/nxnH/lDCKJasJQjMs0277r3QGU21GxKkEdEL0RKP/yqqkpmVZnhPjWjeURMtmbc8IjM8QOix7t7r4YAgN+nYGVbPQBg9+GJI/5810BhxwwpIgQx72ktKCKjU2lksuWvq4MTCQDu7syKmIx5Z+O79cJaM6Uffo0KPY3vSohXrRlR/UIjrM0ytyIiR7y7kVIjvNmcircGqDVDENVCU10NmFDL8o6Kkc7m0FNQUJe31blxNADGEd7S19lsTsV0muWIiGrNlL7nGO9HpIhISLpgVnW7NcNUCpHBO2YVkRFJFt4ZKWVYzSeu5hAO+rCstd6LoxEE4SJ+n6L7RMoYVg+NTiObUxEO+tARCbt1PFMjvNMG/4Y4RaRcIZJ/fUUBgn7v2+/FqOpChPv4rkkHsxsJgFGTUzNjEmWIMI4qoYgwf8jR7REp/CwEQYiHFSLlYt67C0nMy1vr4XPx2mBmAy9beKco/FsjTBEpFxmRNIzuyroSo7oLkQzn1oz2ppirEBFvqmKFSCJdfsad/XB7He9uxFiIGFcC7KREVYKoOlrr84bVoTKG1X1D+UJkWat7bRnAsG+mjCLCFt7V1wS4FwJmIt5lDzMDqrwQSQqLePe+NdNg+N7lVJHRKXnCzBgr2urhU/K5AIcNFx/aMUMQ1YeZLBFWiCxvc7dla0YR0T2B/P0ZZnyJepK3vLd7eU/mAmx8l1+gmTmzqr6uWpwi4vcp2l6Dcv1LGc2q4aAfXS35J5vdg5M4NDaNh7fux/P7RgAAx9LoLkFUDW0m0lX3DU8BAFa47B0z4xHRM0T4P3haUURELll1SlVHU6azedmf364Zsx4R8YoIkJ+ciaeyZSdnRqZYjog8HhEgb1jdPzyF637worYLB8j/HR+/hAoRgqgWWk2YVdm2bikVkZQ4RUSbmilnVq0ARaSqCxGvds24lQAYCQfQN16+Wh8rtGbcnL03w7GLIvjdzkGMTqXhU4B1y5px9jHt+KsTFkk14UMQhFhaGsqnq6azORxko7suKyJmPCJTgjJEAL24yOZUZLI5BIqo+7KHmQFVXIhkc6oWIc47R2SuBUhuKSJm9s3IaFYFgI+fsRI1fj9WLKjHWUe3UfFBEFUKM6uWSlc9WBjdrQ360RENuXk0a4oI5wwRYGa7JZHJoaFoIVIwq5IiIh/GYsH98V22rlqsIjLXvpl0NqedRSazKpAvjG48/2ivj0EQhMe0zaGIGCdm3B5P1SLeTYzvilBEjA/RyXRW86wY0TbvSuwRkbdEEoyxEOFmVg3OPTWTzua0gBu3FJFSHhGWIaIoQKMEC+8IgiBm06rtmyleiHQXCpEVLvtDAEOyahnVmY3vivCI+HyK9iBdyidSCYqIvCcTTCprLET4VNHa1EyZHBFjL7FBeCFS3tGt7ZmpDVJAGEEQUsLGdyeTmaLTIcyo6kXaspldM2zhnYipGQAIB8pPzmgeEcoRkQ+jUZWXnGemNcOKgtqgn5sSU4q59s2w0V3Z2jIEQRCMSCigPSwWS1fVRndd3DHDaDATaMbMqgI8IsDcQZpubHt3SvUWIqxK5FgMsEIklc0hl1OLfk7MhXh3hllFRDajKkEQBENRFCxszO+Pmb32ATCEmXmgiDBVO57KasMPs5nSxncFKSIsS6SEJUAb3yVFRD5SnFNVAb0yBWa2foy4NTGTf43yUzMj8fzHmyXLECEIgjBy6vJWAMBze4dnfDyVyeHgKFNEvPOIAPp0zGz0iHdBisgcloBKGN+t+kKEZ3vEaAYq9aaYcCFVlTHX1IyM8e4EQRCzOX1VvhB5ds/MQuTg6BRyat4IuiDi7ugukL/ms7ZRqfYMm5qpE+URmUMRYa0ZMqtKCO8wMwAI+BQwz2epyRk3FRFtA2+yvEdEtjAzgiAII6cVCpFXD47N8LwZjapebJZVFEX3iZQwrBqX3okgHCi/gVc3q5IiIh0iWjOKosy5bybmUqoqMLdHZIRNzZAiQhCExCxuqsWKtnrkVOD5vSPax7uHvDOqMuYKNZsSGGgGGGMjaHy34tAUEc6TK3NlibiqiLCpmenyOSIt9eQRIQhCbpgq8uc9Q9rHvDSqMlioWenWjDuKyJzju1SIyIcIRQTQ/7ETc3pE3J2aUdUjHd0jNL5LEESFwHwizxl8ItqyOw8LkcicrRlxS+8APTG1lCKije9Sa0Y+tEKEtyKitWbmUkTcaM3kXyOTU4sWRmM0vksQRIVw2sp8IbKzfwJDhb0zXm3dNaK1ZuZSRASZVUNmA81IEZGPtACzKmAINSupiLjXmqmv8Wvm2WKhZqSIEARRKbQ2hHDswggAYOveYaQyORxiW3e99IgUCoxi+2ZUVdUUEWHju0HWminhEUmTWVVahLVm5jAOxVwc31UUpWSWSCabQ0xbeEceEYIg5Of0VW0A8mO8PYXR3foaPxY0uD+6yyiniCQzObCcM1Hju3qid4nxXTKryktSlFnVdGvGncXH7HVis35IxqZp4R1BEJWF0Seib931ZnSXoXtEjlSd4waVpFaQIhE2qYiQR0RCRJtVSykibppVAX1MePbkDMsQaawNIiB45w1BEAQPTl3ZAp+S37jLws28SFQ1Ui5HZKqQIVIb9AtbLKoFmpX0iJAiIi3MI8J78ZxZj4gbOSJA6SwR8ocQBFFpRMNBnNDZBAD46YsHAXjrDwH01kyxvCbNHyIoQwTAnNlVZFaVGHGKiGytGeYRmflDMjpFe2YIgqg8WHuG5SAt83B0FyiviLDNu6IW3gFzKyIJas3IS0pQlVjOrJrO5jBdeLO4YVYFgGgt84jMas3QnhmCICoQVogwvG7NRMqYVacEZ4gAZhQRas1Ii4hdM0B5j4jxjeq2R2T21MwoZYgQBFGBrF/WMmPIwMswM8CQrFpGERGVIQKY8YjQ+K606Nt3+RqIQmUWELH2SDjo4+5NKUUpjwgtvCMIohKprfHjnUubAOTbIm0N3l7DynlE3FBEtGTVIr7EXE7V7nVhUkTkQ981w/cNUk4RcTNDhFFqamYknv99E3lECIKoMFieyLLWOk9Hd4E5PCKCN+8C5XNE2H0OIEVESkQHmhWTyfSJGXfaMkBpRYTFu7eQR4QgiArjw+uWYPWiKK7YsNTro+gekeSRO72mkmI37wLlc0SMKonMHhH37oiSIaoQYeFgY0U23k54oIiUmpoZIY8IQRAVSmdzHX5z45leHwOArohkCzu9ag1tGDcUkXCZje8sVdWnAAFBOSY8kLdEEoyoQqQjGgYADMaSR/yZ26O7QJmpGcoRIQiCcExdjR+sOzQxK13VDUWE+RLLKSLhoN/zFlY5qrcQ0TwifP9xFkTyOw8GJhJH/BlTRNwKMwPmzhFpqSePCEEQhF0URdF9IrOus24qIokiikgljO4CVVyIiNq+K5siou+a0Sv1TDaH8WlmViVFhCAIwgmREoZVV3NEiikiWl6WvEZVoIoLEfYPxHtqpr2giEwmMzMWHgH6mmhXWzNhfca9Z2QKALQiBACaaOEdQRCEI0pt4HUjRyRkUERmm2XZ0ARTTWRF7tMJRJRHpCEU0KrfwYmZqogXZtW2hhqc1NkIVQU+9t1tGJ9Oa2FmtPCOIAjCOaw1M+GBIsKmZlQVSGdnFiKkiEiOqEJEURRDe2amTyQ27b4ioigKvn3VeiyMhrF7cBL/38PbtbYR7ZkhCIJwTgNTnj3wiBj9H7N9IppHhBQROUllxSSrAkbD6kxFxItAMwBY2BjGd64+BfU1fjy7Zxifefw1ADS6SxAEwYOSHhEXpmZq/D5tamd2fhWbpCGzqqQws6qIf6BSiogXZlXGmsVRfPPKk+H3Kdg7FAdAYWYEQRA8KJWuOuWCIqIoip6uOsuwyhQRmTfvAlVciKQEmVUB3bBa2iPiTY7c2ce04wsfOE77PU3MEARBOKfUvpl4wSNSL1ARAQz7Zma3ZkgRkRtRHhEA6IgWWjMlFBE3c0Rmc+WGZfjk2asAAGu7Gj07B0EQxHxBV0RmB5rlC4M6gYoIAIRLhJpVilmVIt6FtmZmKyLetWaM3HLRsfiHM1agrSHk6TkIgiDmA5Ei47upTE7zIopszQC6GXW2IsI8I2RWlZSkK2ZVXRFJZ3OYLrwp3DarFoOKEIIgCD4U84hMp/SioFbg+C5Q+YpIVRYiqqoKS1YFdEXksEERMVbKXisiBEEQBD+KeUSYP6TG7xNynzFSavEdRbxLTCanggXQhQSaVScM6arsDRoO+hCkEDGCIIh5QzFFRAszE2xUBUovvtPGd6k1Ix/MHwKIUUSKpat6lSFCEARBiEXziBgKES3eXbA/BDDEvKeLKyJhas3Ih+hCpFi6qixGVYIgCIIvDaEjk1XjLsS7M7TFd5lZHhFSROSF+UN8CuD38TerAkemq3qxZ4YgCIIQj+YRMbZm2OiuwIV3jHAJRSRRIWbVqnw8Twoc3WWUUkSipIgQBEHMK5hHJJXJIZnJ4sX9Y7jnd28DAOpdUET0QLPZikhlbN+tyrsim+2uEWganZ2u6nWqKkEQBCGGBoPq8Xf/93n8pXsEQL4tc/mpS4W/PpuKOdIjQoqItOhhZuL+cWanq2oekRC1ZgiCIOYTfp+Cuho/plJZ/KV7BAGfgis2LMUN5x6ttelFwhSRI3NEKmN8t7oLEQFhZozZ6aqsd0iKCEEQxPzjqPYG7Dg4jvedtBj/9J53YHlbvWuvrS29OyJZNX+vk33pXVXeFUWGmTFmp6uSWZUgCGL+8v1rNiA2nUZXS53rr11aEamMpXdVWYiI3DPDmJ2uGmNm1dqq/CsnCIKY1zTWBtFY682DJiWrViBJFxQRVohMJDOYSmUMOSKkiBAEQRD80HJEZisiWo4ItWak451dTfjR/z5NaJXI0lWnUlkMxpI0NUMQBEEIoVSOiJasSuO78tFUV4NTV7QIf52OaBjdQ3EMxBKITVMhQhAEQfBnzmRVycd3XSmTkskk1q5dC0VR8PLLL7vxklJgTFfVA82oNUMQBEHwo7QiUhlmVVdOd8stt2Dx4sVuvJRUGNNVadcMQRAEIYJQkWTVbE7VwjtlH98VXoj85je/wZYtW3D33XeLfinp6CgoIr1jCUwXKlUyqxIEQRA8KZasalzuKrsiIvTxfGBgAJ/4xCfw+OOPo65u7tnqZDKJZDKp/T4Wi4k8nnDaC+mqew5Pah8jRYQgCILgiZYjYhjfNY7yyl6ICDudqqq4+uqrce2112L9+vWmvmbz5s1obGzUfnV1dYk6niuw1szuwXwhEg76EBS434YgCIKoPsJFxndZuJnfpyAg+X3H8unuuOMOKIpS9tcLL7yAe++9F7FYDJs2bTL9vTdt2oTx8XHtV09Pj9XjSQUzqx4amwZAbRmCIAiCP6EiZlVtdFdyNQSw0Zq5/vrrcfnll5f9nOXLl+POO+/E1q1bEQrNXPizfv16XHnllXjooYeO+LpQKHTE51cyTBFhUFuGIAiC4I3emtEVEW1iRnKjKmCjEGlra0NbW9ucn/eNb3wDd955p/b73t5eXHjhhXjsscewYcMGqy9bkRxZiJAiQhAEQfCFeUBSmRxUVYWiKJo6Irs/BBBoVl26dOmM3zc0NAAAVq1ahc7OTlEvKxXGdFUAiJIiQhAEQXDGOJ47kcwgGg5qiojso7tAle6acROjKkKtGYIgCII3dUE/ljTVAgC++fvdAIypqvLf5l074fLly6GqKtauXevWS0oBM6wCQCRErRmCIAiCLz6fgi984DgAwH/8aS9e6RmrmM27ACkiwiFFhCAIghDNeas78IG1i5FTgVt+skNL85Z9zwxAhYhwOoyKCJlVCYIgCEHc/r7j0Fpfg10DE/jmH/ItmpDkm3cBKkSEw9JVAVJECIIgCHG01Nfg84UWzduFIE1SRAhqzRAEQRCu8d4TFuGCNR3a70kRIWaaVak1QxAEQQhEURTceenxWlxEmBQRwqiIUI4IQRAEIZr2aBj/50MnoL7Gj9NWtXp9nDmhO6NgZrZmSBEhCIIgxHPJiYtx8fGL4PcpXh9lTkgREUxDKIDW+hoAM9s0BEEQBCGSSihCAFJEXOFbV63DYCyJhY3huT+ZIAiCIKoIKkRc4JTlLV4fgSAIgiCkhFozBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4htTbd1VVBQDEYjGPT0IQBEEQhFnYfZvdx8shdSEyMTEBAOjq6vL4JARBEARBWGViYgKNjY1lP0dRzZQrHpHL5dDb24tIJAJFUbw+jufEYjF0dXWhp6cH0WjU6+PMW+jv2R3o79kd6O/ZPejvWkdVVUxMTGDx4sXw+cq7QKRWRHw+Hzo7O70+hnREo9Gqf5O7Af09uwP9PbsD/T27B/1d55lLCWGQWZUgCIIgCM+gQoQgCIIgCM+gQqSCCIVCuP322xEKhbw+yryG/p7dgf6e3YH+nt2D/q7tIbVZlSAIgiCI+Q0pIgRBEARBeAYVIgRBEARBeAYVIgRBEARBeAYVIgRBEARBeAYVIhVOMpnE2rVroSgKXn75Za+PM6/Yt28frrnmGqxYsQK1tbVYtWoVbr/9dqRSKa+PNi+47777sGLFCoTDYaxbtw5/+tOfvD7SvGLz5s045ZRTEIlE0N7ejksvvRS7du3y+ljzns2bN0NRFNx0001eH6VioEKkwrnllluwePFir48xL9m5cydyuRy+/e1v4/XXX8fXvvY1fOtb38Ktt97q9dEqnsceeww33XQTbrvtNrz00ks488wzcfHFF+PAgQNeH23e8Mwzz+C6667D1q1b8dRTTyGTyeCCCy5APB73+mjzlm3btuGBBx7AiSee6PVRKgoa361gfvOb3+Dmm2/GT3/6Uxx33HF46aWXsHbtWq+PNa+56667cP/992Pv3r1eH6Wi2bBhA04++WTcf//92sdWr16NSy+9FJs3b/bwZPOXw4cPo729Hc888wzOOussr48z75icnMTJJ5+M++67D3feeSfWrl2Lr3/9614fqyIgRaRCGRgYwCc+8Ql8//vfR11dndfHqRrGx8fR0tLi9TEqmlQqhe3bt+OCCy6Y8fELLrgAzz77rEenmv+Mj48DAL1/BXHdddfhve99L84//3yvj1JxSL30jiiOqqq4+uqrce2112L9+vXYt2+f10eqCvbs2YN7770XX/nKV7w+SkUzNDSEbDaLjo6OGR/v6OhAf3+/R6ea36iqiptvvhlnnHEGjj/+eK+PM+949NFH8eKLL2Lbtm1eH6UiIUVEIu644w4oilL21wsvvIB7770XsVgMmzZt8vrIFYnZv2cjvb29uOiii3DZZZfh4x//uEcnn18oijLj96qqHvExgg/XX389duzYgUceecTro8w7enp6cOONN+Lhhx9GOBz2+jgVCXlEJGJoaAhDQ0NlP2f58uW4/PLL8atf/WrGRTubzcLv9+PKK6/EQw89JPqoFY3Zv2d2Uent7cU555yDDRs24Lvf/S58PqrfnZBKpVBXV4cf//jH+OAHP6h9/MYbb8TLL7+MZ555xsPTzT9uuOEGPP744/jjH/+IFStWeH2cecfjjz+OD37wg/D7/drHstksFEWBz+dDMpmc8WfEkVAhUoEcOHAAsVhM+31vby8uvPBC/OQnP8GGDRvQ2dnp4enmF4cOHcI555yDdevW4eGHH6YLCic2bNiAdevW4b777tM+tmbNGnzgAx8gsyonVFXFDTfcgJ///Od4+umncfTRR3t9pHnJxMQE9u/fP+NjH/vYx3Dsscfi05/+NLXCTEAekQpk6dKlM37f0NAAAFi1ahUVIRzp7e3F2WefjaVLl+Luu+/G4cOHtT9buHChhyerfG6++WZcddVVWL9+PU477TQ88MADOHDgAK699lqvjzZvuO666/DDH/4Qv/jFLxCJRDT/TWNjI2praz0+3fwhEokcUWzU19ejtbWVihCTUCFCECXYsmULdu/ejd27dx9R4JGQ6IyPfOQjGB4exhe+8AX09fXh+OOPxxNPPIFly5Z5fbR5AxuNPvvss2d8/MEHH8TVV1/t/oEIogTUmiEIgiAIwjPIdUcQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGf8P6ZyOHzGZunmAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {
      "filenames": {
       "image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_41_2.png"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "backend = np\n",
    "\n",
    "def function(x):\n",
    "    return x*backend.sin(x**2 + 1)\n",
    "\n",
    "def analytical_gradient(x):\n",
    "    return backend.sin(x**2 + 1) + 2*x**2*backend.cos(x**2 + 1)\n",
    "\n",
    "\n",
    "x = np.linspace(-5, 5, 100)\n",
    "plt.plot(x, function(x), label=\"f(x)\")\n",
    "\n",
    "descent_x, descent_y = gradient_descent(1, 0.01, 300, solver=\"analytical\")\n",
    "\n",
    "# Change the backend to JAX\n",
    "backend = jnp\n",
    "jax_descend_x, jax_descend_y = gradient_descent(1, 0.01, 300, solver=\"jax\")\n",
    "\n",
    "plt.scatter(descent_x, descent_y, label=\"Gradient descent\", marker=\"v\", s=10, color=\"red\") \n",
    "plt.scatter(jax_descend_x, jax_descend_y, label=\"JAX\", marker=\"x\", s=5, color=\"black\")"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}