Morten Hjorth-Jensen
1501 lines
200 KiB
Plaintext
1501 lines
200 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "59b40432",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html exercisesweek41.do.txt -->\n",
|
|
"<!-- dom:TITLE: Exercises week 41 -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0c67e680",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Exercises week 41\n",
|
|
"**October 9-13, 2023**\n",
|
|
"\n",
|
|
"Date: **Deadline is Sunday October 15 at midnight**"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c36ec93d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Overarching aims of the exercises this week\n",
|
|
"\n",
|
|
"The aim of the exercises this week is to get started with implementing\n",
|
|
"gradient methods of relevance for project 2. This exercise will also\n",
|
|
"be continued next week with the addition of automatic differentation.\n",
|
|
"Everything you develop here will be used in project 2. \n",
|
|
"\n",
|
|
"In order to get started, we will now replace in our standard ordinary\n",
|
|
"least squares (OLS) and Ridge regression codes (from project 1) the\n",
|
|
"matrix inversion algorithm with our own gradient descent (GD) and SGD\n",
|
|
"codes. You can use the Franke function or the terrain data from\n",
|
|
"project 1. **However, we recommend using a simpler function like**\n",
|
|
"$f(x)=a_0+a_1x+a_2x^2$ or higher-order one-dimensional polynomials.\n",
|
|
"You can obviously test your final codes against for example the Franke\n",
|
|
"function. Automatic differentiation will be discussed next week.\n",
|
|
"\n",
|
|
"You should include in your analysis of the GD and SGD codes the following elements\n",
|
|
"1. A plain gradient descent with a fixed learning rate (you will need to tune it) using the analytical expression of the gradients\n",
|
|
"\n",
|
|
"2. Add momentum to the plain GD code and compare convergence with a fixed learning rate (you may need to tune the learning rate), again using the analytical expression of the gradients.\n",
|
|
"\n",
|
|
"3. Repeat these steps for stochastic gradient descent with mini batches and a given number of epochs. Use a tunable learning rate as discussed in the lectures from week 39. Discuss the results as functions of the various parameters (size of batches, number of epochs etc)\n",
|
|
"\n",
|
|
"4. Implement the Adagrad method in order to tune the learning rate. Do this with and without momentum for plain gradient descent and SGD.\n",
|
|
"\n",
|
|
"5. Add RMSprop and Adam to your library of methods for tuning the learning rate.\n",
|
|
"\n",
|
|
"The lecture notes from weeks 39 and 40 contain more information and code examples. Feel free to use these examples.\n",
|
|
"\n",
|
|
"In summary, you should \n",
|
|
"perform an analysis of the results for OLS and Ridge regression as\n",
|
|
"function of the chosen learning rates, the number of mini-batches and\n",
|
|
"epochs as well as algorithm for scaling the learning rate. You can\n",
|
|
"also compare your own results with those that can be obtained using\n",
|
|
"for example **Scikit-Learn**'s various SGD options. Discuss your\n",
|
|
"results. For Ridge regression you need now to study the results as functions of the hyper-parameter $\\lambda$ and \n",
|
|
"the learning rate $\\eta$. Discuss your results.\n",
|
|
"\n",
|
|
"You will need your SGD code for the setup of the Neural Network and\n",
|
|
"Logistic Regression codes. You will find the Python [Seaborn\n",
|
|
"package](https://seaborn.pydata.org/generated/seaborn.heatmap.html)\n",
|
|
"useful when plotting the results as function of the learning rate\n",
|
|
"$\\eta$ and the hyper-parameter $\\lambda$ when you use Ridge\n",
|
|
"regression.\n",
|
|
"\n",
|
|
"We recommend reading chapter 8 on optimization from the textbook of [Goodfellow, Bengio and Courville](https://www.deeplearningbook.org/). This chapter contains many useful insights and discussions on the optimization part of machine learning."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "38303814",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Code examples from week 39 and 40"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f68b9c20",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Code with a Number of Minibatches which varies, analytical gradient\n",
|
|
"\n",
|
|
"In the code here we vary the number of mini-batches."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "4dfe5e82",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[3.97594511]\n",
|
|
" [2.99866581]]\n",
|
|
"Eigenvalues of Hessian Matrix:[0.39890447 3.73456649]\n",
|
|
"theta from own gd\n",
|
|
"[[3.97594511]\n",
|
|
" [2.99866581]]\n",
|
|
"theta from own sdg\n",
|
|
"[[3.94814932]\n",
|
|
" [2.97879245]]\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABTeElEQVR4nO3deXwTZeIG8Ce9OQstlJ7Qgtzlvu9WOQRSgS6ioCis7qrc4u4Kq66gqyz8lEPxwnXBq4jacgVFOVpAAQUpyn1owQKt5UyhpaVN398fIaFpkjZ3ZjLP9/PJBzozmbyTNM2T91QJIQSIiIiIFMrP2wUgIiIi8iaGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIglatWoVVCqV8RYQEICoqCg8+OCDOHXqlNfKNW/ePKhUKq89vhycOXMGKpUKr732mreLQkQ2CvB2AYjIupUrV6JNmzYoKSnB999/j1deeQWZmZk4fvw4GjZs6O3iERH5BIYhIglLTExE9+7dAQBJSUnQ6XR48cUXsW7dOkyePNnLpSNvKisrM9YaEpFz2ExGJCOGYPTHH38Yt5WUlOCZZ55B586dERoairCwMPTp0wfr1683u79KpcK0adPw8ccfo23btqhduzY6deoEjUZjduymTZvQuXNnBAcHIyEhwWqzT0lJCebOnYuEhAQEBQUhJiYGU6dOxbVr10yOi4+Ph1qthkajQZcuXVCrVi20bdvW+NirVq1C27ZtUadOHfTs2RP79++v8fkwNCdmZmbiqaeeQqNGjRAeHo7U1FRcuHDB7NrnzZtndo74+HhMmjTJ7Jzbt2/HX/7yF4SHh6N+/fp45JFHUFRUhPz8fIwbNw4NGjRAVFQU/va3v6GsrMzsvBUVFXjllVfQtGlThISEoHv37ti2bZvZcadOncKECRMQERGB4OBgtG3bFm+99ZbJMVlZWVCpVPj444/xzDPPICYmBsHBwTh9+jSKi4vxt7/9DQkJCQgJCUFYWBi6d++O1atX1/j8EZEev1IQyUhOTg4AoFWrVsZtpaWluHLlCv72t78hJiYGt27dwtatW5GamoqVK1fikUceMTnHpk2bsG/fPrz00kuoW7cuFi1ahDFjxuDEiRNo3rw5AGDbtm0YNWoU+vTpg88++ww6nQ6LFi0yCWEAIITA6NGjsW3bNsydOxcDBgzAL7/8ghdffBF79uzBnj17EBwcbDz+559/xty5c/Hcc88hNDQU8+fPR2pqKubOnYtt27bh1VdfhUqlwrPPPgu1Wo2cnBzUqlWrxufl8ccfx8iRI5GWlobc3Fz8/e9/x8MPP4zt27c7/Fw//vjjSE1NxWeffYbs7Gz885//RHl5OU6cOIHU1FT89a9/xdatW7Fw4UJER0dj9uzZJvdfvnw5mjVrhqVLl6KiogKLFi3C8OHDsWPHDvTp0wcAcPToUfTt2xdNmzbF66+/jsjISHzzzTeYMWMGLl26hBdffNHknHPnzkWfPn3w7rvvws/PDxEREZg9ezY+/vhj/Pvf/0aXLl1QVFSEw4cP4/Llyw5fO5HiCCKSnJUrVwoAYu/evaKsrExcv35dbN68WURGRoqBAweKsrIyq/ctLy8XZWVl4rHHHhNdunQx2QdANGnSRBQWFhq35efnCz8/P7FgwQLjtl69eono6Ghx8+ZN47bCwkIRFhYmKv/Z2Lx5swAgFi1aZPI4a9asEQDEihUrjNuaNWsmatWqJc6dO2fcdvDgQQFAREVFiaKiIuP2devWCQBiw4YNNj1PU6ZMMdm+aNEiAUDk5eWZXPuLL75odo5mzZqJRx991Oyc06dPNzlu9OjRAoBYvHixyfbOnTuLrl27Gn/OyckRAKw+f4MHDzZuGzZsmIiNjRVardbknNOmTRMhISHiypUrQgghMjMzBQAxcOBAs/InJiaK0aNHm20nItuxmYxIwnr37o3AwEDUq1cP9957Lxo2bIj169eb9RP54osv0K9fP9StWxcBAQEIDAzEBx98gGPHjpmdMzk5GfXq1TP+3KRJE0RERODs2bMAgKKiIuzbtw+pqakICQkxHlevXj2kpKSYnMtQ81K5mQkA7r//ftSpU8esWahz586IiYkx/ty2bVsA+v5QtWvXNttuKFNN7rvvPpOfO3bsaNf9LVGr1SY/G8o0cuRIs+2WHsfa87dz507odDqUlJRg27ZtGDNmDGrXro3y8nLjbcSIESgpKcHevXtNzvmnP/3J7HF69uyJr7/+GnPmzEFWVhZu3rzp8DUTKRXDEJGEffTRR9i3bx+2b9+OJ554AseOHcP48eNNjsnIyMC4ceMQExODTz75BHv27MG+ffvw5z//GSUlJWbnDA8PN9sWHBxs/BC9evUqKioqEBkZaXZc1W2XL19GQEAAGjdubLJdpVIhMjLSrKkmLCzM5OegoKBqt1sqvyVVr8nQNOdMMLCnrJbKae35u3XrFm7cuIHLly+jvLwcb775JgIDA01uI0aMAABcunTJ5P5RUVFm53zjjTfw7LPPYt26dUhOTkZYWBhGjx7t1SkYiOSGfYaIJKxt27bGTtPJycnQ6XT473//iy+//BJjx44FAHzyySdISEjAmjVrTOYAKi0tdegxGzZsCJVKhfz8fLN9VbeFh4ejvLwcFy9eNAlEQgjk5+ejR48eDpXBHYKDgy0+J+7qW2Pt+QsKCkLdunURGBgIf39/TJw4EVOnTrV4joSEBJOfLc3xVKdOHcyfPx/z58/HH3/8YawlSklJwfHjx11zMUQ+jjVDRDKyaNEiNGzYEP/6179QUVEBQP8BGRQUZPJBmZ+fb3E0mS0Mo7kyMjJMajyuX7+OjRs3mhx7zz33ANAHssrS09NRVFRk3C8F8fHx+OWXX0y2bd++HTdu3HDL41l7/gYMGAB/f3/Url0bycnJyM7ORseOHdG9e3ezm6VavOo0adIEkyZNwvjx43HixAkUFxe7+rKIfBJrhohkpGHDhpg7dy7+8Y9/IC0tDQ8//DDUajUyMjIwZcoUjB07Frm5uXj55ZcRFRXlcFPJyy+/jHvvvRdDhgzBM888A51Oh4ULF6JOnTq4cuWK8bghQ4Zg2LBhePbZZ1FYWIh+/foZR5N16dIFEydOdNWlO23ixIl44YUX8K9//QuDBg3C0aNHsXz5coSGhrrl8fz9/TFkyBDMnj0bFRUVWLhwIQoLCzF//nzjMcuWLUP//v0xYMAAPPXUU4iPj8f169dx+vRpbNy40abRcL169YJarUbHjh3RsGFDHDt2DB9//DH69Olj0g+LiKxjGCKSmenTp2P58uV46aWXMH78eEyePBkFBQV499138b///Q/NmzfHnDlzcO7cOZMPXnsMGTIE69atw/PPP48HHngAkZGRmDJlCm7evGlyTpVKhXXr1mHevHlYuXIlXnnlFTRq1AgTJ07Eq6++ajKs3tv+/ve/o7CwEKtWrcJrr72Gnj174vPPP8eoUaPc8njTpk1DSUkJZsyYgYKCArRv3x6bNm1Cv379jMe0a9cOBw4cwMsvv4znn38eBQUFaNCgAVq2bGnsN1STu+++Gxs2bMCSJUtQXFyMmJgYPPLII3juuefccl1EvkglhBDeLgQRERGRt7DPEBERESkawxAREREpGsMQERERKRrDEBERESkawxAREREpGsMQERERKZoi5hmqqKjAhQsXUK9ePYvT2RMREZH0CCFw/fp1REdHw8/PffU3ighDFy5cQFxcnLeLQURERA7Izc1FbGys286viDBUr149APons379+l4uDREREdmisLAQcXFxxs9xd1FEGDI0jdWvX59hiIiISGbc3cWFHaiJiIhI0RiGiIiISNEYhoiIiEjRGIaIiIhI0RiGiIiISNEYhoiIiEjRGIaIiIhI0RiGiIiISNEYhoiIiEjRGIaIiIhI0RiGiIiISNEYhoiIiEjRGIaIiIhI0RiGiIiISNEYhoiIiEjRGIaIiIhI0RiGiIiISNG8HoZ27tyJlJQUREdHQ6VSYd26dVaPfeKJJ6BSqbB06VKPlY+IiIh8m9fDUFFRETp16oTly5dXe9y6devwww8/IDo62kMlIyIiIiUI8HYBhg8fjuHDh1d7zPnz5zFt2jR88803GDlypIdKRkRERErg9ZqhmlRUVGDixIn4+9//jvbt23u7OERERORjvF4zVJOFCxciICAAM2bMsPk+paWlKC0tNf5cWFjojqIRERGRD5B0zdBPP/2EZcuWYdWqVVCpVDbfb8GCBQgNDTXe4uLi3FhKIiIikjNJh6Fdu3ahoKAATZs2RUBAAAICAnD27Fk888wziI+Pt3q/uXPnQqvVGm+5ubmeKzQRERHJiqSbySZOnIjBgwebbBs2bBgmTpyIyZMnW71fcHAwgoOD3V08IiIi8gFeD0M3btzA6dOnjT/n5OTg4MGDCAsLQ9OmTREeHm5yfGBgICIjI9G6dWtPF5WIiIh8kNfD0P79+5GcnGz8efbs2QCARx99FKtWrfJSqYiIiEgpvB6GkpKSIISw+fgzZ864rzBERESkOJLuQE1ERETkbgxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoXg9DO3fuREpKCqKjo6FSqbBu3TrjvrKyMjz77LPo0KED6tSpg+joaDzyyCO4cOGC9wpMREREPsXrYaioqAidOnXC8uXLzfYVFxfjwIEDeOGFF3DgwAFkZGTg5MmTuO+++7xQUiIiIvJFKiGE8HYhDFQqFdauXYvRo0dbPWbfvn3o2bMnzp49i6ZNm9p03sLCQoSGhkKr1aJ+/fouKi0RERG5k6c+vwPcdmY30Wq1UKlUaNCggdVjSktLUVpaavy5sLDQAyUjIiIiOfJ6M5k9SkpKMGfOHEyYMKHahLhgwQKEhoYab3FxcR4sJREREcmJbMJQWVkZHnzwQVRUVODtt9+u9ti5c+dCq9Uab7m5uR4qJREREcmNLJrJysrKMG7cOOTk5GD79u01thsGBwcjODjYQ6UjIiIiOZN8GDIEoVOnTiEzMxPh4eHeLhIRERH5EK+HoRs3buD06dPGn3NycnDw4EGEhYUhOjoaY8eOxYEDB6DRaKDT6ZCfnw8ACAsLQ1BQkLeKTURERD7C60Prs7KykJycbLb90Ucfxbx585CQkGDxfpmZmUhKSrLpMTi0noiISH4UM7Q+KSkJ1eUxCU2DRERERD5INqPJiIiIiNyBYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUzeuTLhIREREZ6XTArl1AXh7goVUjGIaIiIhIGjIygJkzgXPnPPqwDENERETkfRkZwNixgBeW4WKfISIiIvIunU5fI+Sl9UgZhoiIiMi7du3yeNNYZQxDRERE5F15eV59eIYhIiIi8q6oKK8+PMMQERERedeAAUBsLKBSeeXhGYaIiIjIu/z9gWXL9P/3QiBiGCIiIiLvS00FvvwSiInx+EOrhPDSODYPKiwsRGhoKLRaLep7aDZLIiIickClGagL69dHqFrt9s9v1gwRERGRonEGaiIiIpIGLsdBREREisXlOIiIiEixuBwHERERKRqX4yAiIiJF43IcREREpGhcjoOIiIgUjctxEBERkaJxOQ4iIiJSPC7H4V5cjoOIiEgmvLAcByddJCIiIunw9weSkvT/Lyz0yEOymYyIiIgUjTVDRETkPpWaPBAVpR815O/v7VIRmWAYIiIi97C06GZsrH7UUGqq98olNwyUbsdmMiIicj3DoptVl1g4f16/PSPDO+WSm4wMID4eSE4GJkzQ/xsfz+fPxTiajIiIXEun039gW1trSqXS1xDl5MizhsNTNTXWVnE3zMPz5Zc+X8Pmqc9v1gwREZFr1bTophBAbq7+OKnR6YCsLGD1av2/Op3pfk/V1FS3irth26xZ5uUjhzAMERGRa9m66KaXF+c0U1PQ8WTTn5wDpQx5PQzt3LkTKSkpiI6Ohkqlwrp160z2CyEwb948REdHo1atWkhKSsKRI0e8U1giIqqZrYtuenlxThM1BZ0vv/RsTY2tQXH9etc8nsJ5PQwVFRWhU6dOWL58ucX9ixYtwuLFi7F8+XLs27cPkZGRGDJkCK5fv+7hkhIRkU1qWnRTpQLi4vTHSYEtTVJTpni2psbWoLh0KTtTu4DXw9Dw4cPx73//G6kWOoEJIbB06VI899xzSE1NRWJiIj788EMUFxcjLS3NC6UlIqIaVbfopuHnpUul03naliapixdtO5ermv7sWcWdfYec5vUwVJ2cnBzk5+dj6NChxm3BwcEYNGgQdu/ebfV+paWlKCwsNLkREZEHWVt0MzZWeqOgXNl3yVVNf4ZAacuAbx/uO1RWXOaRx5F0GMrPzwcANGnSxGR7kyZNjPssWbBgAUJDQ423uLg4t5aTiIgsSE0FzpwBMjOBtDT9vzk50gpCgO0BplEjzzb9pabqa31sIbXO6E64fOoKPp3yPcY3240WUUUeeUxJhyEDVZVfPiGE2bbK5s6dC61Wa7zl5ua6u4hERGSJYdHN8eP1/0qlaawyW/s4vf32nZ+r7gfc0/Q3apRtx0mpM7qdRIXA0Q2nsWhEFgaE/oyIVqF4+J1++Oz3vtCigUfKIOnlOCIjIwHoa4iiKr3QBQUFZrVFlQUHByM4ONjt5SMiIh9gaJIaO1YfbCo3TVUOOqmp+mMtLTFi2O9qhqB2/rzlJjPDBJZS6Yxuo9LCUux8+zA0n93AxiMJyCm/C8Bdxv0dQ05A3TUPyeOCMWSW+8sj6TCUkJCAyMhIbNmyBV26dAEA3Lp1Czt27MDChQu9XDoiIvIZhj5ONQWd1FR9bY2n1gqzNahJscatioIjF/HVkhPY+HUAvr3QHjfQzbgvGCW4u/EhqO8uhnrmXWjapzWA1vo+v7PcXzavh6EbN27g9OnTxp9zcnJw8OBBhIWFoWnTppg1axZeffVVtGzZEi1btsSrr76K2rVrY8KECV4sNRER+Rxbg46h6c+T5bIlqEmMqBD45cuT2LgiD5q9jfBjUTsI9Dfuj/T7A+qWJ6BODcbgWYmoE9HDa2X1+tpkWVlZSE5ONtv+6KOPYtWqVRBCYP78+Xjvvfdw9epV9OrVC2+99RYSExNtfgyuTUZERE7z9urx3n58G9y8chPb3zgMzRc3oTlxF87pok32d6t9FCnd8/Bg/3No2TYAfrEx1V6Hpz6/vR6GPIFhiIiInJKRYblmZtkyydbMeMqFA/nQLD4JzdZgbP2jA26itnFfLRRjSOQhpAy9hREzWyL6zG67nkeGIRdiGCKSIRl8CyaF4OrxJirKK/DTp8eh+V8BNPua4MDNtib74/zPQ93mV6SMq4WkaYmoFVZLv8OB55FhyIUYhohkht/CSSp0Ov1irdZmqDaM5srJ8emwfiP/BrYuOwJNxi1sOt0K+RV3RnSrUIFedY8gpc9lqP8ajQ6pLaHyqzL9gIPPo6c+v73egZqIyIS1b4+VF8xkICJPsWf1eE92qvaAs9+fg2bZr9Bk1kbmpQ4oRS/jvnooxLCYI1AP12H4rNaIaN+h+pNJ/HlkGCIi6ahpwUyVSj8j76hRPv0tnCTE1pmdfWAGaN0tHX5YeRSaDy9Dkx2NQyWtAMQa9zcPOIuUxDNQj6+HgVMSEVS3j+0nl/jzyDBERNIh8W+PpEC2zuws0xmgtb9r8e3So9Cs1+GrnDa4JO7U8PhBh/6hh6Dudw3qp+LQZkRzqPyaOfZAEn8eGYaISDok/u2RFMgHZ4A+ve0sNMvPYOOOeth5tQPKcaeGp4HqGoY3PQL1SODe2e0Q1qJz9SezdaCDxJ9HhiEikg6Jf3skBfKBGaDLisuw+/0j2PjxNWgONcWJW80B3KnhaRP0K9Qdc6F+uAH6/qU9Amv3s+3E9gx0kPjzyNFkRCQdhhEnNX179PGROyRBlj744+IkOwP0lV+vYvPio9ioUWFzbjtcEw2M+wJQhkEND0E96DrU0+Jx1z0ONH05Ot2Anc8jh9a7EMMQkYwY/sgClr89cjSZOc7J5BkSfp5FhcDxr37Dxrdzofm+Ib4vTEQF7pStkeoSRiQch3qUP4bOaofQpqGOP5iz0w3Y8TwyDLkQwxCRzMjsW7hXcU4mxbp14xZ2LD8EzWc3oDkSj9/KTWt4OoSchLrLBaRMCkfPSe3gH+Si4JaVBVhYRstMZqbTAx04zxARKZenVwaXK87JpDiGld81m/3x7fn2uF5p5fcglOLuRr8g5Z5ijJzeAs36tQLQyvWF8MGBDgxDRCRNnl4ZXG44J5MiGFZ+17yfB83ecPxwo73Zyu8j7zqJlD8F4Z4Z7VE30gMrv/vgQAeGISIiOeKcTN7hgX5DN6/cROabt1d+P94CubrWAFob93etdQwpPf+A+s8R6DqhDfwCmlg/mTtIfJi8IxiGiIjkyAebKiTFUuhZv95t/bMuHMjHpqUnodkSjK35iSjGnRqeWijG4CaHkDK0FCNmtERM97YA2lo/mbtVN0we0P/8+uuyqpFkGCIikiMfbKqQDEud0sPDgcuXzY91sH9WRXkFDqQZVn6PwE/F7QBEGvcbVn5X318LydMTUSusl/WTeUNqqv6aqz5PBrNn68OQTPqscTQZEZEccU4m97DWKb06Nj7XRQVF2Lr0MDQZpdh0qhXyKu6EH8PK7+rel6H+SxQ6jm1lvvK7FH35JXD//ebbXTQVBofWuxDDEBH5JM7J5Fo1zZ9TEwtDyc/uPIvseWtxYv8NbLveE9twj3H+n7q4jmExh6G+V7/ye5PExs6V39OcnW/IBhxaT0RE1bPWVBEbyzmZHFFTp/Sa5OVBd0uHH1cdxcZVl3Fz/2HMLluI0dCf81kAeYjCV82eQtMnR2DglEQE17dj5Xep8aFO/AxDRERyxjmZXMfJzuYLnrmExQ9dxSXRAWOQgS8xA4Bp40ukKh+P/f4i0Ko9UL+b5RPJhQ914mcYIiKSO87J5BoOdjavgArnEIvn86agAv5oiCt43/9JqHQCVXv9qHxpDigf6sTv5+0CEBGRD9Lp9Ms2rF6t/1en83aJamaYP0dlueOyQNV6Hn0QAoD/C5iLp7vtQtbSgyj4+gDCdRfNgtCdE1VqPpKzGp4vqFT6ZXRkMN8QwxAREblWRoa+Y21yMjBhgv7f+Hj9dikzzJ8DmH3AG0LPZYSbbL9RqxHy//U23ix7Cq/tT8KgmZ0RcPWibY8ng+ajalXzfBl/XrpUFrVfDENERFIgx5oUSwwj3Kp2rDXMxyPhQCQqBI4FdcK6LvNQgAiTfecQi0lYhWcS0pE1djmK3/gvkJmJ+tfzED3/SdMT+VDzUY0MnfhjYky3x8bKajQjh9YTEXmbr6w874Gh1q5268Yt7Hz7MDSrr0NzOB6/3l753Q86DMAu9AzIRstWKrSfloRej3WwbeV3Jc4B5aZlSjjPkAsxDBGRZFmb5E/KcwVZ++DLytI3idXEwnw8bi9bJQVHLuLrpSeg+dof35xvj+u487kQhFIkhx9Cyt1FGDmjBeL7xzpWDs4B5RKcZ4iIyNfJceX56mqxSkttO4e7+spYKZtYshSH0AGaFRewcY/5yu9N/AqgvusE1KlBGDyzPepGdne+LJwDSlYYhoiIvEVuk9ZZq8Uy9AeaN8+287ijr4yVsolz5yDuvx/z8CXW4k4A6VLrGFJ66Fd+7/ZQG/gFRFQ9o/M4B5RsMAwREXmLnCats6UW6/339TUfNfWVcXaoddWmsL59gZkzIYSFeX2gHw6/DDNRERGJEUPKMXKWB1d+5xxQssAwRCQXbuqgSF4kp1FHttRinTsHzJ+vryFSqSz3lXF0qLXh93/9euDTT4GLd4avFwWGok6Z1uq8Pn4QiMM5rFtzi8GELOLQeiI5kOu8LVQ9OU1aZ2vtVMuWrh9qXfn3f+lSkyAEALXKtLadRwo1bCRJrBkikrqa+mlwVIp8GSatGzvW9TUprmZPLVZSkuv6ymRkQPxpLADzJjADm7/VS6GGjSSJQ+uJpEyG87aQAyyNgoqLk9aoIw/OnWNY+X3TqgJM2/MwIpDvXDOGEt4nPtqMzqH1RCS/0UbkGDmMOnJzLVbhuUJ8u+QINOt1+Oq31rgoOmAQsvBv5DtXblfVsEk5bPjKpJ1exDBEJGVyGm1EzpHDqCMXz53z6/az0Cw/A82OuthxpQPK0Me4LxRapIbtAK7YWcZGjYBLl0zLtngxEBamX+rEkSAj5bDBZnSXYDMZkZRJYUZfoqocrCUpLynH7vePQPPxVWh+jsOxWy1M9rcKzIG641mkPNwA/f7aHoE/fm/b7z9wpyns9Glg9+47Zbt4EZg92/EgI+UZwhXQjM7lOFyIYYhkS4lrHJFPuZpzDZsXH4Vmo8DXv7fDVdHQuC8AZRjQ4DBSBmkx8qlmaDUswfTONf3+G1gLJs4GGamHDQV8WWKfISKS12gjIuhXfj+xOQeat3/Hxu8a4HttInToa9wfrrqMEQnHoL7PD0NntUeDZl2sn6y63//KLDXTuWKpEyn22atcK3f0qG33saUZXcp9ojyAYYhI6rjGEUncrRu3sOsd/crvGw/F49fy5gCaG/e3Dz6FlC7noX4kDL0faw//oP7WT1aVtd//xo2Bhx7ShxlLH9yuCDJS67Nnqe+SLWqaUkDKfaI8RPJhqLy8HPPmzcOnn36K/Px8REVFYdKkSXj++efh58c5I0kh5DDaiBTl4rFL+HrJcWz8yrDye1fjPsPK7+rkIoyc3hwJA1sCaOn4gzny+++KICOlGcKtNflVx5blT9gBG4AMwtDChQvx7rvv4sMPP0T79u2xf/9+TJ48GaGhoZg5c6a3i0fkHHuqpuUw2sgXKLy5wBpRIXB47Sn9yu+7w7HXwsrvI1ucgHpMIAbPbI960S5Y+b0ye3//XRFkDDOEu3uttZpU1+RnjS3N6K5oSvQRkg9De/bswahRozBy5EgAQHx8PFavXo39+/d7uWRETmLVtPQ485r4YIgquVaCrOWHsXFNETTHWuB3XSsArYz7u9Q6BnX3P6Ce3BjdJ7Z1z8rvjnJFkJFKn72amvwssaUZXYp9orxFSNyCBQtEs2bNxIkTJ4QQQhw8eFBERESItLQ0m8+h1WoFAKHVat1VTCL7pKcLoVIJof9zc+emUulv6eneLqHyOPOapKcLERtrer/YWFm+jhey88X7j+wUoyL3itq4YXJJISgW6ogfxLsTdojcHy94u6g1M7ymVV9Xe99nll7fuDjPvb5paea/l5Zuzz+vPzYzU4jycted147PW1fz1Oe35MNQRUWFmDNnjlCpVCIgIECoVCrx6quvVnufkpISodVqjbfc3FyGIZKO8nLzP6xV/1DHxdn2x4xcw5nXRObBtkJXIX765KiYl5Qputc+YnYZMX4XxBNtd4iNL/wgii4Webu49nNVkCkv14cMe8KGq2Rm2hZaMjPtK7Oj5/UghqHbVq9eLWJjY8Xq1avFL7/8Ij766CMRFhYmVq1aZfU+L774ogBgdmMYIkmQwR8gxXHmw0aGwbboYpFY/8+94i9tdohovwtmxe5Z55B46e5McSDtmKi4Vea9EOAq3gwyrmD4PbMUuq39ntlSW+nIeT2MYei22NhYsXz5cpNtL7/8smjdurXV+7BmiCRNBlXTiuPoayKjYHt29znx9oM7xIjGP4oQFJsUrw6uizFRe8QHk3aKvJ//uHMnH2r+kz17mvzsqa10VVOim3gqDEm+A3VxcbHZEHp/f39UVFRYvU9wcDCCg4PdXTQix0hpuC7pOfqaSG0emkoqyivw46qj0Ky6BM1PUfi5pDWAGOP++IBcqNv+hpTxdTFoaiKC6/c2PQGHXEuLrfON2TtCjPOYAZDBchyTJk3C1q1b8d5776F9+/bIzs7GX//6V/z5z3/GwoULbTqHbJfj8MHRKQQusSFFjr4mElsOofBcIbYsOwLNunJs+rUNLorGxn1+0KFPvSNI6XcF6idj0S6lBVR+KssnkvoyFEpW0+eCo7+TEv288djnt1vrnVygsLBQzJw5UzRt2lSEhISI5s2bi+eee06UlpbafA5ZjiZj9bRvk3jVtCI58ppIoM/Fr5lnxbLULDE4bL8IRKnJw9fHNTEu7nvx8ZPfiYvHL9l+Umeb/+TeR0fOfKwZnn2GXEh2YUjmo1PIRt4erkvmHHlNPBxsy26WiZ1vHhT/6Jkp2gadNvsz0TLwNzG7W6bY/voBcavolmMP4swHKr/IeZeM+rHZwlOf35JvJnMFWTWTsXpaWSRaNa1ojrwmliZrjItzWZ+LqznX8M2So9i4wXzld3+UY2CDQ1AP1EI9xcLK745wtKnF2VXiLeF7xD4+1gzvqc9vhiGpkVgfBCKykQs/tEWFwMlvcrDxrd+h+a4BvtMmQldpwYAw1RWMiD8KdYofhs1ujwbNQl11FXqOfKC644scZ2l3jCGUAqavnzOh1Es89flt12iy3NxcxMXFuassBEh6dAoRVcPJteMqr/yuOdwMp8vMV35Xd7698vuf2yEgxI6V3+3lyDIUrl7agaPZHMcRYnazKwy1adMGs2fPxpw5c1CnTh13lUnZOOyaSDEunbiMr14/Bs3X/vjmXDsUVln5PSn8ENRJRRg5LQHNk5xc+d1e9n6guvKLHBcQdV5qqv75YROjTexqJtu9ezeefvpp5Obm4pVXXsHkyZPdWTaXkVUzmY+19xLRHaJC4Mj609j47nlo9oRhz/VEqCAwALsQhTyUoBbC7mqAkWNCMGRWe9SLruftItve/OfKJn52F6DbJNlM1rdvX/zwww/46KOP8Nxzz+GNN97AkiVLkMRfRteRyirJROQShpXfNZ8XQXO0Bc7q7tTwjEEG3sZURCK/0h1igd7LgOjelk/oabY2/7lilXgDdhcgD/Or+RBzjzzyCE6ePImUlBSMHDkSY8aMwenTp11dNuUyVE/HxJhuj41lOzmRDOQd/AMfTNqFMdF70ahhOYa/0B1vHRqEs7pYhOAmRkb8iM39/410jDUNQsCdPjEZGd4pvKMMX+SAO1/cDOz9IsfuAuRhDo8mKy4uxoEDB5Ceno433ngDgYGBmDp1KubNm4d69SRQtVuJrJrJKuOQUnni66Y4okIge/VxaD74A5ofG2NfUXuT/dF+eVC3OoWU+0Nw94xE1G4Y7LtTaLhimgF2F6DbJDm0/t1338W+ffuwb98+HDt2DP7+/ujYsSN69+6Nzp0749NPP8XJkyexdu1adO/e3W2FtpdswxDJD4cCS4+bwmnxpWJsW3YYmi9LoDnZEhcqTGspetQ5gpReF6F+PBKdH2htuvSFr/eJccVz7kPDw8lxkgxDcXFx6N27t/HWvXt3swVRX331VaSlpeHw4cMuL6yjGIbII9wx4Rw5x8XhNPeHC9i07DQ020KwraADSlDLuK8ObmBI1GGkDCvDiKdbI7JjhPUTrV4NTJhQ8wOmpQHjx9tdTp/h5sksSfokGYZs8ccffyA6Oho6nc6Vp3UKwxC5HWcOlx4XhNOK8grs+1C/8vvG/YaV3+9o5n8OKe1+hfqBOhg0NREhDUJsK5uv1wy5EpudFU22YUgIgZ07d2LQoEGuPK1TGIbI7fjhJi1OhNPrF65jy9Ij2Li2DF/92gYFlVZ+V6ECfeodRkrfK1A/EYP2o+6yvvK7LeVjnxiiaklyaL0tVCqVpIIQkUdwKLC02Dkbcs7OXGje+A0bM+si60oHlOHOsPb60OLeuCNQj6jA8KfbolHrjs6Xj1NoEEmKy8MQkSJxKLC02Bg60x7fhlfOxeJo6V0A7iw1dFfgGaR0OAP1hFD0f6I9gur2dX0ZuWQCkWQwDBG5gisnnCPn2Rg6V/x6D47iLvijHAMaHIK6vxbqKU3RenhzAPFuLSIALplAJBEMQ0SuwGYPaXV0vR1OxbnzUME8nFZAhQuIRlwzP6y+bzeGPd0ODRO6eKGgcHqBVyJynkMzUBORBUqeOTwjQ98hODlZP2Q8OVn/s4dnUS4rLsP217Mxu+cuTMv7JwT0wacyAUAFIOrT1/Hxqgo82OcsGp49qA9zRKRILh9NJkUcTUYeJaUaEk/w8vxKl05cxtdLjkHzlR8257ZHIUKN++7H53hLNR2NRcGdO8TFAQ8+qJ/rh5NjEkmabIfWSxHDEJGbeGF+JcPK75r3zkOzOwx7rrdHBe6cu7HqIka2OI6UMYH6ld+b1DYNp5cuAePGcXJMIhlgGHIhhiFyG6XVAlXlofmVSgtLkfXmIWjWFEFzrDnOlMeZ7O8UcgLqbnlImdwIPR5tB78AKz0AODkmkazIdp4hIsXgOmRunV8p/5cCfLXkBDTfBOLbvEQU4c56h8EowT2NDyFl8E2MnHkX4nq1BtDa+skM7Jx/iIiUgWGIyBHW+smcP6/frpSmFhfOryQqBA6uOYGN7+dXWvn9zvpehpXf1WNDcPf09qgT0cP+8nJyTCKygGGIyF46nb5GyFILsxD6ppZZs/Tzx/h6U4uT8ysVXyrG9jcOY+MXJdh0siXOV7QB0Ma4v3vto0jpVQD1Y03QZXwbqPycnLSSk2MSkQUMQ0T2YlPLHQ7Mr3RuXx42LT2FjVsNK7/3NO6rjSIMjToE9dAyjJjVClGd2wFoZ3+5rPXl4uSYRGQBwxCRvdjUYqqGZSUq7huN/SuPYOP/LkLzUyQO3mwD4E7NS1PDyu/j6iBpWiJCGvQ2fwx71NSXS+mTYxKRGYYhInuxqcVclWUligNDsXlvKDRzKrBp7GUUiPbGQw0rv6v7XIH6r9FIHNMSKr/YO+dyZoSerX25uCYYEVXCofVE9jIMz66pqUVhw7MNK79rsuog63IH3EKwcV89FOLe2CNQD9dh+NNt0LhtI8sncWaEnr3D5pU+LQKRDHCeIRdiGCKXM9RAAJabWhQwmkx3S4c9/z0CzUdXoDkYgyOlLU32twg4q1/5fXw9DHgqEUF1g6o/obMzWXtoziMi8hzOM0QkZQptarl2VotvFh+BZmMFvjrTDldER+M+f5Sjf+hhqPtfQ8rUpmg1LAEqv2a2ndgVI/TYl4uIHMQwROSoKv1kfLWp5eQ3OdC8fRaanaHYdS0R5ehr3NdQdRXDmx5Fyn2q2yu/d3bsQVwxQo99uYjIQQxDRAaO9CHx9/e5Jpey4jJ89+5haNK02PhLM5wqSwCQYNzfNuhXqDvlIuWRhujzeHsEhPRz/kFdUatz8WLN94+L47B5IjLDMEQEyHtpDRd0BL586gq+XnwMmk0qbM5tDy26GPcF4hYGhR2CetANqKfFo8XdLQC0cO01OFuro9MBs2fXfP/Fi32u5o6InMcwRCTnpTUcDHGiQuDoxl+hefccNn5vWPn9Tg1PI9UljGx+HClj/DFkZnvUj+3mzqtwfjLEmprZDBpZGcVGRIrGMETKJuelNewMcaWFpdjx1mFsXH3j9srvdwG4y7i/Y8gJpHTLg3pSI/R4pC38g/p76ELg0EzWJth5moicwDBEyibXpTVsDHF/tOiLTUtPVVr5/U4Nj3Hl9+QbGDPwEpqEld9uZmvrneDnzAg9dp4mIicwDJGyybVGwcYQ90Dn49iBJACAH3RIVWXg3qiDaJccic7/9zDq7MnVh4/PJdJXytERelxzjIicwDBEyuZMjYI3ZzC2MZxFIQ/dax/F3xK+xJi8txF05Q/gAoBPAWz+F3D5svmdvN1XypERes42sxGRovl5uwA+Q6fTz4C7erX+X53O2yUiWxhqFAwfmFWpVJaHY2dk6Jd+SE4GJkzQ/xsfr9/uARe1NczmfNub7wVj38fH8cDRefogVJmlIATcCRKzZsnr99jQzBYTY7o9NlbaneCJyOu4HIcryHlYNlW/tIYQwPz5QMuWd2p/1q93btkIB1SUV2D/x8egWXkRmv1N8PPNVjiDeMTgPPxQTbPQ6dNAixa2jbSyxNNLV7iito1rjhH5DI8tpyVk4Ny5c+Khhx4SYWFholatWqJTp05i//79Nt9fq9UKAEKr1bq+cOnpQqhUQug/Gu/cVCr9LT3d9Y9JrpeeLkRsrOlrGB6uv1XeFhNjvq3q6x4XJ0R5udNFup53XWT8Y4/4c8udoonfH6YPA52YE7JYVACiAirrv3uZmdbLasstLc3559ZWll6D2Fi+h4gUzK2f35VIvmbo6tWr6NKlC5KTk/HUU08hIiICv/76K+Lj49GihW0Tv7ktWdq7SjZ5X3W1BpX3nToFzJtnuTOuLRysUTnz3Tlolv0KTWYdZF7ugHIEYAB2IQp50KI+6kWHYsQIYMTs2yu/W6qVjIu7M/pq9Wp9M56jPFUz5OwirUTkk7hQ620LFy5EXFwcVq5cadwWHx/vvQJVJtdh2UpVU3OmoeOuIeQ68z3Bxg7Ouls67P1Av/L7xmzDyu+xAIAxyMBbmIYoVDqXXywwfBnQ9vYcQDWNvnJmKHnjxkDfvjUf5yw5z/VERD5B8h2oN2zYgO7du+P+++9HREQEunTpgvfff7/a+5SWlqKwsNDk5hZyHZatRIaah6rh1TByqnLHZ1tnM65ONSFE+7sWnz+9G4+0+A5NQq6h/5SO+M/eJBwpbQl/lGNg6EGs6/oS0jEWkajyu2OpvIYQN368/t/KgaGmDuLVuXhR39/I3Z3C7flSQUTkBpIPQ7/99hveeecdtGzZEt988w2efPJJzJgxAx999JHV+yxYsAChoaHGW1xcnHsKx4ne5KGmmgfAdOSUM+HVyuizU1vOYMnoLNzdMBuNmtXGA0v74uPf+uOyCEcD1TWMb/Y90qbtRsHp69hxuQNGFbwPFQTMIoy9I70MQ84NZataVgAID7d+f0vhy9X4pYKIvM2tPZJcIDAwUPTp08dk2/Tp00Xv3r2t3qekpERotVrjLTc31z0dsMrL9R08LXWgdnFnWnKCrZ2IMzPtO76aTvO3im6JzCXZYna3TNEq8DezQ9sEnRZ/654pdrxxUJTdLHOuvLaw1Dk5Lk6/vbRUiMaNPdIp3CJ3XK9UlZfrryMtTf8v/zYQVctTHagl32coKioK7dq1M9nWtm1bpKenW71PcHAwgoOD3V007070xuHDtrO35sGW2YzDwoBatUyad3SR0fiuwxS8MysSm3OLoUVn477KK7+PnBqPu+6pZuV3d9SUVNe3KCtL3yRmjbv7vill9mhOwUEkWZIPQ/369cOJEydMtp08eRLNmjXzUomqcGY9JUfxj6p97G3OtCXkrlgBkXIfzi5ag18+P44tp5vh3bxHUJ53ZzJEw8rv6lH+GPq0HSu/u6v51drMzt5uplLC7NF2LqpLRB7m1nonF/jxxx9FQECAeOWVV8SpU6fEp59+KmrXri0++eQTm8/hkWo2T1V/u2peI1+vrq98fVu36ucHsrc500LTUkVMrMiesEhM75glEgLOmp2qY8hx8c++mWL3e7+I8lIHn1NPN79KpZmquqY8OTO8nt5qhiSSMU81k0k+DAkhxMaNG0ViYqIIDg4Wbdq0EStWrLDr/p56Mt3OVX9UvTW5nScDo6UJFA3PkT0hsrxcXP7fWpE56F/ihfC3RH1cNbl7MG6K4Y1/FG89kCXOfJfr2mswlK1qeQEhZs1y3XMopb5vvhjSpRI2iWSIYciFfCYMueKPqrdmzPZUAKvu+iqHompqHip0FSL7s+Pi5XsyRa86h4QKOpO7RPrli8db7xDr5u4V1/Ouu7b8Va+l6nPm7++e57C68MWZ1J2Tlmbb+9aTs30TyQRnoHYhj61t4m62ziaclqafc6Yqb82Y7anZhW25vpgYYNUqoKAAiIjQby8oQGmdMGzdHwpN+i1oTtyFc7pok7t2q30U6h4FUP85Al0ntIFfgIdmpTB0lF+/Xt9vpipXPoc1zWZNjsnK0i/kWxNPrwNHJAOe+vxmGHIFT43scvaPqjf+KHsygNlzfVeuoPypaQgouNMpOBexmIllWItU1EIxhkQegnpIKUbOaoXorpHOlc0ZnnwOOUrR9QyvX02j5bhsD5EZLschF54c2eXsEGRvjBry5JIlNpZ798Nvoff5dLPV3mNwHun4Ew6MW4h270xHrbBezpXHVTz5HFobcUaOU8JoOSKZk/wM1JJmzxIPrmDLbMLV/VH1xozZngxgNpb7rvNZAITZL78fBFQqFbrtWY5aoUEW7ukl3h76Ts4zTMERE2O6PTaWw+qJJIBhyFH2LvHgKs78Ua1pnSorS0k4xZMB7Pb1WVjEAgBQARUK0AgRuGT9F79yLYtUcNkX35CaCpw5o2+mTUvT/5uTwyBEJAEMQ47y5uKSjv5RdbZmyREeCGC6Wzrsfu8Q5vbfhVn5z0JAH3wqEwBUAMKnWuhYbomUalm8EWLJPapbVJeIvIZhyFHebrpw9I+qp6vr3RTAqq783u/JDvjP3iQsK5+Gcfgcl1QRpg8VFwdV+pfwH2vj9UmplsUbIZaISEE4mqwm1kbXyH24rKdHDdkzbNtK2U5vO4uNb+ZAs7M+dl7tgHIEGu/SQHUN98YdRYpa4N7Z7RAWX9/y9cl5ZA+HvhORwnBovQs5/GRWN1Js1Cj5fqh6S+WQU2mOH5OwYuE5vxbYGC+I+Vhe/hT8oMMA7EIU8hAYUIHITpEYOTEcff/SHoG1A608cBWGju+A5ZE9Uu7QyqHvRKQgDEMu5NCTactEgXv3Av/3f5bvr1JJ+0PVm6yFzPHjIV57DRCmXaAN/X9ex9N4VPUJIkSB6f0cmcaAtSzuxdBGRC7AMORCdj+ZtkxyFxYGXL5s/Rx//zuwaJFD5fVpVkJm5Z8sdRMW1exzOHjyA9s9PDn3FhH5NIYhF7L7ybS1P1B14uLYRFbV7ZApzp2zMvjdCXy+pcFTS68QkSJ4KgxxNJklrhgBJrW5arys4MhFbL7vbcAdQQjg8y0F3pp7i4jISQxDlrhqWLWU5qrxMFEh8PPnJ/DKkCz0rnsYkYnh+PCrRu590PPn3Xt+qp43594iInIC1yazpKY1wGzlrblqvNQX5uaVm8h88zA2fn7z9srvrQG0Nu6vF3QLuOXGAly86MaT+yBX/554e+4tIiIHMQxZUt3CirYKC/POjMAe7rx64UA+Ni09iY3fBmPrHx1wEz2M+2qhGIObHIJ6cClGzmqJmC4PA/HPOx8yrWnc2PXnBHyzo7U7fk+4bAgRyZVQAK1WKwAIrVZr3x3T04WIjRVC/9GtvzVubPqztdv8+e65mKrKy4XIzBQiLU3/mCqVeVlUKv0tPd3ph9OV6cS+D4+Ifw3MFF1rHTV7qFj/8+LJdjvEpnk/iuLLxeYnSE+/Ux5bnkd7nvPMTKevz2J5q/4OxMa65Ln0GsNr4Orfk/Jy/XNj7bVVqYSIi9MfR0RkA4c/v+3E0WQ1qVor0Lcv0KJF9bUb4eHAH3+4v/bA0rd7a5yYBLKooAhblhyGJuMWNp1uhfyKJndOiwr0rHMUKX0uQf2XKHQc2woqvxq6SNtT7vnzgTlz9M95dce7YzSZL46MunVLvxTLpUuW9zs7WaicJ7QkIsnx1Ggy1gxZU7nGJTPT9NtsdbUbLqqBqZG1b/cuqj05812uWH5/lri30Y8iGDdNTlEXheJPMbvFysd2ifxDBY6Vv7xciK1bhQgLs17WyjUJ1p5zV9RmWHqdDbUctpRNLtLTPVPLZqk2LS5O3rVpROQVnqoZYhiyxJamEW/+wa/pg7q6W1qa5VOWlovv3/1FzO2TKTqEnDC7W0LAWTGjU5b4dsF+UaItcc11ZGba98Hs6ue8utfZ3rJJnb3h2crvic2q+zJBRGQjT4UhdqCuylrTyPnz+u2Gav7UVP36ZN7oWFvTEObqVOq8qv1di2+XHoVmvQ5f5bTBJdHBuM8POvSrfxjqflehfjIWbdUtoPJr6mzJTdk7+siVz3lNr/PMmfaVTcqqm//HGmc7Ofv7S3OBYiIiCxiGKqtp0jiVSj9p3KhR+j/23vqDv369/fe53Rfk9K2m0IzZAc2OethxtQPK0cd4SCi0GN70CNQjBe59ui3CW3ZyYaEtcGT0kSuec1te508/tb9sUmVPeDb0GfLGSEgiIi9hGKrMnknjvPWtV6ez/YP6NgH99AAz8uZi+bDmAJob97UO+g3qDr9D/XAD9PtrewTW7uviAlejpvmc3PXBbMvrfPGifqj+pUueLZs72Ft7tXSp/KcOICKyA2egrkwOk8bt2mX35IK5iMWfkI7l5U8hAGW4u+EBLB6VhZPfnsHx0uZ4bX8SkmZ1RmDtQDcV2grDfE7AndFGBoaf3fHBbOvr99BDpmUxcGfZ3MHW2qvGjTnai4gUiWGoMls/NE6dcm85qmPjB/nn+BPGIw1JyER37Efd5hFYM2s3Lp0txrYrXfH0uiS0HBLv3rLaIjVV/wEcE2O6PTbWfR/Mtr7Oo0Z5vmzuYKiBqxrqKmvcWF9bJpdrIiJyIc4zVNntVdVtmiE5Pd07HxxZWUByco2HTQ78GJHdYpEyKRy9JreDf5DEazA8OctzTa9z1bl2fGEGas7/Q0Qy5Kl5hhiGqrI2yqgyZyemc1DBkYvYvPgoRqwah7CKi/CDeRkFAF1ENAIu/C6/D2xPqikcrFmjry2RcwCqytJkl3Fx+uY+BiEikiCGIRey+8l86SXgxRdrPi4z060dqUWFwKGMU9j43gVo9objhxvtIeCHMcjAl9B/kJsEIn7Lt4+1cPDgg8Dq1R5b382jfKGWi4gUw1NhiKPJLGnZ0rbj3NCRuuRaCbYvOwTNFzehOd4CubpWAFoZ93epdQyJ3cPwW6claLHuNfMPbH7Lt52leYsuXQLGjat5nim54vw/RERmGIYs8fDq24aV3zVbgrE1PxHFlVZ+D8FNDI44hJQhJRgxsyVie7QF0BZAErB0Gr/lO6tyODD0JbJ1nikiIvIJDEOWuHn+m4ryChxIOw7N/wqg2ReBn4rbAYg07o/1vwB169NQ318LydPao3ajnpZPxG/5riWHeaaIiMjlGIYsMcx/M3asPvhY6mBr5xwzRQVF2Lr0MDQZpdh0qhXyKtoBaGfc37POYaT01q/83un+VlD5RbvmWsh2cphnioiIXI5hyBrD/DdVO9ja0S/n7PfnsOnNX6HZXhvbL3ZAKXoZ99XFdQyNPgL1sDKMmN0GTRIT3XARZBcPN48SEZE0MAxZo9MBYWHAf/5zZ2mGmJhq++Xobunw46qj0Hx4GRsPRONQSSsAscb98QG5SGn3G9QP1sWgqYkIrt/bQxdDNvHW8iBERORVDEOWWBpybRhaXSUIFZ4rxLdLjuhXfv+tNS5WWfm9b/3DUPfVr/zeLqUFVH5xnroKspcbmkeJiEj6OM9QVdYmXaw0h8+vDbpBs/wMNmbVw86riShDkPGwUGhxb5x+5ffhs9sivGWYi69GQbw1Jw4nJyQikgROuuhCdi/HYWVEkQCQjyjEIhcVuPOh3CowB+qOZ6GeEIr+TyZ6fsFTX1Rd7ZwnAgknJyQi8jpOuuhO1j7oahharQIQhTwkIROiQTjUA7VQT2mGVsMSACR4rPg+z1rtnCcnPuS0BUREiiG7VesXLFgAlUqFWbNmOXaCL74AIiP1i51OmKD/Nz4e4st0nP/qoE2n2LDsLLZf7YLZ65NuByFyGZ1OXyNkbeJDQD/xoU7n0WIREZHvklUY2rdvH1asWIGOHTs6doIXXtAvtXDpkslmce4cxP33493/K7TpNHU6tnDs8alm9kx8SERE5AKyCUM3btzAQw89hPfffx8NGzZ07CRvvGFhnXd98xcg8FeswEVVhMVjjBo31jfXZGWxdsIdOPEhERF5mGzC0NSpUzFy5EgMHjy4xmNLS0tRWFhocjNQWbmPH4A4nEfjeVOhUqnujB6r6uJF4OGHjc1ryMiw+1p8mk6nD4qrVzsWGDnxIREReZgswtBnn32GAwcOYMGCBTYdv2DBAoSGhhpvcXF2zO3TsqW+g25MTM3HGjr02huInA0MnmJvOTMy9AGxSn8su54fw8SH1sKoSqUf5s6JD4mIyEUkH4Zyc3Mxc+ZMfPLJJwgJCbHpPnPnzoVWqzXecnNzbX/AqCj9SKUzZ4DMTOCTT/RNY5Y40qHXFYHBE+wtp2EEWNX+PvYGRsPEh4B5IOLEh0RE5AaSn2do3bp1GDNmDPwrffjpdDqoVCr4+fmhtLTUZJ8lxnkKAFQ7S0Hjxvq+KJXPl5WlDwI1ycyseSi2DRM6mg0Z98Z8N/aWs4b5mYzLWOTk2F52TnxIRKR4nHTxtuvXr+Ps2bMm2yZPnow2bdrg2WefRaINC5zaHIZmzADGjDENHKtX62tGapKWBowfb32/I4HBGxMPOlJOVwbGqmXhxIdERIrFSRdvq1evnlngqVOnDsLDw20KQnZ54w39rXLgcFWHXnuGjCcleW/iQXvLCbhvBBgnPiQiIg+QfJ8hl/r4Y33QqUnlfi6u6tBrT2Dw5sSDjgQbjgAjIiIZk2UYysrKwtKlS+2/4333mXaMbtTI8nGVAwdgvUOv4djFi2tuvrEnMHhz4kFHgg1HgBERkYzJMgw5xdD0EhNjNhO1icqBIzW1+uH2Tz9d82gpewKDNycedCTYcAQYERHJmPLCkIG9gSM1VV8DZMm5c8Cf/qRf98waewKDN5udHA021gJjbKxnFlYlIiJykHLDkL2BQ6cDZs+u/tjx4/Uf/NbYGhi83ezkaLCpPD9TWpr+35wcBiEiIpI0yQ+tdwWLQ/MMQ8jPn7fcUbnqEHJbh48DQHp69QHAliHjGRn62iZHH8MVOLSdiIi8iEPr3c3QHDR2rD74VA5ElpqD7OmfM2sWMGqU9eAglyHjciknERGRE5TXTFZ5va2wMGDNGtuag+zpn+PsSC/D0HprVCr3Da0nIiJSGGXVDG3YAMydaz6j85Il+mH21TUHGfrxVDfkvTJnRno5MvEhEREROURZNUMTJ1peSHTcOODKFX0H6KQky81blUdZ2cKZkV7eHFpPRESkMMoKQ5bYM6Nzairw+efVdyJ2xUgvzuhMRETkMQxDgH0zOt9/P/DZZ5b3uWqCQW8PrSciIlIQhqHKbG12GjtWP7S96jpnrppg0F0zOlfuPJ6VxQ7YREREYBgyZU+zk7snGHT1jM4ZGfp5lZKTgQkT9P/Gx9e8jAgREZGPU9akiwAsTtlUdYJFKXHFxIcZGfrarKovtaGWictlEBGRBHlq0kWGIV8PBIaZtq0N1ZdyECQiIkXzVBhiM1lYmO8GIcC+OYuIiIgUiGGoVi390hm+inMWERERVYth6Nw5364V4ZxFRERE1VLWchzWSLFWxFUrxhvmLDp/3rwDNXCnzxDnLCIiIoVizRAgvVoRVw6Dd9ecRURERD5C2WFIijM5G4bBW1pDbexYxwKRq+csIiIi8iHKHVovxSH17h4G76qmNyIiIg/w1NB65fYZio3VNw9JJQgB9g2DT0qy//z+/o7dj4iIyIcpKwxpNEBhoXRrRTgMnoiIyOOUFYYGDADcWM3mNA6DJyIi8jhld6CWGsMw+Kqjvgyk2OGbiIhI5hiGpITD4ImIiDyOYUhqOAyeiIjIo5TVZ0guUlP166VxGDwREZHbMQxJFYfBExEReQSbyYiIiEjRGIaIiIhI0RiGiIiISNEYhoiIiEjRGIaIiIhI0RiGiIiISNEYhoiIiEjRGIaIiIhI0RiGiIiISNEkH4YWLFiAHj16oF69eoiIiMDo0aNx4sQJbxeLiIiIfITkw9COHTswdepU7N27F1u2bEF5eTmGDh2KoqIibxeNiIiIfIBKCCG8XQh7XLx4EREREdixYwcGDhxo030KCwsRGhoKrVaL+vXru7mERERE5Aqe+vyW3UKtWq0WABAWFmb1mNLSUpSWlhp/LiwsdHu5iIiISJ4k30xWmRACs2fPRv/+/ZGYmGj1uAULFiA0NNR4i4uL82ApiYiISE5k1Uw2depUbNq0Cd999x1iY2OtHmepZiguLo7NZERERDLCZrIqpk+fjg0bNmDnzp3VBiEACA4ORnBwsIdKRkRERHIm+TAkhMD06dOxdu1aZGVlISEhwdtFIiIiIh8i+TA0depUpKWlYf369ahXrx7y8/MBAKGhoahVq5aXS0dERERyJ/k+QyqVyuL2lStXYtKkSTadg0PriYiI5Id9hm6TeFYjIiIimZPV0HoiIiIiV2MYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkVjGCIiIiJFYxgiIiIiRWMYIiIiIkWTTRh6++23kZCQgJCQEHTr1g27du3ydpGIiIjIB8giDK1ZswazZs3Cc889h+zsbAwYMADDhw/H77//7u2iERERkcyphBDC24WoSa9evdC1a1e88847xm1t27bF6NGjsWDBghrvX1hYiNDQUGi1WtSvX9+dRSUiIiIX8dTnt+Rrhm7duoWffvoJQ4cONdk+dOhQ7N6920ulIiIiIl8R4O0C1OTSpUvQ6XRo0qSJyfYmTZogPz/f4n1KS0tRWlpq/Fmr1QLQJ0wiIiKSB8PntrsbsSQfhgxUKpXJz0IIs20GCxYswPz58822x8XFuaVsRERE5D6XL19GaGio284v+TDUqFEj+Pv7m9UCFRQUmNUWGcydOxezZ882/nzt2jU0a9YMv//+u1ufTKkpLCxEXFwccnNzFdVXitfN61YCXjevWwm0Wi2aNm2KsLAwtz6O5MNQUFAQunXrhi1btmDMmDHG7Vu2bMGoUaMs3ic4OBjBwcFm20NDQxX1S2RQv359XreC8LqVhdetLEq9bj8/93ZxlnwYAoDZs2dj4sSJ6N69O/r06YMVK1bg999/x5NPPuntohEREZHMySIMPfDAA7h8+TJeeukl5OXlITExEV999RWaNWvm7aIRERGRzMkiDAHAlClTMGXKFIfuGxwcjBdffNFi05kv43XzupWA183rVgJet3uvWxaTLhIRERG5i+QnXSQiIiJyJ4YhIiIiUjSGISIiIlI0hiEiIiJSNFmGobfffhsJCQkICQlBt27dsGvXrmqP37FjB7p164aQkBA0b94c7777rtkx6enpaNeuHYKDg9GuXTusXbvWXcV3mD3XnZGRgSFDhqBx48aoX78++vTpg2+++cbkmFWrVkGlUpndSkpK3H0pdrHnurOysixe0/Hjx02O87XXe9KkSRavu3379sZj5PB679y5EykpKYiOjoZKpcK6detqvI8vvL/tvW5feX/be92+8v6297p95f29YMEC9OjRA/Xq1UNERARGjx6NEydO1Hg/T7zHZReG1qxZg1mzZuG5555DdnY2BgwYgOHDh+P333+3eHxOTg5GjBiBAQMGIDs7G//85z8xY8YMpKenG4/Zs2cPHnjgAUycOBE///wzJk6ciHHjxuGHH37w1GXVyN7r3rlzJ4YMGYKvvvoKP/30E5KTk5GSkoLs7GyT4+rXr4+8vDyTW0hIiCcuySb2XrfBiRMnTK6pZcuWxn2++HovW7bM5Hpzc3MRFhaG+++/3+Q4qb/eRUVF6NSpE5YvX27T8b7y/rb3un3l/W3vdRvI/f1t73X7yvt7x44dmDp1Kvbu3YstW7agvLwcQ4cORVFRkdX7eOw9LmSmZ8+e4sknnzTZ1qZNGzFnzhyLx//jH/8Qbdq0Mdn2xBNPiN69ext/HjdunLj33ntNjhk2bJh48MEHXVRq59l73Za0a9dOzJ8/3/jzypUrRWhoqKuK6Bb2XndmZqYAIK5evWr1nEp4vdeuXStUKpU4c+aMcZscXu/KAIi1a9dWe4yvvL8rs+W6LZHj+7syW67bV97flTnyevvC+1sIIQoKCgQAsWPHDqvHeOo9LquaoVu3buGnn37C0KFDTbYPHToUu3fvtnifPXv2mB0/bNgw7N+/H2VlZdUeY+2cnubIdVdVUVGB69evmy12d+PGDTRr1gyxsbFQq9Vm3yy9yZnr7tKlC6KionDPPfcgMzPTZJ8SXu8PPvgAgwcPNpulXcqvtyN84f3tCnJ8fztDzu9vV/CV97dWqwWAahdh9dR7XFZh6NKlS9DpdGar1Tdp0sRsVXuD/Px8i8eXl5fj0qVL1R5j7Zye5sh1V/X666+jqKgI48aNM25r06YNVq1ahQ0bNmD16tUICQlBv379cOrUKZeW31GOXHdUVBRWrFiB9PR0ZGRkoHXr1rjnnnuwc+dO4zG+/nrn5eXh66+/xuOPP26yXeqvtyN84f3tCnJ8fzvCF97fzvKV97cQArNnz0b//v2RmJho9ThPvcdlsxxHZSqVyuRnIYTZtpqOr7rd3nN6g6NlXL16NebNm4f169cjIiLCuL13797o3bu38ed+/fqha9euePPNN/HGG2+4ruBOsue6W7dujdatWxt/7tOnD3Jzc/Haa69h4MCBDp3TWxwt46pVq9CgQQOMHj3aZLtcXm97+cr721Fyf3/bw5fe347ylff3tGnT8Msvv+C7776r8VhPvMdlVTPUqFEj+Pv7m6W9goICs1RoEBkZafH4gIAAhIeHV3uMtXN6miPXbbBmzRo89thj+PzzzzF48OBqj/Xz80OPHj0k803CmeuurHfv3ibX5MuvtxAC//vf/zBx4kQEBQVVe6zUXm9H+ML72xlyfn+7itze387wlff39OnTsWHDBmRmZiI2NrbaYz31HpdVGAoKCkK3bt2wZcsWk+1btmxB3759Ld6nT58+Zsd/++236N69OwIDA6s9xto5Pc2R6wb03xgnTZqEtLQ0jBw5ssbHEULg4MGDiIqKcrrMruDodVeVnZ1tck2++noD+tEap0+fxmOPPVbj40jt9XaEL7y/HSX397eryO397Qy5v7+FEJg2bRoyMjKwfft2JCQk1Hgfj73Hbe5qLRGfffaZCAwMFB988IE4evSomDVrlqhTp46xV/2cOXPExIkTjcf/9ttvonbt2uLpp58WR48eFR988IEIDAwUX375pfGY77//Xvj7+4v//Oc/4tixY+I///mPCAgIEHv37vX49Vlj73WnpaWJgIAA8dZbb4m8vDzj7dq1a8Zj5s2bJzZv3ix+/fVXkZ2dLSZPniwCAgLEDz/84PHrs8be616yZIlYu3atOHnypDh8+LCYM2eOACDS09ONx/ji623w8MMPi169elk8pxxe7+vXr4vs7GyRnZ0tAIjFixeL7OxscfbsWSGE776/7b1uX3l/23vdvvL+tve6DeT+/n7qqadEaGioyMrKMvm9LS4uNh7jrfe47MKQEEK89dZbolmzZiIoKEh07drVZFjeo48+KgYNGmRyfFZWlujSpYsICgoS8fHx4p133jE75xdffCFat24tAgMDRZs2bUzeXFJhz3UPGjRIADC7Pfroo8ZjZs2aJZo2bSqCgoJE48aNxdChQ8Xu3bs9eEW2see6Fy5cKFq0aCFCQkJEw4YNRf/+/cWmTZvMzulrr7cQQly7dk3UqlVLrFixwuL55PB6G4ZOW/u99dX3t73X7Svvb3uv21fe3478nvvC+9vSNQMQK1euNB7jrfe46nYBiYiIiBRJVn2GiIiIiFyNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIZGn16tUICQnB+fPnjdsef/xxdOzYEVqt1oslIyK54dpkRCRLQgh07twZAwYMwPLlyzF//nz897//xd69exETE+Pt4hGRjAR4uwBERI5QqVR45ZVXMHbsWERHR2PZsmXYtWsXgxAR2Y01Q0Qka127dsWRI0fw7bffYtCgQd4uDhHJEPsMEZFsffPNNzh+/Dh0Oh2aNGni7eIQkUyxZoiIZOnAgQNISkrCW2+9hc8++wy1a9fGF1984e1iEZEMsc8QEcnOmTNnMHLkSMyZMwcTJ05Eu3bt0KNHD/z000/o1q2bt4tHRDLDmiEikpUrV66gX79+GDhwIN577z3j9lGjRqG0tBSbN2/2YumISI4YhoiIiEjR2IGaiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgU7f8BfRJ6s1wfRskAAAAASUVORK5CYII=\n",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"filenames": {
|
|
"image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_5_1.png"
|
|
}
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Importing various packages\n",
|
|
"from math import exp, sqrt\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"x = 2*np.random.rand(n,1)\n",
|
|
"y = 4+3*x+np.random.randn(n,1)\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"theta_linreg = np.linalg.inv(X.T @ X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(theta_linreg)\n",
|
|
"# Hessian matrix\n",
|
|
"H = (2.0/n)* XT_X\n",
|
|
"EigValues, EigVectors = np.linalg.eig(H)\n",
|
|
"print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
|
|
"\n",
|
|
"theta = np.random.randn(2,1)\n",
|
|
"eta = 1.0/np.max(EigValues)\n",
|
|
"Niterations = 1000\n",
|
|
"\n",
|
|
"\n",
|
|
"for iter in range(Niterations):\n",
|
|
" gradients = 2.0/n*X.T @ ((X @ theta)-y)\n",
|
|
" theta -= eta*gradients\n",
|
|
"print(\"theta from own gd\")\n",
|
|
"print(theta)\n",
|
|
"\n",
|
|
"xnew = np.array([[0],[2]])\n",
|
|
"Xnew = np.c_[np.ones((2,1)), xnew]\n",
|
|
"ypredict = Xnew.dot(theta)\n",
|
|
"ypredict2 = Xnew.dot(theta_linreg)\n",
|
|
"\n",
|
|
"n_epochs = 50\n",
|
|
"M = 5 #size of each minibatch\n",
|
|
"m = int(n/M) #number of minibatches\n",
|
|
"t0, t1 = 5, 50\n",
|
|
"\n",
|
|
"def learning_schedule(t):\n",
|
|
" return t0/(t+t1)\n",
|
|
"\n",
|
|
"theta = np.random.randn(2,1)\n",
|
|
"\n",
|
|
"for epoch in range(n_epochs):\n",
|
|
"# Can you figure out a better way of setting up the contributions to each batch?\n",
|
|
" for i in range(m):\n",
|
|
" random_index = M*np.random.randint(m)\n",
|
|
" xi = X[random_index:random_index+M]\n",
|
|
" yi = y[random_index:random_index+M]\n",
|
|
" gradients = (2.0/M)* xi.T @ ((xi @ theta)-yi)\n",
|
|
" eta = learning_schedule(epoch*m+i)\n",
|
|
" theta = theta - eta*gradients\n",
|
|
"print(\"theta from own sdg\")\n",
|
|
"print(theta)\n",
|
|
"\n",
|
|
"plt.plot(xnew, ypredict, \"r-\")\n",
|
|
"plt.plot(xnew, ypredict2, \"b-\")\n",
|
|
"plt.plot(x, y ,'ro')\n",
|
|
"plt.axis([0,2.0,0, 15.0])\n",
|
|
"plt.xlabel(r'$x$')\n",
|
|
"plt.ylabel(r'$y$')\n",
|
|
"plt.title(r'Random numbers ')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9671e6bc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"In the above code, we have use replacement in setting up the\n",
|
|
"mini-batches. The discussion\n",
|
|
"[here](https://sebastianraschka.com/faq/docs/sgd-methods.html) may be\n",
|
|
"useful."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1d74a9cd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Momentum based GD\n",
|
|
"\n",
|
|
"The stochastic gradient descent (SGD) is almost always used with a\n",
|
|
"*momentum* or inertia term that serves as a memory of the direction we\n",
|
|
"are moving in parameter space. This is typically implemented as\n",
|
|
"follows"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8abf1570",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\mathbf{v}_{t}=\\gamma \\mathbf{v}_{t-1}+\\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t) \\nonumber\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "17a32c59",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation} \n",
|
|
"\\boldsymbol{\\theta}_{t+1}= \\boldsymbol{\\theta}_t -\\mathbf{v}_{t},\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2956cf09",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where we have introduced a momentum parameter $\\gamma$, with\n",
|
|
"$0\\le\\gamma\\le 1$, and for brevity we dropped the explicit notation to\n",
|
|
"indicate the gradient is to be taken over a different mini-batch at\n",
|
|
"each step. We call this algorithm gradient descent with momentum\n",
|
|
"(GDM). From these equations, it is clear that $\\mathbf{v}_t$ is a\n",
|
|
"running average of recently encountered gradients and\n",
|
|
"$(1-\\gamma)^{-1}$ sets the characteristic time scale for the memory\n",
|
|
"used in the averaging procedure. Consistent with this, when\n",
|
|
"$\\gamma=0$, this just reduces down to ordinary SGD as discussed\n",
|
|
"earlier. An equivalent way of writing the updates is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9c55e40e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"$$\n",
|
|
"\\Delta \\boldsymbol{\\theta}_{t+1} = \\gamma \\Delta \\boldsymbol{\\theta}_t -\\ \\eta_{t}\\nabla_\\theta E(\\boldsymbol{\\theta}_t),\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b60fe832",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"where we have defined $\\Delta \\boldsymbol{\\theta}_{t}= \\boldsymbol{\\theta}_t-\\boldsymbol{\\theta}_{t-1}$."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "51c2673d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Algorithms and codes for Adagrad, RMSprop and Adam\n",
|
|
"\n",
|
|
"The algorithms we have implemented are well described in the text by [Goodfellow, Bengio and Courville, chapter 8](https://www.deeplearningbook.org/contents/optimization.html).\n",
|
|
"\n",
|
|
"The codes which implement these algorithms are discussed after our presentation of automatic differentiation."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1801af52",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Practical tips\n",
|
|
"\n",
|
|
"* **Randomize the data when making mini-batches**. It is always important to randomly shuffle the data when forming mini-batches. Otherwise, the gradient descent method can fit spurious correlations resulting from the order in which data is presented.\n",
|
|
"\n",
|
|
"* **Transform your inputs**. Learning becomes difficult when our landscape has a mixture of steep and flat directions. One simple trick for minimizing these situations is to standardize the data by subtracting the mean and normalizing the variance of input variables. Whenever possible, also decorrelate the inputs. To understand why this is helpful, consider the case of linear regression. It is easy to show that for the squared error cost function, the Hessian of the cost function is just the correlation matrix between the inputs. Thus, by standardizing the inputs, we are ensuring that the landscape looks homogeneous in all directions in parameter space. Since most deep networks can be viewed as linear transformations followed by a non-linearity at each layer, we expect this intuition to hold beyond the linear case.\n",
|
|
"\n",
|
|
"* **Monitor the out-of-sample performance.** Always monitor the performance of your model on a validation set (a small portion of the training data that is held out of the training process to serve as a proxy for the test set. If the validation error starts increasing, then the model is beginning to overfit. Terminate the learning process. This *early stopping* significantly improves performance in many settings.\n",
|
|
"\n",
|
|
"* **Adaptive optimization methods don't always have good generalization.** Recent studies have shown that adaptive methods such as ADAM, RMSPorp, and AdaGrad tend to have poor generalization compared to SGD or SGD with momentum, particularly in the high-dimensional limit (i.e. the number of parameters exceeds the number of data points). Although it is not clear at this stage why these methods perform so well in training deep neural networks, simpler procedures like properly-tuned SGD may work as well or better in these applications.\n",
|
|
"\n",
|
|
"Geron's text, see chapter 11, has several interesting discussions."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c0c6253b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Using Automatic differentation with OLS\n",
|
|
"\n",
|
|
"We conclude the part on optmization by showing how we can make codes\n",
|
|
"for linear regression and logistic regression using **autograd**. The\n",
|
|
"first example shows results with ordinary leats squares."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "dcd56458",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[4.09527217]\n",
|
|
" [3.02622906]]\n",
|
|
"Eigenvalues of Hessian Matrix:[0.34998197 4.32265589]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"theta from own gd\n",
|
|
"[[4.09527217]\n",
|
|
" [3.02622906]]\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABVq0lEQVR4nO3dd3wUZeIG8GfSCSUkQEIaEJAeeo3USJOyghFRUEBO7zwFNICecJwCnsjJTymKylkOVAyiEsAFFSkJoICCBOnVAKGEziYEEpLN+/tj2SWbbM+W2Z3n+/nkozszO/vObpZ58lZJCCFAREREpFB+ni4AERERkScxDBEREZGiMQwRERGRojEMERERkaIxDBEREZGiMQwRERGRojEMERERkaIxDBEREZGiMQwRERGRojEMEcnQ0qVLIUmS4ScgIADR0dF4/PHHcfz4cY+Va+bMmZAkyWOv7w1OnToFSZLw9ttve7ooRGSjAE8XgIjMW7JkCZo1a4bCwkL88ssvmD17NjIyMnDkyBGEh4d7unhERD6BYYhIxhITE9GxY0cAQO/evaHVajFjxgysXr0a48aN83DpyJOKi4sNtYZEVDlsJiPyIvpgdPHiRcO2wsJCTJkyBW3btkVYWBgiIiKQlJSENWvWVHi+JEmYMGECvvjiCzRv3hyhoaFo06YN1q5dW+HYdevWoW3btggODkZCQoLZZp/CwkJMmzYNCQkJCAoKQmxsLMaPH48bN24YHdegQQMMGTIEa9euRbt27VClShU0b97c8NpLly5F8+bNUbVqVXTu3Bm7d++2+n7omxMzMjLw3HPPoXbt2qhVqxZSUlJw/vz5Ctc+c+bMCudo0KABnnrqqQrn3Lx5M/7617+iVq1aqFGjBsaMGYOCggLk5uZixIgRqFmzJqKjo/HSSy+huLi4wnlLS0sxe/Zs1KtXDyEhIejYsSM2bdpU4bjjx49j1KhRiIyMRHBwMJo3b47333/f6JjMzExIkoQvvvgCU6ZMQWxsLIKDg3HixAncunULL730EhISEhASEoKIiAh07NgRy5cvt/r+EZEO/6Qg8iLZ2dkAgCZNmhi2FRUV4dq1a3jppZcQGxuLO3fuYOPGjUhJScGSJUswZswYo3OsW7cOu3btwuuvv45q1aph7ty5ePjhh3H06FE0bNgQALBp0yYMHToUSUlJ+Oqrr6DVajF37lyjEAYAQggMGzYMmzZtwrRp09CjRw/s27cPM2bMwI4dO7Bjxw4EBwcbjv/jjz8wbdo0TJ8+HWFhYZg1axZSUlIwbdo0bNq0CW+++SYkScIrr7yCIUOGIDs7G1WqVLH6vjzzzDMYPHgw0tLSkJOTg5dffhlPPvkkNm/e7PB7/cwzzyAlJQVfffUVsrKy8M9//hMlJSU4evQoUlJS8Le//Q0bN27EW2+9hZiYGEyePNno+YsWLUL9+vWxYMEClJaWYu7cuRg4cCC2bNmCpKQkAMChQ4dw//33o169enjnnXdQt25drF+/Hi+88AKuXLmCGTNmGJ1z2rRpSEpKwuLFi+Hn54fIyEhMnjwZX3zxBd544w20a9cOBQUFOHDgAK5everwtRMpjiAi2VmyZIkAIHbu3CmKi4tFfn6++PHHH0XdunVFz549RXFxsdnnlpSUiOLiYvH000+Ldu3aGe0DIKKiokReXp5hW25urvDz8xNz5swxbOvSpYuIiYkRt2/fNmzLy8sTERERouw/Gz/++KMAIObOnWv0OitWrBAAxEcffWTYVr9+fVGlShVx9uxZw7a9e/cKACI6OloUFBQYtq9evVoAEN99951N79Pzzz9vtH3u3LkCgLhw4YLRtc+YMaPCOerXry/Gjh1b4ZwTJ040Om7YsGECgJg3b57R9rZt24r27dsbHmdnZwsAZt+/vn37GrYNGDBAxMXFCY1GY3TOCRMmiJCQEHHt2jUhhBAZGRkCgOjZs2eF8icmJophw4ZV2E5EtmMzGZGMde3aFYGBgahevToefPBBhIeHY82aNRX6iXzzzTfo1q0bqlWrhoCAAAQGBuLTTz/F4cOHK5wzOTkZ1atXNzyOiopCZGQkTp8+DQAoKCjArl27kJKSgpCQEMNx1atXh0qlMjqXvualbDMTADz66KOoWrVqhWahtm3bIjY21vC4efPmAHT9oUJDQyts15fJmoceesjocevWre16vilDhgwxeqwv0+DBgytsN/U65t6/rVu3QqvVorCwEJs2bcLDDz+M0NBQlJSUGH4GDRqEwsJC7Ny50+icjzzySIXX6dy5M3744QdMnToVmZmZuH37tsPXTKRUDENEMvb5559j165d2Lx5M5599lkcPnwYI0eONDomPT0dI0aMQGxsLJYtW4YdO3Zg165d+Mtf/oLCwsIK56xVq1aFbcHBwYab6PXr11FaWoq6detWOK78tqtXryIgIAB16tQx2i5JEurWrVuhqSYiIsLocVBQkMXtpspvSvlr0jfNVSYY2FNWU+U09/7duXMHN2/exNWrV1FSUoL33nsPgYGBRj+DBg0CAFy5csXo+dHR0RXO+e677+KVV17B6tWrkZycjIiICAwbNsyjUzAQeRv2GSKSsebNmxs6TScnJ0Or1eKTTz7Bt99+i+HDhwMAli1bhoSEBKxYscJoDqCioiKHXjM8PBySJCE3N7fCvvLbatWqhZKSEly+fNkoEAkhkJubi06dOjlUBlcIDg42+Z64qm+NufcvKCgI1apVQ2BgIPz9/TF69GiMHz/e5DkSEhKMHpua46lq1aqYNWsWZs2ahYsXLxpqiVQqFY4cOeKciyHycawZIvIic+fORXh4OF577TWUlpYC0N0gg4KCjG6Uubm5JkeT2UI/mis9Pd2oxiM/Px9qtdro2D59+gDQBbKyVq5ciYKCAsN+OWjQoAH27dtntG3z5s24efOmS17P3PvXo0cP+Pv7IzQ0FMnJycjKykLr1q3RsWPHCj+mavEsiYqKwlNPPYWRI0fi6NGjuHXrlrMvi8gnsWaIyIuEh4dj2rRp+Mc//oG0tDQ8+eSTGDJkCNLT0/H8889j+PDhyMnJwb///W9ER0c73FTy73//Gw8++CD69euHKVOmQKvV4q233kLVqlVx7do1w3H9+vXDgAED8MorryAvLw/dunUzjCZr164dRo8e7axLr7TRo0fj1VdfxWuvvYZevXrh0KFDWLRoEcLCwlzyev7+/ujXrx8mT56M0tJSvPXWW8jLy8OsWbMMxyxcuBDdu3dHjx498Nxzz6FBgwbIz8/HiRMnoFarbRoN16VLFwwZMgStW7dGeHg4Dh8+jC+++AJJSUlG/bCIyDyGISIvM3HiRCxatAivv/46Ro4ciXHjxuHSpUtYvHgx/ve//6Fhw4aYOnUqzp49a3TjtUe/fv2wevVq/Otf/8Jjjz2GunXr4vnnn8ft27eNzilJElavXo2ZM2diyZIlmD17NmrXro3Ro0fjzTffNBpW72kvv/wy8vLysHTpUrz99tvo3Lkzvv76awwdOtQlrzdhwgQUFhbihRdewKVLl9CyZUusW7cO3bp1MxzTokUL7NmzB//+97/xr3/9C5cuXULNmjXRuHFjQ78hax544AF89913mD9/Pm7duoXY2FiMGTMG06dPd8l1EfkiSQghPF0IIiIiIk9hnyEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0RcwzVFpaivPnz6N69eomp7MnIiIi+RFCID8/HzExMfDzc139jSLC0Pnz5xEfH+/pYhAREZEDcnJyEBcX57LzKyIMVa9eHYDuzaxRo4aHS0NERES2yMvLQ3x8vOE+7iqKCEP6prEaNWowDBEREXkZV3dxYQdqIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0j4ehrVu3QqVSISYmBpIkYfXq1WaPffbZZyFJEhYsWOC28hEREZFv83gYKigoQJs2bbBo0SKLx61evRq//vorYmJi3FQyIiIiUoIATxdg4MCBGDhwoMVjzp07hwkTJmD9+vUYPHiwm0pGRERESuDxmiFrSktLMXr0aLz88sto2bKlp4tDREREPsbjNUPWvPXWWwgICMALL7xg83OKiopQVFRkeJyXl+eKohEREZEPkHXN0O+//46FCxdi6dKlkCTJ5ufNmTMHYWFhhp/4+HgXlpKIiIi8mazD0LZt23Dp0iXUq1cPAQEBCAgIwOnTpzFlyhQ0aNDA7POmTZsGjUZj+MnJyXFfoYmIiMiryLqZbPTo0ejbt6/RtgEDBmD06NEYN26c2ecFBwcjODjY1cUjIiIiH+DxMHTz5k2cOHHC8Dg7Oxt79+5FREQE6tWrh1q1ahkdHxgYiLp166Jp06buLioRERH5II+Hod27dyM5OdnwePLkyQCAsWPHYunSpR4qFRERESmFx8NQ7969IYSw+fhTp065rjBERESkOLLuQE1ERETkagxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAxDREREpGgMQ0RERKRoDENERESkaAGeLgARERHJhFYLbNsGXLgAREcDPXoA/v6eLpXLMQwRERERkJ4OvPgicPbsvW1xccDChUBKiufK5QZsJiMiIlK69HRg+HDjIAQA587ptqene6ZcbsIwREREpGRara5GSIiK+/TbUlN1x/kohiEiIiIl27atYo1QWUIAOTm643wUwxAREZGSXbjg3OO8EMMQERGRkkVHO/c4L8QwREREpGQ9euhGjUmS6f2SBMTH647zUQxDRERESubvrxs+D1QMRPrHCxb49HxDDENERERKl5ICfPstEBtrvD0uTrfdx+cZ4qSLREREpAs8Q4dyBmoiIiJSMH9/oHdvT5fC7dhMRkRERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESK5vEwtHXrVqhUKsTExECSJKxevdqwr7i4GK+88gpatWqFqlWrIiYmBmPGjMH58+c9V2AiIiLyKR4PQwUFBWjTpg0WLVpUYd+tW7ewZ88evPrqq9izZw/S09Nx7NgxPPTQQx4oKREREfkiSQghPF0IPUmSsGrVKgwbNszsMbt27ULnzp1x+vRp1KtXz6bz5uXlISwsDBqNBjVq1HBSaYmIiMiV3HX/DnDZmV1Eo9FAkiTUrFnT7DFFRUUoKioyPM7Ly3NDyYiIiMgbebyZzB6FhYWYOnUqRo0aZTEhzpkzB2FhYYaf+Ph4N5aSiIiIvInXhKHi4mI8/vjjKC0txQcffGDx2GnTpkGj0Rh+cnJy3FRKIiIi8jZe0UxWXFyMESNGIDs7G5s3b7babhgcHIzg4GA3lY6IiIi8mezDkD4IHT9+HBkZGahVq5ani0REREQ+xONh6ObNmzhx4oThcXZ2Nvbu3YuIiAjExMRg+PDh2LNnD9auXQutVovc3FwAQEREBIKCgjxVbCIiIvIRHh9an5mZieTk5Arbx44di5kzZyIhIcHk8zIyMtC7d2+bXoND64mIiLyPYobW9+7dG5bymIymQSIiIiIf5DWjyYiIiIhcgWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgULcDTBSAiIiIfpdUC27YBFy4A0dFAjx6Av7+nS1UBwxAREZEceUmQMCs9HXjxReDs2Xvb4uKAhQuBlBTPlcsENpMRERHJTXo60KABkJwMjBql+2+DBrrt3iA9HRg+3DgIAcC5c7rtMrsOhiEiIiI58bIgUYFWq6sREqLiPv221FTdcTLBMERERCQXXhgkKti2rWKQK0sIICdHd5xMMAwRERHJhRcGiQouXHDucW7AMERERCQXXhgkKoiOdu5xbsAwREREJBdeGCQq6NFDN2pMkkzvlyQgPl53nEwwDBEREcmFs4OEVgtkZgLLl+v+646+Rv7+uuHzQMXr0D9esEBW0wQwDBEREcmFM4OEJ4fnp6QA334LxMYab4+L022X2TxDkhCmuqz7lry8PISFhUGj0aBGjRqeLg4REZFlpiYsjI/XBSFbgoR+eH75W7w+ULkrkFRy4kh33b8ZhoiIiOTI0SCh1epqgMyNSpMkXQ1NdrasmqpMcdf9m8txEBERyZG/P9C7t/3Ps2d4viPn90EMQ0RESubt618pmbnPzheG57sZwxARkVJ50UKaVI6lz86bh+eXD3ht2rjlZRmGiIiUyFwHW/36VzIc8UN3WfvsVqzQBaNz50wv66HvMySjeX4AmAx4hTXquOWlPT60fuvWrVCpVIiJiYEkSVi9erXRfiEEZs6ciZiYGFSpUgW9e/fGwYMHPVNYIiJf4AvrXymVLZ/dlCnA/Pm6//eSeX70AU+U6+sUlHfZLS/v8TBUUFCANm3aYNGiRSb3z507F/PmzcOiRYuwa9cu1K1bF/369UN+fr6bS0pE5CPksv6VJyYE9Ha2fna1a3vNPD/Hvj+B/Cf+DiEEyk816a6Q4vFmsoEDB2LgwIEm9wkhsGDBAkyfPh0pdz+4zz77DFFRUUhLS8Ozzz7rzqISEfkGOXSwZX8lx6xZY9txFy4AI0cCQ4fKroN8SWEJfvnvAai/uAH1vvqILj6LTLinBsgcj4chS7Kzs5Gbm4v+/fsbtgUHB6NXr17Yvn272TBUVFSEoqIiw+O8vDyXl5WIyGt4uoMt+ys5RqsFli2z7Vj9Z+fo8Hwnu3Fagx/fOQj1dwI/nGmB66KtYV9n/Oy5gt0l6zCUm5sLAIiKijLaHhUVhdOnT5t93pw5czBr1iyXlo2IyGvp178y18EWAOrU0e3PzHRubYK1Pi+SpOuvNHSox2swZGfbNuDKFevH1aljuXO0m6ZTOLHpNNTvZUO9JQzbbiSiBPcb9tWSrmJQwmGohvpjUFIEMMLpL28XWYchPalcBzAhRIVtZU2bNg2TJ082PM7Ly0N8fLzLykdE5FX0618NH64LH6aCyeXLwJNP6v7fmc1XnBDQMVotsGmTbcc+8YT5cOPC5smSwhLs+OQg1J9fh/qPeBy50whAfcP+5kEnoWqbA9XYCCQ90xL+Qd11O7Ra6+HcxWQdhurWrQtAV0MUXaa69tKlSxVqi8oKDg5GcHCwy8tHROS19Atplr8xmuLM5is59FfyNqYCjCXh4ebP4+TmSc0ZDdbPPwj1mlJ8f6oFrol78wIFoBg9w/dD1SsfqokN0OiBRgAaVTyJLeHcxTw+msyShIQE1K1bFxs2bDBsu3PnDrZs2YL777/fwjOJiMiqlBTg1CkgI0PXF6WOmTldnDnc3tP9lcyR68g2fYCxNQgBwIwZFVemd+J0Cn9mnsHClC3oG7EHteuH4rEF92NZdndcExEIl67jiQa/4KsXtuPyqVvYdK09Ulf1QqMH6ls+qblV7ss/dhXhYfn5+SIrK0tkZWUJAGLevHkiKytLnD59WgghxH/+8x8RFhYm0tPTxf79+8XIkSNFdHS0yMvLs/k1NBqNACA0Go2rLoOIyLtlZAihuy1a/snIqNzrlJQIERcnhCSZPr8kCREfrzvOXVau1JWpbDni4nTbPUn/XtnyuVh7Dyvx+ZYUlYht7/8hXumSIVoEH6/wlKZBJ8VLHTPElnf3iuLbxZW/5owMIdLShMjIEJpr19xy//Z4M9nu3buRnJxseKzv6zN27FgsXboU//jHP3D79m08//zzuH79Orp06YKffvoJ1atX91SRiYh8j7uaryw1iXhiQkA5j2yz1r/KHFP9ruz8fPPO5mH9vINQr9Hi++zmuCpaGw7xRwl61tyPIT01UE1ogMb9GgJoaF8ZzXXiLj/6zV2jwV0atWSCNUNERFa4q2ZIz1RtTHy8e2tjrNW8eKKWqqy0NPtrhcr+pKXdO5eNn++3PeaLvhG7RSCKjHbVlK6LUfV/Fssn/iKun7pRueuyoybOXfdvSQgPdd12o7y8PISFhUGj0aBGjRqeLg4RkfxotUCDBtbXs8rOdu4we09OCJiZCZRpmTArI8MzI9tsLZ85Zctt5fMthYSziEMCslEK3WfQJDAbqtanoRpdE92eTURASCUbk7RaYPZsXZ+m8vS1guVq4tx1//Z4MxkREcmAJ5qvPD0hoNxHttkyH5QpphZivfv5iuHDdYeUObz07qPJeAc9wvZD1eMGVBPqo8mABAAJlb8OwPqIOA/PMcUwREREOuaG28fF6YKQr80KLceRbeVry+bPB0aMsH3IuYngevqXs1AvOAl1Rj2EieV4By8hHvc+3wK/6vizw6P4aFkyIprUdv41meuXVZ4H55hiMxkRERnzdPOVu+ibjix1Uo6Pd27ToCXmJkQcOVI35L/s9vh44PHHTW4vfWceftM0g3rJFaj3xGB/YROjl2kScBILar+BPpp0BN0u00HZFWvD2fIel5eWprtmsJmMiIg8xdPNV+7i76+76f7f/5k/5vHHnRuEzAVNS6Pa3n4bWLFCNw9U+efNmQNs24bCw9nYtfU2lu5ugbWPtcQlcW/OKD9o0a3GAai6X4dqfD00vbUX0ojP3DOCzpERce6eYwqsGSIiIqVyd82QuZqf+fOBSZPMl8NM5/UzO85h7cITUG8KxeYrrXEH91ZeqAENHow/CNVggYGTm6NW4wjdDmvX7OyO8suXA6NG2XasiddmzRAREZEr2VJr4aw+LJZqfh591PJz7/alKc3cgt1noqD+32Wod0fjj8KmAO7N0Nww4DRUiaegGlUdPZ5LRFA1Eys1uHttOHtredw5x1QZDENERKRM7hpNZstSGDYY3+8YFosHDI/9oEVS9YNQdbsG1XNxaD6kESS/+pZP4u4RdLaOiHNFfyU7MAwREZEyuWs0maMzSZdzWDRDdeRhQOxBqAZpMWhKc9Ru2tr6E8ty9wg6WxZhnTULmD7do530GYaIiEiZrNVamJqvxxE21rIIGM//o1cKCZrAOpg+MxS9XghBULUkx8vi6DVXZoShuSkb4uNlM2UDwxARESmTuyaatLGWRQAQkOAHUWabBEkCwr/6EP1SOleuHIB916wPQGvWAMuWAVeu3DvW3matlBTdZIoynbLBz9MFICIi8hh9rUVsrPH2uDjnDTG/WxsjTNb76Gp+chCLeeFv4HZV40kPpfg4SM5eLNaWa05P1406S07WhaOyQQi4Nww/Pd3219VP2TBypO6/MglCAIfWExERuWSiSVEqsCftCNSfXMTNHfsw904qAJSr+dEp+eIrBD75mHsnvLR3zqPyXLFeXTnuun8zDBERETnJ7Wu3sWnBfqi/KcTaY41xvvReE1kKvsUH0gREiYv3niCjfjMAHJsx2oUL2XKeISIiIi9wYe9FrJ13FOqfgrHxYivcxr2+PVVxE/2jD0D1YDEGTe6FqObnZNtvBoBjI988tZCtEzEMERER2UGUCuxdcRTqj3Oh/jUSu2+1ABBl2B/vfw6q5iegeqwqek9IREjNrsYnkPNSJ44EGw8sn+FsDENERERWFN4oxOaF+6H++jbWHr0PZ7XNADQz7O9c9QBUXa9A9bdotB7eBJJfrPmTyZk9wcZZUw/IAMMQERGRCbn7LmHdvKNQrw/ChtxE3EInw75QFKBf3QNQDbiDwZObom7rRA+W1IlsnTFaz0PLZzgbwxARERF0zV9/fHMMaz++APXO2vitIBFApGF/nP95DGl6AqoRVfDAi60QUrOL5wrrKrbMGA3Ir+N3JTEMERGRYhXeKETGu3ebv440Qo62KYCmhv2dqh7EkM6XofprXbR9rCkkvxjPFdZdzM0YXacO8MQTuskT5dbxu5I4tJ6IiKxz5/w3LnbxwGWse+cI1OsDseFCIm6jCnpgG6JxAdcQjtCoGhg8QIvBk5ogum2U9RP6Khl85hxaT0RE8pCeXrGWwMOrjNtDlArsTz8O9X/PQ72jNn4raAEBXaffh5GORZiAGJQZRRUYBwxdCLT1/o7BlaKfMVoBWDNERETmmZuNWL+OlbOXinCSorwiZL63H+oVBVh7qBFOa+OM9ncIPYSXE77FiIMzgfILZcj82pSEM1A7EcMQEVWaDJoM3M7abMRuWI7BHpcPX9E1f/0QgJ/Ot8RNVDfsC8Ft9I3cD1W/QgxObYzYdpFedW1KxWYyIiK58PJmIodZm41YCCAnR3ecteYUF639dXDNCagXn4N6ey3svNkSAt0N+6P9cjGkyTGohoegz4uJCK1dZtX3zEznXRt5PYYhIiJLzDUT6Vft9uWmFFtnI7Z2nBPD5J2bd7Bl0X6ol9+E+lBDnCppDKCxYX+7Koeh6nQRqqcj0X5UM/gF1HWszPYeR16NYYiIyBytVncTN9WbQAhdU0pqqm6osS82pdg6G7Gl45wQJq8cvYrv3zkM9ff+WH+uJfLRwbAvGIXoU2c/VH1vY8ikxojr1BxA88qV2ZHjyKuxzxARkTmZmUBysvXjXLhqt0fp+wyZm43YWr8aB/sciVKBw2tPQv3hWah/icCO/JYoxb39UX6XMOS+o1A9EoS+qYmoGlnV/ddGbsE+Q0REnqb0phRLsxHrR1xZWo7Bjj5Hdzrej20fHoA6LR/qAw3wZ8l9AO4zHNom5Cge6nAWI7udQdNWQfCLiwV6dHQ8qFT22sinMAwREZkjl6YUT45kMzcbcVyc9eUYbAyJ3z60FN/nn8SfaIRt6IFS+CMIRXig9j6o+tzCkBfvQ70LB3Vl+MWJndjtuTYljiZUEDaTERGZI4emFLmMZHMkDNjazFjGtYA6OD5kMlp+OAHV6lbTbXT1XEfWrk0un4ECcZ4hJ2IYIiKH6W/EgOmmFFeMJtPfnNes0dVQlOctkwJqtRD16wPnzkNCxVuNfovFCQ89PdeRl0466Svcdf/2c9mZiYh8gb4pJTbWeHtcnGtuhOnpupt/crLpIATcuzGnpurCgsxcO3kdXz7/Cx5P+BVjzs2BAFBqHHlMByGg4rXZM9eRs1kbTVi2nOTV2GeIiMialBTd8HlX9xkxVwthij4EzJwJ9Onj8T4sR3/4E+r3z0D9c038okmEFt3u7rkfASjBO/6vIEJ72XB8hRBUVtmA48lO7M6cdLI89kGSFYYhIiJbuHrRSku1EJa88Ybux819WEoKS/Dz4gNQL7sB9b76OF7cEEBDw/7E4ONQtT8H1VO10PmpMfD3H3Pv5n/okK7M1uiDgi1c0YndVUHMUh8kd4RuqoBhiIhIDqzVQljjhhmxr2ffwI/zDkGtFvjhTEvcEG0N+wJxB70j9kOVfBNDXmiIhJ7GM0MDuBcmMzNtC0P6MBAXZ70Tew8XrDDviiBmaRLKRx4BatUCrl69t50dtd2CHaiJiORg+XJg1KjKncMFnYmPbzgF9aJTUG8Nw7YbraAt8zd0LekqBicchmqYP/pPaokacTb++2rvKD1PdGJ3pJy2ns+e0KvwjtrsQE1EpCTOaOZxQmfiksISbH3vD7zcKRPNgv9Ek/4NMOW73si80Q5aBKBF8Am80iUTP3+wDxcLa+Kzk90x/J0k24MQcG/CQ+DezV7P1ISH7u7E7mg5rXGk9o8dtd1C9mGopKQE//rXv5CQkIAqVaqgYcOGeP3111FaWurpohEpj1ara+JYvlz3X/7j7Dz65qDyN11H2NmHRXNGgxUvbseTCb8gMjQfvV5og7d398bROw0RgGL0Cd+DBQ9vwcmMMzhYeB/+s7M3uj3XGv5Blah9sjfgpKQAp07plj5JS9P9Nzvb9bUlzgxijnbyduWIOQLgBX2G3nrrLSxevBifffYZWrZsid27d2PcuHEICwvDiy++6OniESkHJ55zLUvLQ9jLhlqmE5tOY+2iU1BvqY6t11uhBPcb9kVI1zCowSGohvphwKSWCKvX3vGyWGLvKD1Xd2I3x1mjCStb++ery77IgOz7DA0ZMgRRUVH49NNPDdseeeQRhIaG4osvvrDpHOwzRFRJnHjOfUyFTn9/22rhLPRhKSkswY5PDkL9+XWo/4jHkTuNjPY3DzqJIW1yoBoTjqRnWiIgxM6/lV01VNyXhqBb64Nkja8uCGyB2+7fQubmzJkj6tevL44ePSqEEGLv3r0iMjJSpKWl2XwOjUYjAAiNRuOqYhL5rpISIeLihND9813xR5KEiI/XHUfOUVIiREaGEGlpuv9+/bXufZYky5+DJAmxcqXhNDdO3xArUn8RTyZsExHSVaPDA3BHJNfcI+YNzRDHN56qXHlXrqz4OxIXZ1QWWZ3Xk1autP5Z8jtm4K77t+zDUGlpqZg6daqQJEkEBAQISZLEm2++afE5hYWFQqPRGH5ycnIYhogclZFh2z/YGRmeLqlvMxUMyv7ExwuxcqU4mXFaLHg4U/QJ/10EosjokHDpmhhV/2exfOIv4vqpG/a9fvmApr8x62/uNoQzu6/XFeeVA1OfZa1a967P1663EhiG7lq+fLmIi4sTy5cvF/v27ROff/65iIiIEEuXLjX7nBkzZgjoZns3+mEYInJAWpptYciO2lqfZy44OPO8GzcKsXGj0H6+TOxL/URM7bxRtAg+XuFjaRp0UrzUMUNseXevKL5d7Njrmquh+fpr19QaKqE20tTviKn3+W7IVSp3hSHZ9xmKj4/H1KlTMX78eMO2N954A8uWLcORI0dMPqeoqAhFRUWGx3l5eYiPj2efISJH2LryuAL7M5jkho7meWfz8NP8g1Cv1uL77Ga4Imob9vmjBD1q7oeqpwaqCQ3QuF+Dyr2Ypf5itt4+7P3dUPLvnC/1kXICd/UZkv1oslu3bsHPz3gGAH9/f4tD64ODgxEcHOzqohEpgydnAPY2lmYXruTs0Kd+Pgv1gpNQZ1RD5rVWKEaSYV9N6QYG1jsIlUrCg5NbIDyhXWWu4h6tFnjhBcsLldrC3lFQnlyPzNM8NWJO4WQfhlQqFWbPno169eqhZcuWyMrKwrx58/CXv/zF00UjUgZLQ74dmXjOV1lb4VySdBPnDR1q03ulvaPFb0sPQb30KtR7YnGgqDGAOMP+xoHZULU+DdWTNdHtby0RGNrN/MkcNXu2LshVlr1Dym09/uJF3ZxXrEGhSpJ9M1l+fj5effVVrFq1CpcuXUJMTAxGjhyJ1157DUFBQTadg0PriZzAVPNPfLwuCHFYvVOadm7m3sRP8w5AvaoY6042w2VRx7DPHyXoFnYAqu43oBpfD00HNjR5DqdJT9etlVUZji4PYssQ9PLTDXDOK5/krvu37MOQMzAMETkJ+zOYZ+vaYmlpwMiRhodndpyDev4JqDdXRcbVVriDe038YdDgwfiDUA0RGDilBSIahbui5BU5uoaWM9cNM7cemaXXr8zrkSwxDDkRwxARuZyNNUOlGzdh16lIqJdcgfr3aOwrbGq0v1HAaahanYLqiRro8VwiAkMDXVRgC2yt5QJ0tYPz5gGTJjm/1tDeCShdsFAteRbDkBMxDBGRy1lp2hEArgVEopV2Ly6Ie31i/KDF/TUOQNXtOlTPx6PZoIaQ/JywPlll2FrLBQArV+oCjztmoL54URe6rPHFUWYKJcvRZDk5OYiPj3dVWYjIm/hak1llr8dCR/NS6MLNX0s+xAVEozry8GDcQagGazFoSgvUatzG2VdTObZ2YJ41617Nj6tGQZU97/Lltj3HF0eZkUvZtWp9s2bN8Oqrr6KgoMBV5SEib5CerqsFSU7W1SAkJ+sep6fr9nvb6vbWrscGpSWl+E3TFMubzkSuiDLadxZxGO/3Ieq1rYUNb+3BlfwQfJ2ThNGLu6NW4wjnXosz6KdTkCzUUMXFAdOnu69MgO0hrbILopLi2NVMtn37dkyaNAk5OTmYPXs2xo0b58qyOQ2byYicyNqirS+9pAtB3rK6fSUWoS24VICNCw5AvfIO1p1ogtxSXQjygxa9kImeIbvRpFUw2k4biOZDm3i++cse1jowz5qlC0PurA20NsqMfYZ8jqz7DH3++eeYPn06ateujfnz56O3zNtmGYaInMSRUUaAfEf62HI9dero9t+dyuPsrgtYO/841BurYPPlRBSiiuHQ6sjDgNiDUA3SYtCU5qjdtJaLL8DFTHVgLssTIddcSJPr7xhViuxXrb9165Z49dVXRWhoqBg2bJg4fvx45RYGcSGuWk/kJLYu2uot60nZeD13qoeL5S1miXZVDlXY3SDgjJjYOlP8NGe3KMov8vQVOV9JiRCzZpn/TD2xiCjX8FIMd92/7eozVC5EoX///vjb3/6G7777DomJiZgyZQry8/Odl9SISF4q0zFVCCAnR9dJWS5svB7//OsYcWgmGtw+DAmlSKq2H2/2z8T+9OP4sygO7/7RC/2mdkBQNdsmgvU6H39seru+ZiY11b39wlJSgFOndKPG0tJ0/83OZo0QOcyu0WSLFy/Grl27sGvXLhw+fBj+/v5o3bo1xo8fj7Zt2+LLL79EixYtsGrVKnTs2NFVZSYiT3FGx1Q5jfSx8Xr8AJRC4POqz+HW9vsR2bqVa8slJ9u2WW5GLBty3dllgmt4kRPZFYZmz56Nrl27YuzYsejatSs6duxotCDqX/7yF7z55pt46qmncODAAacXlog8zNqirbaQyUgfUSqQlVMbjYLroHrRFfjB8vX4AahWcAnVrh0BUNctZZQFJS+aSoph9zxD1jz99NN49dVXHS4QEcmYv79utuERI+x/rgxWt7997TY2v3sA6q9vY+3RxjhXmoiHsRjfYjhKYeNcI0q76Tt7OLuvzU9FPsHhPkPmREZGYvPmzc4+LRHJQXo6MHmy6X3x8cDLL+tCT/n5aTy4uv2FvRfxydhtGBr9K2rVEhgyqxP+e7gnzpVGIxQFENExyEh+HSLcxpFfMqnZchtrcw5Jku6ztyXkOmE+JyJX4HIcRAD/WrWFufl49L75Rrffw6vbi1KBvSuOQv1xLtS/RmL3rRZG++P9z2FIs5NQjaiC5BdaIaRmiG7HnTu6m/7ly6ZPrOQ5bJwxnL0S8zmRcsl6niFvwzBEFpm6ect5kkBPsDYfT/mg4OZwWXijEBnv7sd3K25j7dH7cFYbY7S/U9WDUHW5DNXfotHmUQuTH3IOG/MqE3Lt/f0huothyIkYhsgs/rVqG1tXMXfjApm5+y5h3byjUK8PwobcRNxCVcO+KriFfnX3Q9X/DgZPaoLotlEWzlSOh2u2ZM3RkCvD3x/yDrJcqJXIp2i1upueqb8HhNAFotRUYOhQ/rUqgxFFolRg37fHoP7oAtQ7a+O3gkQAkYb9sX4XoGp2XNf8NTERVSK6OPZCKSm6z5zNphU5OpxdBr8/RJYwDJFyyXX+FDny0AKZRXlFyHh3P9QrCrD2cCOc0TYF0NSwv2PoIai6XILqr3XR9rGmkPyc9Pr6m76+JuTrrxmKKoMLrJLMMQyRcvGvVdtZm1/IicPmLx28jHXvHIH6x0D8dCERBbg3gWsV3ELfqP1Q9S/C4NQmiGnfAkAL8yerDKX3JXNmvy83/v4QOYJhiJSLf63azt9fFwKGD9fduEx1LnZw2LwoFTiw6jjU/z0P9Y5a+PVmSwjcuynG+F3AkCbHoXo0BA+8kIjQ2g42f9nDXF+yc+d02329L5mzg6ALf3+InIEdqEm59CNcrP21yhEu9zipc3FRXhG2vH8A6uU3oT7UCKe1cUb721c5DFXni1A9E4X2o5qZH/3lCkof+eTKQQXu7JzO6TJ8AkeTORHDEJnFodT2c/Amc/nwFXw/7wjUP/hj/blE3ER1w74Q3EafyP1Q9S3EkEmNEdvRg7VxSh755I4g6I6QovQmTh/C0WRE7pCSogs8pv7h5FBq02wcUSRKBQ6pT0L94Vmot0dgR34iBLob9tf1u4ghjY9C9Ugw+k5qhdDwDrqb5PFM4KYH/5JXcl8ydwwqcPUCq0pv4iSHMAwRcSi109y5eQdbPzgAdVo+1AcTkF1yH4D7DPvbVjkCVcdcqJ6ORIcnmsEv4O78P3L6S17Jfcm8PQhyugxyEMMQEeD6v1adTUb9Ia4cvYrv3zmMtT/4Y/3ZFshDe8O+YBTigTr7oepzG0NS70N8l2YAmhmfQG5/yVsb+QQAERG6z0Cr9a2bqrcHQU6XQQ5iGCLvJ6Ng4BYerkURpQKH195t/volAjvyW6K0TPNXlN8lDG50FKpHgtBvUiKqRnYyfzI5/iVvaeST3rVrQN++vtcPxduHwHt7zRZ5jlAAjUYjAAiNRuPpopCzrVwpRFycELp/unU/cXG67b5o5UohJMn4egHdNkly2XXfKbgjNs79XbzYNlM0DDhV4eXbhBwR/+qeIX793wGhLdbafuKMjIrXYuonI8Ml12WRqd8tN7/vHqH/HSv/e+YN1yrn3ydyiLvu3xxNRt5LaeuKuXnI99Xj1/DDvMNQr5PwY05L5CHMsC8IRXig9j6o+tzCkBfvQ72kWMdeZPlyYNQo68elpQEjRzr2GpWh1epGl40YoasNMsUXh9p76/psnC7D53A0GZElcmxecTUX94cQpQJHf8yG+v0zUP8cjl/yElGKbob9kdJlDL7vCFQPB6LfpERUq2uh+ctWcu+j4u+v+zEXhADf7IfirYMKOLkjOYhhiLyTEjtKuqA/RPGtYvy8+ADUX2qg3t8AJ4obAmho2N8q5BhU7c9DNa42Oj/VAn4BTu4r4g19VJTaD8XbBhXocboMcgDDEHknJd6gnFSLcj37Bn54+yDUayX8mNMCN0Q7w75A3EFyrX1QJRdgyIuN0KB7EwBNKlFoK7zhL3m5115RRd5as0UewzBE3kmJN6hK1KIcW58N9aLTUG+riZ81idCWaf6qLV3B4IZHoHrYH/0nJaJ6TMcKz3cpuf8l7w21V1SRt9ZskUewAzV5J6V2lLRx+ZCSwhL88t8DUH9xA+p99XGsOMHoNC2Dj0PV7hxUT9VCl3Et4B8kg/fI0hQJnp4+gcu2EHkE1yZzIoYhH6XUG5SZkT4F02dDfbAR1N8J/HCmBa6LcMPuQNxBr4j9UPW+CdWLDZHQM94DBXeQXGan9tYRVkRejGHIiRiGfJhSb1B3a0ou/LQP27aU4r8Hu2OLpi20ZVq+a0lXMSjhMFRD/TFgckvUiJP5776p2p81a+Q1fYKna6iIFIZhyIkYhnycgm5QJYUl2PHJQag/vw71H/E4cqeR0f7mQSehapsD1dgIJD3TUh7NX7YwV/tz+zZw9arp5/hqUygRGXCeISJb+XhHSc0ZDX5856Cu+et0c1wTbQz7AlCMnuH7oeqVD9XEBmj0QCMAjcyfzBGuDpvmJs+0NHUC4JvTJxCRRzAMEcnQyc2noX7vFNRbqmPr9VYowf2GfRHSNQxqcAiqoX4YMKklwuq1t3AmK6wFHVf317E0eaatfGn6BCLyCIYhInu5oKZEe0era/767BrUe+Nx+E4jAPUN+5sFnYSqTQ5UY8KR9ExLBIR0N38yW1kLOu5YTd7a5Jm28KXpE4jIIxiGiOzhxJqSvLN5WD/vINRrtPg+uzmuitaGff4oQc+a+6HqpYFqYgLu6+Pk5i9rQWfFCmDyZNcvd1KZWh1r8/soqC8ZEVWOV4Shc+fO4ZVXXsEPP/yA27dvo0mTJvj000/RoUMHTxeN/+C6glzfUyfUlGRvzYF64Z9QZ1bDlmutUIwkw75w6ToG1jsE1UMSHpzSEjXrt7NwpkqwZV238eOBy5fNn8NZ/XUcrdWxNju1XIbjE5FXkH0Yun79Orp164bk5GT88MMPiIyMxMmTJ1GzZk3XvaitN2P+g+v84CLX99TBhWG1d7T4dckhqJdehTorFgeLGsMPMeiBbXgEKxHgD9RtE4khY2qh27OJCAjpVvH8zmbLum6WglBZle2vY8vszhERQEiI7hg9S7NTu6N5j4h8i5C5V155RXTv3r1S59BoNAKA0Gg01g9euVKIuDghdP+U6n7i4nTbyx8nScbHAbptklTxeF9k63tlz/nk+p5mZFQsl6mfjAyRdy5PfPvSdjG20TZRW7pstPsRfC0uSlHOe88ckZZm27XYeL2Vpv/cy3/2ZT/3khLda6Wl6f5bUmL6XCUlFX8ny58zPt7884lIVuy6f1eC7MNQ8+bNRWpqqhg+fLioU6eOaNu2rfjoo48sPqewsFBoNBrDT05Ojm1vpq03Y/6D6/zgIvf31MYA8Wa1N0QQCo02h+GGeLzeL2LLgNmiVA5hz9ZgV6eO6c/YFZ+HqWAdH2//e2JHaCUi+WMYuis4OFgEBweLadOmiT179ojFixeLkJAQ8dlnn5l9zowZMwSACj8W30x7bsZK/wfXFcFF7u+pjeXrhQwBCHFfYLaY1D5DbH5nj7hTcEdeYU9fFmtB55tvrNfYOLtcttT+WGJrrVdamnPLTkQuwTB0V2BgoEhKSjLaNnHiRNG1a1ezz3GoZsiem7HS/8F1RXCR+3taUiK00bGi1Ey5tJBErhQl/m/gRnF43UlRqi01fr7cwp4tTVP645xRY+MucnufiahS3BWG/DzTU8l20dHRaNGihdG25s2b48yZM2afExwcjBo1ahj9WGVrR1B9R2Fb+Or8J7a+V2vW2H5Omb6nZ3acw/sjtuDBqCw8fuEdCEgohWR0jICun2/Utx/gpe/7oNmghpD87h6j1QKZmcDKlba9oLsmEExJ0XUkjo013h4XZ9zBOCUFOHUKyMgA0tJ0/83Olm8HZH2HbEkyvV+SdGvXmRuOT0SKJPvRZN26dcPRo0eNth07dgz169d37gvZczO2ZQSMpflP3MVVQ9Rtfa++/BJ4+23bXlMm72lpSSl2fXYI6iVXoP49GvsKmwLQB4aOqO13A7P9X0V48b3RVpK5hWFNjYyzxp1hLyVFN/rN2u+INy134u+vG3k4fLjud6bs75K14fhEpFwurXdygt9++00EBASI2bNni+PHj4svv/xShIaGimXLltl8Dpuq2WztR6Hvx2BrM4OnOHukV1klJbrOtc5ujqjMe1qJ/iY3L94Uq6buFH9pvFVE+V00emk/lIjuNfaKtwZmiEPqE7rmL1tey1wHczn0GVICb2veIyKT2GeoDLVaLRITE0VwcLBo1qyZ1dFk5dn8Ztp7M5brP7juGKKemuqaPj6OvKcOBL8zO8+JDx7fIgbW+U0E47bRU6tDIx6N2y4+f3abuHzkin3lF8J6Z2lXfi50jzM6ZBORR7krDElCmGqT8C15eXkICwuDRqOx3n/IVNOGuWYQQH6zJWu1QIMG5ptm9M1N2dmVK2dmJpCcbP24jAz7m1jseU/NTbCnbxK52/+ltKQUv395BOpPL0G9uy723m5mdHhCwBmoWmZDNao6ej6fiKBqQfaVuSxb3xs9S79fREQKZtf9uxIYhkyRW8CxhytDSlnuCl2VKIOQJNyuHonUusuhPtECuaVR94qHUiRVPwDV/degei4OLVSN7nV6rqzly4FRo6wfN2EC8Mgj3vX7RUTkRu4KQ7LvQO1W5UPQiBHed5OyZ1RcZZTtqAp4pqOqlWUlJCEQmncRx/Ik5CIK1ZCPAbEHoBqoxaDJzVCneWuzz60UWztBP/KI93RMJiLyYbIfWu826em6WobkZN1f9cnJusfp6Z4umX3cOUTd3PDs2Fhg5kygqEhXU6XVVv61TLEx0D0Vvxnr3/wdVzRB+PZsEsZ+3B11mtd2TZkADu8mIvIybCYDbO534hX0TUfWhqg7s/mqbI3a8ePAxx+7dKHV29duY9OC/Ti6dAem5KRaf0JlmwQdof+dAkzXmnnT7xQRkYewz5ATWXwz5dD3xdk8dSN2Yai8sPci1s47CvVPwdh4sRVuIxR+0OIUGiAW5+AHM7/G8fGe++zs7YxPRERGGIacyOKb6a4Ox+7m7huxk0OlKBXIWn4E6k8uYu1vkdh9y3gW8nj/c1A1P4FnEn9F2xVTIZn7NX75ZWDuXDsvxom8uTM+EZGHsQO1u7irw7G72Tq7sLNY6cwMIYCcHN1xZkLl7Wu3sfndA1B/fRtrjzbGudLmAJob9neuegCqrleg+ls0Wg9vAskvFkAvIP4K8H//Z/p1334b6NrVczUx3jR7MxGRQjEMyXRNLKdw543YwVCZu+8S1r5zFOqfgrAxNxG30MmwLxQF6B+9H0P6F2Pw5Kao2zqx4vm0Wt1QdktSU3XBkDUyRERkgrLCkH7RzLI1JTJZE8vr2RgWRVRd/LHiKNQfXYD61zrYVdASQKRhf5z/eaianYBqRBUkv9AKITW7Wj6hE2qkiIhI2ZQVhhITgfPn7z3Wj3Liwo6VZyVUCgDXAyLRvl8jnC6tB6CpYV+nqgeh6nIZqr9Fo82jTSD5xdj+ur7azElERG6jrHmGygYhQHfj1o+6MjVfTlycsoZA62vOli+3f34g/SSMQIX5dUohQUDCMyUf4nRpPVTBLTxU91d8PGYbzmddxG83W+LVTb3R9rGm9s8C7cvNnERE5BbKGk0GoEJf9LKjnADljvwxNfrMjvmBRKnA/vTjOPbvr9Bj/weIEhcN+84gHv+WZsC/WWOoRlTBAy8kokpEFeeU2xPzKhERkVtwaL0TWQxDet42dL6yyk+UOGNGxWOszA9UlFeEzPf2Q72iAOpDjXBGGwcA8IMWPbAN3YJ2oXHzQLSe0hftnmjpvLW/yuMEh0REPolD691NSX1KTNUCmSKELlCUGY116eBlfD//KNQ/BOCn8y1xEx0Nh4fgNvpG7oeqXyGGTG6GmPa9XXoZBvplQUzVbNk7r5Kz5wXiPENERLLHMKSnlD4l5maJNufuaKwvu76H94/0wc6bLSHQ3bA72i8XQ5ocg2p4CPq8mIjQ2p1dVHArnDGvUiWbCl1+PiIicgk2k/lKnxJbaiCszRJtwUik4SuMBAC0q3IYqk4XoXo6Eu1HNYNfgA398OVeQ+LspUR8ab07IiIPYZ8hJzIbhuR+Y7I1QNhaA2Hr0iMmTA37EA0GtcCQSY0R18nOWjS515A4e306X1zvjojIA9wVhpQ1tD6m3Pw1ch46n56uu6EmJwOjRun+26CBbnv544YPr3jj1U8bUPZ4B/pFCQClMXH4z9W/4u9pPe0LQlot8PrrwCOP2FY+T7Fn4kZPnI+IiFxKWWHowAHdqLG0NN1/s7PlG4RsCThara7GxVTlnn5baqphvqDimrXtK4ckQZIk+L230P4ajPR0oH5906PUzJTPY5w9cSMngiQi8irK6kDtDYtmWgs4ZUd32VgD8dPDH+CTrI7YcLYD9iEOsTgHP9jQOurIaCzA9k7aclkqw9kTN3rrRJBy79dFROQiyqoZ8gb2NLHYWLOwRF0b35xNwg1E4DW8DgnCehSqUwc4ccL+IGQpzJnj6RoS/VIikpl5kCQJiI+3fX06Z5/PHWxtliUi8kEMQ3JjTxOLjTULoYHFmN4tEzs/OYBPi8dCmjULVqc/vHwZ2L7dtrKUZS3MmeLpGhILS4k4tD6ds8/navb0OyMi8kEMQ67i6DpfNgaDvNJqSPvKD1f966DUTLQRAEoiY/Dp7Sfwxs+90eXpRN0w+MaNbSuLIzU29jxHTjUk+okbnbU+nbPP5yp29jsjIvJFyuoz5C6VGUpuw+rvN6RwTHzyOs6iOtLxAb7GCJRCMu4HJEmQAAR8+F7FGghX9mmx9zlyqiFxxsSNrjyfK9jTLCv3/nZERA5S1jxDLp6nAIBzJtu7ew4BQCpznlIA0t0fvRtBdVDQazBiDm2EdK7MTS0+3nznZ1cubmrt3HpymmdIyZYv1/URsiYtDRg50vXlISIqg/MMuZKjTVi2nLeSTQ7Xs29geUZdzK/9Bs4L41oWU41hNYuvIHbjZ5Dmz7N92gBX9mmxdG69WbOAU6cYhOTAW0e+ERE5kfJqhjZudN1syLbO8JyRYdTkcGx9NtSLTmPtz2HYdqMVtHdbL/2gxRCsxYN1fse4/PcQXHjDfMfnOnV01xQUZHt5TTXnWapRKsvaMOzKnJvcx5W1hERElcTlOJzI8GZ+8QVqjBnjuvWibGxy0C79Aj/fSIT6ixtQ76uPY8UJRvtbBJ+Aqu1ZqMZGoOvTLeG/fZttISssDBg3TtdPxda+KY7MLWNrnyjOW+Md9E27gPF3Q+7L1RCRz2MYciLDmxkTgxrnz5s+yBl/Ab/+uvkZl8tQ4TushcrwOBB30CtiP4b0ugnVCwlo2Lue8RNs7ddRlqv65HABUt/EmjwikiGGISeyuGp9eeWasGyWnq5bg8uCUkg4izgkIBvh0g0MSjgM1VB/9E9tgbB6Yeaf6MgCq64IJ1yA1LexJo+IZMZdYYhD68tzZG6dux2nBUx3cgZgGPSeed8z2DLpIJKeaQn/oO62nd/KcHvTL1hu6Q5n3NQ4DNu3ecNyNURELqDM0WSW2DlqRnNGg4zHPgTOnrU4q7MEwG/WTIw5/hq6P98a/kF2hJOyI7Ts4ezV0bkAKRER+SBlhaGYGKesF/Vn5hksTNmCvhF7ULt+KD5aWcu217d15mdT9DMa17Zz5XnAeeGEw7CJiMgHKauZ7K23gDFjdMHH1KgZM3PraO9osfPTg1B/dg3qvXE4VHQfgHudnAMDBFBiw+tXNiSkpADVqgEDBtj3PGeFE2vNdfo+Q3JYXoOIiMhGyqoZeughm9eLyjubh2+n7MDYRj+jbsh1dH++Nd76tTcOFd0Hf5Sgd80svPNQJo79dAqfFwy3XGPjzDW4rl61/VhHXtfShJTetgApERGRDZRVMwRYXC/q1M9noV5wEuqMasi81grFSDI8raZ0AwPrHYRKJeHByS0QntBOtyM9HWjUA7hyxfTrOTskHD9u3/H2vK4t8wfpm+tMHcdh2ERE5IWUNbS+3NA87R0tflt6COqlV6HeE4sDRcZ9ehoHZkPV+jRUT9ZEt7+1RGBooPGJzc25U5Yz52qx5fX07J1nyN75gzgMm4iIXIzzDDlR2TdTuinhp/kHoF6lxfd/NsVlUcdwnD9K0C3sAFTdb0A1vh6aDmxo/qTW5twBHFsiozKvpzdrFjB9uu3hhPMHERGRDHGhVjPmzJkDSZKQmppq93MfbrAHtWODMPztJHx2sjsuizoIgwaPxW/Hsud+waUT+dhyoy1eWtvbchACrM+5AwCXLwPbt9tdTodfD9AFoddesy+02DN/EBERkY/xqj5Du3btwkcffYTWrVs79PzN19sDCMZ9gaeganUKqifC0P3viQgMvd/+k7l7zh1bz+PI8H3OH0RERArmNTVDN2/exBNPPIGPP/4Y4eHhDp3j9X7bcHjdnzhWWB/zfu+N5MntKvYDspW759xx5etx/iAiIlIwrwlD48ePx+DBg9G3b1+rxxYVFSEvL8/oBwBe/LYHmg1qCMnP0lzRNtLPueOESRw9/nruvhYiIiIZ8Yow9NVXX2HPnj2YM2eOTcfPmTMHYWFhhp/4+HjrT7I0v44p7p5zx5Wvx/mDiIhIwWQfhnJycvDiiy9i2bJlCAkJsek506ZNg0ajMfzk5ORYfkJ6um40VXIyMGqU7r8NGui2W6Kfc8eGSRydwpWv5+5rISIikgnZD61fvXo1Hn74YfiXqZXQarWQJAl+fn4oKioy2meKxaF59s6vY4q759xx5es549ycg4iIiJyA8wzdlZ+fj9OnTxttGzduHJo1a4ZXXnkFiYmJVs9h9s3k/DrOZ8ss1kRERDZwVxiS/dD66tWrVwg8VatWRa1atWwKQhbZM79O796Vey0lMFfLdu6cbjub24iISIZk32fIpTi/jvNotboaIVMVjfptqanWO6YTERG5mexrhkzJzMx0zok4v47zsJaNiIi8lLJrhji/jvOwlo2IiLyUssMQ59dxHtayERGRl1J2GAI4v46zsJaNiIi8lFf2GXK6lBRg6FDOjVMZ+lq24cN1wadsR2rWshERkYwxDOn5+7Njb2Xpa9lMzTO0YAFr2YiISJYYhsi5WMtGRERehmGInI+1bERE5EXYgZqIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgUjWGIiIiIFI1hiIiIiBSNYYiIiIgULcDTBSAX0GqBbduACxeA6GigRw/A39/TpSIiIpIlhiFfk54OvPgicPbsvW1xccDChUBKiufKRUREJFNsJvMl6enA8OHGQQgAzp3TbU9P90y5iIiIZIxhyFdotboaISEq7tNvS03VHUdEREQGDEO+Ytu2ijVCZQkB5OTojiMiIiIDhiFfceGCc48jIiJSCIYhXxEd7dzjiIiIFIJhyFf06KEbNSZJpvdLEhAfrzuOiIiIDBiGfIW/v274PFAxEOkfL1jA+YaIiIjKYRjyJSkpwLffArGxxtvj4nTbOc8QERFRBZx00dekpABDh3IGaiIiIhsxDPkif3+gd29Pl4KIiMgrsJmMiIiIFI1hiIiIiBSNYYiIiIgUTfZhaM6cOejUqROqV6+OyMhIDBs2DEePHvV0sYiIiMhHyD4MbdmyBePHj8fOnTuxYcMGlJSUoH///igoKPB00YiIiMgHSEKYWuZcvi5fvozIyEhs2bIFPXv2tOk5eXl5CAsLg0ajQY0aNVxcQiIiInIGd92/vW5ovUajAQBERESYPaaoqAhFRUWGx3l5eS4vFxEREXkn2TeTlSWEwOTJk9G9e3ckJiaaPW7OnDkICwsz/MTHx7uxlERERORNvKqZbPz48Vi3bh1+/vlnxMXFmT3OVM1QfHw8m8mIiIi8CJvJypk4cSK+++47bN261WIQAoDg4GAEBwe7qWRERETkzWQfhoQQmDhxIlatWoXMzEwkJCR4ukhERETkQ2QfhsaPH4+0tDSsWbMG1atXR25uLgAgLCwMVapU8XDpiIiIyNvJvs+QJEkmty9ZsgRPPfWUTefg0HoiIiLvwz5Dd8k8qxEREZGX86qh9URERETOxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREiuY1YeiDDz5AQkICQkJC0KFDB2zbts3TRSIiIiIf4BVhaMWKFUhNTcX06dORlZWFHj16YODAgThz5oyni0ZEREReThJCCE8XwpouXbqgffv2+PDDDw3bmjdvjmHDhmHOnDlWn5+Xl4ewsDBoNBrUqFHDlUUlIiIiJ3HX/Vv2NUN37tzB77//jv79+xtt79+/P7Zv3+6hUhEREZGvCPB0Aay5cuUKtFotoqKijLZHRUUhNzfX5HOKiopQVFRkeKzRaADoEiYRERF5B/1929WNWLIPQ3qSJBk9FkJU2KY3Z84czJo1q8L2+Ph4l5SNiIiIXOfq1asICwtz2fllH4Zq164Nf3//CrVAly5dqlBbpDdt2jRMnjzZ8PjGjRuoX78+zpw549I3U27y8vIQHx+PnJwcRfWV4nXzupWA183rVgKNRoN69eohIiLCpa8j+zAUFBSEDh06YMOGDXj44YcN2zds2IChQ4eafE5wcDCCg4MrbA8LC1PUL5FejRo1eN0KwutWFl63sij1uv38XNvFWfZhCAAmT56M0aNHo2PHjkhKSsJHH32EM2fO4O9//7uni0ZERERezivC0GOPPYarV6/i9ddfx4ULF5CYmIjvv/8e9evX93TRiIiIyMt5RRgCgOeffx7PP/+8Q88NDg7GjBkzTDad+TJeN69bCXjdvG4l4HW79rq9YtJFIiIiIleR/aSLRERERK7EMERERESKxjBEREREisYwRERERIrmlWHogw8+QEJCAkJCQtChQwds27bN4vFbtmxBhw4dEBISgoYNG2Lx4sUVjlm5ciVatGiB4OBgtGjRAqtWrXJV8R1mz3Wnp6ejX79+qFOnDmrUqIGkpCSsX7/e6JilS5dCkqQKP4WFha6+FLvYc92ZmZkmr+nIkSNGx/na5/3UU0+ZvO6WLVsajvGGz3vr1q1QqVSIiYmBJElYvXq11ef4wvfb3uv2le+3vdftK99ve6/bV77fc+bMQadOnVC9enVERkZi2LBhOHr0qNXnueM77nVhaMWKFUhNTcX06dORlZWFHj16YODAgThz5ozJ47OzszFo0CD06NEDWVlZ+Oc//4kXXngBK1euNByzY8cOPPbYYxg9ejT++OMPjB49GiNGjMCvv/7qrsuyyt7r3rp1K/r164fvv/8ev//+O5KTk6FSqZCVlWV0XI0aNXDhwgWjn5CQEHdckk3svW69o0ePGl1T48aNDft88fNeuHCh0fXm5OQgIiICjz76qNFxcv+8CwoK0KZNGyxatMim433l+23vdfvK99ve69bz9u+3vdftK9/vLVu2YPz48di5cyc2bNiAkpIS9O/fHwUFBWaf47bvuPAynTt3Fn//+9+NtjVr1kxMnTrV5PH/+Mc/RLNmzYy2Pfvss6Jr166GxyNGjBAPPvig0TEDBgwQjz/+uJNKXXn2XrcpLVq0ELNmzTI8XrJkiQgLC3NWEV3C3uvOyMgQAMT169fNnlMJn/eqVauEJEni1KlThm3e8HmXBUCsWrXK4jG+8v0uy5brNsUbv99l2XLdvvL9LsuRz9sXvt9CCHHp0iUBQGzZssXsMe76jntVzdCdO3fw+++/o3///kbb+/fvj+3bt5t8zo4dOyocP2DAAOzevRvFxcUWjzF3Tndz5LrLKy0tRX5+foXF7m7evIn69esjLi4OQ4YMqfCXpSdV5rrbtWuH6Oho9OnTBxkZGUb7lPB5f/rpp+jbt2+FWdrl/Hk7whe+387gjd/vyvDm77cz+Mr3W6PRAIDFRVjd9R33qjB05coVaLXaCqvVR0VFVVjVXi83N9fk8SUlJbhy5YrFY8yd090cue7y3nnnHRQUFGDEiBGGbc2aNcPSpUvx3XffYfny5QgJCUG3bt1w/Phxp5bfUY5cd3R0ND766COsXLkS6enpaNq0Kfr06YOtW7cajvH1z/vChQv44Ycf8Mwzzxhtl/vn7Qhf+H47gzd+vx3hC9/vyvKV77cQApMnT0b37t2RmJho9jh3fce9ZjmOsiRJMnoshKiwzdrx5bfbe05PcLSMy5cvx8yZM7FmzRpERkYatnft2hVdu3Y1PO7WrRvat2+P9957D++++67zCl5J9lx306ZN0bRpU8PjpKQk5OTk4O2330bPnj0dOqenOFrGpUuXombNmhg2bJjRdm/5vO3lK99vR3n799sevvT9dpSvfL8nTJiAffv24eeff7Z6rDu+415VM1S7dm34+/tXSHuXLl2qkAr16tata/L4gIAA1KpVy+Ix5s7pbo5ct96KFSvw9NNP4+uvv0bfvn0tHuvn54dOnTrJ5i+Jylx3WV27djW6Jl/+vIUQ+N///ofRo0cjKCjI4rFy+7wd4Qvf78rw5u+3s3jb97syfOX7PXHiRHz33XfIyMhAXFycxWPd9R33qjAUFBSEDh06YMOGDUbbN2zYgPvvv9/kc5KSkioc/9NPP6Fjx44IDAy0eIy5c7qbI9cN6P5ifOqpp5CWlobBgwdbfR0hBPbu3Yvo6OhKl9kZHL3u8rKysoyuyVc/b0A3WuPEiRN4+umnrb6O3D5vR/jC99tR3v79dhZv+35Xhrd/v4UQmDBhAtLT07F582YkJCRYfY7bvuM2d7WWia+++koEBgaKTz/9VBw6dEikpqaKqlWrGnrVT506VYwePdpw/J9//ilCQ0PFpEmTxKFDh8Snn34qAgMDxbfffms45pdffhH+/v7iP//5jzh8+LD4z3/+IwICAsTOnTvdfn3m2HvdaWlpIiAgQLz//vviwoULhp8bN24Yjpk5c6b48ccfxcmTJ0VWVpYYN26cCAgIEL/++qvbr88ce697/vz5YtWqVeLYsWPiwIEDYurUqQKAWLlypeEYX/y89Z588knRpUsXk+f0hs87Pz9fZGVliaysLAFAzJs3T2RlZYnTp08LIXz3+23vdfvK99ve6/aV77e9163n7d/v5557ToSFhYnMzEyj39tbt24ZjvHUd9zrwpAQQrz//vuifv36IigoSLRv395oWN7YsWNFr169jI7PzMwU7dq1E0FBQaJBgwbiww8/rHDOb775RjRt2lQEBgaKZs2aGX255MKe6+7Vq5cAUOFn7NixhmNSU1NFvXr1RFBQkKhTp47o37+/2L59uxuvyDb2XPdbb70lGjVqJEJCQkR4eLjo3r27WLduXYVz+trnLYQQN27cEFWqVBEfffSRyfN5w+etHzpt7vfWV7/f9l63r3y/7b1uX/l+O/J77gvfb1PXDEAsWbLEcIynvuPS3QISERERKZJX9RkiIiIicjaGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISIiIlI0hiEiIiJSNIYhIiIiUjSGISLySsuXL0dISAjOnTtn2PbMM8+gdevW0Gg0HiwZEXkbrk1GRF5JCIG2bduiR48eWLRoEWbNmoVPPvkEO3fuRGxsrKeLR0ReJMDTBSAicoQkSZg9ezaGDx+OmJgYLFy4ENu2bWMQIiK7sWaIiLxa+/btcfDgQfz000/o1auXp4tDRF6IfYaIyGutX78eR44cgVarRVRUlKeLQ0ReijVDROSV9uzZg969e+P999/HV199hdDQUHzzzTeeLhYReSH2GSIir3Pq1CkMHjwYU6dOxejRo9GiRQt06tQJv//+Ozp06ODp4hGRl2HNEBF5lWvXrqFbt27o2bMn/vvf/xq2Dx06FEVFRfjxxx89WDoi8kYMQ0RERKRo7EBNREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESKxjBEREREisYwRERERIrGMERERESK9v9tH8Nw7VTzCAAAAABJRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"filenames": {
|
|
"image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_16_2.png"
|
|
}
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Using Autograd to calculate gradients for OLS\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import autograd.numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from autograd import grad\n",
|
|
"\n",
|
|
"def CostOLS(beta):\n",
|
|
" return (1.0/n)*np.sum((y-X @ beta)**2)\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"x = 2*np.random.rand(n,1)\n",
|
|
"y = 4+3*x+np.random.randn(n,1)\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(theta_linreg)\n",
|
|
"# Hessian matrix\n",
|
|
"H = (2.0/n)* XT_X\n",
|
|
"EigValues, EigVectors = np.linalg.eig(H)\n",
|
|
"print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
|
|
"\n",
|
|
"theta = np.random.randn(2,1)\n",
|
|
"eta = 1.0/np.max(EigValues)\n",
|
|
"Niterations = 1000\n",
|
|
"# define the gradient\n",
|
|
"training_gradient = grad(CostOLS)\n",
|
|
"\n",
|
|
"for iter in range(Niterations):\n",
|
|
" gradients = training_gradient(theta)\n",
|
|
" theta -= eta*gradients\n",
|
|
"print(\"theta from own gd\")\n",
|
|
"print(theta)\n",
|
|
"\n",
|
|
"xnew = np.array([[0],[2]])\n",
|
|
"Xnew = np.c_[np.ones((2,1)), xnew]\n",
|
|
"ypredict = Xnew.dot(theta)\n",
|
|
"ypredict2 = Xnew.dot(theta_linreg)\n",
|
|
"\n",
|
|
"plt.plot(xnew, ypredict, \"r-\")\n",
|
|
"plt.plot(xnew, ypredict2, \"b-\")\n",
|
|
"plt.plot(x, y ,'ro')\n",
|
|
"plt.axis([0,2.0,0, 15.0])\n",
|
|
"plt.xlabel(r'$x$')\n",
|
|
"plt.ylabel(r'$y$')\n",
|
|
"plt.title(r'Random numbers ')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6502c1ea",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Same code but now with momentum gradient descent"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "fa89613b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[4.]\n",
|
|
" [3.]]\n",
|
|
"Eigenvalues of Hessian Matrix:[0.28837459 4.42584543]\n",
|
|
"0 [-15.52856208] [-17.65196615]\n",
|
|
"1 [-0.3842967] [0.3228044]\n",
|
|
"2 [-0.3592571] [0.30177145]\n",
|
|
"3 [-0.335849] [0.28210895]\n",
|
|
"4 [-0.3139661] [0.26372759]\n",
|
|
"5 [-0.29350903] [0.2465439]\n",
|
|
"6 [-0.27438488] [0.23047985]\n",
|
|
"7 [-0.25650679] [0.21546249]\n",
|
|
"8 [-0.23979359] [0.20142361]\n",
|
|
"9 [-0.22416937] [0.18829946]\n",
|
|
"10 [-0.20956318] [0.17603044]\n",
|
|
"11 [-0.19590868] [0.16456084]\n",
|
|
"12 [-0.18314387] [0.15383855]\n",
|
|
"13 [-0.17121077] [0.1438149]\n",
|
|
"14 [-0.1600552] [0.13444436]\n",
|
|
"15 [-0.14962649] [0.12568438]\n",
|
|
"16 [-0.13987729] [0.11749517]\n",
|
|
"17 [-0.13076331] [0.10983954]\n",
|
|
"18 [-0.12224317] [0.10268273]\n",
|
|
"19 [-0.11427818] [0.09599224]\n",
|
|
"20 [-0.10683216] [0.08973767]\n",
|
|
"21 [-0.0998713] [0.08389064]\n",
|
|
"22 [-0.09336399] [0.07842458]\n",
|
|
"23 [-0.08728068] [0.07331468]\n",
|
|
"24 [-0.08159374] [0.06853772]\n",
|
|
"25 [-0.07627734] [0.06407201]\n",
|
|
"26 [-0.07130734] [0.05989727]\n",
|
|
"27 [-0.06666117] [0.05599455]\n",
|
|
"28 [-0.06231773] [0.05234611]\n",
|
|
"29 [-0.0582573] [0.0489354]\n",
|
|
"theta from own gd\n",
|
|
"[[3.81114345]\n",
|
|
" [3.15863713]]\n",
|
|
"0 [-0.05446143] [0.04574692]\n",
|
|
"1 [-0.05091289] [0.04276619]\n",
|
|
"2 [-0.046531] [0.03908546]\n",
|
|
"3 [-0.04218461] [0.03543455]\n",
|
|
"4 [-0.03813208] [0.03203047]\n",
|
|
"5 [-0.03443175] [0.02892224]\n",
|
|
"6 [-0.03107818] [0.02610528]\n",
|
|
"7 [-0.02804715] [0.02355925]\n",
|
|
"8 [-0.02531037] [0.0212604]\n",
|
|
"9 [-0.02284019] [0.01918548]\n",
|
|
"10 [-0.02061094] [0.01731293]\n",
|
|
"11 [-0.01859922] [0.01562311]\n",
|
|
"12 [-0.01678384] [0.01409821]\n",
|
|
"13 [-0.01514564] [0.01272215]\n",
|
|
"14 [-0.01366733] [0.01148039]\n",
|
|
"15 [-0.01233332] [0.01035984]\n",
|
|
"16 [-0.01112952] [0.00934865]\n",
|
|
"17 [-0.01004321] [0.00843617]\n",
|
|
"18 [-0.00906293] [0.00761275]\n",
|
|
"19 [-0.00817834] [0.0068697]\n",
|
|
"20 [-0.00738008] [0.00619918]\n",
|
|
"21 [-0.00665974] [0.0055941]\n",
|
|
"22 [-0.00600971] [0.00504808]\n",
|
|
"23 [-0.00542313] [0.00455536]\n",
|
|
"24 [-0.0048938] [0.00411073]\n",
|
|
"25 [-0.00441613] [0.0037095]\n",
|
|
"26 [-0.00398509] [0.00334743]\n",
|
|
"27 [-0.00359612] [0.0030207]\n",
|
|
"28 [-0.00324512] [0.00272586]\n",
|
|
"29 [-0.00292838] [0.0024598]\n",
|
|
"theta from own gd wth momentum\n",
|
|
"[[3.99083639]\n",
|
|
" [3.00769731]]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Using Autograd to calculate gradients for OLS\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import autograd.numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from autograd import grad\n",
|
|
"\n",
|
|
"def CostOLS(beta):\n",
|
|
" return (1.0/n)*np.sum((y-X @ beta)**2)\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"x = 2*np.random.rand(n,1)\n",
|
|
"y = 4+3*x#+np.random.randn(n,1)\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(theta_linreg)\n",
|
|
"# Hessian matrix\n",
|
|
"H = (2.0/n)* XT_X\n",
|
|
"EigValues, EigVectors = np.linalg.eig(H)\n",
|
|
"print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
|
|
"\n",
|
|
"theta = np.random.randn(2,1)\n",
|
|
"eta = 1.0/np.max(EigValues)\n",
|
|
"Niterations = 30\n",
|
|
"\n",
|
|
"# define the gradient\n",
|
|
"training_gradient = grad(CostOLS)\n",
|
|
"\n",
|
|
"for iter in range(Niterations):\n",
|
|
" gradients = training_gradient(theta)\n",
|
|
" theta -= eta*gradients\n",
|
|
" print(iter,gradients[0],gradients[1])\n",
|
|
"print(\"theta from own gd\")\n",
|
|
"print(theta)\n",
|
|
"\n",
|
|
"# Now improve with momentum gradient descent\n",
|
|
"change = 0.0\n",
|
|
"delta_momentum = 0.3\n",
|
|
"for iter in range(Niterations):\n",
|
|
" # calculate gradient\n",
|
|
" gradients = training_gradient(theta)\n",
|
|
" # calculate update\n",
|
|
" new_change = eta*gradients+delta_momentum*change\n",
|
|
" # take a step\n",
|
|
" theta -= new_change\n",
|
|
" # save the change\n",
|
|
" change = new_change\n",
|
|
" print(iter,gradients[0],gradients[1])\n",
|
|
"print(\"theta from own gd wth momentum\")\n",
|
|
"print(theta)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ac5271c6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## But noen of these can compete with Newton's method"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "0def25cd",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[3.83190841]\n",
|
|
" [2.9998542 ]]\n",
|
|
"Eigenvalues of Hessian Matrix:[0.27971414 4.455592 ]\n",
|
|
"0 [-12.5786304] [-15.41021561]\n",
|
|
"1 [-2.79550688e-15] [-2.24968001e-16]\n",
|
|
"2 [-2.2941718e-16] [-4.20820528e-16]\n",
|
|
"3 [4.95697233e-16] [5.72742343e-16]\n",
|
|
"4 [4.95697233e-16] [5.72742343e-16]\n",
|
|
"beta from own Newton code\n",
|
|
"[[3.83190841]\n",
|
|
" [2.9998542 ]]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Using Newton's method\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import autograd.numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from autograd import grad\n",
|
|
"\n",
|
|
"def CostOLS(beta):\n",
|
|
" return (1.0/n)*np.sum((y-X @ beta)**2)\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"x = 2*np.random.rand(n,1)\n",
|
|
"y = 4+3*x+np.random.randn(n,1)\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"beta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(beta_linreg)\n",
|
|
"# Hessian matrix\n",
|
|
"H = (2.0/n)* XT_X\n",
|
|
"# Note that here the Hessian does not depend on the parameters beta\n",
|
|
"invH = np.linalg.pinv(H)\n",
|
|
"EigValues, EigVectors = np.linalg.eig(H)\n",
|
|
"print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
|
|
"\n",
|
|
"beta = np.random.randn(2,1)\n",
|
|
"Niterations = 5\n",
|
|
"\n",
|
|
"# define the gradient\n",
|
|
"training_gradient = grad(CostOLS)\n",
|
|
"\n",
|
|
"for iter in range(Niterations):\n",
|
|
" gradients = training_gradient(beta)\n",
|
|
" beta -= invH @ gradients\n",
|
|
" print(iter,gradients[0],gradients[1])\n",
|
|
"print(\"beta from own Newton code\")\n",
|
|
"print(beta)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4a8fec1d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Including Stochastic Gradient Descent with Autograd\n",
|
|
"In this code we include the stochastic gradient descent approach discussed above. Note here that we specify which argument we are taking the derivative with respect to when using **autograd**."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "00658ea8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[3.93988393]\n",
|
|
" [3.06578047]]\n",
|
|
"Eigenvalues of Hessian Matrix:[0.34100913 4.27547557]\n",
|
|
"theta from own gd\n",
|
|
"[[3.93988393]\n",
|
|
" [3.06578047]]\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAkMAAAHFCAYAAADxOP3DAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABV9klEQVR4nO3deXwT1fo/8M+0pS1bCwW6F1qRvWwCAkKhKIuyBKyIwhVxu6IsCkUvcL0KqMjFnyAqLhflwr36ZblKhQBuKC2LLIKA7AhYoCyllKWFQrf0/P6ICU2bNEsnmUnm8369+tLMTCZnkoZ5es5zniMJIQSIiIiINMpP6QYQERERKYnBEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1jMERERESaxmCIiIiINI3BEJEKLV26FJIkmX8CAgIQFRWFRx99FMePH1esXTNnzoQkSYq9vjc4deoUJEnCO++8o3RTiMhBAUo3gIhsW7JkCVq2bInCwkL8/PPPmD17NtLT03H06FHUr19f6eYREfkEBkNEKpaYmIjOnTsDAJKTk2EwGDBjxgysXr0aTz75pMKtIyWVlJSYew2JqHo4TEbkRUyB0cWLF83bCgsLMWXKFHTo0AGhoaEICwtD9+7dsWbNmkrPlyQJEyZMwOeff45WrVqhVq1aaN++PdatW1fp2PXr16NDhw4ICgpCQkKCzWGfwsJCTJ8+HQkJCQgMDERMTAzGjx+Pa9euWRwXHx+PwYMHY926dejYsSNq1qyJVq1amV976dKlaNWqFWrXro27774bu3fvtvt+mIYT09PT8fzzz6Nhw4Zo0KABUlJScP78+UrXPnPmzErniI+PxxNPPFHpnBs3bsRf//pXNGjQACEhIXj88cdRUFCA7OxsjBgxAvXq1UNUVBReeukllJSUVDpvWVkZZs+ejcaNGyM4OBidO3fGTz/9VOm448ePY9SoUQgPD0dQUBBatWqFDz/80OKYjIwMSJKEzz//HFOmTEFMTAyCgoJw4sQJ3Lx5Ey+99BISEhIQHByMsLAwdO7cGcuXL7f7/hGREf+kIPIimZmZAIDmzZubtxUVFeHKlSt46aWXEBMTg+LiYvz4449ISUnBkiVL8Pjjj1ucY/369di1axdef/111KlTB2+//TYefPBBHDt2DHfccQcA4KeffsLQoUPRvXt3rFixAgaDAW+//bZFEAYAQggMGzYMP/30E6ZPn46kpCTs378fM2bMwPbt27F9+3YEBQWZj//tt98wffp0vPLKKwgNDcWsWbOQkpKC6dOn46effsJbb70FSZIwdepUDB48GJmZmahZs6bd9+WZZ57BoEGDsGzZMmRlZeHll1/GY489ho0bN7r8Xj/zzDNISUnBihUrsHfvXvz9739HaWkpjh07hpSUFDz77LP48ccfMXfuXERHRyM1NdXi+QsXLkSTJk2wYMEClJWV4e2338YDDzyATZs2oXv37gCAw4cP45577kHjxo0xb948REZG4vvvv8cLL7yA3NxczJgxw+Kc06dPR/fu3fHJJ5/Az88P4eHhSE1Nxeeff44333wTHTt2REFBAQ4ePIjLly+7fO1EmiOISHWWLFkiAIgdO3aIkpIScf36dfHdd9+JyMhI0atXL1FSUmLzuaWlpaKkpEQ8/fTTomPHjhb7AIiIiAiRn59v3padnS38/PzEnDlzzNu6du0qoqOjxa1bt8zb8vPzRVhYmCj/z8Z3330nAIi3337b4nVWrlwpAIhFixaZtzVp0kTUrFlTnD171rxt3759AoCIiooSBQUF5u2rV68WAIRer3fofRo3bpzF9rffflsAEBcuXLC49hkzZlQ6R5MmTcSYMWMqnXPixIkWxw0bNkwAEPPnz7fY3qFDB3HXXXeZH2dmZgoANt+/vn37mrcNGDBAxMbGiry8PItzTpgwQQQHB4srV64IIYRIT08XAESvXr0qtT8xMVEMGzas0nYichyHyYhUrFu3bqhRowbq1q2L+++/H/Xr18eaNWsq5Yl8+eWX6NGjB+rUqYOAgADUqFEDixcvxpEjRyqds0+fPqhbt675cUREBMLDw3H69GkAQEFBAXbt2oWUlBQEBwebj6tbty6GDBlicS5Tz0v5YSYAePjhh1G7du1Kw0IdOnRATEyM+XGrVq0AGPOhatWqVWm7qU326HQ6i8ft2rVz6vnWDB482OKxqU2DBg2qtN3a69h6/zZv3gyDwYDCwkL89NNPePDBB1GrVi2UlpaafwYOHIjCwkLs2LHD4pwPPfRQpde5++678e2332LatGnIyMjArVu3XL5mIq1iMESkYv/973+xa9cubNy4EWPHjsWRI0cwcuRIi2PS0tIwYsQIxMTE4IsvvsD27duxa9cuPPXUUygsLKx0zgYNGlTaFhQUZL6JXr16FWVlZYiMjKx0XMVtly9fRkBAABo1amSxXZIkREZGVhqqCQsLs3gcGBhY5XZr7bem4jWZhuaqExg401Zr7bT1/hUXF+PGjRu4fPkySktL8cEHH6BGjRoWPwMHDgQA5ObmWjw/Kiqq0jnff/99TJ06FatXr0afPn0QFhaGYcOGKVqCgcjbMGeISMVatWplTpru06cPDAYDPvvsM3z11VcYPnw4AOCLL75AQkICVq5caVEDqKioyKXXrF+/PiRJQnZ2dqV9Fbc1aNAApaWluHTpkkVAJIRAdnY2unTp4lIb3CEoKMjqe+Ku3Bpb719gYCDq1KmDGjVqwN/fH6NHj8b48eOtniMhIcHisbUaT7Vr18asWbMwa9YsXLx40dxLNGTIEBw9elSeiyHycewZIvIib7/9NurXr4/XXnsNZWVlAIw3yMDAQIsbZXZ2ttXZZI4wzeZKS0uz6PG4fv061q5da3HsfffdB8AYkJW3atUqFBQUmPerQXx8PPbv32+xbePGjbhx44ZbXs/W+5eUlAR/f3/UqlULffr0wd69e9GuXTt07ty50o+1XryqRERE4IknnsDIkSNx7Ngx3Lx5U+7LIvJJ7Bki8iL169fH9OnT8be//Q3Lli3DY489hsGDByMtLQ3jxo3D8OHDkZWVhTfeeANRUVEuD5W88cYbuP/++9GvXz9MmTIFBoMBc+fORe3atXHlyhXzcf369cOAAQMwdepU5Ofno0ePHubZZB07dsTo0aPluvRqGz16NF599VW89tpr6N27Nw4fPoyFCxciNDTULa/n7++Pfv36ITU1FWVlZZg7dy7y8/Mxa9Ys8zHvvfceevbsiaSkJDz//POIj4/H9evXceLECaxdu9ah2XBdu3bF4MGD0a5dO9SvXx9HjhzB559/ju7du1vkYRGRbQyGiLzMxIkTsXDhQrz++usYOXIknnzySeTk5OCTTz7Bv//9b9xxxx2YNm0azp49a3HjdUa/fv2wevVq/OMf/8AjjzyCyMhIjBs3Drdu3bI4pyRJWL16NWbOnIklS5Zg9uzZaNiwIUaPHo233nrLYlq90l5++WXk5+dj6dKleOedd3D33Xfjf//7H4YOHeqW15swYQIKCwvxwgsvICcnB23atMH69evRo0cP8zGtW7fGnj178MYbb+Af//gHcnJyUK9ePTRr1sycN2TPvffeC71ej3fffRc3b95ETEwMHn/8cbzyyituuS4iXyQJIYTSjSAiIiJSCnOGiIiISNMYDBEREZGmMRgiIiIiTWMwRERERJrGYIiIiIg0jcEQERERaZom6gyVlZXh/PnzqFu3rtVy9kRERKQ+Qghcv34d0dHR8PNzX/+NJoKh8+fPIy4uTulmEBERkQuysrIQGxvrtvNrIhiqW7cuAOObGRISonBriIiIyBH5+fmIi4sz38fdRRPBkGloLCQkhMEQERGRl3F3igsTqImIiEjTGAwRERGRpjEYIiIiIk1jMERERESaxmCIiIiINI3BEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1jMERERESaxmCIiIiINI3BEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTFA+GNm/ejCFDhiA6OhqSJGH16tU2jx07diwkScKCBQs81j4iIiLybYoHQwUFBWjfvj0WLlxY5XGrV6/Gzp07ER0d7aGWERERkRYEKN2ABx54AA888ECVx5w7dw4TJkzA999/j0GDBnmoZURERKQFivcM2VNWVobRo0fj5ZdfRps2bZRuDhEREfkYxXuG7Jk7dy4CAgLwwgsvOPycoqIiFBUVmR/n5+e7o2lERETkA1TdM/Trr7/ivffew9KlSyFJksPPmzNnDkJDQ80/cXFxbmwlEREReTNVB0NbtmxBTk4OGjdujICAAAQEBOD06dOYMmUK4uPjbT5v+vTpyMvLM/9kZWV5rtFERETkVVQ9TDZ69Gj07dvXYtuAAQMwevRoPPnkkzafFxQUhKCgIHc3j4iIiHyA4sHQjRs3cOLECfPjzMxM7Nu3D2FhYWjcuDEaNGhgcXyNGjUQGRmJFi1aeLqpRERE5IMUD4Z2796NPn36mB+npqYCAMaMGYOlS5cq1CoiIiLSCsWDoeTkZAghHD7+1KlT7msMERERaY6qE6iJiIiI3I3BEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1jMERERESaxmCIiIiINI3BEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1jMERERESaxmCIiIiINI3BEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1jMERERESaxmCIiIiINI3BEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1jMERERESaxmCIiIiINI3BEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1TPBjavHkzhgwZgujoaEiShNWrV5v3lZSUYOrUqWjbti1q166N6OhoPP744zh//rxyDSYiIiKfongwVFBQgPbt22PhwoWV9t28eRN79uzBq6++ij179iAtLQ2///47dDqdAi0lIiIiXyQJIYTSjTCRJAlff/01hg0bZvOYXbt24e6778bp06fRuHFjh86bn5+P0NBQ5OXlISQkRKbWEhERkTt56v4d4LYzu0leXh4kSUK9evVsHlNUVISioiLz4/z8fA+0jIiIiLyR4sNkzigsLMS0adMwatSoKiPEOXPmIDQ01PwTFxfnwVYSERGRN/GaYKikpASPPvooysrK8NFHH1V57PTp05GXl2f+ycrK8lAriYiIvJDBAGRkAMuXG/9rMCjdIo/yimGykpISjBgxApmZmdi4caPdccOgoCAEBQV5qHVEREReLC0NePFF4OzZ29tiY4H33gNSUpRrlwepvmfIFAgdP34cP/74Ixo0aKB0k4iIiHxDWhowfLhlIAQA584Zt6elKdMuD1O8Z+jGjRs4ceKE+XFmZib27duHsLAwREdHY/jw4dizZw/WrVsHg8GA7OxsAEBYWBgCAwOVajYREZF3MxiMPULWJpULAUgSMGkSMHQo4O/v8eZ5kuJT6zMyMtCnT59K28eMGYOZM2ciISHB6vPS09ORnJzs0Gtwaj0REVEFGRmAlftvJenpgIP3W7lpZmp9cnIyqorHVFQGiYiIyHdcuCDvcV5M9TlDRERE5AZRUfIe58UU7xkiIiIiBSQlGWeNnTtnPW9Ikoz7k5JcO7/BAGzZYuxZiooynkeluUfsGSIiItIif3/j9HnAGPiUZ3q8YIFrAUxaGhAfb8xJGjXK+N/4eNXOTmMwREREpFUpKcBXXwExMZbbY2ON212pM+SF0/UVn03mCZxNRkREVAW5hrQMBmMPUMVAyMQ09JaZ6dD5NTObjIiIiBTm7y/P9PktW2wHQoAxNykry3icQtP1reEwGREREcnDS6frMxgiIiIieXjpdH0GQ0RERCQP03T9irPTTCQJiItzfbq+mzAYIiIiInm4c7q+GzEYIiIiIvm4Y7q+m3E2GREREckrJcW42r2XVKBmMERERETyk2u6vgdwmIyIiIg0jcEQERERaRqDISIiItI0BkNERESkaQyGiIiISNM4m4yIiIiUYTCoYvo9gyEiIiLyvLQ04MUXLVe5j401VrD2cGFGDpMRERGRZ6WlAcOHWwZCAHDunHF7WppHm8NgiIiIiORhMAAZGcDy5cb/GgzWj3nxRUCIyvtM2yZNsv5cN2EwRERERNWXlgbExwN9+gCjRhn/Gx9fuZdny5bKPULlCQFkZRmP8xAGQ0RERFQ9zgx7Xbjg2DkdPU4GDIaIiIjIdc4Oe0VFOXZeR4+TAYMhIiIicp2zw15JScZZY5Jk/XhJAuLijMd5CIMhIiIicp2zw17+/sbp80DlgMj0eMECj9YbYjBERERErnNl2CslBfjqKyAmxvKY2Fjjdg/XGZKEsDbI51vy8/MRGhqKvLw8hISEKN0cIiIi32EwGGeNnTtnPW9IkoxBTmZm5d4eOxWoPXX/ZgVqIiIicp1p2Gv4cGPgUz4gMg17zZtnPejx9weSk62f1xQoeQCHyYiIiKh6qhr2euklIDXVfv2h8kw1iwYPdmerzThMRkRERPKoOOyVmwuMGFF5+MzUY2QtP8hUs0gI5AMIBdx+/2YwRERERPIz5RLZmnZvLZeownM8FQxxmIyIiIgqc2Sdsaq4suyGvee4CROoiYiIyFJamrGqdPnAJDbWmCjt6LR3V5bd8OASHOWxZ4iIiIhuc2adsaq4Un/Ig0twlMecISIiIi0rn/QcHg488YRzeT5VndfZ+kMVnsOcISIiInIv0xR207T3vn2dz/OxxZVlN6p6jhspHgxt3rwZQ4YMQXR0NCRJwurVqy32CyEwc+ZMREdHo2bNmkhOTsahQ4eUaSwREblfdRN3yTG2hsMc4WhujyvLbth6jhspHgwVFBSgffv2WLhwodX9b7/9NubPn4+FCxdi165diIyMRL9+/XD9+nUPt5SIiNyuYk+FIwX6fIUng0CDwZgg7WqmjDO5PSkpwKlTQHo6sGyZ8b+ZmVUnYqekIPf7X7Gh31zX2uckVeUMSZKEr7/+GsOGDQNg7BWKjo7GpEmTMHXqVABAUVERIiIiMHfuXIwdO9ah8zJniIjIC5QrtmehqgJ9vkKO2VvOyMgwBprOciZnyAXHvv0D+oVnoN9aH9vyE1GGAgDuv38r3jNUlczMTGRnZ6N///7mbUFBQejduze2bdtm83lFRUXIz8+3+CEiIhWrqqfCtG3SJN8cMpNr9pYzXJnCbivPpxpKC0uxZeFveLlLBloEZqLlwDvwt2+SsTW/Pcrgj8Sg47K8jj2qDoays7MBABERERbbIyIizPusmTNnDkJDQ80/cXFxbm0nERFVkysF+nyBUkGgK1PYq8rzccL189ex6uXtGNN0KyJr5aHXxPZ4Z3cyfi9JQA0Uo3+D3Vj48Cac2noWP+c0q9ZrOcorii5KFTLKhRCVtpU3ffp0pKammh/n5+czICIiUjNXCvT5AmeCQFuru7siKckY3FQ17T0mBli6FMjJsVxp3gVnd13A2vnHof+xJjbmtkMxupv31ZeuYlD8IeiG+WNAahuExHY27/PUyI6qg6HIyEgAxh6iqHJRbE5OTqXeovKCgoIQFBTk9vYREZFMXCnQ5wuUCgJNU9iHDzcGPuUDIlNnw3vvAffd59LpRZnAvpXHoF+UDf3OCOy51QrA7c+uacBpDG2fCd3oeugxNhEBwT2rcTHVp+pgKCEhAZGRkdiwYQM6duwIACguLsamTZswd65nMsyJiMgDHOmpiI01HucNKq7ebqtXRckg0DSF3Vri9oIFTg+HFeUXIeODA9CvKID+SDOcNbQE0BIAIKEM3esehK7HFejGx6HlwDsg+TWR71qqSfFg6MaNGzhx4oT5cWZmJvbt24ewsDA0btwYkyZNwltvvYVmzZqhWbNmeOutt1CrVi2MGjVKwVYTEZGsHOmpkDFx162cmRmmdBCYkgIMHepY4GbF5eNX8M28w9Cv88d35xJxA7eHuGqhAP2jDkB3fwkGTWmJ8Dbt3HMNchAKS09PFwAq/YwZM0YIIURZWZmYMWOGiIyMFEFBQaJXr17iwIEDTr1GXl6eACDy8vLccAVERCSbVauEiI0VwhgaGH/i4ozbvcGqVUJIkmX7AeM2SbJ+HabnVHxeVc9R0O8/ZIp3BqeLXqF7hR9KLZoc5XdBPNtyk1g34xdx8/LNar+Wp+7fqqoz5C6sM0RE5EUcHWJSG9O6Wo6u61X+Oo8fBxYtMvYQmcTFuTRcJTdDsQE7Fh+CfukV6PfF4WhxU4v97YKPQdf5AnTPhKPTX1rCL0C+ieqeun8rPkxGRERkwd9f3plTnuLMzLArV6wPpc2aBTRrpngQeCP7Bn6YfxBrvy7BupOtkCtuD3EFoATJYfuh63MDQyY1RXzPFgBaKNJOuTAYIiIikoOjM77WrDHmD1UcmDl3Dpg505jUrEAweG53uenvl9qiCN3M++pJ1zCw8SHohkq4f0obhDbu5PH2uRODISIiIjk4OuPr//7PdpFFSTIWWRw61O29QqJMYP9Xv0P/rwvQ7wjH7putUX76+x0Bp6Frewq6x0LQ87lE1KjVw63tURKDISIiIjk4MjOsYUPg0iXb53BXkcU/Fd8oNk5/X34Daw83xRnD7SEuCWXoWucQdPdcxtDxsWg1uKmqpr+7E4MhIiIiOThSHuAvfzEmRdsjY5HFKyev4pt3DkO/zg8/nG2JDriOKFxAAgQuIxT3RR6GbkAxBqW2QGS7trK9rjdhMERERCQXe4UMw8IcC4aqWWTxxE+noX8/E/pN9bA1LxEG9MCDSMMBjEAcbrerLDoGfh+8r/iMNaVxaj0REZHcbJUHME2/t1dk0TT93tGXKzZg55LD0C+5DP3eOBypMP19Uo2FmF8y0fgSFV8PkGUBVnfw1P2bwRAREZEnpaUZh9KAygGRJDkcmBTkFGDDuwehX1WMdSda4pJoZN4XgBL0rn8Auj7XMWRCEyQ8nuR4/SMVYZ0hIiIiX2QaSnv2WeDyZct9YWFVPvX8nmysm/879BuC8WNOOxShq3lfKPIwsMlB6HTG6e/1mtxl3JGR4Xj9I2+s7yQDBkNEROQdlK5MLffrVwyEAGMxxuHDzb1DokzgQNpx6D85D/2ORthV0AZApPnw+IAsDE38A7rHQpD0vI3p744mY8uYtO1tGAwREZH6ObP4qdpf32AwnssaISAkCUVPjcPUmfWw5nBznDY0B9DcfEjX2gehuycXuudj0GbonZD84qp+PUeTsauZtC2b8kGnh1JbmDNERETqZsqxsZZfA7g/+Vfu18/IAPr0sXtYMtKxCckIxi30i9hvnP4+uTmiOkQ4/lqA25K23aJC0JkPIBRw+/1bvtXUiIiI5GbqRbFVsRkwVmw2GLzn9R0cjhoVmY41f9+Jy5cE9Nld8cx/kpwPhIDb9Y+A2wGcienxggXqCISGD686v8lNGAwREZF6Obr46cyZxh4XuYMiZxZftaOstAw7PjuIT9+86NBLP7u8D3Szu6JWw1qOttY2U9J2TIzl9thYdUyrryro9ADmDBERkXo5mtT75pvGH7nziKqTfGwwoHDdBhxasgsbfw3F/LMjkI1E+KEV7sc8xOAc/FDFsFVSUvXaXlFKinHNMyWT0G2xF3S6GYMhIiJSL2eTes+ds5iN5bHXL3dc9v4cHHxhETps+QANy3LQCUAnAI/i/2E63kJpXFNktn4esT/8A4CNZTvcNWzl76/O6fMKz2TjMBkREamXafHTirkutsidR2Tv9SUJIi4OB3IiMLtfBrrWOYhx7bfi3k2vIawsx+LQWJzF59IYrFiQjV7f/R2SmoetPE3hmWycTUZEROpWVcXmqqSny9MLYuP1Tf/3vN+/8K+yZwEAfjDgFOIRg7PWexsqztxSunaSWtiY8cbZZERERIDt5F975Bp6+fP1yyItey+yEIeHsAr/KnsWwbiFweG/YHXfhYizFQgBlROuTcNWI0ca/6vFQAioesabBzAYIiIi9UtJAU6dMvb2/OMfjj3H2tCLwWCcdbZ8uUOzzzI3Z+G9lE2475l41LxwEslIx0gsQzLScTd+Qf1mjbB6+k7kXizD2ot3Y8hT4Y61TcPVnm1yNeiVAYfJiIjIu9grIggAjRoZZycFBt7e5kAV6bLSMuz6z2Ho/50L/a8xOFjUzOK0rYNOQNfxLHRPNsDdT7SGf2CFnhwHCyrKNoTni8oNHeaHhCB08GCuWi8HBkNERD7GkTyi8oFOFVWkBYDdD83BogPdsfZ4S1wsu927449SJNU7AF3vPAyZmIA772tSdbu8qdqzF/DU/ZvBEBEReSdrPT3lmXJPVq4EUlNtHlcGCWcRiwRkogz+qIt8PBB3ELrBAg9MaY2wpvWdb5e1QM1Ty4f4EAZDMmIwRETko4qLjTkmubnW90sS0LAhcOmS3VO9l/AuWj+bhN4T2iKwTqDd46tkLVCLizPWD2Ig5DBP3b9ZdJGIiLzXtm22AyHA2DPjQCAEAC/OjgBGdpKnXWqu9uwsDUz/ZzBEROqigX94SUZyzsqSu/CfWqs9O8OBpHNfwKn1RKQeaWnG5NM+fYBRo4z/jY83blcrJ6dqk8wcDGCKataDqKKKNOLi5F8LzNvZWkXetOSJmr+XTmIwRETq4I3/8Hpj8OYDykrL8MuSQ3g1KQMd7w9HFmJRBuuBjvgz0An676fGIyoGRO5eC8xbVbWKvNxLnqgAgyEiUp43/sPrjcGbF7t15RbWz9yFsa02IzYoB12faoM3tyZjX1FrTMa7kIDK679LkjEAWrDg9uKtXAvMMfZWka9YSdvLMWeIiJTnzD+8asjBsBe8SZIxeBs6lL0N1ZBz6BLWzzsK/Xc18MOFtriJLuZ9dXAd98cchG6wAQOn3AvpwFfWc1vKz95SIqnZW3PgHM3FWrNGHd/JamIwRETKc/QfXrUsYeBtwZuXEGUCR9adhP6js1i7LQzbrydC4HYeT6z/eehaHYfu0dpIntgWQSHdbz+5mYOBjieTmr05+djRZPIFC4zvs9qvxw4GQ0SkPEf/4ZV7to+rvC14U7HSwlJs/eQg9J9fg35/Ak6W3gngTvP+u2oega7rReiejUSHR1pA8ou2fTI1zd6yVfHaNIxacWhObT1ISUnGwK2qoB/wmV5QBkNEpDzTP7z2ljBQy2wfbwveVCb/bD6+m3cQ+tUC35xujauig3lfIIpwX6P9GHLfLQxJbYbYLq0AtFKsrS5xdhhVjT1IplXkH3qo6uN8pBeUwRARKc/0D+/w4cYbhbUlDNQ028fbgjcVOP3zWaxdcBL6jXWQcaUtSnCPeV8D6TIG33EEupQA9JvUBnWju1RxJi/gzDDqlSvO9SB5UkqKMWhbsMD+sV7eC8pgiIjUISXF+A+/vSRYoHpDCnIMR3hb8OZONt7PstIy7Fl2FPrPcqDfFYXfClsAiDU/rUXgH9C1PwPdE2Ho/kwb+Af2VO4a5OZoYHDuHDBtmroT8YcOdSwY8vZeUKEBeXl5AoDIy8tTuilEZE9pqRDp6UIsW2b8b2mp5f5Vq4SIjRXCeLsw/sTGGrfbU53nOnq+uDjXz6dWtj4TK9d/MyRcfBzzhoj2O2/xtvihVCSF7BP/b1C6OPrNSSWvxv3S0y1/J2z9vPuuY8elpyt3LaWlxs9Ykqy3TZKMv/MVv6cy8dT9mwu1EpH3sJWUam018Io9FpcuAY884thznSFn4qvakmgB2/ksI0cC77wDIYRFuUNT8cPh+Ao/oD/ujzkA3UADBk5phYYtGni27UoxGIzFN+0No86ZAzz2mP3zLVtmfL+VYvreAdZ7Qd04lMdV62XEYIjIB5huMLZyMUw3mMxMY+2Tijdwf3/bRRvLP1ep4EONSbQ2gs/yj6zVfRYACkMjIJ08ieAGtd3ZQvnJFZA6EkCEhRmrltuTnq58crK138+4uMpD2DLz2P3brf1OMigpKRGvvPKKiI+PF8HBwSIhIUHMmjVLGAwGh8/BYTIiH+Do0MOsWba79Ks7HGFvCM9Vq1ZZb7MkGX+UGHYzDY+48j6qYXjHFZ4eRlV4CMpp7vr9r4Kn7t+qD4befPNN0aBBA7Fu3TqRmZkpvvzyS1GnTh2xYMECh8/BYIjIByxb5tgNOCzM9Zv3smW2X1/uG6WJvaBDoRtiwZfrqxcI2Xs/1cZdAakjOXCm11BLIKwinrp/q35tsu3bt2Po0KEYNGgQ4uPjMXz4cPTv3x+7d+9WumlE5EmOzla5ckX+13DnOmQqWgMqa+d5fPToJtzfcDfGPpxb/RN6ywwjd66NZyoEOXKk8b8Vh9xMsyi5ZpqiVD+1vmfPnvjkk0/w+++/o3nz5vjtt9+wdetWLHBkqh8R+Q5HavvUr+9aMFRVXSB3r0Pm6WrW5XJiREQk9p5rBP3iXOh/icDeW60AGCs898YN11/D2+osKb28ihJrppEF1QdDU6dORV5eHlq2bAl/f38YDAbMnj0bI6vIrC8qKkJRUZH5cX5+vieaWjU1zhIh8iaO1PZ58UVgxgznzmuvLpC7b5SerGadlgYx8QVI588BMCY/N0Is9uM97EUyJJThnroHoet5BbrnYyDGxUKyFXya+EKdJTUsr6KmpUS0yK2DcDJYvny5iI2NFcuXLxf79+8X//3vf0VYWJhYunSpzefMmDFDwDihweJHsZwhd+UaEGlRVUmp9hJSASH8/W0ntFrjaK6Sq/kxHkiivXQ0V2y8701RBghDhfMbIIkyQPx035si5/Alyyfay2d5+WXfqLPkaHK+tyWE+wAmUP8pNjZWLFy40GLbG2+8IVq0aGHzOYWFhSIvL8/8k5WVpVwwpMZZIkTerqqkVHs38C+/dG5GjCdulG5Ioj36zUnx9sB00TNknwhAkTiD2EqBkEMBlyMzojw8w0h23jarS0MYDP0pLCxMfPTRRxbb3nrrLdGsWTOHz6HYbDKVzhIh8nlyVob21I2ymm0uuVUiNn+wT7zUOV00r/GHxWl6I716AZ0vBDz2cFaXKnnq/q36nKEhQ4Zg9uzZaNy4Mdq0aYO9e/di/vz5eOqpp5Rumn1KJ+URaZWcCameWofMhTZfP38dP7x7EPo0A9ZntsJl0d68rwaK0afBfujuLcCI1geBWQ60wVZOjBbyWZxZG498juqDoQ8++ACvvvoqxo0bh5ycHERHR2Ps2LF47bXXlG6afWpIyiPSKjlv4J66UTrQ5rO7LmDt/OPQ/1gTG3PboRjdzfvqS1cxKP4QdMP8MSC1DUJiOxt3ZAjHgiFvmQrvLpzVpVlcjsOdMjK8p9Q6eR/OUPQ8Bd5zUSawb+Ux6BdlQ78zAntutbLY3zTgNIa2z4RudD30GJuIgGArf+M6ulaWksuREFnhqfu36nuGvJojdVG8qRYHqYca17HSAg8NFxXlFyHjgwPQryiA/kgznDW0BNASACChDN3rHoSuxxXoxseh5cA7IPk1sd9uTwz1+Tr+AeKz2DPkbgqu9ks+ypmV28lrXD5+Bd/MOwz9On98dy4RN1DXvK8WCtA/6gB095dg0JSWCG/TyLUXUWixTZ/AP0AUocpV67OyshAXF+e2xriL4qvW8x8gkoszK7fzL1bVO77hFPTvn4J+Sz1szWuLMtz+zKL8sjGk+e/QPVIT976QiJphNeV5UfZuOI9/gChGlcFQ7dq1kZqaimnTpqF27dpua5TcFA+GAP4DRPJgHppXMxQbsGPxIeiXXoF+XxyOFje12N8u+Bh0nS9A90w4Ov2lJfwCVL98pHWu/nunxn8n+QeIolSZM7RhwwZMnjwZixcvxuzZs/Hkk0+6q12+RwtTU8n95JihqMYbjg+7kX0DG949CH1aCdadbIVc0c68LwAlSA7bD12fGxgyqSnie7YA0EK5xjrC3u+Pq8NJah2GYokUbXClONF//vMfERsbKzp06CDSvaA8uWJFF4nkVt1qyHItDeONRfg82Oazu86LT0ZtEgMb/SKCcMvi7a4nXRWjmmwVK174WVw7fc1tbXALe78/rlbcV3Olfncvx0JVUn0F6ps3b4pXX31V1KpVSwwbNkwcP35cznbJisEQ+YzqVEOW64ajtrX2HAly3NzmMkOZ2LfyqHj93nTRudahSm9xQsBpMaljhtg4b48oLiiW5TU9zt7vz5dfulZxX+2V+rlumaJUHwwVFBSILVu2iEmTJgk/Pz8RFBQkUlNTRX5+vpztkwWDIfIpriwbINcNR21/wTsS5LipzUXXi8QPc3aLCW0zRGP/LMtTwyC61dkv3uqfLg6uPi7KDGUyXbBCHPn9adTItaBB7cEG1y1TlCqDoY8//lg89dRTom3btiIgIEAEBQWJLl26iIkTJ4rFixeLe++9V8TGxopdu3a5q70uYTBEPsfZdazkuOGo7S94R4Icmdt8+cQV8flzW8WIuJ9FXeRZnKomCoQucof4bMxmceG3i26+eDex1cvm6O+PIz8Vh5O8YRiK65YpRpXBUGxsrBg+fLh45513xNatW0VhYWGlY2bPni3atGkjWwPlwGCIfJIzOTBy3HDU9Be8o0HOjz9Wu83Hfzwl5unSRe/QvcIfJRZPi/C7KJ5psUno/7FTFFwqcP91u1NVvWyO/v648l6r6feqKnIu/ksOU+VCrVlZWXaPefrpp/Hqq686mcZNRE5zZoaio2tOVXWcmtbac3SGz9Kljp2vXJsNxQbsXHIY+iWXod8bhyPFTQE0Me9vG/w7htx1HrqnGqLLmNbwC+jl2jWoia06OufOGbfPnOnYeRo2BC5frnwewHbFfW+p1M91y3ya7MtxhIeHY+PGjXKfloiqQ44bjhwBlVwcDbi++MKhw24F18P303dCv6oY6060xCXR1rwvACXoXf8AdH2uY8iLdyChV3MAzV1otEoZDMYp7dZ+L4Qw/m58+qljvz/z5gGPPOLckh/etFQIS6T4Lrf2O6kEh8mIRPXzHtSUSCpTDksZIHL8IkRN3LDYFYprYmSTrWL5xJ/F1VPX5G+/mkoTOPpezprl2O+Pq8NJHIYiKzx1/+baZERaUt2lYdSy1p69VdgdUAZjm4fjK3yNFMQHZGFo4h/QPRaCpOcTUaNWDRkbXI7aigsuXw6MGmX/uGXLgKAgx35/fKkCNSlKlctxeCsGQ0TlVPeGo5a19mwFZg46gzh8FJSKkF4doHs+Bm2G3gnJT5K5kRWkpQEPPWR7/6pVng+InF3ihQELeRCDIRkxGCKSmVpuiNYCMwfsafs4opbMQVSnaDc1zAqDAYiIMCYY29KgAXDxomffS3u9bFx7ixTkqfu3l64CSESKMiWSjhxp/K9SN8mUFODUKZx/dwW+7fwqFgZPcehpd73/pGcDIcDYA1NVIAQY92dkeKI1t5kSmIHbw50maktgJnITBkNE5HXKSsuw47OD+Ps9GUis/QdiJj+Cgbtfx4uFc5GFWHM+UCWSZBzSU2KatqNBjqeDIcAYVH71FRATY7k9NtZzeWBECpJ9aj0RaZybhtBu5t7Ej+8egP6rYqw93hI5ItG8zx+l6FXvAHTJ+ajR4R/wm/U8AA9O01bLsGF1sI4OaRiDISKSj8wzpbL352DdvGPQfx+IDRfboRBdzftCkIcH4g5BN0TggZfaoH5Cxz/39AbaNrLeDnckeTt6zcnJwJtv2j+fknVsWEeHNIoJ1EQkD1tVjJ2Ydi/KBA6tOQH9x+eg39YQOwsSLfY38T8LXZuT0P2lLnqNS0RgnUDbJ7PWWwPI2/PhzDWrNYGaSMU4m0xGDIaI3Ky42NgbcumS9f1VzEgquVmCzR8egP7/rmPtoQRklja22N+l9iHoul2C7rlotE1p5vr0d7nr+5hmYdmayWbtmtU4tZ5IxRgMyYjBEJEbpaUBY8cCubn2j/2zVs2103n49p1D0OuBb8+0QR5CzYcEoRB9w/dD168Qg1ObI/quSHnaWM1eq0qcrc9Tvi0vvGCcym6iZNFFIhXz1P2bOUNE5DpbQYYN3768Ee+cDMHmq21RinvM2xtJlzD4zqPQPRSIfpMTUTv8bvna6MjaW5MmGZOHnRmecnXhWiYqE6kOgyEick1VQYYNc3ffi024CwDQOugEdB3PQvdkA9z9RGv4B7ppurujK9xv2eJc8nB1Fq5lojKRqjAYIvJl7pzybS/IKKcMEs4iBv6hdTA/OQNDJibgzvvuBHCnPG2piqs9OEDV719SkmMruStR04iInMJgiMhXuXtBUAeDDAFAgkD9D2fjp3Gdq/+6znK1B8fe+2eq3Dx8uDHw8VRNIyKSHStQE/kiUy5PxZ6bc+eM29PSjI8NBmMi8PLlxv8aDHZPbZr+/n/zLzrUFKlRI0irVqHuuMeduwa5mHpwKi41YWKtKrWj7x8rN7v0O0SkNpxNRuRrHJ3yPX8+MHmyQz1HJTdLsPWTg9B/kQ/9gXj8UdoEfjDgFOIRg3Pwg41/Rho1Mp4/sIp6QJ5ga4V7W/WAnJ0y787hSDVXt3Z37yNpnsfu30ID8vLyBACRl5endFOI3C89XQjjLd/5H0ky/qxaJa6dviZWvPCzGNVkq6gnXbU4LAi3xMBGv4jvk94QZfjzOTbOoxqrVgkRG2vZzri4ym109P1LT1emzbGx6nhfV62q/Lmr9bMnr+Wp+zdzhoh8jaMJw9YIAQHg0vBxiBOnUFxu+ntDKReDmx6BLqUG+k1ORJ3ILgC6AGmtPbf0RXU4OqW9OgnXcrJVtsA0VKfkMJy7yhUQKYTBEJE1ah6asMfRhGEbJADh4iK6YwcuBsZB1yELuifC0O3pNtanv3tT3RxHprRXZ8q8XNQebLirXAGRQhgMEVXkTXkQ1oI2O1O+jbO77Fs+9TdE/TMZQFP7B/tS3Rw1TJlXe7Chlt4zIplwNhmplxKzVBydRaQGaWnGRN8+fYBRo4z/jY8H1qwxBm5ApRlUZYCtVOdKou5vL2NjvYhpyjxQeQaap6bMqz3YUEPvGZGMGAyROtm60bszGLE3NAEYhybUMHXYVtB29iwwfDhEmcDZaQuRH9jQcjfiMAL/w3lE2Q6KrE011xqlp8yrPdhwpVwBkYpxaj2pjzsW1XSEqwtvepqdqd8CwFXURyMYV5BPwhZE4QLqBhYjtnsshoyNQYcahyCNePjPJ9iZaq5lSuWOmT5je0N15af3e5oz5QqIXMSFWknd3HWTUDJxVO1DEyZ28kkkAGG4ilcxCzsbDoGurx+GpCYjtkv5XoSWxpuVN8wCU5JSuVDeUN3a1HvG3yHyAQyGyHnuTDBWMnFU7UMTf7qUcRCNHDjutXoL4Zc9w/YN05tmgWmRNwQb/B0iH8FhMnKOu4ewli835gjZ88UXxnwOOf8BVunQRFlpGfYsOwr9ZznQ74pCvcILyIADw3mA8kN6VH3eXOaBqJo4TFbOuXPnMHXqVHz77be4desWmjdvjsWLF6NTp05KN01bPDGE5Wivy6RJQG7u7cem5SUaNXL9pqGioYnCa4XY+N4B6FfewtpjzXC+rDWA1gCAACQgD6EIRZ79Eyk9pEfV50tlC4hUSvXB0NWrV9GjRw/06dMH3377LcLDw3Hy5EnUq1dP6aZpjyeGsOzVeDEpHwgBxnaNGGG5zZWhOwWHJi4dycX6eUeh/yYAP1xIRAG6mPfVxg3cH3MAuoEGDJzSCqErU4EZM+yflFObiYjsUv0w2bRp0/Dzzz9jy5YtLp+Dw2QycXQIa9kyYORI11+nqlkqzvy6VmfozgNDE6JM4Nh3mdAvPAP91jBsu54IUa7aRYzfBehaHYfu0VpInpCI4HrBlu2LiAAuX7Z+cjXMNiIiqiZP3b9VHwy1bt0aAwYMwNmzZ7Fp0ybExMRg3Lhx+Otf/2rzOUVFRSgqKjI/zs/PR1xcHIOh6vLk1HNrSdqNGgGXLjl3HpUFBaWFpdj26SHo/3MV+v1NcLwkwWJ/x5pHoLv7InR/jUDHkS0h+VVRK/rLLyv3hgGc2kxEPoPB0J+Cg41/DaempuLhhx/GL7/8gkmTJuFf//oXHn/8cavPmTlzJmbNmlVpO4OhavJ0gnHF3plz54DHHnPtXAomEl8/fx3fzzsI/WoD1me2xhURZt5XA8W4t8F+6PoWYPCLd6Jx95gqzlSOtWDRJC5OPbONiIiqgcHQnwIDA9G5c2ds27bNvO2FF17Arl27sH37dqvPYc+QGylZaM3Rnilrqjt056Ssneex9t3j0P9YG+mX26IYQeZ9YdIVDEo4DN0wf/Sf3AYhsU7+Ttqa0Wfy5Ze3PyMiIi/G2WR/ioqKQuvWrS22tWrVCqtWrbL5nKCgIAQFBdncT9WgZO0TR5OrrXFzIrEoE9i7/Cj0n16E/pcI7L3VCkC0ef+dNU5haLtT0I2pj3v+2gYBwT0dO3HF3rF77rE9ow8wBqWpqcCDD6piWJCIyBuoPhjq0aMHjh07ZrHt999/R5MmTRRqESlWaK2qqe+2uHGF8aL8IqS/fwD6FTex7vAduENcRBQuIASAP+5Et7pHoOt5BboJjdHi/gRIfvHOvYC1obCGDSvPpCtP6dXMiYi8kOqDocmTJ+Oee+7BW2+9hREjRuCXX37BokWLsGjRIqWbpm1K1T6x1TNljRtqA+Ueu4xv5h2B/ht/fH8uETfQGQ8iDT/jL4jD7fYYIqLh/9EHrveU2RoKqyoQKo/1hYiIHKb6nCEAWLduHaZPn47jx48jISEBqampVc4mq4hT66vgrdVtK7Y7NxeYPNkyQJIpkfjYt39Av/AM1m6tj5/zE1GG2+/P09JifCqeAWBcE8ysutP6q1iI1SGsPE1EPoAJ1DJiMGSDO9cYU4JMgZ2h2IBtiw4ap7//1gS/V5j+3j74GHRdLkD3VAN0enUgJFtBi6uz66qTKK6yUgJERNXBBGp38tbeEGtcvRZbwzDnzhm3e2ONmmoM3V0/fx0/vHsQ+jQD1me2wmXR3ryvBoqRHHYAuntvYMikpmjSowWAFsagxR0VuV0d4lLLauZERF5Ge8GQL/WGfPklMG5c5TW67F2LJ9YY8wJnd13A2vnHof+xJjbmtkMxupv31ZeuYlD8IeiG+WNAahuExFpZB8/RoMXZ4MbRmW8Vi1CqaTVzIiIvoq1gSK8HHn/cN3pD/vY34P/9v8rbz561fy2eWGNMhUSZwL6Vx6BflA39zgjsudUKwO3Ao2nAaQxtnwnd6HroMTbR/vR3R4MWZ6f12yshYBoKO3EC2LZN+R5OX+ppJSJN0lbOUHQ0Qs6ft36QN+VafPUV8PDDVR8TF2f7Wjy1xpgKFOUXIeODA9CvKMDaI3ciy3C7wrOEMnSvexC6HlegGx+HlgPvqHr5i4rcWZFbyeKWzvClnlYiUh3mDLmDrUAIcG9viJx/ORsMxqExe6q6lur2aKi8J+Dy8Sv4Zt5h6Nf547s/p7+b1EIB+kcdgO7+Egya0hLhbdq5/kJV1T2qbv6OksUtq1L+sz9+HJg50zd6WolI24QG5OXlCQAiz/jPdtU/y5bJ++KrVgkRG2v5GrGxxu2uSE+3fw32rqW01NgGSbL+PEkSIi7OeJy7r0cmv/+QKd4ZnC56he4Vfii1aF6U3wXxbMtNYt2MX8TNyzflf3Fr70lcnDzvSWmp8TNftsz4X2ufiadYu05bP1X9DhEROch8/87Lc+vraKtnyBFyLttga8bW2bPAQw/dTlJ2pmfFmWRcW9fiao+GimagGYoN2LH4EPRLr0C/Lw5Hi5sCiDfvbxd8DLrOF6B7Jhyd/tISfgGR7muMOytyK1XcsiJ766FV5KN5Z0Tkm7SXM3ThgudWXHe0cJ4zORaO1qBp1Mh4Y67qWqzle9gqVGjvejyQc3Uj+wY2vHsQ+rQSrDvZCrmioXlfAEqQHLYfuj7G6e/xPWPd0gZNqk4RSB/IOyMi5TBnyB3mzjXOJpM7v8MaezO2ynOmZ8U008jeuT/80P61ONOjodAMtHO7L2Ddu8eh31ATP11qiyJ0M++rJ13DwMaHoBsq4f4pbRDa2Mr0d6o+Z36XK3LzArlERHLQVjCk03kuKdWZ4SxnavuUH+Ky1an38sv2Z5uVP58jwYu7aupUIMoE9n/1O/T/ugD9jnDsvtka5ae/JwScwdC2mdA9FoKezyWiRq0e1Xo9coArn6kbF8glIpKbtoIhwHMrrjv7F7EzPSu2Zho1agR89NHtKdlycldNHQDFN4qxaeEB6JfdgP5wU5wx/FnhGcbp713rHILunsvQjYtF6yFNIfk1dvo1qBqc/UxZCZuIvIy2coY8uTaZvRo0tjiTY+HJKe4y19S5cvIqvp13GPp1Er7NSsR13P5cauIm+kUegG5AMQaltkBku3AZL4Sc5uzvskwL5BIRMWfIW5UPUP76V2DGjMo5SlVx5q9wT840kqGmzomfTmPtB5nQbwrFlmttYcDtIa4IvxwMaXYUuoeDcd+LiajVsKubLoScZu+zFwKYNQto1kyVdaeIiOxhz5CcrM3OatDA+N/Ll6t+riM9K+7uCXLk/E7MQDMUG/DL0sNY8+/L0O+Nw5Hiphb7E4OOQ9fpHHRPNUSXMa3hF+Dn/vaT65yZfUhEJANP3b8ZDMnFVh0WU6/JzJnA1avGG4etnpWqZpO5e9kDZ85fRdBRkFNgnP6+qhjrTrTEJdHI/LQAlKBX/QPQJV/HkBcScEeyjLk/1tofEwM8+yx7LOTEgJOIPIjBkIzc/mY6U4NnzRrn/7q2F2hVt9hhNc9/fk821s3/HfoNwfgxpx2KEGzeF4o8PND4EHQ64IGX2qBek1DX2+ls+yvimllERF6FwZCM3P5mOloIMT3dmOPjzF/X7i526ML5RZnAgbTj0H9yHvodjbCroI3FU+IDsjA08Q8MGVUXSc8nIrBOoPPtkqv95altkVMiIqoSE6i9ibM1eJxJfHZ3sUMHz1/yw0ZsOtAA+v+7Dv2hpjhtaA6gufmwrrUPQndPLnTPx6DN0Dsh+cU53xZXOFMQsHw9p8GDgW3bONxDREQMhmThxho8bi926ODzxg48gyXoZ34cjFvoF7HfOP19cnNEdUh07fWry9nrNgWPMTFAbu7t7RxCIyLSLAZDcjAtkWGvBo8r1XjdGWg58bw/0BTh0iXj9Pfhgeg7ua06pr+7et3lAyFAkcVmiYhIHZgzJBdTEi/g/Eyxqshc7LCisqISlETFIfDqRUjW9kNCfmBDHP1gA+5+qm31p7/LzdXiltZ4YLFZIiJynKdyhlR2Z/NipiUyYmIst8fGVq+3wVTwDrgdWJm4uOzBzdyb0L+yE8+02ILoWlcx8upHEJBQViEcEpDgJwH1ln+Cbs+2V18gBFT9/jirfP4VERFpBnuG5GZvppirdVqqWfAue38O1s07Bv33gdhwsR0KUdO8LwR5eK3Bh3iucAFqF1xy6fyKs/b+uMqZJVGIiMhtOLVeRoqsTWZNdQonGgzGKfwZGcbHycnGHxuBlCgTOLTmBPQfn4N+W0PsLLBMcG7ifxa6Nieh+0td9Br35/R3tRXUc7Y95Y8/fhz49NPKC9leumT7+SamEghERKQoBkMyUkUwVJ3Chg4GUSU3S7Dl44PQf5EP/cEEZJZaVnjuUvsQdN0uQfdcNNqmNIPkV81hJXeSo+J2xWDqnnuApk3dln9FRETyYjAkI8WDoeoUTrQTRBV8/B/oDzbF2rUC35xORB5uV3gOQiH6hu+Hrl8hBqc2R/RdkTJelBu5s+K2OxPd1dSrRkTkAxgMyUjxYMjZCtUmdoKoMgDnEIt4nEIZjDfeRtIlDL7zKHQPBaLf5ETUDq9d3dZ7lrsrbgPyLzjq7nXjiIg0ihWofYmrhRPtVFf2AxCHs3gsYAWiOsVA92QDdH2yNfwDXahnpBburrgNGAOUoUPl6cmx1YvFukVERF6DwZAnuFA48WbuTRx5bws6OfC0//zXDxiZ7FLTVMfdFbdNnFkSxRaDwdgjZK1ztfzSH0OHcsiMiEjFVFg4xgeZKlTbqoMjSUBcHC7Wb4nFT2zB0KidaNgImLLawR4eV6swq5G7K27LyZleLCIiUi32DHmCqTDg8OHGwKdcT4KA8fH0S5PxdodwCNxOcj7tl4Cr/o1Qr+SS1erQ1VrmQ63cubSJ3DzVi0VERG7FniFP+bNCtYiOttichVg8hFWYWzgZAn7oXOswXr83A/tWHsMfJY1Rf8UnkCTJdvXpefOMPQ/LlxsTtQ0Gz1yPu7ih4rbbeFMvFhER2cTZZB6QdyYP3807BP0age9Ot0RbHEAULuACovALuqBPo8PQ9buFwZObIaazlRunrdlPjz5qDIJ8cRaT3DO+3MHN68YREWkdp9bLSIlg6NTWs1i74CT06XWQcaUdSlHDvK+hlIvBTY9Al1ID/SYnok5kHfsnrFjHJjcXGDHCPbV41MJW7R411fRxV90iIiJiMCQnT7yZZaVl+PX/jkL/WQ70u6Owv7CFxf6WgSeh65AF3RNh6PZ0G/gHVuPm7YlaPGqlVE2fqgIwb+jFIiLyQgyGZOSuN/PWlVvY+P5B6Ffewtrfm+NC2e3kZz8Y0DP0AHRJ1zBkQhM0H5Ag2+u6XMTR27mzMrW917UXgKmpt4qIyEew6KJK5Ry6hPXzjkL/XQ38cKEtbqKLeV8dXMf9MQehG2zAwCmt0aBZB/c0wpVZTN5+s1aqpo+jRRXlqFtERESKYDBkhygTOPrNH9B/mAX9z2HYfj0RArendcf6n4eu1XHoHq2N5IltERTS3f2NcnYWky8sF+GJytQVsagiEZEmeN3U+jlz5kCSJEyaNMltr1FaWIpN7+3DlM4ZaB58Gq2HNMW075Kx7Xo7CPjhrppHMDM5A3uWHcWZ4ih8eKA3BrzSGUEhQW5rkwUHizgiKel2z0bFQMLUs5GW5v72ykGJmj4sqkhEpAle1TO0a9cuLFq0CO3atZP93Pln8/H9/EPQrzZg/ak2uCo6mPcFogj3NtwPXd9bGJLaDLFdWgFoJXsbHFZFEUeLWjyA7/RsKFHTh0UViYg0wWt6hm7cuIG//OUv+PTTT1G/fn1Zznlm+zl8OGITBjTcjYZxwRjxbnd8kdkTV0V9NJAu4/E7tuKrl7Yj91wxvr3UBc8v74XYLiopoPdnEUfExFhuj429ncfiSz0bzvSGyYVFFYmINMFreobGjx+PQYMGoW/fvnjzzTerPLaoqAhFRUXmx/n5+QCM0993//ewcfr7rij8VtgCwO1gonmNTOjan4ZuTH10f6YNAoJ7ytN4dyUv21t93R09G0olYjvaGyZnW7xpaRAiInKZVwRDK1aswJ49e7Br1y6Hjp8zZw5mzZpVaXurhjnIFq0BtAZgnP5+T8hB6HpehW5CY7R44A4AMk6BB9yfvFzVLCa5ezaUTsQ29YZZa4M7avooEYAREZHHqb7OUFZWFjp37owffvgB7du3BwAkJyejQ4cOWGDKi6nAWs9QXFwcgDzUhh8GRu/DUx324p4+QQjp3Nx9vRtK1cUxkXO5CKWvpTxP906xqCIRkSJYdPFPq1evxoMPPgj/cjc7g8EASZLg5+eHoqIii33WmN7Mr15Jx5AW2Qj8+8vu791QS5VoOZaLUMu1KMnb6zQREXkhBkN/un79Ok6fPm2x7cknn0TLli0xdepUJCYm2j2H+c38/HOEPP64Z3o31FQluro9G2q6FiIi0gxWoP5T3bp1KwU8tWvXRoMGDRwKhCxMneq5aeZqmpZtL9HaHjVdCxERkcxUHwzJ6vx52/vkrmCstmnZ1VkuwtE2hocbe5E4lERERF5E9cNkcjB3swGw28m2bBkwcmT1X1TO5GWlOXItYWFAzZreveQHERGpiqeGybym6KLHyNVTY5qWDVQuFOht07LtXYsQwOXL3r/kBxERaZK2gqHoaM9WMHakSrS3sHUtMTFAgwbWn2PqRZo0ydi7REREpELaGiYzzSYDXJ9m7gpfmpZd8VoMBqBvX/vP40wzIiJyEmeTuYNO59kKxibVSV5Wm4rXsny5Y8/jTDMiIlIpbQVDQPWnmZMltc2aIyIicpL2giHAt3pqlMbFTImIyMtpK4Ga5OdLs+aIiEiTGAxR9fnSrDkiItIcbQ6TkfyYi0VERF6KwZAvTXtXGnOxiIjIC2k7GLK2mjuXkCAiItIU7eYMpaUZl4rgEhJERESaps1gyGAw9ghZmwrOJSSIiIg0RZvB0JYtlXuEyhMCyMoyHkdEREQ+TZs5Q44uDcElJJhgTkREPk+bwRCXkHAME8yJiEgDtDlMZlpComLFZBNJAuLitL2EBBPMiYhII7QZDHEJiaoxwZyIiDREm8EQUP0lJAwGICMDWL7c+F9fCgyYYE5ERBqizZwhE1eXkPD1XBommBMRkYZoOxgCnF9CwpRLU3EIyZRL4wsLkzLBnIiINEQSwlpiiG/Jz89HaGgo8vLyEBISYv0gR6aQGwxAfLztISRJMvYQZWZ6d76R6TrPnbOeN+Qr10lERKrm0P1bBtrNGSovLc148+/TBxg1yvjf+PjKM6a0kkvDBHMiItIQBkPOTCHXUi5NdRPMiYiIvIS2c4bsTSGXJOMU8qFDjb0gWsulcTXBnIiIyItoOxhyZtgrOfl2sUZ7uTS+VKzR2QRzIiIiL6PtYTJnh72YS0NERORztB0MuTLsxVwaIiIin6LtqfXVmULO1dyJiIjcylNT67WdM2Qa9ho+3Bj4lA+I7A17MZeGiIjIJ2h7mAzgsBcREZHGabtnyIRTyImIiDSLwZAJh72IiIg0icNkREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1jMERERESapvpgaM6cOejSpQvq1q2L8PBwDBs2DMeOHVO6WUREROQjVB8Mbdq0CePHj8eOHTuwYcMGlJaWon///igoKFC6aUREROQDvG5tskuXLiE8PBybNm1Cr169HHqOp9Y2ISIiIvlwbTIb8vLyAABhYWE2jykqKkJRUZH5cX5+vtvbRURERN5J9cNk5QkhkJqaip49eyIxMdHmcXPmzEFoaKj5Jy4uzoOtJCIiIm/iVcNk48ePx/r167F161bExsbaPM5az1BcXByHyYiIiLwIh8kqmDhxIvR6PTZv3lxlIAQAQUFBCAoK8lDLiIiIyJupPhgSQmDixIn4+uuvkZGRgYSEBKWbRERERD5E9cHQ+PHjsWzZMqxZswZ169ZFdnY2ACA0NBQ1a9ZUuHVERETk7VSfMyRJktXtS5YswRNPPOHQOTi1noiIyPswZ+hPKo/ViIiIyMt51dR6IiIiIrkxGCIiIiJNYzBEREREmsZgiIiIiDSNwRARERFpGoMhIiIi0jQGQ0RERKRpDIaIiIhI0xgMERERkaYxGCIiIiJNYzBEREREmsZgiIiIiDSNwRARERFpGoMhIiIi0jQGQ0RERKRpDIaIiIhI0xgMERERkaYxGCIiIiJNYzBEREREmsZgiIiIiDSNwRARERFpGoMhIiIi0jQGQ0RERKRpDIaIiIhI0xgMERERkaYxGCIiIiJNYzBEREREmsZgiIiIiDSNwRARERFpGoMhIiIi0jQGQ0RERKRpDIaIiIhI0xgMERERkaYxGCIiIiJNYzBEREREmsZgiIiIiDSNwRARERFpGoMhIiIi0jSvCYY++ugjJCQkIDg4GJ06dcKWLVuUbhIRERH5AK8IhlauXIlJkybhlVdewd69e5GUlIQHHngAZ86cUbppRERE5OUkIYRQuhH2dO3aFXfddRc+/vhj87ZWrVph2LBhmDNnjt3n5+fnIzQ0FHl5eQgJCXFnU4mIiEgmnrp/q75nqLi4GL/++iv69+9vsb1///7Ytm2bQq0iIiIiXxGgdAPsyc3NhcFgQEREhMX2iIgIZGdnW31OUVERioqKzI/z8vIAGCNMIiIi8g6m+7a7B7FUHwyZSJJk8VgIUWmbyZw5czBr1qxK2+Pi4tzSNiIiInKfy5cvIzQ01G3nV30w1LBhQ/j7+1fqBcrJyanUW2Qyffp0pKammh9fu3YNTZo0wZkzZ9z6ZqpNfn4+4uLikJWVpalcKV43r1sLeN28bi3Iy8tD48aNERYW5tbXUX0wFBgYiE6dOmHDhg148MEHzds3bNiAoUOHWn1OUFAQgoKCKm0PDQ3V1C+RSUhICK9bQ3jd2sLr1hatXrefn3tTnFUfDAFAamoqRo8ejc6dO6N79+5YtGgRzpw5g+eee07pphEREZGX84pg6JFHHsHly5fx+uuv48KFC0hMTMQ333yDJk2aKN00IiIi8nJeEQwBwLhx4zBu3DiXnhsUFIQZM2ZYHTrzZbxuXrcW8Lp53VrA63bvdXtF0UUiIiIid1F90UUiIiIid2IwRERERJrGYIiIiIg0jcEQERERaZpXBkMfffQREhISEBwcjE6dOmHLli1VHr9p0yZ06tQJwcHBuOOOO/DJJ59UOmbVqlVo3bo1goKC0Lp1a3z99dfuar7LnLnutLQ09OvXD40aNUJISAi6d++O77//3uKYpUuXQpKkSj+FhYXuvhSnOHPdGRkZVq/p6NGjFsf52uf9xBNPWL3uNm3amI/xhs978+bNGDJkCKKjoyFJElavXm33Ob7w/Xb2un3l++3sdfvK99vZ6/aV7/ecOXPQpUsX1K1bF+Hh4Rg2bBiOHTtm93me+I57XTC0cuVKTJo0Ca+88gr27t2LpKQkPPDAAzhz5ozV4zMzMzFw4EAkJSVh7969+Pvf/44XXngBq1atMh+zfft2PPLIIxg9ejR+++03jB49GiNGjMDOnTs9dVl2OXvdmzdvRr9+/fDNN9/g119/RZ8+fTBkyBDs3bvX4riQkBBcuHDB4ic4ONgTl+QQZ6/b5NixYxbX1KxZM/M+X/y833vvPYvrzcrKQlhYGB5++GGL49T+eRcUFKB9+/ZYuHChQ8f7yvfb2ev2le+3s9dt4u3fb2ev21e+35s2bcL48eOxY8cObNiwAaWlpejfvz8KCgpsPsdj33HhZe6++27x3HPPWWxr2bKlmDZtmtXj//a3v4mWLVtabBs7dqzo1q2b+fGIESPE/fffb3HMgAEDxKOPPipTq6vP2eu2pnXr1mLWrFnmx0uWLBGhoaFyNdEtnL3u9PR0AUBcvXrV5jm18Hl//fXXQpIkcerUKfM2b/i8ywMgvv766yqP8ZXvd3mOXLc13vj9Ls+R6/aV73d5rnzevvD9FkKInJwcAUBs2rTJ5jGe+o57Vc9QcXExfv31V/Tv399ie//+/bFt2zarz9m+fXul4wcMGIDdu3ejpKSkymNsndPTXLnuisrKynD9+vVKi93duHEDTZo0QWxsLAYPHlzpL0slVee6O3bsiKioKNx3331IT0+32KeFz3vx4sXo27dvpSrtav68XeEL3285eOP3uzq8+fstB1/5fufl5QFAlYuweuo77lXBUG5uLgwGQ6XV6iMiIiqtam+SnZ1t9fjS0lLk5uZWeYytc3qaK9dd0bx581BQUIARI0aYt7Vs2RJLly6FXq/H8uXLERwcjB49euD48eOytt9Vrlx3VFQUFi1ahFWrViEtLQ0tWrTAfffdh82bN5uP8fXP+8KFC/j222/xzDPPWGxX++ftCl/4fsvBG7/frvCF73d1+cr3WwiB1NRU9OzZE4mJiTaP89R33GuW4yhPkiSLx0KIStvsHV9xu7PnVIKrbVy+fDlmzpyJNWvWIDw83Ly9W7du6Natm/lxjx49cNddd+GDDz7A+++/L1/Dq8mZ627RogVatGhhfty9e3dkZWXhnXfeQa9evVw6p1JcbePSpUtRr149DBs2zGK7t3zezvKV77ervP377Qxf+n67yle+3xMmTMD+/fuxdetWu8d64jvuVT1DDRs2hL+/f6VoLycnp1JUaBIZGWn1+ICAADRo0KDKY2yd09NcuW6TlStX4umnn8b//vc/9O3bt8pj/fz80KVLF9X8JVGd6y6vW7duFtfky5+3EAL//ve/MXr0aAQGBlZ5rNo+b1f4wve7Orz5+y0Xb/t+V4evfL8nTpwIvV6P9PR0xMbGVnmsp77jXhUMBQYGolOnTtiwYYPF9g0bNuCee+6x+pzu3btXOv6HH35A586dUaNGjSqPsXVOT3PlugHjX4xPPPEEli1bhkGDBtl9HSEE9u3bh6ioqGq3WQ6uXndFe/futbgmX/28AeNsjRMnTuDpp5+2+zpq+7xd4Qvfb1d5+/dbLt72/a4Ob/9+CyEwYcIEpKWlYePGjUhISLD7HI99xx1OtVaJFStWiBo1aojFixeLw4cPi0mTJonatWubs+qnTZsmRo8ebT7+jz/+ELVq1RKTJ08Whw8fFosXLxY1atQQX331lfmYn3/+Wfj7+4t//vOf4siRI+Kf//ynCAgIEDt27PD49dni7HUvW7ZMBAQEiA8//FBcuHDB/HPt2jXzMTNnzhTfffedOHnypNi7d6948sknRUBAgNi5c6fHr88WZ6/73XffFV9//bX4/fffxcGDB8W0adMEALFq1SrzMb74eZs89thjomvXrlbP6Q2f9/Xr18XevXvF3r17BQAxf/58sXfvXnH69GkhhO9+v529bl/5fjt73b7y/Xb2uk28/fv9/PPPi9DQUJGRkWHxe3vz5k3zMUp9x70uGBJCiA8//FA0adJEBAYGirvuustiWt6YMWNE7969LY7PyMgQHTt2FIGBgSI+Pl58/PHHlc755ZdfihYtWogaNWqIli1bWny51MKZ6+7du7cAUOlnzJgx5mMmTZokGjduLAIDA0WjRo1E//79xbZt2zx4RY5x5rrnzp0rmjZtKoKDg0X9+vVFz549xfr16yud09c+byGEuHbtmqhZs6ZYtGiR1fN5w+dtmjpt6/fWV7/fzl63r3y/nb1uX/l+u/J77gvfb2vXDEAsWbLEfIxS33HpzwYSERERaZJX5QwRERERyY3BEBEREWkagyEiIiLSNAZDREREpGkMhoiIiEjTGAwRERGRpjEYIiIiIk1jMERERESaxmCIiIiINI3BEBEREWkagyEi8krLly9HcHAwzp07Z972zDPPoF27dsjLy1OwZUTkbbg2GRF5JSEEOnTogKSkJCxcuBCzZs3CZ599hh07diAmJkbp5hGRFwlQugFERK6QJAmzZ8/G8OHDER0djffeew9btmxhIERETmPPEBF5tbvuuguHDh3CDz/8gN69eyvdHCLyQswZIiKv9f333+Po0aMwGAyIiIhQujlE5KXYM0REXmnPnj1ITk7Ghx9+iBUrVqBWrVr48ssvlW4WEXkh5gwRkdc5deoUBg0ahGnTpmH06NFo3bo1unTpgl9//RWdOnVSunlE5GXYM0REXuXKlSvo0aMHevXqhX/961/m7UOHDkVRURG+++47BVtHRN6IwRARERFpGhOoiYiISNMYDBEREZGmMRgiIiIiTWMwRERERJrGYIiIiIg0jcEQERERaRqDISIiItI0BkNERESkaQyGiIiISNMYDBEREZGmMRgiIiIiTWMwRERERJr2/wGKGk4Rgk7WQgAAAABJRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"filenames": {
|
|
"image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_22_1.png"
|
|
}
|
|
},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"theta from own sdg\n",
|
|
"[[3.92648983]\n",
|
|
" [3.07020234]]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Using Autograd to calculate gradients using SGD\n",
|
|
"# OLS example\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import autograd.numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from autograd import grad\n",
|
|
"\n",
|
|
"# Note change from previous example\n",
|
|
"def CostOLS(y,X,theta):\n",
|
|
" return np.sum((y-X @ theta)**2)\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"x = 2*np.random.rand(n,1)\n",
|
|
"y = 4+3*x+np.random.randn(n,1)\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(theta_linreg)\n",
|
|
"# Hessian matrix\n",
|
|
"H = (2.0/n)* XT_X\n",
|
|
"EigValues, EigVectors = np.linalg.eig(H)\n",
|
|
"print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
|
|
"\n",
|
|
"theta = np.random.randn(2,1)\n",
|
|
"eta = 1.0/np.max(EigValues)\n",
|
|
"Niterations = 1000\n",
|
|
"\n",
|
|
"# Note that we request the derivative wrt third argument (theta, 2 here)\n",
|
|
"training_gradient = grad(CostOLS,2)\n",
|
|
"\n",
|
|
"for iter in range(Niterations):\n",
|
|
" gradients = (1.0/n)*training_gradient(y, X, theta)\n",
|
|
" theta -= eta*gradients\n",
|
|
"print(\"theta from own gd\")\n",
|
|
"print(theta)\n",
|
|
"\n",
|
|
"xnew = np.array([[0],[2]])\n",
|
|
"Xnew = np.c_[np.ones((2,1)), xnew]\n",
|
|
"ypredict = Xnew.dot(theta)\n",
|
|
"ypredict2 = Xnew.dot(theta_linreg)\n",
|
|
"\n",
|
|
"plt.plot(xnew, ypredict, \"r-\")\n",
|
|
"plt.plot(xnew, ypredict2, \"b-\")\n",
|
|
"plt.plot(x, y ,'ro')\n",
|
|
"plt.axis([0,2.0,0, 15.0])\n",
|
|
"plt.xlabel(r'$x$')\n",
|
|
"plt.ylabel(r'$y$')\n",
|
|
"plt.title(r'Random numbers ')\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"n_epochs = 50\n",
|
|
"M = 5 #size of each minibatch\n",
|
|
"m = int(n/M) #number of minibatches\n",
|
|
"t0, t1 = 5, 50\n",
|
|
"def learning_schedule(t):\n",
|
|
" return t0/(t+t1)\n",
|
|
"\n",
|
|
"theta = np.random.randn(2,1)\n",
|
|
"\n",
|
|
"for epoch in range(n_epochs):\n",
|
|
"# Can you figure out a better way of setting up the contributions to each batch?\n",
|
|
" for i in range(m):\n",
|
|
" random_index = M*np.random.randint(m)\n",
|
|
" xi = X[random_index:random_index+M]\n",
|
|
" yi = y[random_index:random_index+M]\n",
|
|
" gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
|
|
" eta = learning_schedule(epoch*m+i)\n",
|
|
" theta = theta - eta*gradients\n",
|
|
"print(\"theta from own sdg\")\n",
|
|
"print(theta)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "584dbef3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Same code but now with momentum gradient descent"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "2908775e",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[4.12998822]\n",
|
|
" [2.78886274]]\n",
|
|
"Eigenvalues of Hessian Matrix:[0.3303366 4.33020191]\n",
|
|
"theta from own gd\n",
|
|
"[[4.12921833]\n",
|
|
" [2.78951443]]\n",
|
|
"theta from own sdg with momentum\n",
|
|
"[[4.10156593]\n",
|
|
" [2.7748567 ]]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Using Autograd to calculate gradients using SGD\n",
|
|
"# OLS example\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import autograd.numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from autograd import grad\n",
|
|
"\n",
|
|
"# Note change from previous example\n",
|
|
"def CostOLS(y,X,theta):\n",
|
|
" return np.sum((y-X @ theta)**2)\n",
|
|
"\n",
|
|
"n = 100\n",
|
|
"x = 2*np.random.rand(n,1)\n",
|
|
"y = 4+3*x+np.random.randn(n,1)\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(theta_linreg)\n",
|
|
"# Hessian matrix\n",
|
|
"H = (2.0/n)* XT_X\n",
|
|
"EigValues, EigVectors = np.linalg.eig(H)\n",
|
|
"print(f\"Eigenvalues of Hessian Matrix:{EigValues}\")\n",
|
|
"\n",
|
|
"theta = np.random.randn(2,1)\n",
|
|
"eta = 1.0/np.max(EigValues)\n",
|
|
"Niterations = 100\n",
|
|
"\n",
|
|
"# Note that we request the derivative wrt third argument (theta, 2 here)\n",
|
|
"training_gradient = grad(CostOLS,2)\n",
|
|
"\n",
|
|
"for iter in range(Niterations):\n",
|
|
" gradients = (1.0/n)*training_gradient(y, X, theta)\n",
|
|
" theta -= eta*gradients\n",
|
|
"print(\"theta from own gd\")\n",
|
|
"print(theta)\n",
|
|
"\n",
|
|
"\n",
|
|
"n_epochs = 50\n",
|
|
"M = 5 #size of each minibatch\n",
|
|
"m = int(n/M) #number of minibatches\n",
|
|
"t0, t1 = 5, 50\n",
|
|
"def learning_schedule(t):\n",
|
|
" return t0/(t+t1)\n",
|
|
"\n",
|
|
"theta = np.random.randn(2,1)\n",
|
|
"\n",
|
|
"change = 0.0\n",
|
|
"delta_momentum = 0.3\n",
|
|
"\n",
|
|
"for epoch in range(n_epochs):\n",
|
|
" for i in range(m):\n",
|
|
" random_index = M*np.random.randint(m)\n",
|
|
" xi = X[random_index:random_index+M]\n",
|
|
" yi = y[random_index:random_index+M]\n",
|
|
" gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
|
|
" eta = learning_schedule(epoch*m+i)\n",
|
|
" # calculate update\n",
|
|
" new_change = eta*gradients+delta_momentum*change\n",
|
|
" # take a step\n",
|
|
" theta -= new_change\n",
|
|
" # save the change\n",
|
|
" change = new_change\n",
|
|
"print(\"theta from own sdg with momentum\")\n",
|
|
"print(theta)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d5b4691f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## AdaGrad algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
|
|
"\n",
|
|
"<!-- dom:FIGURE: [figures/adagrad.png, width=600 frac=0.8] -->\n",
|
|
"<!-- begin figure -->\n",
|
|
"\n",
|
|
"<img src=\"figures/adagrad.png\" width=\"600\"><p style=\"font-size: 0.9em\"><i>Figure 1: </i></p>\n",
|
|
"<!-- end figure -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9e0f6972",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Similar (second order function now) problem but now with AdaGrad"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "5c39eaa6",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[2.]\n",
|
|
" [3.]\n",
|
|
" [4.]]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"theta from own AdaGrad\n",
|
|
"[[1.9978254 ]\n",
|
|
" [3.01066976]\n",
|
|
" [3.98914003]]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Using Autograd to calculate gradients using AdaGrad and Stochastic Gradient descent\n",
|
|
"# OLS example\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import autograd.numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from autograd import grad\n",
|
|
"\n",
|
|
"# Note change from previous example\n",
|
|
"def CostOLS(y,X,theta):\n",
|
|
" return np.sum((y-X @ theta)**2)\n",
|
|
"\n",
|
|
"n = 1000\n",
|
|
"x = np.random.rand(n,1)\n",
|
|
"y = 2.0+3*x +4*x*x\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x, x*x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(theta_linreg)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Note that we request the derivative wrt third argument (theta, 2 here)\n",
|
|
"training_gradient = grad(CostOLS,2)\n",
|
|
"# Define parameters for Stochastic Gradient Descent\n",
|
|
"n_epochs = 50\n",
|
|
"M = 5 #size of each minibatch\n",
|
|
"m = int(n/M) #number of minibatches\n",
|
|
"# Guess for unknown parameters theta\n",
|
|
"theta = np.random.randn(3,1)\n",
|
|
"\n",
|
|
"# Value for learning rate\n",
|
|
"eta = 0.01\n",
|
|
"# Including AdaGrad parameter to avoid possible division by zero\n",
|
|
"delta = 1e-8\n",
|
|
"for epoch in range(n_epochs):\n",
|
|
" Giter = 0.0\n",
|
|
" for i in range(m):\n",
|
|
" random_index = M*np.random.randint(m)\n",
|
|
" xi = X[random_index:random_index+M]\n",
|
|
" yi = y[random_index:random_index+M]\n",
|
|
" gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
|
|
" Giter += gradients*gradients\n",
|
|
" update = gradients*eta/(delta+np.sqrt(Giter))\n",
|
|
" theta -= update\n",
|
|
"print(\"theta from own AdaGrad\")\n",
|
|
"print(theta)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "234e9738",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Running this code we note an almost perfect agreement with the results from matrix inversion."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "53f908ed",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## RMSProp algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
|
|
"\n",
|
|
"<!-- dom:FIGURE: [figures/rmsprop.png, width=600 frac=0.8] -->\n",
|
|
"<!-- begin figure -->\n",
|
|
"\n",
|
|
"<img src=\"figures/rmsprop.png\" width=\"600\"><p style=\"font-size: 0.9em\"><i>Figure 1: </i></p>\n",
|
|
"<!-- end figure -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ab2867c0",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## RMSprop for adaptive learning rate with Stochastic Gradient Descent"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "cb5dcea3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[2.]\n",
|
|
" [3.]\n",
|
|
" [4.]]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"theta from own RMSprop\n",
|
|
"[[1.99984215]\n",
|
|
" [2.99767262]\n",
|
|
" [3.99944037]]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n",
|
|
"# OLS example\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import autograd.numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from autograd import grad\n",
|
|
"\n",
|
|
"# Note change from previous example\n",
|
|
"def CostOLS(y,X,theta):\n",
|
|
" return np.sum((y-X @ theta)**2)\n",
|
|
"\n",
|
|
"n = 1000\n",
|
|
"x = np.random.rand(n,1)\n",
|
|
"y = 2.0+3*x +4*x*x# +np.random.randn(n,1)\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x, x*x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(theta_linreg)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Note that we request the derivative wrt third argument (theta, 2 here)\n",
|
|
"training_gradient = grad(CostOLS,2)\n",
|
|
"# Define parameters for Stochastic Gradient Descent\n",
|
|
"n_epochs = 50\n",
|
|
"M = 5 #size of each minibatch\n",
|
|
"m = int(n/M) #number of minibatches\n",
|
|
"# Guess for unknown parameters theta\n",
|
|
"theta = np.random.randn(3,1)\n",
|
|
"\n",
|
|
"# Value for learning rate\n",
|
|
"eta = 0.01\n",
|
|
"# Value for parameter rho\n",
|
|
"rho = 0.99\n",
|
|
"# Including AdaGrad parameter to avoid possible division by zero\n",
|
|
"delta = 1e-8\n",
|
|
"for epoch in range(n_epochs):\n",
|
|
" Giter = 0.0\n",
|
|
" for i in range(m):\n",
|
|
" random_index = M*np.random.randint(m)\n",
|
|
" xi = X[random_index:random_index+M]\n",
|
|
" yi = y[random_index:random_index+M]\n",
|
|
" gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
|
|
"\t# Accumulated gradient\n",
|
|
"\t# Scaling with rho the new and the previous results\n",
|
|
" Giter = (rho*Giter+(1-rho)*gradients*gradients)\n",
|
|
"\t# Taking the diagonal only and inverting\n",
|
|
" update = gradients*eta/(delta+np.sqrt(Giter))\n",
|
|
"\t# Hadamard product\n",
|
|
" theta -= update\n",
|
|
"print(\"theta from own RMSprop\")\n",
|
|
"print(theta)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4c415477",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## ADAM algorithm, taken from [Goodfellow et al](https://www.deeplearningbook.org/contents/optimization.html)\n",
|
|
"\n",
|
|
"<!-- dom:FIGURE: [figures/adam.png, width=600 frac=0.8] -->\n",
|
|
"<!-- begin figure -->\n",
|
|
"\n",
|
|
"<img src=\"figures/adam.png\" width=\"600\"><p style=\"font-size: 0.9em\"><i>Figure 1: </i></p>\n",
|
|
"<!-- end figure -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5bf0aa76",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## And finally [ADAM](https://arxiv.org/pdf/1412.6980.pdf)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "17bd80f0",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Own inversion\n",
|
|
"[[2.]\n",
|
|
" [3.]\n",
|
|
" [4.]]\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"theta from own ADAM\n",
|
|
"[[1.99988325]\n",
|
|
" [3.00072412]\n",
|
|
" [3.99932732]]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Using Autograd to calculate gradients using RMSprop and Stochastic Gradient descent\n",
|
|
"# OLS example\n",
|
|
"from random import random, seed\n",
|
|
"import numpy as np\n",
|
|
"import autograd.numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from autograd import grad\n",
|
|
"\n",
|
|
"# Note change from previous example\n",
|
|
"def CostOLS(y,X,theta):\n",
|
|
" return np.sum((y-X @ theta)**2)\n",
|
|
"\n",
|
|
"n = 1000\n",
|
|
"x = np.random.rand(n,1)\n",
|
|
"y = 2.0+3*x +4*x*x# +np.random.randn(n,1)\n",
|
|
"\n",
|
|
"X = np.c_[np.ones((n,1)), x, x*x]\n",
|
|
"XT_X = X.T @ X\n",
|
|
"theta_linreg = np.linalg.pinv(XT_X) @ (X.T @ y)\n",
|
|
"print(\"Own inversion\")\n",
|
|
"print(theta_linreg)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Note that we request the derivative wrt third argument (theta, 2 here)\n",
|
|
"training_gradient = grad(CostOLS,2)\n",
|
|
"# Define parameters for Stochastic Gradient Descent\n",
|
|
"n_epochs = 50\n",
|
|
"M = 5 #size of each minibatch\n",
|
|
"m = int(n/M) #number of minibatches\n",
|
|
"# Guess for unknown parameters theta\n",
|
|
"theta = np.random.randn(3,1)\n",
|
|
"\n",
|
|
"# Value for learning rate\n",
|
|
"eta = 0.01\n",
|
|
"# Value for parameters beta1 and beta2, see https://arxiv.org/abs/1412.6980\n",
|
|
"beta1 = 0.9\n",
|
|
"beta2 = 0.999\n",
|
|
"# Including AdaGrad parameter to avoid possible division by zero\n",
|
|
"delta = 1e-7\n",
|
|
"iter = 0\n",
|
|
"for epoch in range(n_epochs):\n",
|
|
" first_moment = 0.0\n",
|
|
" second_moment = 0.0\n",
|
|
" iter += 1\n",
|
|
" for i in range(m):\n",
|
|
" random_index = M*np.random.randint(m)\n",
|
|
" xi = X[random_index:random_index+M]\n",
|
|
" yi = y[random_index:random_index+M]\n",
|
|
" gradients = (1.0/M)*training_gradient(yi, xi, theta)\n",
|
|
" # Computing moments first\n",
|
|
" first_moment = beta1*first_moment + (1-beta1)*gradients\n",
|
|
" second_moment = beta2*second_moment+(1-beta2)*gradients*gradients\n",
|
|
" first_term = first_moment/(1.0-beta1**iter)\n",
|
|
" second_term = second_moment/(1.0-beta2**iter)\n",
|
|
"\t# Scaling with rho the new and the previous results\n",
|
|
" update = eta*first_term/(np.sqrt(second_term)+delta)\n",
|
|
" theta -= update\n",
|
|
"print(\"theta from own ADAM\")\n",
|
|
"print(theta)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f16e27bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Introducing [JAX](https://jax.readthedocs.io/en/latest/)\n",
|
|
"\n",
|
|
"Presently, instead of using **autograd**, we recommend using [JAX](https://jax.readthedocs.io/en/latest/)\n",
|
|
"\n",
|
|
"**JAX** is Autograd and [XLA (Accelerated Linear Algebra))](https://www.tensorflow.org/xla),\n",
|
|
"brought together for high-performance numerical computing and machine learning research.\n",
|
|
"It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "279eb2a4",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### Getting started with Jax, note the way we import numpy"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "a88c09b3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import jax\n",
|
|
"import jax.numpy as jnp\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"from jax import grad as jax_grad"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cd958e85",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### A warm-up example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "41d78257",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/jax/_src/numpy/lax_numpy.py:173: UserWarning: Explicitly requested dtype float64 requested in asarray is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.\n",
|
|
" return asarray(x, dtype=self.dtype)\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[<matplotlib.lines.Line2D at 0x11df37280>]"
|
|
]
|
|
},
|
|
"execution_count": 11,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABQCUlEQVR4nO3deXxU1eH+8c+dmaxkhZCFECDsS9hBFNkURami4K6tim1tUyOVUmu/WlutbeXXRUstYLRWXLHugDu4sCj7JoussgWSAAnJTNZJMnN/fwxEEQQCydyZyfN+veYFTG7Iw4iZh3POPccwTdNERERExE9sVgcQERGR5kXlQ0RERPxK5UNERET8SuVDRERE/ErlQ0RERPxK5UNERET8SuVDRERE/ErlQ0RERPzKYXWA7/J6veTn5xMbG4thGFbHERERkTNgmiZlZWW0adMGm+3UYxsBVz7y8/PJyMiwOoaIiIichby8PNq2bXvKawKufMTGxgK+8HFxcRanERERkTPhcrnIyMiofx8/lYArH8emWuLi4lQ+REREgsyZLJnQglMRERHxK5UPERER8SuVDxEREfErlQ8RERHxK5UPERER8SuVDxEREfErlQ8RERHxK5UPERER8auA22RMREREmojXA3uXQvlBiEmB9kPBZvd7jAaNfEydOpXBgwcTGxtLcnIy48ePZ9u2bcddM3HiRAzDOO5x/vnnN2poEREROXMz188k9+PJMC0Lnr8S3vyJ78dpWeR+PJmZ62f6NU+DyseiRYvIyclh+fLlLFiwgLq6OsaMGUNFRcVx111++eUUFBTUP95///1GDS0iIiJnzla0nRkHPiHXVn7c87m2CmYc+ARb0Xa/5mnQtMuHH3543K9nzZpFcnIya9asYcSIEfXPR0REkJqa2jgJRURE5Ox5PWRv+Ahs5cxITAAgu9RFbkIcMxLjySlxkl08Hy72+G0K5pzWfDidTgBatmx53PMLFy4kOTmZhIQERo4cyV/+8heSk5NP+nu43W7cbnf9r10u17lE+v6slbW8unofOw+V87fr+jbJ1xAREQk4e5eCK5/so7+ckZhAbkI8HsMgp6SU7FIX4PRdlzncL5HO+m4X0zSZMmUKw4YNIysrq/75sWPH8vLLL/Ppp5/y2GOPsWrVKi6++OLjCsa3TZ06lfj4+PpHRkbG2UY6pRqPl//3wVZeW72fvCOVTfI1REREAk75wfqfjqisAsBjGISZ5tHiceJ1Te2sy8fdd9/Nhg0beOWVV457/sYbb+SKK64gKyuLcePG8cEHH7B9+3bee++9k/4+999/P06ns/6Rl5d3tpFOqXVsBOd3bAXAexsLmuRriIiIBJyYlPqf/qWVb6bCME1qDYPchLiTXtfUzmraZdKkScybN4/FixfTtm3bU16blpZG+/bt2bFjx0k/HhERQURExNnEaLAr+qSx9Oti3t2QT/bITn75miIiIpZqPxTi2jDDVsGGSN/77YyDh9kcEX50DYhBtjfGd52fNGjkwzRN7r77bt566y0+/fRTMjMzT/s5xcXF5OXlkZaWdtYhG8vYrDTsNoNNB1zsKao4/SeIiIgEO5ud3D6XkZsYD0DrujouqKomu9RFTomTGYnx5PYZ49f9PhpUPnJycnjppZeYPXs2sbGxFBYWUlhYSFWVbw6pvLyce++9l2XLlrFnzx4WLlzIuHHjSEpKYsKECU3yB2iIli3CGdrJN/Xy7oZ8i9OIiIj4hzepKxm1BgBXllfWT3tke2PISR+NN6mrX/MYpmmaZ3yxYZz0+VmzZjFx4kSqqqoYP34869ato7S0lLS0NC666CL+9Kc/nfFCUpfLRXx8PE6nk7i4uNN/QgO9tiqP+97cQPfUWD6cPOL0nyAiIhLkiot3cMk7E6gzDP4adwM/6Nyj0Xc4bcj7d4PWfJyup0RFRfHRRx815Lf0uzG9UnjgbYOthWXsPFRO5+QYqyOJiIg0qVcW/pU6w6BjNVx8ywMQ5v8t1b+t2R0slxAdzvAuSYCmXkREpHn4pGQlAN1sfYm0uHhAMywfAFf2aQPAuxsKTjuaIyIiEsy2bn+PnWEmDtNkRN9fWh0HaKbl49JeKYTbbew8VM62g2VWxxEREWky/1vlOzSuR2Uklw4YaHEan2ZZPuIiwxjRtTUA723QhmMiIhKaamsr+cy9B4AOkRcR4bB+ygWaafkAGNfXt++Ipl5ERCRUfb76SY7YbSR4vFx8Xo7Vceo12/IxukcKEQ4bu4sq2JzfNIfZiYiIWOnNbW8B0Lm8NSO6t7M4zTeabfmIiXBwcXffSbvvaupFRERCjLN0D0vxnT6fnnA94Y7AecsPnCQWOHbXyztf5mvqRUREQsp7y/9BrWHQwQ1jzr/B6jjHadbl4+LuybQIt3OgtIq1+0qtjiMiItJo5uZ/DkDrii71R4sEimZdPqLC7Vza03eE8DtfasMxEREJDbt2f8JXdg8O06Rd+k9x2APr7T6w0ljgqn6+qZf3Nhbg8WrqRUREgt/ba2YA0K0igivOO9/iNCdq9uVjWOfWxEeFcbjMzYpdxVbHEREROSeeuhrec20HIKJ6MAPbJVqc6ETNvnyEO2z8oHcqAPM09SIiIkFu+bqnOWw3iPd46dr1Z9hsJz+R3krNvnwAjOvrm3r5YFMhNXVei9OIiIicvTnbXgcgs6wlVw/oZHGak1P5AIZktiI5NgJnVS1Ldhy2Oo6IiMhZKXMd4LNa3xICb91l9GoTZ3Gik1P5AOw2gyv6+LZb110vIiISrD5a8Rhum0FGjcmArOswjMCbcgGVj3rHpl7mf3WQqhqPxWlEREQabs7+hQC0cnbg6v7p1oY5BZWPo/pnJJDRMorKGg+fbD1odRwREZEG2bt3CV/aarGZJtWOm+icHGt1pO+l8nGUYRiM+9Z26yIiIsFk3trpAHSvDOPifhdYnObUVD6+5djUy2fbDuOqrrU4jYiIyJnxeup4p2QzAKazP1ceXccYqFQ+vqV7aixdkmOoqfPy4aZCq+OIiIickVVfPkuB3SDG48WMu4mMltFWRzollY9vMQyDq49utz5vvaZeREQkOMzd8j8AOpcncGX/rhanOT2Vj++4up9vdfDSr4s45Kq2OI2IiMipVZQXsqDmEADFzlH1W0cEMpWP78hoGc2Adgl4TXhnQ4HVcURERE5p/orHqbYZtKkxSWozjqSYCKsjnZbKx0mMP3pv9Nz1ByxOIiIicmrz8j4FINnZngkD2lqc5syofJzED3qnYbcZbNjvZNfhcqvjiIiInNT+/ctZbbgxTJM95Vdwac9UqyOdEZWPk0iKiWBY5yQA5mrhqYiIBKh3Vj8BQLeqMPp1u4CYCIfFic6Mysf3GN//6F0vX+ZjmqbFaURERI7n9dQx98hGAGylWVx9dK+qYKDy8T0u7ZlKZJiN3UUVbNjvtDqOiIjIcdZufIkDdoj2etlTO54RXVtbHemMqXx8j5gIR/3cmaZeREQk0Mzb8hIAXcriGd27K+GO4HlLD56kFjg2hPXOhnw8Xk29iIhIYKisLOKjat9O3CWlwxjfL3BPsD0ZlY9TGNG1NQnRYRwuc7Ps62Kr44iIiADwyYppVNoMUmu9HHFcyqD2iVZHahCVj1MId9j4QW/fTnFztOeHiIgEiHn75gOQ5szg6v4Z2GyGxYkaRuXjNI4NZX24qZCqGo/FaUREpLkrLFjHCrMSgJ3OsfVnkgUTlY/TGNQ+kfSEKMrddXy85aDVcUREpJl7Z9U/MQ2DrpV24lr2p0danNWRGkzl4zRsNqN+z4856zT1IiIi1jG9XuYWrQMg3NWTCf2Da6HpMSofZ+DYf9xF2w9TXO62OI2IiDRXX371P/baIcrrZYPzyvqT2IONyscZ6JwcS+/0eOq8Ju/qpFsREbHIvE0vANC1PJZ+HTuSGh9pcaKzo/Jxho6NfrytqRcREbFAdVUJH1btB8BZOjTo9vb4NpWPMzSubxvsNoP1eaU66VZERPxu4aonKLMZJNd62VlzEZdnBccJtiej8nGGWsdGMLyL76TbOdpuXURE/GzO7vcBSHe14ZKebYmNDLM40dlT+WiAY1Mvc9Yd0Em3IiLiN4cObmKZWQHArtLLmdA/+Pb2+DaVjwa4tGcK0eF29h2pZO2+EqvjiIhIM/HuysfxGgadqwyqwvowvEvwnGB7MiofDRAd7qifY9PCUxER8QfT62Xe4dUARDl7Mq5vG8Lswf32HdzpLXBs6uXdDQXU1HktTiMiIqHuq21v87XdJMJrstF5BeODdGOxb1P5aKChnZJIiYugtLKWz7YdsjqOiIiEuDkb/gtAt4poWrfMoG/beIsTnTuVjway24z6HeXeWrvf4jQiIhLKatxlfFC5D4Dy0vOZ0D8dwwiuE2xPRuXjLFw7oC0An249RElFjcVpREQkVC1a9W+cNoOkOi8by0czYUDwT7mAysdZ6ZYaS682cdR6TN7ZoD0/RESkaczb9S4AGa5UzuuYTNvEaIsTNQ6Vj7N0bPTjzTWaehERkcZXVLSVJV4XAHtLL61/3wkFKh9n6ap+bXDYDL7c72TnoTKr44iISIh5f+XjeAyDTtUGh7z9GNs7zepIjUbl4ywlxUQwqptvk5e31mrPDxERaVzzDq4AoIWzK5dnpRIT4bA4UeNR+TgH1xwdAnt73QE8Xm23LiIijWPrtnlss3kJM002ll7JNSGy0PQYlY9zMLpHMnGRDgqc1SzfVWx1HBERCRFzNzwDQPfyKKJjMhjaKcniRI1L5eMcRDjsjOvrO9xHC09FRKQx1NZW8n75LgCqnIMZ3z8duy349/b4NpWPc3TtQN/UywebCqlw11mcRkREgt3nq2dyxGaQWOdlQ9mlXBtiUy6g8nHO+mckkJnUgqpaDx9sKrQ6joiIBLm5O98GoENZMr3atqZLSqzFiRpfg8rH1KlTGTx4MLGxsSQnJzN+/Hi2bdt23DWmafLwww/Tpk0boqKiGDVqFJs3b27U0IHEMIz6VqqpFxERORelJbtZ5HECsL90dEjt7fFtDSofixYtIicnh+XLl7NgwQLq6uoYM2YMFRUV9df87W9/4/HHH2f69OmsWrWK1NRULr30UsrKQncvjAkD2mIYsGxXMXlHKq2OIyIiQer9Ff+gzjDo4Ia82gH16wpDTYPKx4cffsjEiRPp1asXffv2ZdasWezbt481a9YAvlGPadOm8bvf/Y5rrrmGrKwsnn/+eSorK5k9e3aT/AECQXpCFEM7tQK054eIiJy9uQVfABDv7Mzo7im0bBFucaKmcU5rPpxO39BQy5YtAdi9ezeFhYWMGTOm/pqIiAhGjhzJ0qVLT/p7uN1uXC7XcY9gdP3ADADeWJuHV3t+iIhIA+3Y+SFf2Tw4TJNNpVdy/aDQnHKBcygfpmkyZcoUhg0bRlZWFgCFhb4FlykpKcddm5KSUv+x75o6dSrx8fH1j4yMjLONZKnLeqUSG+Eg70gVK3YfsTqOiIgEmXnrnwKge0UEjqj2jOza2uJETeesy8fdd9/Nhg0beOWVV074mGEcfz+yaZonPHfM/fffj9PprH/k5eWdbSRLRYXbubKvb9/9N7TwVEREGqCutpp3XTsAqHEO4poB6TjsoXtD6ln9ySZNmsS8efP47LPPaNv2m2Gh1NRUgBNGOQ4dOnTCaMgxERERxMXFHfcIVtcd3fPj/Y0FlGvPDxEROUPL1j1Nkd0g3uNlg2tM/ftJqGpQ+TBNk7vvvpu33nqLTz/9lMzMzOM+npmZSWpqKgsWLKh/rqamhkWLFjF06NDGSRzABrRLpOPRPT/e31BgdRwREQkSc7e/AUBHVxI926bQNQT39vi2BpWPnJwcXnrpJWbPnk1sbCyFhYUUFhZSVVUF+KZbJk+ezKOPPsrbb7/Npk2bmDhxItHR0dxyyy1N8gcIJIZh1O94qqkXERE5E07nPj6r860VLCi9mOsGBefax4ZoUPl48skncTqdjBo1irS0tPrHq6++Wn/Nfffdx+TJk7nrrrsYNGgQBw4cYP78+cTGhnaLO+baAW2xGbByzxH2FFWc/hNERKRZ+2j5Y9QYBu3cJnvrBnFVn9Dc2+PbGjztcrLHxIkT668xDIOHH36YgoICqqurWbRoUf3dMM1Banwkw7v4Vihr9ENERE5nbv4SABKdHRnTM4346DCLEzW90F1Ka6FjC4XeXLsfj/b8EBGR77F7z0I22GqxmyZbSsdxfTOYcgGVjyZxac8U4iIdFDirWfp1kdVxREQkQM1bMx2AbpXhhLXoyLDOSRYn8g+VjyYQGWbn6n6+w+ZeXRWc+5aIiEjT8tTV8I5zKwDe0v5cMyAdu+3ke2KFGpWPJnLD0aGz+ZsPUlJRY3EaEREJNCvWP8NBu0Gsx8uXZWPr3zeaA5WPJpKVHkePtDhqPF7mrtdhcyIicrx5214HoFNZS/p3SKdDUguLE/mPykcTMQyDG48eCvTq6v2YphaeioiIT3lZAZ/UHAbgUOkIbhzcfEY9QOWjSY3vn064w8aWAhebDgTnab0iItL45q94jGqbQdsak3zvMMZmpVkdya9UPppQQnQ4l/XynXfz6up9FqcREZFAMXf/ZwAkOdszrl9bosLtFifyL5WPJnbj0QVEc9fnU13rsTiNiIhYLS/vC9YaNdhMk62lV9S/TzQnKh9NbGinVrRNjKKsuo4PNumwORGR5m7e6mN7e4TRKimLPm3jLU7kfyofTcxmM7h+oK/VvrZK262LiDRnXk8d80o2AWA4+3DDoAwMo3ns7fFtKh9+cN2gthgGLNtVzN5iHTYnItJcrdnwPPl2aOH18lXFOCb0T7c6kiVUPvwgPSGqfsvc11dr9ENEpLmas2U2AF3K4hnRsyOJLcItTmQNlQ8/OXYP9+tr8qjzeC1OIyIi/lZZfogF7oMAFJWOaFY7mn6XyoefXNozhZYtwjnocrNo+2Gr44iIiJ99vHIaVTaDNjUmzrBLms0hciej8uEnEQ471xyd23tlpfb8EBFpbubumw9AsqsdNw5u32wOkTsZlQ8/uuk83xDbp1sPUeistjiNiIj4S37+alYabgzT5OvSsdwwuK3VkSyl8uFHnZNjOa9DS7wmvL46z+o4IiLiJ/NWTQOga5WDHp3OJy0+ytpAFlP58LNjox+vrs7D69VhcyIioc70eplX/CUADmcWN53XzuJE1lP58LMf9E4jLtLB/pIqPt9ZZHUcERFpYus2vkSeHaK9XvK913BRt9ZWR7KcyoefRYbZ6zeV0cJTEZHQN++rlwDoUhbHVQN74bDrrVevgAVuHuIbclvw1UEOl7ktTiMiIk2lqvIIH1XnA+B0Xli/51Nzp/Jhge6pcfTLSKDOa/LmWu14KiISqj5dOY1ym0FKrZeEtAlktIy2OlJAUPmwyM3HFp6uysM0tfBURCQUzd37IQBtXOncPKSjxWkCh8qHRa7s04YW4XZ2F1WwbFex1XFERKSRFRauZ7lZCcDh6qu4pEeKxYkCh8qHRVpEOLi6fuGp9vwQEQk1766ahmkYdK2yMaLfaMIdess9Rq+EhW45eq/3h5sKKCrXwlMRkVBher3MPbwGgHBnT27W3h7HUfmwUFZ6PH0zEqj1mLyxRgtPRURCxcYtr7PHDpFek6iWt9EhqYXVkQKKyofFfni0Dc9esU87noqIhIg5G58DoGt5C64/v5+lWQKRyofFruybRmykg31HKvnia+14KiIS7NzVTj6s8m0iaVYN45KeWmj6XSofFosOd3DtAN/phi8v146nIiLBbuGqJyiz2Whd52VAnzsI046mJ9ArEgBuObbj6ZaDHHRVW5xGRETOxVtfvwtAW1caN2lvj5NS+QgAXVNiGdwhEY/X5LVVuu1WRCRYHT60mRVUABAfcyNtE7Wj6cmofASIHw5pD/gOm/No4amISFCat+JxPIZB52qD8cOusjpOwFL5CBCXZ6WSGB1GvrOahdsOWR1HREQayPR6mXdoFQCtKnsxqluyxYkCl8pHgIgMs3PdQN/C09krtPBURCTYbNk+j10Ok3CvSe/ud2G3GVZHClgqHwHklqNTL59uO0TekUqL04iISEO8tuYpALpXRHPThYMtThPYVD4CSGZSC4Z1TsI0fWs/REQkONS6K/jY7fu+nR4xmpS4SIsTBTaVjwDzo/N9ox+vrsrDXeexOI2IiJyJBSv+hdNuo1Wdl6tGTLI6TsBT+Qgwl/RIJi0+kuKKGj7YWGh1HBEROQNv75gLQNeqNlzYNc3iNIFP5SPAOOy2+tNuX1y+1+I0IiJyOsXFO1ht9+3tMTDzdgxDC01PR+UjAN14XgYOm8GavSVszndaHUdERE7G64HdS3j93V9SZxh0rIabR99odaqgoPIRgJJjI7k8KxWAl3Tei4hIQJm5fia5H0+GaVnw/JV86t4FwLUVFcz+/DfMXD/T2oBBQOUjQN12QQcA5qw7gLOq1towIiJSz1a0nRkHPiHXVs62sDC2RITjME2O2D3MOPAJtqLtVkcMeCofAWpwh0S6pcRSVevhrbX7rY4jIiIAXg/ZGz4ip6SUGYkJ/DkpEYD2tbX8NyGOnBIn2Rvm+6Zk5HupfAQowzD40QW+225fXL4X09R5LyIiltu7FFz5ZJe6+EVJKesjfft5fB0eTk5JKdmlTnAd8F0n30vlI4BN6J9OTISDXYcrWPZ1sdVxRESk/GD9TzvW1tX/PMw0yS51nfQ6OZHKRwCLiXBwzYB0AJ5ftsfaMCIiAjEpAJjAX1smAGAzTWoNg9yEuBOuk5NT+Qhwtx7d8XTBVwfZX6LzXkRELNV+KMS14cGklhQ5HDi8XhbtO1C/BiQ3IR7i0n3XyfdS+QhwXVJiubBzK7ymbrsVEbGczU5un8uYFxsDwE1l5SR4vWSXusgpcTIjMZ7cPmPAZrc4aGBT+QgCtx+97fZ/q/ZRXasV1CIiVsqr821/4DBNbnOW1T+f7Y0hJ3003qSuVkULGg6rA8jpje6RQtvEKPaXVDFvfT43DM6wOpKISLNVdGAtOKBfZSypP5oD5Yd8azzaDyVbIx5nRCMfQcBuM+rXfjy3dI9uuxURsciu3YtYZveNdgzPvBsjcwT0vg4yh2uqpQFUPoLEjYMziAyz8VWBi9V7S6yOIyLSLOUufgTTMMiqDOMmneNy1lQ+gkRCdDjj+/luu31u6R5rw4iINEOHD23mY9O3f0dW/E1Eh2vlwtlS+Qgitw/tAMCHmwopdFZbG0ZEpJl55tMHqTUMOlcZ3HrZ3VbHCWoNLh+LFy9m3LhxtGnTBsMwmDNnznEfnzhxIoZhHPc4//zzGytvs9YjLY7zMlvi8Zq8vGKv1XFERJqNMtcB5lb5DozrZLuUdq2iLU4U3BpcPioqKujbty/Tp0//3msuv/xyCgoK6h/vv//+OYWUb0w8Ovoxe8U+3HW67VZExB9mf/Y7Kmw22taYjBt5r9Vxgl6DJ6zGjh3L2LFjT3lNREQEqampZx1Kvt+YnimkxUdS4KzmnS8LuG5gW6sjiYiENHe1k1eKV4PdIKNyICO66v3tXDXJmo+FCxeSnJxM165dufPOOzl06ND3Xut2u3G5XMc95Ps57DZuO7rp2Kwvduu2WxGRJjZv8R8pthu0rvMyfND/YRiG1ZGCXqOXj7Fjx/Lyyy/z6aef8thjj7Fq1Souvvhi3G73Sa+fOnUq8fHx9Y+MDG2gdTo3n+e77XZzvouVu49YHUdEJGR56mp4Nm8BAO2d3bh2cBeLE4WGRi8fN954I1dccQVZWVmMGzeODz74gO3bt/Pee++d9Pr7778fp9NZ/8jLy2vsSCEnITqcawf4plue/WK3xWlERELXJ8v+xn4HxHq8dOtyr26vbSRNfqttWloa7du3Z8eOHSf9eEREBHFxccc95PTuuLADAPO/Osi+Yp12KyLS2Eyvl2d2vAFAF2c6tw3vb3Gi0NHk5aO4uJi8vDzS0tKa+ks1K52TYxnZtTWmCc8v22N1HBGRkLNi3X/YYvcQ4TVp2WoSbRKirI4UMhpcPsrLy1m/fj3r168HYPfu3axfv559+/ZRXl7Ovffey7Jly9izZw8LFy5k3LhxJCUlMWHChMbO3uwdG/14dVUeZdW11oYREQkxz2x8BoAerkR+NHKYxWlCS4PLx+rVq+nfvz/9+/uGn6ZMmUL//v35wx/+gN1uZ+PGjVx99dV07dqV22+/na5du7Js2TJiY2MbPXxzN6JLazq1bkG5u4431uy3Oo6ISMj4auscVhjV2E0TbBMZ0C7B6kghpcErZ0aNGnXK2zs/+uijcwokZ85mM7jjwkwenLOJ55bu4bYLOmC36RYwEZFz9ezqxwHoVd6C64ZfrttrG5nOdgly1wxIJz4qjL3FlXy29fv3UxERkTOTl/cFC+p82xhUVYznB721ZrGxqXwEuehwBzef1w6AZz7fZXEaEZHgN+uLP+E1DHpWOBh93vWE2fVW2dj0ioaA24e2x2EzWL7rCJsOOK2OIyIStIqKtjK32reGrqbkEm45+o87aVwqHyEgLT6KK/r4hgWfWaLRDxGRszV78e+pMQw6VRv07PEjEluEWx0pJKl8hIg7h3cE4N0NBRQ4qyxOIyISfMrLCnjFuQUAR/EQfjqis8WJQpfKR4jISo/n/I4tqfOaPPfFHqvjiIgEnTcW/Z5ym0F6jUli+k/ITGphdaSQpfIRQo6NfsxeuY9yd53FaUREgkeNu4wXDi0HIPFIb+4c0dXiRKFN5SOEXNQtmY6tW1BWXcerq3RAn4jImXp3yR85bDdIqvPijv4xgzskWh0ppKl8hBCbzeCnw3yjH89+vps6j9fiRCIigc/rqWPWvvkApJd04o4RvbSpWBNT+Qgx1wxIp1WLcA6UVvHh5kKr44iIBLzPlv+DPXaTGI+XgrqJjM1KtTpSyFP5CDGRYXZ+dH57AP6zZPcpt8IXEWnuTK+XZ7a/CkAXZxtuvrAfDm0q1uT0CoegWy9oT7jDxpd5pazaU2J1HBGRgLX6y1lsstUR4TXZ7bqFGwdnWB2pWVD5CEFJMRFcO6AtAE8v/triNCIigeu/G54GoIcrgbGDzyc2MsziRM2DykeIunN4JoYBH285xI6DZVbHEREJONu2v8sXVGIzTfaVXMuPL8y0OlKzofIRojq2juGynr5FU08v1pbrIiLf9eyqfwCQVR7Neb0uJjU+0uJEzYfKRwj7+Ujfbbdz1h+g0FltcRoRkcCxf/9yPqwtAuBQ0RX8bERHixM1LyofIax/u0TOy2xJrcdk1he7rY4jIhIwnv/iEbyGQY9KB+0zf0CXlFirIzUrKh8hLvvo6MfLK/bhqq61OI2IiPWKi7bzduU+AKqKLuLnIztZnKj5UfkIcaO6JtM1JYZydx2zV+yzOo6IiOVmL/k9bptBx2pwtJzAoPbaSt3fVD5CnM1m8LMRvlb/7Oe7cdd5LE4kImKdyvJD/K90MwBhxUPIHtVFW6lbQOWjGbiqbxtS4yI5VOZm7rp8q+OIiFjmjcUP4rIZtKkxKQm7kUt7pFgdqVlS+WgGwh02fjLMd/967uKv8Xi15bqIND+17gpeKFwKQKsjWdw5shs2m0Y9rKDy0UzcPKQdcZEOdh2uYL4OnBORZui9zx/hoN2gVZ2XPbU/ZHz/dKsjNVsqH81ETISDiUM7ADBj4U4dOCcizYrXU8esvR8AkFGSyR0jehEZZrc4VfOl8tGMTLwwk6gwO5sOuFiyo8jqOCIifrNo5TR22U1aeL18XfFDbh7SzupIzZrKRzPSskU4N5/n+x9uxmc7LU4jIuIfptfLf7fOBqBraSrXDx1ITITD4lTNm8pHM3PniEzC7AYrdh9hzd4jVscREWlyaze+yJe2WsK9JrtcN9dPQYt1VD6ambT4KCYcXWQ187OvLU4jItL0nv0yF4CeZfFcNnAoiS3CLU4kKh/NUPbIThgGfLL1EFsKXFbHERFpMtt3fsBisxybabLvyLXcOVwHyAUClY9mqGPrGH6QlQbAkws1+iEioWvWir8B0Ks8igt6X0JqfKTFiQRUPpqtX4zybbn+7oZ89hZXWJxGRKTx5eev5oPawwAUF4/VAXIBROWjmcpKj2dk19Z4TY1+iEhoeuHzP+IxDLpV2unSZTyZSS2sjiRHqXw0Y5Mu7gzAm2v3c6C0yuI0IiKNp+TI17xZsRuAmuKR3H30+50EBpWPZmxQh5Zc0LEVtR6TXI1+iEgIeWXRg1TbDDq4ISXjh3RPjbM6knyLykczN2m0718Dr67O46Cr2uI0IiLnrrKyiNmlGwGIKh7MPZd0sziRfJfKRzN3QcdWDGqfSE2dl6cW7bI6jojIOXt70e9x2gzSak1iUn9KVnq81ZHkO1Q+mjnDMJg0ugsAs1fupajcbXEiEZGzV1tbyXMFSwBIOtKTSaN7WJxITkblQxjRJYm+beOprvXynyUa/RCR4PXh53+h0G6QWOclvGU2/dslWh1JTkLlQ3yjHxf7Rj9eXLaXkooaixOJiDSc6fXy7O53AGhf0oG7RvezNpB8L5UPAWB0j2R6psVRWePh2S92Wx1HRKTBlqz6FzvtJtFeL7T4OedltrQ6knwPlQ8Bjo1++O58ee6LPTgray1OJCLSMP/56kUAupWmkH3pUIvTyKmofEi9y3ql0j01ljJ3Hf/9XGs/RCR4rN/4MutttYSZJkb4TxjaqZXVkeQUVD6kns1mcM/RO1+e/WIPpZVa+yEiweE/a6cD0NMVx88uuwzDMCxOJKei8iHHOTb6Ue6u45klWvshIoHv668XsJhyDNMk0v4jjXoEAZUPOY7NZjD5kq4AzPpit+58EZGA959ljwLQqyKSH192g0Y9goDKh5xgTM8UeqTFUVHj0b4fIhLQCgvW8VHdYQDiuZYLNOoRFFQ+5AS+0Q/f2o/nl+7hiEY/RCRA/XfRH6gzDLpW2rn98p9p1CNIqHzISY3pmUKvNhr9EJHA5Szdw9wq39q0NuYYjXoEEZUPOSnD+Gbtx/NL91CsM19EJMA898n9VNkMOrjhljG/sTqONIDKh3yvS3ok0zs9nsoaD08t1uiHiASOqsojvOHcAEDH2qFc0Lm1xYmkIVQ+5HsZhsGvLvWt/Xhh2R4OuaotTiQi4vPyxw9QareRUuvl5ssftjqONJDKh5zSRd2SGdAugepaL9M/22l1HBER6mqrefXw5wBk1fbj/E5pFieShlL5kFMyDIN7L+sGwCsr95F3pNLiRCLSbHk9sHsJb71xO4UOgwSPl9vGTLU6lZwFlQ85raGdkriwcytqPSZPfLLD6jgi0szMXD+T3I8nw7QszOev5LWy9QDc6Kpi5e7Hmbl+pqX5pOFUPuSM/HqMb/TjzbX72XW43OI0ItKc2Iq2M+PAJ+TayvkiKpJtEeFEe714jDpmHPgEW9F2qyNKA6l8yBkZ0C6R0d2T8Zrwz481+iEifuL1kL3hI3JKSpmRmMCfWiUC0LWmhmcS4skpcZK9Yb5vSkaCRoPLx+LFixk3bhxt2rTBMAzmzJlz3MdN0+Thhx+mTZs2REVFMWrUKDZv3txYecVCU8b49v1458t8thS4LE4jIs3C3qXgyie71MU1rnLyw8LANFkfGUlOSSnZpU5wHfBdJ0GjweWjoqKCvn37Mn369JN+/G9/+xuPP/4406dPZ9WqVaSmpnLppZdSVlZ2zmHFWr3axHNFH9+q8sfma5hTRPyg/CAAHmBLRLjvOcMgzDTJLnWdcJ0EhwaXj7Fjx/LnP/+Za6655oSPmabJtGnT+N3vfsc111xDVlYWzz//PJWVlcyePbtRAou1fnVJV2wGfLzlIGv2llgdR0RCXUwKAHNiWtSXjzDTpNYwyE2IO+E6CQ6NuuZj9+7dFBYWMmbMmPrnIiIiGDlyJEuXnnxIzO1243K5jntI4OqcHMN1A9sC8NcPt2KapsWJRCSktR+KMy6N/3d0rceFlVWs3ZNXvwYkNyEe4tKh/VCLg0pDNGr5KCwsBCAl5fgGmpKSUv+x75o6dSrx8fH1j4yMjMaMJE1g8iVdCXfYWLn7CAu3HbY6joiEMpudOxNiqLbZSKzz8O+Dvu852aUuckqczEiMJ7fPGLDZLQ4qDdEkd7t890hj0zS/95jj+++/H6fTWf/Iy8trikjSiNokRDFxaAfAN/rh9Wr0Q0SaxradC9hq8x3t8Mcj1YR962PZ3hhy0kfjTepqTTg5a47G/M1SU1MB3whIWto3290eOnTohNGQYyIiIoiIiGjMGOIHd43qxCsr97G1sIy5Xx5gQv+2VkcSkRBjer08uuh+TIdB78oIhkxaDQUrfYtLY1Kg/VCyNeIRlBp15CMzM5PU1FQWLFhQ/1xNTQ2LFi1i6FDNx4WShOhwfjGqE+C788Vdp3vsRaRxzV/6D9Y63ISZJqM7/Y7oyAjIHA69r/P9qOIRtBpcPsrLy1m/fj3r168HfItM169fz759+zAMg8mTJ/Poo4/y9ttvs2nTJiZOnEh0dDS33HJLY2cXi90xNJOUuAj2l1Qxe8U+q+OISAipcZfxz20vADCwLI3bRl9tcSJpTA0uH6tXr6Z///70798fgClTptC/f3/+8Ic/AHDfffcxefJk7rrrLgYNGsSBAweYP38+sbGxjZtcLBcVbuee0b651n9/upOy6lqLE4lIqPjv+zkccBi0qvPyg/MeI8yuDblDiWEG2L2SLpeL+Ph4nE4ncXFxp/8EsVSdx8uYfy5mV1EFv7y4M1OOngEjInK2Dh3cxJXv30SVzWC4azDTc/6LzXbymxYkcDTk/VtVUs6Jw27jN5f5Csd/luzmoKva4kQiEuz+Pv9uqmwGnaoNbhrzqIpHCFL5kHN2eVYqA9snUlXr4XFtuy4i52D9ptl86C0GoB0/YkS3VIsTSVNQ+ZBzZhgGD/ygBwCvrcnToXMicla8njoeXfk3APq5Ysm5+hcWJ5KmovIhjWJg+0Su6J2GacLUD7ZaHUdEgtCcz+5ni91DC6+X9kn/R7dU3agQqlQ+pNHcd3k3wuwGi7cfZvF2bbsuImeuzHWAf+37AIBuRzrzyyvGnOYzJJipfEijad+qBbee3wGAR9/fgkfbrovIGXryo7s4YjdIrzHp2+sRkuMirY4kTUjlQxrVpIs7ExfpYGthGW+u3W91HBEJArv3LOSViq8BSCodzc8v6mVxImlqKh/SqBJbhDPp4i4APDZ/G5U1dRYnEpFA99dFv6XOMOhZEcaVI39Ni4hGPXZMApDKhzS624a2J6NlFAddbnIX7bI6jogEsMUr/skXVOIwTQz3T7h+UIbVkcQPVD6k0UU47Nw/1nfr7VOLvuZAaZXFiUQkENW4y/jr5mcB6F2STM6VN2DXhmLNgsqHNImxWamcl9kSd52Xv+rWWxE5iZfm38M+OyTWeYlO+A3Du7S2OpL4icqHNAnDMPjDlT0xDJj3ZT5r9h6xOpKIBJDDhzbzVNFKANoW9eO+K4dbnEj8SeVDmkxWejw3DPTN3z7yzld4deutiBw1bcEvqbQZdKw26NHjt3RsHWN1JPEjlQ9pUvde1o2YCAdf7nfy9roDVscRkQCwYfOrzKs7BICjZDy/vLSHxYnE31Q+pEm1jo0g56LOAPzto61UuHXrrUhz5vXUMXXl/wOgr6sFE0b9hPioMItTib+pfEiT+/GwDrRrGc1Bl5snF35tdRwRsdA7ix5kk62OaK8Xd10ONw3WrbXNkcqHNLkIh73+1Nunl+xib3GFxYlExAoV5YVM2/MuAF2LM5k8biwOu96GmiP9Vxe/uKxXCsO7JFFT5+VP735ldRwRscBTH95Fkd0grcZLfPJ9XNg5yepIYhGVD/ELwzB4aFwvHDaDj7cc4tOtB62OJCJ+tHfvEl4s3w5AQtFI7r9qsMWJxEoqH+I3nZNj+PGwTAD++M5XVNd6LE4kIv7yt4X3UWcY9KhwMGzQPaQnRFkdSSyk8iF+NeniziTHRrC3uJL/fr7b6jgi4gdLVj7BYspxmCaeion8bFRnqyOJxVQ+xK9iI8PqF59O/3Qn+Tr3RSSk1bor+Oum/wDQu7Q1d115E5FhdotTidVUPsTvru7XhsEdEqmq9fCX97dYHUdEmtDLC+5h79HzW6LifsPoHilWR5IAoPIhfmcYBn+8KgubAe9tKODzHUVWRxKRJlBUtJWnDi8DIKO4D/dfNdLiRBIoVD7EEj3bxHHr+e0B+P3cTVp8KhKCps2/m3Kbjcxq6Nf7QToktbA6kgQIlQ+xzK8v60br2Ah2F1Xw1KJdVscRkUa0ecubzKspBCCmbAKTLulucSIJJCofYpm4yDB+f2VPAGYs3MmeIu18KhIKTK+Xvyz7M6Zh0McVzZ1X3q1FpnIclQ+x1Lg+aQzr7Nv59PdzN2GaptWRROQcvbPo92y01xHl9dI69tdc1D3Z6kgSYFQ+xFKGYfDI1b0It9tYsqOI9zYWWB1JRM5BRXkh/9w9F4CeJR347YRxFieSQKTyIZbr2DqGX4zqBMAj73xFWXWtxYlE5Gw9+f7R81tqTS7s/yhp8drJVE6k8iEB4RejOtGhVTSHytw8Nn+71XFE5Czs2/c5syt9//92qLiYO0b0tjiRBCqVDwkIkWF2Hrk6C4Dnl+1h3b4SixOJSEM9+sm91BoGPSod/Oyqh3DY9RYjJ6e/GRIwRnRtzYT+6Zgm3P/WRmo9XqsjicgZWrR8Ol/YKnCYJr0SJjEos5XVkSSAqXxIQHnwih4kRoextbCMpxdr7w+RYFBbW8nfNz0FQP+y1kyZcKvFiSTQqXxIQGkVE8Efxvn2/vjXJzvYdbjc4kQiclJeD+xeAhvf4L+v3cLeMEjweLl26D+JjQyzOp0EOIfVAUS+a3y/dN5el8/i7Ye5/62NvHLn+dhshtWxRASYuX4mtqLtZG/4CFz5HLHZeKFtG7DbyKhLYK99KdDP6pgS4DTyIQHHMAz+Mj6LqDA7K3Yf4dXVeVZHEpGjbEXbmXHgE3JtvlHJJxITKLPbSKqrY2OEC1uR7laT01P5kICU0TKaX4/pCsCj72/hkKva4kQigtdD9oaPyCkpZUZiAn9qlchbsb7D4oocDnJKnGRvmO+bkhE5BZUPCVh3XJhJ37bxlFXX8cDb2npdxHJ7l4Irn+xSF3eVlPJaXCym4ZsSzSkpJbvUCa4DvutETkHlQwKW3Wbw1+v6EGY3+HjLQeZ9mW91JJHmrfxg/U+jvd/8YyDMNMkudZ30OpGTUfmQgNY9NY5fXtwFgIfmbeZQmaZfRCwTkwLA5vAwHm+ZAIDdNKk1DHIT4k64TuT7qHxIwMse1YlebeIorazlQU2/iFin/VDK49L4eWoyXsOgY00N6/bk1a8ByU2Ih7h0aD/U6qQS4FQ+JOCF2W384/q+OGwG8786yDsbdPKtiCVsdn4UH4XTbifG4+GFgkMYQHapi5wSJzMS48ntMwZsdquTSoBT+ZCg0CMtjknHpl/mbuJwmdviRCLNz/8++i1fO+owTJPHit3Ee785AiHbG0NO+mi8SV0tTCjBQpuMSdC466JOfLi5kC0FLn4/ZxNP/mgAhqHNx0T8Ye/eL/hn/rtgszG8sgPn/2oO5C3zLS6NSYH2Q8nWiIecIY18SNDwTb/0wWEz+HBzIW+vO2B1JJFmodZdwa8/votKm42uVTZ+de2L2BwOyBwOva/z/ajiIQ2g8iFBpVebeO4ZfWz6ZTMHSqssTiQS+v7+9i1sc3iJ9XgZ23EqnVMSrY4kQU7lQ4LOL0Z1on+7BMrcddz72pd4vbr7RaSpLF75b15x+06YPs99CT++ZKzFiSQUqHxI0HHYbTx+Qz+iwuws21XMrKV7rI4kEpKKirby4KZcAAa6Evm/W/6qQx6lUah8SFDKTGrB767oAcBfP9zK9oNlFicSCS1eTx3/985tlNhttHObXH3BU6TGR1odS0KEyocErR8Oaceobq2pqfPyq1fXU1PnPf0nicgZmfX+z1lhqyLCa9IrfBITBvWwOpKEEJUPCVqGYfC3a/uQEB3G5nwX//xYR3mLNIZNX73O9OIVAPQtyeKBG+6wOJGEGpUPCWrJcZFMndAbgNxFX7N0Z5HFiUSCW0V5Ib9Z9gh1hkFWeQS3X/EvEqLDrY4lIUblQ4Le2N5p3DQ4A9OEX722niMVNVZHEglaf3z7JvY7ILnWS5/0vzGimw6Jk8an8iEh4Q/jetKpdQsOutzc98YGHT4nchbmfvoAH3iLsZsmmZXX8+srR1kdSUKUyoeEhOhwB0/c3J9wu42PtxzkpeV7rY4kElT27l3Co3vnAtDvSAYP3Hwv4Q69RUjTaPS/WQ8//DCGYRz3SE1NbewvI3KCXm3i+e3Y7gD8+b0tbCvU7bciZ6LWXcG9n0yq3z597LDpdGwdY3UsCWFNUmt79epFQUFB/WPjxo1N8WVETvDjCzswqltr3HVeJr2ylqoaj9WRRALe43NvYavdQ5zHS4eoB7hhcEerI0mIa5Ly4XA4SE1NrX+0bt26Kb6MyAkMw+Af1/clKSaC7QfLeWjeJqsjiQS0xSv+xUtVvu3Tu5RcxB9uuEanRUuTa5LysWPHDtq0aUNmZiY33XQTu3bt+t5r3W43LpfruIfIuUiKieCJm/phM+C11ft5c81+qyOJBKSiw1t4cPPTAPQvTWDyDX8mPirM4lTSHDR6+RgyZAgvvPACH330Ef/5z38oLCxk6NChFBcXn/T6qVOnEh8fX//IyMho7EjSDA3tnMQ9o7sC8OCcTezQ9usix/F66rjvXd/26e3dMLLPDPplJFgdS5oJw2ziexIrKiro1KkT9913H1OmTDnh4263G7fbXf9rl8tFRkYGTqeTuLi4powmIc7jNbn92ZV8vrOIzskxzLv7QqLDHVbHEgkIT82dyPTSNUR4TUZ4J/HYj3+m6RY5Jy6Xi/j4+DN6/27y+6hatGhB79692bFjx0k/HhERQVxc3HEPkcZgtxlMu6kfybER7DxUzoNvb9L+HyLAhs2vkVuyGoB+zr48dMuPVTzEr5q8fLjdbrZs2UJaWlpTfymREyTFRPDvm/tjM+CtdQd4dVWe1ZFELFVeVsBvlvu2T+9dHsk9107XOg/xu0YvH/feey+LFi1i9+7drFixguuuuw6Xy8Xtt9/e2F9K5IwM6diKX4/pBsAf5m5mfV6ptYFELGJ6vTz45o3kOwySa72M6fkEvTMSrY4lzVCjl4/9+/dz8803061bN6655hrCw8NZvnw57du3b+wvJXLGfjGyE5f2TKHG4+UXL62hqNx9+k8SCTGvzP8tnxgl2E2TIfbbuH3U+VZHkmaqyRecNlRDFqyINERZdS1XT/+CXUUVnN+xJS/9ZAgOu7aPlhDm9cDepVB+kB1lBfxwWy5VNhvDXJn8M/ttIsPsVieUENKQ928t/ZdmIzYyjKduHcj4GV+wfNcR/vrhVn53RU+rY4k0upnrZ2Ir2k72ho/AlU8N8Ls2qVRFhJNca5IxYJSKh1hK/+yTZqVLSiz/uL4vAP9Zspt3vsy3OJFI47MVbWfGgU/ItZUDMK1lAlsiwon0ejkUZtCyap/FCaW5U/mQZmds7zR+MaoTAPe9sYHN+U6LE4k0Iq+H7A0fkVNSyozEBO5PasmL8b4h8GqbjZwSJ9kb5vumZEQsovIhzdK9Y7oxvEsSVbUe7nx+NYfLtABVQsTepeDKJ7vUxW1OF+/GfnM6bU5JKdmlTnAd8F0nYhGVD2mW7DaD6TcPoGNSC/Kd1fz8xdVU1+pfghICyg8CcMRmY3FUVP3TYaZJdqnrhOtErKDyIc1WfHQYz9w+iLhIB2v3lfLAWxu1A6oEv5gUKg2DnNTW7An3bR4WZprUGga5CXHHXSdiFZUPadY6to5h5g8HYrcZvLXuAE8t/v4TmEWCQW1qX+5JSWZTRAQAtzhdrN2TV78GJDchHuLSof1Qi5NKc6byIc3esC5JPDTOd8vtXz/cyoKvNBwtwcnrqeNXr/yA5VG+4nGdq4z7j5QCkF3qIqfEyYzEeHL7jAGbbrUV66h8iAC3XdCBH53fDtOEX76yjg37S62OJNIgptfLQ6+MY5HdiWGaDPGk8VBt1HHXZHtjyEkfjTepq0UpRXy0yZjIUQ+N68W+I1Us3n6YHz+3irfvupCMltFWxxI5I0+8dStzPPsBuKxmGH/96ZOAt36HU2JSoP1QsjXiIQFAIx8iR4XZbcz84QB6psVRVF7DxFkrKa2ssTqWyGm99P5knqnYAMDI8u786Y4Z2GyGb2olczj0vs73o4qHBAiVD5FviYlwMOuOwaTFR/L14Qp+9uIa3HW6BVcC14dLpvL3Qx8DcL4rlUcmvqyt0yXgqXyIfEdKXCSz7hhMbISDlbuPcO/rG/B6dQuuBJ4V657ldztfxmsYDCiL5YEb36Jli3CrY4mclsqHyEl0T40j99aBOGwG73yZz5/e+0p7gEhA2bbjPSavf4wam0FWRTh3X/EWmcmxVscSOSMqHyLf48LOSfWH0M36Yg/TP91pcSIRn7z9y/j5kvsot9noUmXj1qGzGZyZanUskTOm8iFyCuP7p9fvAfLYgu28uHyvxYmkuSsu2s7PPrqTYruNDLfJdb2e4gf9ulkdS6RBVD5ETuOOCzP55cWdAfjD3E2882W+xYmkuaooL+Rnc69nv8MgudbLVe3+wS3Dz7c6lkiDqXyInIFfXdq1fhOyKa+tZ9H2w1ZHkmamxl3GXa9eyXaHl3iPlzEJD5B9+eVWxxI5KyofImfAMAz+eFUWV/ZJo9Zj8rMXVrP06yKrY0kz4amrYcrsy1nrcBPl9XKx46fcd+0tVscSOWsqHyJnyG4zePyGflzcPRl3nZefPLealbuPWB1LQo3XA7uXwMY3YPcSzLpafj97HItsLhymyUWeCTz0o8kYhmF1UpGzZpgBdv+gy+UiPj4ep9NJXFzc6T9BxM+qaz3c+cJqluwookW4nRd/OoQB7RKtjiVBbub6mdiKtpO94SNwfbOu6MmWScyMjwbT5HL3CB796XTC7Pp3owSehrx/62+wSANFhtn5z22DGNqpFRU1Hm5/dqUOopNzZivazowDn5BrK69/7vXYFr7iAfSqieUvP1HxkNCgv8UiZyEyzM4ztw/ivA4tKauu49b/rmTjfqfVsSRYeT1kb/iInJJSZiQmkJsQx8fRUfypVUsABlZV80q5k3BbQA1Ui5w1lQ+RsxQd7uDZOwYzoF0CzqpabnlmOWv2llgdS4LR3qXgyie71FVfQH6VnIRpGPSsdjOr8BCG64DvOpEQoPIhcg5iIhw8/+PzvjUCsoJlXxdbHUuCTfnB+p+2r60D0wTDwDBNXi44iHGS60SCmcqHyDmKjQzjuR8PZljnJCprPEyctVL7gEjDxKRgArPiY7kvOam+eJiGwTMJccddJxIKVD5EGkF0uINnbh9Ufxvunc+vZsFX+leqnBlP+iD+ktSax1v67prqU13Nuj1531oDEg9x6dB+qMVJRRqHyodII4kMs5P7o4GMzUqlxuMl+6U1vLY6z+pYEuCqKo+Q/eIoXo2NAuDCyipeLjiEHY6uAXEyIzGe3D5jwGa3NqxII1H5EGlE4Q4b/765P9cOaIvHa3LfGxuYuXAnAbadjgSI4uId3Db7IpY7KrGZJgNrW/FkVdhx12R7Y8hJH403qatFKUUan8PqACKhxmG38Y/r+9A6NoLcRV/ztw+3ccjl5g9X9sRm066U4rNrzxLu+uQXHAgziPN4uS7ux0y+dgqG6fXd1VJ+0LfGo/1QsjXiISFG5UOkCRiGwf+N7U7r2Aj+9O5XPLd0D0Xlbh67oS8RDr2RNHervnyZKWsepdRhI6XWy63tHuL2MTf4PmjYIXO4tQFFmpjKh0gT+smwTFrHRvDr19bz7oYCDrnc5N46kJYtwq2OJhaZ99lfeGTPK7jtNjLdcGefpxh3/jCrY4n4ldZ8iDSxq/q2YdbE84iNcLByzxHGz/iCnYfKrI4lTe07B8Th9fDk2z/jwb2v4LYZ9KoM4/6Rb6p4SLOkg+VE/GTHwTJ+/Pwq8o5UERvpYOYPBzC8S2urY0kjO9kBcV7gsVZJvBDnO6flvPIEHrxuLpkpLS1MKtK4dLCcSADqkhLLnLsuZHCHRMqq65g4axUvLtujO2FCzHcPiHMb8NvWreqLR3d3FI/f8bGKhzRrWvMh4ketYiJ46adDuP+tjby19gC/n7uZDfud/Gl8FpFhWoga9I4eEIetnBmJCbgNg3UREayJigTgkvIKHnd7MCL1rVeaN418iPhZhMPOY9f35f/GdsdmwOtr9nNd7lLyjlRaHU3O1bcOiPuh08UzCfH1xePqsnL+ebhYB8SJoPIhYgnDMMge2YkXfzKEli3C2XTAxbjpn7Nw2yGro8m5KD+ICbwe24K3YmPqn3aYJn8uOnLcdSLNmcqHiIUu7JzEO5OG0bdtPKWVtdzx3Coen7+NOo/X6mhyFgrr3PwyOYlHklpRZfN9ew0zTeoMg1wdECdST+VDxGLpCVG8ln0BN5/XDtOEJz7dyY1PL9c0TJCZ8+lfuWHdoyxsEY3t6CLiu0pKWasD4kROoPIhEgAiHHamXtObJ27uT2yEgzV7S/jBE0t458t8q6PJaVRWFvObWZfw+7yXKLHbSKjz4jUMckqc/KLUBeiAOJHv0pJrkQByVd829M9I4J7/rWPtvlImvbKORdsP84dxPYmLDDv9byBNx+s54cyVlRvf4uFVfyQvzHdmz7DKFNr1GUNiVT7ZxR8BzvpPz/bGgA6IEwG0yZhIQKrzeHnikx1M/2wnXhPS4iN59JreXNQt2epozc7JNg3zAM8kJjEzPgqvYdCqzstVsbdwz3UPYD92eOBJyopGPCSUNeT9WyMfIgHIYbcxZUw3hndtzW9e/5I9xZXcMWsV1w1sy++v6El8tEZB/OXYpmHYyskGDjjsPNC6FWsjfbfQtq0xeGjY/zi/R+/vfKIOiBP5Phr5EAlwVTUe/jF/G89+sRvThOTYCB65OovLeqVgGIbV8UKb1wPTssg9umnY6IpKlkdFUnH0TpbRFZU8Xh2BbfJGjWpIs6ft1UVCSFS4nd9f2ZPXf34BHZNacKjMTfZLa7jjuVXsKaqwOl5oO7pp2LDKatJqa/mkRXR98bjV6WLaoSJs2jRMpMFUPkSCxKAOLXn/nuHkXNSJMLvBwm2HGfPPxTw+fxvVtR6r4wWvk5w+e8yevcu4v3Urbk5PpSAsDI4OFIeZJvcdKf3m99CmYSINomkXkSC063A5D83bzJIdRQC0TYziN5d1Y1yfNthsmoo5EydbSApAXBue6DGKJYc2s9tbjPvoSEc3dw3bIsIJM01qDYOcklKyj95Ky+3van2HNHuadhEJcR1bx/DCj8/jyR8OIC0+kv0lVdzzv/VcNeNzPj9aSOTUvnv6LPjuYvllZA3/OfQ5WynBbbPRv9rN9a4ytkWEk6NNw0Qahe52EQlShmEwtncaI7u15tnPd5O7aBebDrj40X9XMLxLEvdd1p3ebeOtjhkYvnvba8aQ406fBehX7eb/Wrei2OH7tphRW8foFuOITK0mt/Azckqc9SMdvh8NZiTGQ/posrXYVKRBNO0iEiKKy91M/2wnLy3fS63H97/1yK6tufvizgzu0NLidNb43qmV6CRyw2vwYLDfYefdbx0CF+718ssSJze7ygi//V1mOjd+z/RMOrl9xuBN6spd/e7y459KJDA15P1b5UMkxOwrrmTax9uZ+2U+Hq/vf+/zMlty90WdGd4lqVndnpv78WRmHPjk+PUZwPSEOJ5KTCCxro4SxzcDwDbTZOG+AyR6jx7sd+1/ofd1vp9r0zCRU1L5EBH2FVfy5KKveWNNXv1ISJfkGG67oD0TBrQlJiIEZ12/XRBatIa3f06uvYIZiQnklJRyk6uce5NbsSIqqv5Twrxeam02LSQVOUcqHyJSr8BZxdOLd/Hqqjwqa3y3kcZEOLhuYFtuGdKOrimxFic8d987vQLkJsTxWXQUX0VE+G6VPTryk1JXR6eaWpZGR9UXjtyEuKNFxek7i0Wbh4mcMW2vLiL10uKjeGhcL351aVfeWrOfF5btZVdRBc8t3cNzS/eQlR7H+H7pXNWvDcmxkVbHPSvf3QIdIM9h5+Gklqz81igHhoFhmjx6uJh9YQ6ePDoiooWkIv6lkQ+RZsbrNfni6yJeWLaXz7Yeou7ouhCbAcO6tOayXimM7p5CanyQFJHvbIE+sqISl93Gushv8ju8XupsNhymSd3RqRUvBjZMsmvCoLL4m99PC0lFzkpATLvMnDmTv//97xQUFNCrVy+mTZvG8OGnnztV+RDxnyMVNby3IZ+31h1g3b7S4z6WlR7H6O4pjOrWmqz0eMLsgbctUE2dl92rPqTbRzcDcGdKa5ZHfzPSkVFbS9ca37bo3zu18sv1kLdCC0lFzpHl5ePVV1/l1ltvZebMmVx44YU89dRTPPPMM3z11Ve0a9fulJ+r8iFijT1FFby3sYCPtxxkfV4p3/7OEB1uZ0C7RM7LbMngDi3plR5HXKT/T9Z1Vtay8YCTlXuOsGr3EdbllTDG8zlPhE8HfCfOjm3bBtMwCDNNflbqrF9s+u27XXIT4pmRGE9O+miyL5nm9z+HSCiyvHwMGTKEAQMG8OSTT9Y/16NHD8aPH8/UqVNP+bkqHyLWKyp38+nWQ3yy5SDLdx3BWVV7wjVtE6PonhpHj7RYOifH0DYxivSEaFrHRmA/hy3ePV6Tg65qDpRWcaCkiu0Hy9haWMbWAhf5zuoTrr80ejv/8T4MUD+qcezOlcFV1ZxXXX1c8QA0tSLSBCwtHzU1NURHR/P6668zYcKE+ufvuece1q9fz6JFi075+SofIoHF6zXZcaiclbuLWbH7CGv3lpy0BBwTZjdIjY8kMTqcuMgwYiMdxEWGERVu59tbjJgmVNbUUVZdh6u6FldVHUcqaih0VdfvT3IybROjGNQ+kfMyW3FeZiKdWkVh/Ks3ubYK32jG902vXD0TKos0tSLSRCy926WoqAiPx0NKSspxz6ekpFBYWHjC9W63G7fbXf9rl8t1wjUiYh2bzaBbaizdUmO59YIOgG/6Y2uhiy0FLrYUlLG7uIIDJVUUuqqp9ZjkHaki70jVWX9Nh80gLSGS9IQoMpNi6JkWS/e0OLqlxp50uie3z2VHNxM7xRbonUaddR4RaVxNdqvtd3dRNE3zpDsrTp06lT/+8Y9NFUNEmkB8dBhDOrZiSMdWxz1f5/FysMxNQWkVzqpaXNW1vpGNqlqqaj0n/D7R4Q7iIh3EHh0hSYgOo01CFMmxkQ2auvEmdSUHyC7+CHDWP5/tjYH00XiTup7tH1VEmoDl0y4nG/nIyMjQtIuINJy2QBexjKXTLuHh4QwcOJAFCxYcVz4WLFjA1VdffcL1ERERRERENHYMEWmObHZthy4SBJpk2mXKlCnceuutDBo0iAsuuICnn36affv2kZ2dffpPFhERkZDWJOXjxhtvpLi4mEceeYSCggKysrJ4//33ad++fVN8OREREQki2l5dREREzllD3r8Db79kERERCWkqHyIiIuJXKh8iIiLiVyofIiIi4lcqHyIiIuJXKh8iIiLiVyofIiIi4lcqHyIiIuJXKh8iIiLiV02yvfq5OLbhqsvlsjiJiIiInKlj79tnsnF6wJWPsrIyADIyMixOIiIiIg1VVlZGfHz8Ka8JuLNdvF4v+fn5xMbGYhiG1XEs53K5yMjIIC8vT2fdNCG9zv6h19l/9Fr7h17nb5imSVlZGW3atMFmO/WqjoAb+bDZbLRt29bqGAEnLi6u2f/F9ge9zv6h19l/9Fr7h15nn9ONeByjBaciIiLiVyofIiIi4lcqHwEuIiKChx56iIiICKujhDS9zv6h19l/9Fr7h17nsxNwC05FREQktGnkQ0RERPxK5UNERET8SuVDRERE/ErlQ0RERPxK5SMIud1u+vXrh2EYrF+/3uo4IWfPnj385Cc/ITMzk6ioKDp16sRDDz1ETU2N1dGC3syZM8nMzCQyMpKBAweyZMkSqyOFlKlTpzJ48GBiY2NJTk5m/PjxbNu2zepYIW/q1KkYhsHkyZOtjhI0VD6C0H333UebNm2sjhGytm7ditfr5amnnmLz5s3885//JDc3lwceeMDqaEHt1VdfZfLkyfzud79j3bp1DB8+nLFjx7Jv3z6ro4WMRYsWkZOTw/Lly1mwYAF1dXWMGTOGiooKq6OFrFWrVvH000/Tp08fq6MEFd1qG2Q++OADpkyZwptvvkmvXr1Yt24d/fr1szpWyPv73//Ok08+ya5du6yOErSGDBnCgAEDePLJJ+uf69GjB+PHj2fq1KkWJgtdhw8fJjk5mUWLFjFixAir44Sc8vJyBgwYwMyZM/nzn/9Mv379mDZtmtWxgoJGPoLIwYMHufPOO3nxxReJjo62Ok6z4nQ6admypdUxglZNTQ1r1qxhzJgxxz0/ZswYli5dalGq0Od0OgH0d7eJ5OTkcMUVV3DJJZdYHSXoBNzBcnJypmkyceJEsrOzGTRoEHv27LE6UrPx9ddf8+9//5vHHnvM6ihBq6ioCI/HQ0pKynHPp6SkUFhYaFGq0GaaJlOmTGHYsGFkZWVZHSfk/O9//2Pt2rWsWrXK6ihBSSMfFnv44YcxDOOUj9WrV/Pvf/8bl8vF/fffb3XkoHWmr/W35efnc/nll3P99dfz05/+1KLkocMwjON+bZrmCc9J47j77rvZsGEDr7zyitVRQk5eXh733HMPL730EpGRkVbHCUpa82GxoqIiioqKTnlNhw4duOmmm3jnnXeO+0bt8Xiw2+388Ic/5Pnnn2/qqEHvTF/rY99M8vPzueiiixgyZAjPPfccNpu6+tmqqakhOjqa119/nQkTJtQ/f88997B+/XoWLVpkYbrQM2nSJObMmcPixYvJzMy0Ok7ImTNnDhMmTMBut9c/5/F4MAwDm82G2+0+7mNyIpWPILFv3z5cLlf9r/Pz87nssst44403GDJkCG3btrUwXeg5cOAAF110EQMHDuSll17SN5JGMGTIEAYOHMjMmTPrn+vZsydXX321Fpw2EtM0mTRpEm+//TYLFy6kS5cuVkcKSWVlZezdu/e45+644w66d+/Ob3/7W01znQGt+QgS7dq1O+7XMTExAHTq1EnFo5Hl5+czatQo2rVrxz/+8Q8OHz5c/7HU1FQLkwW3KVOmcOuttzJo0CAuuOACnn76afbt20d2drbV0UJGTk4Os2fPZu7cucTGxtavp4mPjycqKsridKEjNjb2hILRokULWrVqpeJxhlQ+RL5j/vz57Ny5k507d55Q7DRQePZuvPFGiouLeeSRRygoKCArK4v333+f9u3bWx0tZBy7jXnUqFHHPT9r1iwmTpzo/0Ai30PTLiIiIuJXWkEnIiIifqXyISIiIn6l8iEiIiJ+pfIhIiIifqXyISIiIn6l8iEiIiJ+pfIhIiIifqXyISIiIn6l8iEiIiJ+pfIhIiIifqXyISIiIn6l8iEiIiJ+9f8Bor1Qczd146QAAAAASUVORK5CYII=\n",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"filenames": {
|
|
"image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_39_2.png"
|
|
}
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"def function(x):\n",
|
|
" return x**2\n",
|
|
"\n",
|
|
"def analytical_gradient(x):\n",
|
|
" return 2*x\n",
|
|
"\n",
|
|
"def gradient_descent(starting_point, learning_rate, num_iterations, solver=\"analytical\"):\n",
|
|
" x = starting_point\n",
|
|
" trajectory_x = [x]\n",
|
|
" trajectory_y = [function(x)]\n",
|
|
"\n",
|
|
" if solver == \"analytical\":\n",
|
|
" grad = analytical_gradient \n",
|
|
" elif solver == \"jax\":\n",
|
|
" grad = jax_grad(function)\n",
|
|
" x = jnp.float64(x)\n",
|
|
" learning_rate = jnp.float64(learning_rate)\n",
|
|
"\n",
|
|
" for _ in range(num_iterations):\n",
|
|
" \n",
|
|
" x = x - learning_rate * grad(x)\n",
|
|
" trajectory_x.append(x)\n",
|
|
" trajectory_y.append(function(x))\n",
|
|
"\n",
|
|
" return trajectory_x, trajectory_y\n",
|
|
"\n",
|
|
"x = np.linspace(-5, 5, 100)\n",
|
|
"plt.plot(x, function(x), label=\"f(x)\")\n",
|
|
"\n",
|
|
"descent_x, descent_y = gradient_descent(5, 0.1, 10, solver=\"analytical\")\n",
|
|
"jax_descend_x, jax_descend_y = gradient_descent(5, 0.1, 10, solver=\"jax\")\n",
|
|
"\n",
|
|
"plt.plot(descent_x, descent_y, label=\"Gradient descent\", marker=\"o\")\n",
|
|
"plt.plot(jax_descend_x, jax_descend_y, label=\"JAX\", marker=\"x\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4755b5a9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### A more advanced example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "a74afe50",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/jax/_src/numpy/lax_numpy.py:173: UserWarning: Explicitly requested dtype float64 requested in asarray is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/google/jax#current-gotchas for more.\n",
|
|
" return asarray(x, dtype=self.dtype)\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"<matplotlib.collections.PathCollection at 0x11de12520>"
|
|
]
|
|
},
|
|
"execution_count": 12,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
},
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGdCAYAAAAvwBgXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAAB8SklEQVR4nO29eZhcZZn3/z21dFUvVb2mO0t3VhAStkgCCQjIJsuIijqMDMgMDvq+/AQGZGZwAiqo+ObngAsygjLziigKuKKOiGFU0BEiISxhSyBJJ+mkt/Ra3dVd+3n/qHrOOd2pqj7L85zzVNf9ua5cF+l0dz0k1efc53t/7++tqKqqgiAIgiAIwgN8Xh+AIAiCIIjqhQoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8gwoRgiAIgiA8I+D1AcqRy+XQ29uLSCQCRVG8Pg5BEARBECZQVRUTExNYvHgxfL7ymofUhUhvby+6urq8PgZBEARBEDbo6elBZ2dn2c+RuhCJRCIA8v8j0WjU49MQBEEQBGGGWCyGrq4u7T5eDqkLEdaOiUajVIgQBEEQRIVhxlZBZlWCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCIDyDChGCIAiCkIQ9hyfxrWf2IJHOen0U15B6+y5BEARBVBNf3fIWfv1qH9oaQvjrdZ1eH8cVSBEpQzqbQyab8/oYBEEQRJVweCIJAOgZmfL4JO5BhUgJ4skMTtv8e3zkga1IZagYIQiCIMQTS6QBAIOFgqQaoEKkBPuG4xiaTGL7/lF88w+7vT4OQRAEUQVMJDIAgMFYwuOTuAcVIiVIGlSQb/5hN97si3l4GoIgCKIamCBFhGAk03ohksmpuOUnO8gvQhAEQQhDVVVMJvOKyAApIkQikx+dWtJUi2g4gFcPjeM//6fb41MRBEEQ85V4Koucmv/vockksuw38xwqRErAFJGFjWF85pI1AICvPvUW9hye9PJYBEEQxDwlNp3W/junAsOT1dGeoUKkBMmCIhIK+HDZuk6ceXQbUpkc/vWnO5CrkiqVIAiCcA9mVGUMxKgQqWqYWTUU8EFRFGz+0Amor/Fj275R/NerfR6fjiAIgphvMKMqY3CiOnwirhUimzdvhqIouOmmm9x6SUckC/G64aAfANDZXIdLTlwMAOg+HPfsXARBEMT8hBQRgWzbtg0PPPAATjzxRDdejgtGRYRRF8oXJczIShAEQRC8iM1SRKplckZ4ITI5OYkrr7wS//Ef/4Hm5mbRL8cNvRDxax9j6kg1LSMiCIIg3GG2IlItWSLCC5HrrrsO733ve3H++eeLfimusGIjFNT/isKFoiRJke8EQRAEZ1ghoij531dLuqrQ7buPPvooXnzxRWzbts3U5yeTSSSTegUYi3mXZsqKDaaCAHpRQooIQRAEwRtmVu1qrsOBkSlSRJzS09ODG2+8EQ8//DDC4bCpr9m8eTMaGxu1X11dXaKONyfMrGr0iIQL/02KCEEQBMEbpoisWlAPgDwijtm+fTsGBwexbt06BAIBBAIBPPPMM/jGN76BQCCAbPZIVWHTpk0YHx/XfvX09Ig63pwk0keaVUMFdSRJighBEATBGaaIHNXeAKB60lWFtWbOO+88vPrqqzM+9rGPfQzHHnssPv3pT8Pv9x/xNaFQCKFQSNSRLMECzYytmbDWmiFFhCAIguALU0SWt9XDp+jpqu1Rc12FSkVYIRKJRHD88cfP+Fh9fT1aW1uP+LiMFBvf1c2qpIgQBEEQfGGFSFNtDdoaQhicSGIgNv8LEUpWLUGx8d0QKSIEQRCEIFiOSCQcQHs03x2ohnRVoVMzs3n66afdfDlHlB/fJUWEIAiC4AtTRCLhADoiYbyGmNB01b7xabzRG8OCSAgndjYJe525IEWkBKSIEARBEG4yoSkiQa0dI3JyZuveYVzz0Au467e7hL2GGagQKYG2fdegiLCihHJECIIgCJ6oqorJZF4RiYYDaI+w1ow4RSRVeOCu8XtbClAhUoJi47tsgsarHJHx6TRe7x335LUJgiAIccRTWbBJ3Ug4iI6CIiIyXVUrRALelgKuekQqiWLju6wocVsRSaSzePDP+3D/07sRS2Tww49vwOlHtbl6BoIgCEIcrC0T8CkIB32uKCJJKkTkJjmHIqKqKhS2EEAQmWwOP95+EF//77dmGJb2DU/h9KOEvjRBEAThIkajqqIomiIi0iOSyubvc0GPWzNUiJSg+PZd34w/N6olvBmNp/CRB57DWwOTAIAlTbVoCAWwa2ACk8n0HF9NEARBVBJGoyoAdBTGd1m6qt/H/8E3ncn3grxWRMgjUgLWfgkXMasCumIiihcPjOKtgUnU1/jx2UvW4Pf//G6cuqIFADA5a1U0QRAEUdnEDIoIALQ2hGakq4ogVVi1QmZVCVFVtagiEvQrYEWp6CwR5p4+qasJ15yxAqGAHw2FN+hEkgoRgiCI+cTErELE71PQ1pBXRURliaSKJIh7ARUiRWB9M2Dm+K6iKIYRXrGKyFQqX+jU1ejds4ZQ/r9JESEIgphfzG7NABCerirL1AwVIkUwFhmzK0Vt8Z1gRSReUD3qQ7oiwyrlSVJECIIg5hWzFREA6Igww6qo1owcZlUqRIrA2i6KcmTvTJuc8VIRoUKEIAhiXsEUkegMRUTs5EyKzKryYhzdnT2iq2WJiFZEUgVFpEZXRFghMkGtGYIgiHlFbPpIRUR0lghTRMisKiHFjKoM1xSRZEERCRkUEWrNEARBzEuKKSKi01VThQdqUkQkpNjoLsOtdNViikgklH+DklmVIAiCD8OTSSke7op5RIQrImRWlZdyikiooIiIbs2QIkIQBCGWiUQaZ9/1NM7/yjM4NDbt8VlYIXKkIiLMI0KtGXnRNu8WqRLdas2U84hMJjPIse1IBEEQhC0OjExhIplBfyyBf3hwG2IJ71KrY9r4rmFqZla6Km8oWVViWJFRLMLdLbNqsakZ4xuUFSoEQRCEPcam9MJj18AEPvnwi0hnvdmuXqw1IzpdNUmKiLyYUUREB5oVyxEJBXwIFKJdqT1DEAThjJF4CgDQ2VyLuho//mf3ED7z89egqu4rzsUCzUSnq5JHRGI0j0gRs2q48A8mOuK9mCKiKIruEyHDKkEQhCNGp/KFyAlLGnHv374TPgV47IUe3Pf0HlfPoaqq9nAZDc/cRatNzghIV6WpGYlhEzHFzapsakZ0oNmRighgyBIhRYQgCMIRo/G8CtFcX4PzVnfg8+8/DgBw12934c+7h1w7RzyVBbOAGBURQJ+cEaGIpLP5F6VkVQlhikix8d1woTgRrYjEC1Mz9TUzq2PaN0MQBMEHpog01+Vv/ledthzvO2kxAOCPbx927RysLRPwKUfcd0Smq9LSO4nRk1VLKyIip2ayORXTadaamXkG2jdDEATBB+YRaa6r0T521IIGAO4mWBuNqrPTvEVmiWjju1SIyEdZs6q2fVecIjJt+N71IVJECIIgRMAUkZZ6vRBhD3uxafdGeYsZVRki01U1syq1ZuQjUWZ8V8sRyYhTRKYKaodPObIYaii8UckjQhAE4Qy9NXNkIeKmIhIrMrrLYFkiQhSRwn0sSIqIfJRTRHSzqjhFJJ7S/SGzZTpSRAiCIPhgNKsyorWFhz0Xw82KZYgw2iN5RaSfsyKiqiolq8pMsoyBRzerilNEWIZIXehIRUb3iHiXAMg4PJHEuV95Gvf899teH4UgCMIys82qgKE146pHpHRrpqlOTGHEJmYA8ohIiTa+WyxZ1QVFZCpVfGIGmBnz7jXPvHUYew/H8X//Z6+Q+GGCIAhRJNJZ7Vo7QxEJy6WIaAMSmRzXoLWUIUGWpmYkpJwiEnLBrMri24spIlqOiAStmX1DcQD5J4c3emMen4YgCMI8LN494FMQMQwF6IWI+4pItIgiwu45qjpTxXBKyqDqU46IhGjju0XNqnp1Kgpt824xRUSi8d3u4bj238/ucS/8hyAIwilsdLeprmaGF4+pElOprGt7Z8oqIoYHYp75VawQ8fsU+H3KHJ8tFipEilDWrOqiIlJfU8QjIpFZdf+MQmTYw5MQBEFYQx/dnalCNBiKAbeus+YLEX6FUVoSoypAhUhRyo/vio94n9LMqvIqIqqqYt/QlPb757tHZkh9BEEQMsMKkSbD6C6Qb1PUFq79brVnyplVFUXRzKQ8C5GkJAvvACpEimJm+67QqRnNrCqvR2RoMoXJZAaKknd1T6ezeOXgmKdnIgiCMMtooTXTMqsQAYBoLZucccewWi5HBNDvRUmOSrwsm3cBKkSKUt6syv8NMRu28K6YR0SWiHfWllncWIt3HdUGAHh2N7VnCIKoDEaKZIgwmDLhViGit2aOVEQA3RLA8wFYlgwRgAqRojD/h1fJqtrCu6JTM/k36mQyw3WUyyrdhYmZFW31OH1VKwAyrBIEUTkUyxBhuJ2uqrdm5lBEeBYipIjIjRlFJJXNCcvOKKeIMI9INqcK9anMxb6CIrKstQ7vWpVXRF46MIbplNitxARBEDwotmeGwcZo3do3wwqeaKlCJMhfiSezquTohUhpRST/eWJuuuU8InVBP9ik2YSH6ar7hvNG1RVt9VjWWofFjWGksjls3z/q2ZkIgiDMMlrIEWku4hFxUxFRVbWsWRUQ1JohRURuklqyamlFJP95YhSJclMzPp+ChhrvR3hZmNny1nooioLTCqoItWcIgqgEmFm1ub5Ya8a9ULN4KgsmrrvZmqGpGclJZEqP7wb8PgQK4S8J4YpI8Tel1yO8+dHdQiHSVgcABp8IGVYJgpAfFmhWTBFxc2qGqSF+n6KNDc9GL0Q4Ts1Qa0ZeVFXVJKtS+fuaYVWUIlIm4h3wfgPv4ckk4qksfArQ1ZIvRE4rFCI7Do655jQnCIKwy9hUmULExX0zxjCz2dvWGSEB9xx2nwuSIiIfRumrdCFSCDUTpIiwiPe5FJEJjxSR/QV/yOKmWq13ubipFiva6pFTgef3jnhyLoIgCDMk0llNeS4+vuueR2SuiRlATGuGzKoSY6w4i7VmAGPMuxhFRFt6V8SsCniviBhHd42cRu0ZgiAqALbwzu9Tik6qRF3MEdHCzELFjaqAoNbMHMq/m3h/Aslg/9A+BZoXZDYiRqmMaIpIEbMq4H2oGfOHLGutm/FxyhMhCKISMGaIFGuHuKuIlE9VBWhqpuowju6W7NcxRURAqJmqqmWX3gEGRcTj1szy1pmKyMaV+UJkZ/8EhieTrp+LIAjCDKNljKqAu1MzrDUTrS2jiGgPv5SsWhUkyozuMvTFd/wVkWQmp41yFRvfBfR0Va/2zZRqzbQ1hHDswggAYNs+yhMhCEJORsoYVQHD1IwLgWbmFBH+rZmkZlYt/sDtJlSIzIL944SLhJkxwgJkMkbcoHKUGuXSx3fdn05RVdWQqlp/xJ+z4mQglnD1XARBEGbRwsyKZIgAHikiJcLMADGtGd2sWvpe5xZUiMxC27xbRhEJCVREpgpO7tqgH/4SHpWIh2bVwxNJTBVGd5e21B3x5ywuebggfRIEQciGtnm3yMQMoKsTqWxOyHXeiFeKCHlEJIb14Mo5iYUqIswfUiJDBPA20IxFuy9pri36Bm5tCAEAeUQIgpAWZlZtKtGaaagJaKs0RE/OmCpERHhEqBCRF5YNUmp0FxA7NcM27xZbeMdgZlUvPCLGaPditDUUFJFJUkQIgpATTREpUYj4fIpr19m59swAYqdmZBjfLX23q1KsKCJiWjPlM0QAbxWR7uHyhUhrfUERiZMiQhDVSiqTw7Z9I/jz7iGc1NWEC49b6PWRZjBS8Ig01ZW++UfDQUwkMsILkZhHrRnmEQn6vTerUiEyi3KbdxlsakaMWbV8hgige0TiHhQi+1kh0laiECFFhCCqkqlUBr94uRd/2DmIP+8e0pJLw0EfXrvjQgQkGBNlsHj3Uh4RQC8MRE/O6K0ZM4FmHJfeSTS+S4XILJjKES5rVq1iRWQo7xFZ0XakURUAWsmsShBVyRd+9QYe3daj/b6tIYTx6RQS6RwOjExh5YIGD083E23hXZlCJOrS5IypiHeBu2Zqyjx0u4X3pZBkmFJEBFSnjLk27wLeeURUVdUUkWKju4BuVh2fTmtvdIIg5j/b9+ezgz66cSl+df0ZeP7W8/COjnyu0O7BSS+PdgRzBZoBxnRVdxSRYlHzDJqaqTK08d0y/zhCFZFk+c27gL6TIJnJuXqzHzSM7nY1F1dEmmqDYFPHzJlOEMT8JpnJYm/ByH7dOUfhhM5G+HwKjmrPqyC7D8tTiCQz+sK7UmZVQE86FTk1o6qqpmy73ZrRtu9K4BGhQmQWmlm1XGsmwHJEvFFEjKO9bvpE2MRMZ3NdySra51PQUjCsDtEIL0FUBXsG48jmVETDASyMhrWPH1Vox8ikiLCFdz6lfDvEjX0zU6kssoUobbd3zTCzqgxTM96fQDISmiJSzqzK3hTeKCIBv09LXXXTJ7JvDqMqg0Z4CaK62NkfAwAcuzA6Y0cXU0T2SFSIjBjaMr4SoZGAO4UI+95+n1IySRswREbwbM1kqTUjLZWgiAC6YdVNnwgzqi5vLd6WYbDJmREyrBJEVbCrfwIAcOyiyIyPa4XI4ThUVXX9XMXQNu+WMaoCullV5NSM0ahaaskqYGjNiDCrUsS7fJgb3xWoiJiYmgEMMe8uKiL758gQYVBrhiCqi52FQuSYhTMLkWWt9fD7FEwmM+iXZP/UaLywZ6ZMhgigezZiAh/2zGSIAGIDzUgRkRAz47thzazqTY4I4M3iOxbvvrzE6C6DRngJorrQFJFZhUhNwIdlBQV1z2Dc9XMVY3SOzbsMN6Zm2IPnXAq40O27ZFaVDzOKiN6a8U4R8WKEl428tUfCZT9P94iQIkIQ852xqZSmdrBxXSO6YXXC1XOVYq6Fdwx9akbcNVa3ApS/3ocMIZq8Wlxp8ojIi5nxXaaIiBidNe0R8aA1EzMRvAMYF9+RIkIQ8x3Wlulsri06girbCO/IHAvvGG4oItpuszmKAfZgrKpAOsunEEnR1Iy8JE0sAmJtG69yRABDa8YlRSSdzWGqUCRFy8y7A9SaIYhqolRbhrFKshFeNr7bUl/+OhZ1YSAgYVYRMdyPeLVnyKwqMbpHpFxrpuAREaCITJlURNw2qxoLnrkVEVaIUGuGIOY7pYyqDE0RkcQjwqb55lJE9Ij3tLCJn6RpRcRYiPC575BZVWKsKCJJAYpInJmXTCoibnlEWFumrsY/5/IqbQMvtWYIYt7DMkSOWRgt+uerCoXI0GQS41PumetLwcyq5VJVAX1qJqfqLXPemFVEFEXRCgYehUgupyJTCFIjs6qEmDEPCVVEClMzdXN6RPI/JG4pIrFptg+hvJwJ6IrIVCqrmW8Jgph/5HIq3iooIqtLKCINoQAWNeYN7rsPe29YNZsjEg76ECgEnonyiWgKvAlVQs8ScV4UMX8IQIqIlJgxDzFFJJtTkcnynetmbxCzgWZueUTYD2K0du6FzQ2hgPbmJlWEIOYvh8amEU9lUeP3lU1c1tsz3vtEzOaIKIqiT85Mi7nOMnWjnBWAwTNLhAoRyTGjiBjfNDxVkWmD/FcrWaCZPjEztyKiKAra6ildlSDmO8wfsqq9AcEyLVtZDKupTE67Zs41vguIn5xh6oaZyRWei++ME581c7Ta3UDoCTZv3oxTTjkFkUgE7e3tuPTSS7Fr1y6RL+kYMx4R4z8cz8kZ5g+p8fvmrFK1HBHXCpG5V1UbaSHDKkHMe3b2sR0zxdsyDFkUkbFCW8anmGszi943Y2Y4ghHi6E3UJ2Z8ZaPl3UJoIfLMM8/guuuuw9atW/HUU08hk8ngggsuQDwuh3u6GJqLucwbw+fjaxxiaGFmcxhVAWNrxh3zF9u3YEYRAXTD6hC1Zghi3rJzoPzoLkOWLBFjhki5hXcMbd+MKEVEa82YUUQ4tmYkSlUFAHOPtzZ58sknZ/z+wQcfRHt7O7Zv346zzjpL5EvbRmvNmBinSmVyfBWRpLnRXcD9QDNNETHhEQEMI7xUiBDEvGXXHKO7DFaIHBydRiKdNaUAiMCsP4TBFBFR6aoJrTVjxiPC7+FXplRVwGWPyPj4OACgpaXFzZc1TS6nmk6b0xbfcdw3EzcZ7w7oPyCum1VNKiJtWroqtWYIYj6SSGfRPZRXt48tMbrLaK2vQVNdEKoK7D3snSJuds8MI2LIEhEBG981p4jw2zeTlChDBHCxEFFVFTfffDPOOOMMHH/88UU/J5lMIhaLzfjlJkYn8Vxz3Vq6KsclRNro7hwL7wBdEYmnssjmxK/XZq5x860ZMqsSxHxm9+AksjkVjbVBdERDZT9XURR954yH7Rl2PZprdJehKSLCpmYKiogpjwi/h99UtSoi119/PXbs2IFHHnmk5Ods3rwZjY2N2q+uri63jgdgpvHUbPa/CLNqvQlFpMFgGo27kNURszC+C+iO9KEKLUSyORVv9MaEJSoSRKVjjHY3Y3iUwbA6piki5h6ooi4pIl5NzcgwMQO4VIjccMMN+OUvf4k//OEP6OzsLPl5mzZtwvj4uParp6fHjeNpsH9gv0+ZMz00HBRhVjUXZgbkCyH2JnKjPTNhYXwXqPzWzM0/ehl/9Y0/4eYfveKK4kQQlcYuk0ZVBitE9nhYiIwwj4hFRUTY1IyJ4QgGz9YM84iUG7l2E6FmVVVVccMNN+DnP/85nn76aaxYsaLs54dCIYRC5SU+kZg1qgJAmDmYuZpVzcW7MxrCAYzEU64YVvVk1flvVt26dxi/eLkXAPDzlw5BAXDXZSfBb8JlTxDVwpt95aPdZ7NKAkXEbLw7Q/jUjOYRcTnQzERMhZsIPcV1112Hhx9+GD/84Q8RiUTQ39+P/v5+TE9Pi3xZ21iqTrUNvN4oIoAhS8QFRURvzZj0iDBFJJ6sqPZGNqfiC796AwCwblkz/D4FP3vpEP7lJ6SMEIQRsxMzDOYR6R6Kc02ktsJ4IYag0eR1jLWiRSsiplozWo4Ix9ZMNRQi999/P8bHx3H22Wdj0aJF2q/HHntM5MvaxpYiwtGsasUjArg7wjthMdCMmVXTWdW10DUe/OiFHrzRF0M0HMADV63DvX/7znwx8uIh3PKTHVSMEASA0XgKgxP5tqvZQmRJUy1qg36ksjn0jHrzMMra2A0mr2Oip2asKSL8WjOymVWFt2YqiaSN6pSrImJhagZwb9+MqqqWx3fDQT/qa/yIp7IYnkyZ/jovGZ9O4+7f5pN/bzr/HWhtCOGvTlgEVQX+8dGX8NMXD6Im4MPmD53g8UkJwlveLrRXOptrtQeiufD5FKxcUI/Xe2PYPTiJFWV204hiUmt/my1E3JmacTvQLJmRyyMixykkQY93n7s6lUER0ffNiE1XjaeyYEKA2dYMYGjPVIhh9d7fvY3heAqrFtTjqtOWaR9/74mLcM/la6EowCPPH0DPyJSHpyQI7+kdyysaXc11lr7O650z7Bprtnhyb2rGG7NqVU3NVAp67r8ZRYSN73qviIj2iLB496BfsWRuYobVSoh533N4Et99dh8A4LOXrDniSeGSExfjtJWtAIBfv9rn9vEIQioOFQqRRU1hS1/X2VwLABiIJbifyQxsIMBsIcIUkXgqK8TXYu2eQx6RqsCKIsJuyF7liADueURihraMlQVJbN9MJSy++9Kv30Qmp+LcY9tx9jHtRT/nvScuAgD8egcVIkR10zeeL0SWNNVa+jrdxO7Nw8mk5UJEV4B5X2cz2RwyBak5bOqew39qhgoRCdFT7kyYVYP83hQMy1MzLnlEmOISMWnwYrQVFJERyRWRXf0T+P3OQQR8Cm577+qSn3fRcQvhU4BXD41j/7C8ixsJQjS9Y3lFY1GjxUJES1x2/+Ekk81pCrZZj0hNwKepFbyVZ+O9w8w9h6tZtZrGdysNO/06vkvvrOWIRNxSRKatje4ytCwRydNVt+8fBQBsWNmi9bCL0doQwumr2gBQe4aobphHZLHF1oyX+UJsqShg/hoL6KoI7ywRYyFiThHhmKwqWaCZHKeQBBZOVjGKCMsRcbE1Y4WWQmtmSHKz6o6DYwCAEzub5vxcas8QhLEQsaaIsNUPXjycTBZa33mvm5VCRMzkDHuIrfH74DMRlihk1wwVIvKRtCBXaUvvOCoiUylrikhDoTCQvTUje7rqKwfzW6FP6myc83MvPG4h/D4Fr/fGsG+I2jNE9RFPZhArXBMWNVpTRNjqh9F4CjmXM3msGlUZoiZn2L3DbHtERGuGPCISwgoRK3G7PKdmmHRYb1ERca01Y1ERqQSz6nQqi7cKOzPMKCIt9TU4fRVNzxDVCzOqRsIB07unGM2FaPVMThUWm14KqxkiDFH7ZrQHXxP3G0DQ0jsqROTDSoWqL73jo4hkcyqm06w1Y9Ij4rJZ1ezmXQbrB49I7BF5o28c2ZyKtoaQ6ae7S6g9Q1QxhwpGVasTM0D+xseuW263Z7RUVZuKCO/CycroLkBTM1WDpUAzjv06AFoRApiv2N0e37X69GMsRGSNRn+lR2/LmB1NvmDNQgR8Ct7oi2HvYe8WeBGEF/SxDBGLbRmGvpnb3UIkblMREbVvJmFhpQhgyBHhGfFOHhH5sBTxzqZmOCkiU4UfEp9i/o3JvCTiWzPW9swwmAybU4GxKTlVEStGVUZzfQ3edVR+euYJas8QVYZdoyqjxaMRXvutGTEekaSFJauAoTXD4eE3LdmuGTlOIQkJCwuIeCsi8ZTuDzH7ZF5b8JJMczTMFsOuIhL0+9BUl/8aWUd4dxSMqid2zW1UNcKmZ/6L2jNEldE7nm/N2C1EWJaI24nLTBGJWC1EQqKmZszfbwBBrRlSROTDytQMb0WE/ZDUWZhvry28gVOZnNDWR0zziFhfXKdfdOQzrI5Pp7G3MPlykgVFBAAuXLMQQb+Cnf0Tnu3NIAgvsJshwvDKO6Y97Fm4xgL6dW+C804vKwq88fN4tGaS5BGRF1s5IpwUkamUtYkZYKapVaQqMqFNzVhf1tzqUT/YDK8dyqshnc21mlxslsa6IM4otGd+Q+0ZooroG7eXqspg03RuFyLM4yHN1IxVRSSoT8043WxPZlWJ0cZ3TZlVOSsiKeuKiLGSnk6JK0RiWo6IdUWkTeLJmVcK/hCragjj/DUdAIBn9wxzOhFByI2qqroiYrMQafFIJbWbI6Ilq05znprJ2JuaUVUgnXVYiFCyqrwkLCgieo4IL7OqtVRVAFAURWvPiC1EWMS7dUVES1KUsDWzozAxc6KJILNibFiRzxN58cAotzFugpCZkXgKyUwOigJ0NIZsfQ/PWjN2p2YEKSJ6XIQ1syrgvD1DZlWJsbR9l6NMBljfvMtg7RlRrZlEOqvJePY8IoWYdwkVETYxc1JXk62vX7WgHm0NISQzOc30ShDzGbbsrq0hZCkm3YgWdOhyu9bq5l2GvmtGVGvGmkcEcG5Y1ZbekSIiH9Yi3nWZjMlcTpjSzKrWfkjYOVg8PG/YU4CiAA0W1BqGHvMulyJyeCKJ3vEEFAU4fok9RURRFGxY0QIA2ErtGaIK6B13NroLeLdvhj3sWS9EClMzvAPNMtYUEUVRNAWDVyFCioiEJNPm57p5VqeAcXxXLkWE/fA1hAKmFjPNRlazKlNDjlrQYPnCZGTDynwh8pfuER7HIgip0f0h9iZmAP3hZHTK3X0zkzbNqkwJTmVyXFuwWqCZSUUEMGaJODsHFSISY0URqfH7wOI+ePhEmKJhxSMCALU1Yj0iWry7DaMqoI/vymZWZYvurASZFYP5RLbvH9X6rgQxX+lzmCEC5AMBgfxai3HOBtBy6IFm1h72jA8qPNdpaIFmFlpcvLJEyKwqMdpct4kKVVEUrkl32sI7iz8kmllVlCIyzcLM7KkGXq79LofuD7HXlmEc3d6A5rogptNZ8okQ855DDuPdgfzNr7HW/aBDdo21qoD6fXpLhOd11mqgGcBv8R0pIhKjvTFMVqhalggHuc6pIjIlSBHRJ2bsKSKNdfrCKLfXfpdCVVU9UdWhIuLzKTh1BWvPkE+EmN+wPTN2Ft4ZafVgms7u+C6gP/DxmpI0fi+zgWaAYUjCaWuGds3IixVFBNALlgQPRcSmR0TED4gRvTVjTxFhTz6qyn/8zS4HR6cxEk8h6FewelHE8fdj7Zm/7CWfCDG/YVMzixwWIi0ut2xVVcWkTbMqYPDipfi1X7XcKkuKCKfWjAUbghvIcQoJyOZULSTG9Fw3x22IdqdmRHtEYlqqqj1FJBTwaz/EY9NytGdYkNmxC6O2RxCNMMPqC/tGkCGfCDFPSWdzGJxgHhH7rRlAzxJxa6x/KpUFS1mwalYF9Ac+ntOJCW04woZZ1UEhksnmwMRp8ohIRsrwD2v2jSFGEbFYiATlbs0AQFPha8em3DOmlUNvyzjzhzCOXRhFNBxAPJXF670xLt+TIGRjIJZATgWCfgVt9fbCzBgtLObdpWk61pZRlJmrMcxSK2A6kXkLrTwM8dg3Y0xlJY+IZBhbG2b7ZkwR4To1Y9Gsyn6oRLdm7JpVAb2IcdMhX47Xe/kWIn7yiRBVgHHHjJ1RfiNavlDcHY+INjFjYbu5EREJ1kmLEe8AEOKw48z40E2FiGQwqSvgUxAwWYiEOa5l1iLeLfQLARcUEYetGQBoKhhWxyQpRPYNTQEAjmpv4PY9ySdCzHd6OUzMMNyeprM7McMQoYh4NTWTzOb/HxQlf7+TASpEClhdyQzwVUQSFsLUjNQWWjmixnd5KCJNtfmLzviU9x6RdDaHvkI6ZFdzHbfvy3wiz+8bQVaS6SCC4AkzqjqdmAH0oEO3WjN2M0QYIh74EnbuORxaM9rort9nSx0SARUiBexVp/w8InYc1ABQWyiGhJlVeXhE6uTxiPSOTSOn5n+gF0Sc9bmNrFkURUMogIlEBm/2kU+EmH9oiohDoypgGN91uTXjVBHh2QJPOrjnOFFEjIWILMhzEo+xo4iEOU7N2JkpB8RIhkZi086SVQE9S0SG1syBkXxbZmlLHdengYDfh/XLmwFQ3DsxP+njsGeG4fb4rt3Nu4w6AdOJthQRLUfEfiHCzKqy+EMAKkQ0tHh3C9VpOMhPEUnYVURYa0ZYxLuzZFVAzxKRwaxqLER4o/tEyLBKzD8OFVozixt5tGb0QsSNoMNJh4WItlxUQKCZPY8Ih9YMFSLyoY9SuduvA/Jz3cxXYMVBDRh6l8KW3hUUEUfju/mLjgytmZ6Rgj9ERCFi8InIkiJLELzgqYg01+WvCTnVHaWUKSIRSRQRVVUND79W7jkcWjMFsyoVIhKitUY8UEQShjeV1YAtbXxXgCKSzanak4TdZFVA94iMSxBo1lNQREQUIicsaURt0I+xqTTeHpzk/v0JD9m2DVi0CFiwQP+1aFH+41XAVCqjPUjw8IgE/T7tuuBGzLvT1gzv8d1UNqcFrLmdI5Ikj4i8WNm8y2Cf69TAZPx6qx4RXTLkH59u3DQZcTK+K1GgmcjWTNDvw9quJgDAC/vJJ1LxbNsGtLbm5xxPPRVT/f0IDQ3hgqEhhIaGMNXfD3zwg16f0hXYxEwkFHDkFzPi5gjvhNNChPN0ovHh1VqOiHOPCGvNyJKqClAhomHPrMonRyRp6NlZDQoSsQOBwSZmwkGfIxlPRrNqV4tzebkYzLC6fd+okO9PuEB3N3DqqcCGDZgaGUEIgOLzo2XJaviWnYQ/LTsJvuVr0br8nehJeH1Yd+A5McNg6azDLozw6gvv5BjfZfcbRbGmTPBozchoVrWvt88z7IzvhjltQrQ7MQMYJUP+ikhMM6o6ewIymlVVVfVsdn18Oq0ZZnlmiBhZtyxfiLywnwqRiiMeB771LeDzn0dqYgKrAByqqUXDiRdgwSkfQCDafsSXnK0A1zzxJm46/2jLm7MrCZ7+EIY+OeNGa8ZpoBm/zChgpifRyvVwvppV5+9PjkVsBZpxSla1M0/OqDOM7/K+yeuju87eJk0FY1oqk0MindNGjt2G+UPaGmpsS7RzcfKyZihKXnkZnEigPcLvCZIQSHc38P73I/faa/gsgC/XNSGy7hIsOfkS+MP5BN7s1Diy8VFozX2fHzVtS/HAH/fiiVf78KUPnoB3v2OBd/8PAmETM4s4TMwwWhrca804nZqpDfJuzdgLsOSRrMrMqrJs3gWoENHQPSLWFRHHHhEbOwe0MxRu6jk1b4DisU2WwSPMDADqa/wI+BRkcirGplOorRHTFpkLVoh0ClJDgHzeyjEdEezsn8D2faO4+IRFwl6L4EA8Djz4IHDnnUgNDGABgMwx78Lii2+EL5R/n6SHDyL2/M8w+fofgOzM9uLv3uzHZx9/HQdHp/H333keH1i7GHe87zg0F5725wt9hdbMEq6tmUIh4mprxlmgGb/WTOHh0+L1mueuGTKrSoiuSlhXRBIOx3f11oyNrZCGipp3loge7+6sEFEURYp01Z5RcUZVI8wnQu2ZCuDXv0buhhvwmYEB1PsD8J9/LRZcugm+UB2S/bsx+LMvoff/fhKTO7YcUYQMDAzg3GM7sOVTZ+GaM1bApwC/eLkX//v72+ddzH/vONszI6I1I78iwnu5qD6lae0WzLM1Q2ZVCdFT7ux4RPiYVe0oIkG/D0F/vh3DO11VX3jnXDhrlGByRuTEjJH1y/J5IlSISEw8Dvz7vwO33ILPAfhy00K0ffRuRNddAgAYf+5H6P/ezZh++zlAzSEIYHrVKqgNDVAVBerAANrb856R+lAAn71kDX72yXehvsaP5/eN4D/+tNe7/zcB6Jt3+SkibN/MkAvju85bM0wR4ePF0zyJVhURLq0Z+cyq8pzEYzTzkI2VzE4VEWZ2tfqmZIQFbeDl1ZoB5EhXPVAIMxNdiDDD6uuHxoUl3hIOiMeB227LKyH79+Peleux6Op7EFp4FLJT4xj48e0Y++P3ADWHOgC3LliARDyO8O7dwMQEkMsB7UcaV9d2NeH29x0HAPjKll14o3d+7BxSVRX9rBDhaFZtdVER0QLNbD5U1XIONEvabMdz3TVDhYh8aG8MG+EyjgPNbBRBRkTsQQD4bN5lMMOql6FmmkdE0Oguo7O5Fh3REDI5Fa8cHBP6WoRFWBFyzz14N4Cvn3A+mj78WfhC9UgcfAN9370Rib3bEQRw69KlmNiyBV/q74evzlzxetn6TrxnTQfSWRU3PfYS1yVpXhFLZLSHnIVR/oqIG2ZVNjXjONCMc46I1XZ8iMN+MypEJMZOMaDniDhURGwUQUZ4/5Aw9NaMc0XE61CzbE7FoVF3FBFFUbT2zHZqz8jD4CDwoQ8hd889OAvAjo2Xoe2vboLi82Py1acw8MgmZCeGEFUUTD35JL7U3Q3fe94D+MxfExRFweYPnYC2hhq8NTCJu3+7S9z/j0swNaSpLsh14o15REanUkI9NclMFqls/vreYHPEmj3spbMq0lkOu8WcekScmFVZxDt5ROTDVqAZM6tyUkTsjO8C4hbfaa0ZHh4Rj0PNBmIJpLI5BHwKV8NdKbQ8kX2UsCoFg4PAJZcgs2ULlkPBG+d9As3v/nsAwPjWH2P4iXuAXBZRnw/DU1MIXHihpQLESFtDCF/+8IkAgP/8n248u3uI2/+GF7AMEZ5qCAA0F64JqgqMTYlTRZgaAgD1NgPNjNdmHipX0uaSU2rNzHPsjO+GJAg0A4Dawjl4KyITHBbeMbxefMeMqkuaa+G3mF5rBy1hdf8oLcDzGlaEbNuGVl8A0+/7J0TXfwAAMPK7BzD2zEMAgK5gEMPxOAJh5zfc81Z34G9PXQoA+Kcfv6J5FCqRfgFGVQAI+H1aMSKyPcP+7kMBHwI2VYB88Fj+v3k88Nm95vOYmtGSVUkRkQ87u2a0pXecIt6tLNwzwttIxdAVER5m1byqEvNIEelxaWKGsXpRFLVBP2KJDHYfpgV4nmEsQvxBhD54K+rXnA01m8HhX96FiRd+CQA4o60N+yYnuRQhjM+8dzU6m2vRN57AT188yO37ug2bmFkoQElscSFLZNKhURXIt9zqOLbAbSsimkckB1W194CTJEVEXpiqYeUfhxUtqUzO0VOvnrJnVxHhm/rHEGFWHfPIrCpy624xZizAo70z3mAsQoIhhP/6c6g76lTk0kkM/uyLmHrzGQDAGUuW4Jm+Pvhq+IaQ1YcC+MSZKwEA3312X8UqY6IUEcBoWBU3wut08y6DZ6iZ3Ws+U+xVVVc2rEKtGYlhBiQr/zjGatZJz86ug5rBO/WPoZlVeYzvehxopi27E5iqOhs92Ix8Iq5jLEJqalF72edRu/ydyKWmMfjjO5DYux31AG7buBHP7NsHX0BMyPSH13WiIRTA3sNx/E+FekX6Y0wREVCIuDDCq2WIONwFVFvDTxGxk1uV/3z9/mS3PcOMuxRoJiEpG4WI8U2RclCI2J0pZzDJkOeooKqqiDGPyDyYmnErzMwIM6zS5IzLGIqQtnAD6j5yJ8JdxyOXjGPgsc8i2fMqoj4fxrZswZ1//rOwIgTIR4r/9bpOAMBDz+4T9joiEauI5AuRIRdaM3bj3Rn6glEOrRkbSd7A7ELE3j0nTYqIvLBCImShSgz4FDDfoxPzkPOpGb6pf0C+6mcjdXxzRDzyiLg0umuELcDbPzyFwxPi0yMJzCxC6hpRd/n/QWjxMchOxzDwyK1I9e7MT8bE4whYHM21y9+dtgwA8PtdgzgwPCX89XjTp8W78y9EWurzrRmRG3j11oyz0WOe04l2IxsURdEKCLuFCHvotnKvE408J/EYO30zRVG4jFMlbIwOG9HNqs7n2xls867fp2gz9E5gyaqTyQyXOXwrTKeyWiHgZiHCFuABwHZqz4jHWIRE2lB/xZdR07ES2clRDPxwE1IDe/QihKMpdS5WLmjAWe9YAFUFvvfcPtdelwfxZEZTRkWYVd1pzeSvrw0OlV02nTjFozXjIMRSzxKx2ZohRURe7C4C4pF0l3SqiGhubn6KCJuYiYQDUBTn467GLBK3J2fYsrtIOKB5VdxCzxOh9oxQDEXIgqZFaLjy3xBs7URmfBD9P/w00kP7PSlCGB87fTkA4LEXeipqlJf5QyKhgOPWRjHcaM3om3cdKiKsBc5TEbFxzXf68EuFiMTYXQTEI+bdqUeEZ++SMcFxdBfIZwawFo/boWZuj+4aOXVFPmH1eQo2EwNbXrdhQ74IaVuG+iu/jEBjO9LDB9H/g1uQGe31tAgBgHe/YwGWt9ZhIpHBz1865MkZ7NA/Ls6oCrizgTfOyaxaV8NvOtHu0jvA+eK7JJlV5SWVsT6+C/BJutPDbRzmiHA0q7LWTLSW31NQk0eTM14YVRkbVrQCAF47NK6pTAQnjMvr9u1D6+Jj0HDFZgQaWpAa2Iv+H/5rPrLd4yIEAHw+BVedthxAvj1jNwPCbfoEFyJtbHxX4AbeCU7juzyXi9qNeDd+jd3WDJlVJUabmrHamuGQdKeH2zhTRHiO72qtmRC/VgZLV3V78d0BlzNEjCxsDGNFWz1yKsW9c2XW8rqvrjoVjZd/Cf7aKJKHdmLgkU3ITY1JUYQwLlvfiboaP94amMRze4a9Po4p+gUaVQFdERmbTiMjyDsW5zQ1U8fxgc9OkjfDcWvG5r1OJPKcxGNSNpJVgZlJd3bRwm1sKiLsB4Tn+G6MY5gZgxlW3Z6ccTvMbDYbV+bbM5Vy85GeeBy45RatCHn5pAux4EO3wRcMY2rPNgw8dhtyybhURQiQb3N+6OQlAIDvb93v8WnMITJVFQCaC9N0qiquZasVIg6vZfpQgHOPj5MQS6etGfKISEommwMLPbRsVmXVqQOPiBMHNQCEBQSa8frhNeJVqFnPiPuju0Y2rsy3Z7buJUXEMWyD7n334SwAr77rCrRedEN+g+6OLTj80y9CTSelK0IYl5+S3z/zh12D3FcyiEBkhgiQn8pjRnZRDyiTvJJVOUa8O2nHO1Xh7T50i0Sek3iIMSrXrlnVWWvGmUeE5w4EBi8504gXoWaqqhpSVcVv3S0GK0Re7x33LEdlXmDcoOsL4M2LbkDTGVcAAMaefRTDv/kGoOa4Lq/jzXGLo1jSVItEOoc/V0DSqmiPCGBY/yDousC27zqemuH4wOekHc92ktl9+KVkVUkxpqLaL0ScKyJOA834Ts3weYowwsyqbt6MhyZTmE5noSj5zbte0BENYyX5RJzR3Q1s3JjPCKmNIvWRLyJy0oVQc1kM//abGP/TwwDELK/jiaIoOH91OwDgqTcGPD7N3Gjx7lGRhQi7LojxjnGLeOeYYO3kmu/0nkNmVUlJZvNvLEXJp6Vage/UjM2IdwFTM2IUEfbk455Zlakhi6Jh24oTDzYUVBHyiViEjeeedhpS3d1obFuG+r/7KsJLT0AuOYXBn34Rky//BgDQFYkIWV7Hm/esWQgA+N3OAS29WEYS6aw2ViuqNQPo3jFRigi31gzHB76kg2u+UxU+aWOdiWjkOYmHGMPMrIZ3OR2lUlXV9kpoRlhAjkg8xb8Q8cKsemgs7w/pdHHZXTGYYXVrNxUipmF+kBtuwGcGBtC66hS0fvQuBJsWIj3ah/7v/xMSe18AAHSFQtg7NCR0bwwvTl3RgkgogKHJFF7uGfP6OCUZKKgh4aBP+9kVgejrAnuocmq85zmd6OSa7+ThV1VV3axKrRm5YB4RO9n7jsNlDF/ndHw3mclxe8Jiscg8WzOaWdXFQkQbP2zyVqrXfSIx8omYYXAQuPjivB8ECu497W/Q8uHPwheqQ2L/DvR/72akh3vyG3TXrMG+WAwByZUQRk3Ah7OPlb8906cZVWu5pCuXQmS+UC6naoWD02sZr+nEbE7VfBq2ChHt4df6PceJH1Ik8pzEQ5yMMzltzRjfTLbNqobeJ68RXl6xyEaYWXXcRbOqZrYT2OM2A/OJqCqwrZt8IiWJx4GvfAU47jhkXnwRbaF6JD78WTSf9XdQFB8mXv4NBn70OeQSE6gB8ht0X31V+nbMbN6zpgMA8NQb/R6fpDT9Lv3s6PlC/K8LccOoLa/tu04VEWNLxe3WTMqQ1UKKiGQ4K0Qc9usKX+dTgKDf3lOH8c3Ma4R3UohZVQ8vcgsmL4t0/Ztl4yo2xkvtmaKwVsw//zNuGxpCtGMVGq6+B3VHnYpcOomhJ+7ByG+/CeQyqAEw4eIGXd6cfcwCBHwK9hyOY+/hSa+PU5Q+waO7DF0R4e8dY/4Qv09xPK7KK8Ha+PDpdmsm7WAwQyTynMRDUll78e6AM5kMMGSIBPy25U+fT+Hq6Ab4GbyMGKdm3Iq4dutiagbWnnmOCpGZMEPqKacUWjHAN066EAs+ehcCzA/y8L8g/upTAIAuvx/xeBw1dd76fpwQDQe198N/vylne8atIl4zq4pQRLSJGfvXVwYvsyrbth70K/BbHI4A+Cgifp+91xYFFSLQK0s7c9VOWzMJhwvvGDxn3AFd0owIMKtmc6pW6IimX3AypBU2FhbgvdEXc7U9JTUsJfWGG/CZAwfQVFOH6fffkg8pC9Rg6u2/oO+hm5Ae3AugMJ47NYVABRchDL09I2ch0ic43p0h0qzKvG4RDss764J8lt4ZHz7t4OThV0ajKuBSIXLfffdhxYoVCIfDWLduHf70pz+58bKmYQYeO/84jlszDjNEGDxT/wDDkwTHQiQc9Gt/X26EmmWyOQxO5JdpyaCItEfDWLkg7xOhbbzQskFy992HdwO4a9E70Pixb6B+9VlQsxmM/uFBHP7ZnVCTcQQB3LZxY0WM55rl/EIhsn3/qNClb3Zxq4hnLVsRxbl+HXPudQvX5K9d0+msI0XX6bZ1Jw+/SQkzRAAXCpHHHnsMN910E2677Ta89NJLOPPMM3HxxRfjwIEDol/aNHw8Is4UEW79Sy6jZVmtOONZiADuhpoNTaaQzanw+xRty6fX6HHvVdyeYYbUtWuReu01NEPBjlM/hIVX/huCTQuRGetH/w8/jdjzPwWgIgJg8sknceef/1wR47lmWdJUi+MWR5FTgd/vHPT6OEfgukdEiCLC74GKPeypKp8AS9uKiJPWjAP1XyTCT/PVr34V11xzDT7+8Y9j9erV+PrXv46uri7cf//9ol/aNI4KEYdxu/ryI16KiPOWBzOqAvneKk/0UDPxhQiTljsiIWn6oadVe7CZ0ZAai6Guvhnhv/k8ms/5Byj+AOI7/4Te796IVO8uAMAZ0SjGpqdRc+GFFWlKnYvzV8vZnklnczhcUGlEe0T01Q8p5DgHvLFrGY88pFrDNdrJA58WYGlbEbH/8JvOyrdnBhBciKRSKWzfvh0XXHDBjI9fcMEFePbZZ4/4/GQyiVgsNuOXGzCzqicpd9rCOz57EKZTzldps90MtUE/ApwrZz1LRHy6ar8LezKssqEQbPZmf8zVhFkp6O4GTj1VM6R+beV6LPqHf0ftipORSycw/OS/Y+gXX4aajKMOhVbM8DB8ksa184D5RP709hDX7dlOGZxIQlXz7eqWOrGtsGihEMmpwCSHzbZGmNfNabw7AAT8Pq19P+Xg30oLM7PtEbH/8JuSMFUVEFyIDA0NIZvNoqOjY8bHOzo60N9/5Pz85s2b0djYqP3q6uoSeTwNJ3IVL7Oq49aMNuPOQRER4A9huJmuagxkkoX2SBhHtTdAVfM3n6rAGNO+fz/q/UFMnve/0H7ZHfDXNSI12I3+hz6FyVeeBAB0BYMY37Jl3rViisGW4E2ns1KpZCwIsKMxBJ9gNTEc9Gt+Cd4+kUnOW8R5tMB1FdypImK/NVOVZtXZY1OqqhYdpdq0aRPGx8e1Xz09PW4cDykuZlVn47tOWzO8Uv8AY7w7/90sbm7g7ZcoQ8TIeYWlZ7KObXJlcBB4//u1mPbG1k60/t1XEV3/fgBAbNvj6CukpPoA3Lp6dX5pXYXmg1hFURSceXQbALnGurUiPupOES8q1Iz3zqxaDus09N1iTj0i9qdmggE5WtUMoY8bbW1t8Pv9R6gfg4ODR6gkABAKhRAKuW8qdOYRcbZrRnNQc1NEnBciIhURN82qMmWIGHnP6g58+5m9+MPOQaSzOemMY9zo7gbOOw+Z7m6sBDB64nuw4Pz/DV8wjGx8FENPfB2JvdsBABEAQ088MW+9IOXYuLIVj27rkcrA7HZbs6kuiP5YgvsDSlxbVcHnoYrHglF9z4z7UzNaa0aya47Q09TU1GDdunV46qmnZnz8qaeewumnny7ypS3BI+I95VAR4eYR4aCIiEhVZWjpqi74I5i8LJsi8s6lzWitr0EskcHz8zHufXAQuOoq4KSTkOruRgPLBrn4RviCYUx3v4jeB2/QihDNkHrxxVVXhAD6JNVrh8YRS8iRL+N2Ea+HmvG9LkxwvpaFOcQkJJ0qIkEOrZlq8ogAwM0334z//M//xHe+8x28+eab+NSnPoUDBw7g2muvFf3SpvF0fDfNVxHhMb6rbasU6BFxZ2pGTkXE71O09oxs0xKOYcvqHn4YZ05MoHHhUWidlQ0y+KPbkYuPIQjg1g0b5r0hdS4WNoaxoq0eOYn2EHmhiAD8rwvcWzOaR8S+F09vxzv0iDgJNLNZBIlCuBPsIx/5CIaHh/GFL3wBfX19OP744/HEE09g2bJlol/aNFrEux2PiIPqNP91nALNeCoi88CsmsuphohqecyqjPNXd+BHLxzEU28M4Pb3rRG63dQ1DK2YVgC5tRej/bz/BSUQRGasH4d/dZc2lhtVFAz/5jdV4wWZi40rW9A9FMfWvcM4b/WRbWu3cStVlSHquqD73fhcy/i0ZpxFNszH1owrlvRPfvKT+OQnP+nGS9lCS1Z1sn3XZo6ILtPJE2im91Ur1yMyHE8hnVWhKEB7RI4wMyNnHr0A4aAPh8am8WbfBNYsjnp9JPvE48CDDwJ33onUwAAigRAiF16HhuPPBQBMvfUchp74OtRkHH4AG+vr8fTQEAJVrILMZuPKVjzyfA+27pVNEXHJrFonxqzK+6EqzMGLpweaeTg1I5lZVa6yyCOcjDQ5bc1wU0TYD4j0UzPuBJqxC+mChpCUZtDaGj/OOGoBgApvzxh3xQwMINq8GK1X3Y2G48+Fmsti9A/fweGffwlqMo4aAFNf/jL+Z2yMipBZbFiR94m83jvuipG7HNmcigGXVyM01orZwMu7NVPHdXzX3vWVfV0yk7McNZ+WVBGR6zQe4SR/nxUiqWzOViqg05lyhja+y0ER4W3wMtLkUqAZG92VzR9i5AK29OzNIzN1KgKWklrYFfOVlevQ/vdfQ037CmQnRzHw6G2IPf8zAEAn25h7yy3APM8GsYPRJ/KCx3uIhiaTrq9GEOUR4W285+HFY/cbuwMKzA6gqrqab/W1q86sWgnwiHgH9P6bFZzOlDN4SIYM3k8RRliyaiKdE5okKevEjJFzjm2HogCvHYppPfmKYXAQuOQSLSX1lVMuRfuHPwdfqB6JntfR991/RLLntfyyutWrsX+ebMwVycZC6q7XwWbM5O3magRNKRXUmuGl7vLw4iUctuONX2e1PVO1UzOVACsg7CWrGt4UNnwiTmfKGXU1fFZUA2ILkYaaANi1LSZQgpYxVXU2CyIhnLy0GQDw35XUnilszc1s24ZWfwBTF9+IlnM/DsXnx8Qrv8XAo7chGx9FDQrL6l57bd5szBWJthCx29tCRE9Vda+IZ60ZntcEVVURLzyYNYSCXL4nj7ymhMN2vLGtYtUSoJtV5ZqaoUIEQNpBlRjwKdqN1Y55SF+AxGnpneSBZj6fYsgMEFeIMI9IR1ReRQTQd41sqZRCZHAQuOgipLq7EamNovYjX0LDie+Bmsti5L8fwMiT9wK5DKKKkm/FVGFAmV1YIfJ6b8xTn0jPSL4Q6Wx2T8ES0ZpJZnLI5tgWcU6KSNB5gnXSYTteURTb3kRZk1XpCgG9SgzZUETybwr741ROHdSM2pr813NRRDiPvM1GDzVzQxGpjEJk695hacKsisL2xWzYgMxbb6GpaRFarvoKwl3HIZeYxOBPPo+J7b8EkN8VM0ytGMt0RMNY2VYP1eM8kZ7RKQDA0hb31EQRgWbsgQrgs/QO4NSaYR4RB+14PUvE2jnSDu51IpHrNB7htG/mJEvE6Uw5ozbIrzWjrc7mtChqNqIc8kZk3TMzm1ULGrCyrR7prIo/vnXY6+MUJx4HbrstPxmzbx9aF70DrR+9C8HmRUiP9aPv+/+MRPeLqAdw25o1+V0xNBVjiw2sPeNh3PuBkXwh0uWBIsLTO8auY3U1fm6L+1gh4qQ141QRAQwbeG0qIuQRkRAn23cBvTpN2PCI8Fp6xzNHZJLliHB6ipiNPjkjRgFQVdX1QCYnaO2Z1yVsz7Ai5J57CpMx69F4+f+Bv74Jyf7d6P/+PyMzchA1AMa2bMGdr75KfhAHaIZVCQqRpS3uFSINoYBmjOXVlhLRYuaxXFTziPBQRKgQmT9oBh7bLmYHrZkMn0AzY+Kf1dny2Yg0qwKGFEVBrZnx6bRW4MnuEQGAi09YBAB48vV+jMTF7+Axzawi5KUT3oP2D38Wvpowpvdux8Ajm5CbGkMNgIl4nFJSOXBaQRF5oy8m7OejHLmcioOj+SK+y8VCRFEU7umqIq5jPMyqWoilE0XEZmsmSTki8uK4NeMg6S7JSRFhX5/NqbbGiBnZnKq1d3gZvGbTUp9/Yh4WdNNl/pCW+hrHf69ucFJnI05Y0ohUJodHnj/g9XHyzCpCdpz6YbT91Y1QfH5Mvvo7DP70C1BT07oplfwgXGiPhrFyQd4n8rwHeSKDE0mkMjn4fYrramIT5z1ULA8pyrHFXMumEznkiDi5Ntl9+NXNqnLd+uU6jUc4zd/XPSJ2xnf5BpoBQCJlvxCJG5Y5iZiaAYDWQiEyEk8K+f5aPHUFqCFA/mnw709fDgD4wdb9yDgoJLlgKELOAvDamR9F8zkfAwCMb/0xhp/4GpDLkilVEBs99ImwtsySploEXH5qbqzj6x1j5u9oLZ/RXcAwncghR8RRa8bmPYeSVSXGuSJif9+MPjXj7Mk96PchUOixTqXtb4ZkBq+gX3HcLipFayGtcXhSrCJSCf4QxiUnLkJLfQ16xxPeRr7PKEIUvHHe/0Lj6ZcDAEaf/i7GnnkIAHBGWxuZUgUhQyHipj+E0cR5rJ9lkkQ4KiI8I965tGYo0Gz+4GTXDOCwNcNJEQH4GFbjBoOXqI2wolszlZCqOptw0I+/PbULAPDdZ/d5cwgW2X7PPThL8WHnxf+I6Pr3AwCGt9yH2F9+AgA4Y8kSPNPXR6ZUQWxYkTesvtkXmzGC6gY9bGLGxdFdhrb4jntrhp8iEuagiCS5mFXtPfymtNFhuW79cp3GI9KOzar2ZLJsTtV2BTh5UzJ4yIaa01zQxAwAtDWwQkRMa6YSFREA+OjGZfD7FPylewRv9sXcfXFjZLsvgF3v/xctqGzov76KyZeeAFAoQvbtg4/2xQijIxpGZ3Mtcirw0oFRV19bL0TcV0R4m1WFtGYcPuypqsplv5jtqRmH9zpRyHUaj+DWmrH4pjCOgDmR6Rh8FBEWiSzuRtNSn2/NjAhqzegZIvLGuxdjUWMtLjpuIQDgITdVEVaEbNuGVn8QiQ/eivpjz4SaTePwL/5/xF//PQAqQtxk/bJ89P8L+9wtRLxszfAONYtN5x+qIjzHdwsPe6lszpaXK51VwXajOgo003JE7LVmZNtILtdpPCLpVBFhxiGro1SGwkU6RUTQxAygt2biqayQxXeVqogA0Eyrj798SGjgm4axCAmGEP7rz6HuqFORSycx+NMvYvqt5wBQEeI265bn2zPb97usiIy6H2bG4B3zPpEUp4gA9q6zxsKBj0fE7q4ZuW79cp3GA1RV5egRsaeI1Ph9XJL/eKT+adsqOfZVZxMNBxD05/9/RfhEBsYrI1W1GKcsb8bqRVEk0jk8tq1H7IsZi5CaWtRe9nnULn8ncqlpDP74DiS6XwRARYgXMEXkpQOjrk1RJdJZDMTy7VJPzKp1nFszBUUkWsvvfRsK+MCsc3YKEWPopROfhh6iSWbVeQHzaABOChFnrRlexiEeqX9xzmuzi6EoiqaK8G7PTCTSmCj8P1TK+K4RRVFw9enLAADf37pfW9rFHUMR0hZuQN1H7kS463jkEpMYeOyzSPa8mo9s37iRihAPeEdHBJFQAPFUFjv7J1x5zYMFNSQSCmhFgZs01fLdQTVR8IhEOG3eBfI/n04WjBqv+U6GAZzmiFAhIhlpw9OG24Fm7E3kdPMug0fqnxtmVQBoLfhEhjgbVgcK/pBoOCAsB0U0H1i7BE11QRwcncZPtgtQRYxFSKQVdVd+GaHFxyA7NY6BR29Dqncnoj5fPrL9z3+mIsQD/D4F79R8Iu4Em7Gtu10tdcIm5soR5W5WZYoI36KqzsHiOx5hZoDe1rGsiFBrRk5SGQ6FiOYRsaeI8BjdBfik/sUF7GcoRmuDGEVE94dUllHVSDjox7XvXgUA+OJ/valNMjjGuEF32zYsaO1Ew0fvQk3bMmQmhjDwyCakBvYg6vNhmCLbPUczrLrkEzng4eguYPSI8DKrMo8I32tZmIMi4vSaH7apiPBK8uZN1V9lWIXo9yna0iWr2G/N8J3pri28uZ2YVVkhwjMEqBit9WJGePsq2B9i5BNnrsT6Zc2YTGbwTz96xXmLZvYG3cXHoOHKf0Mg2o70cA/6v/8vSA8d0IsQCirzHFaIuGVY9XJiBtADzWKJjOP3u6qqWo5IhLPfzUlrJqntFnO60sO6IqKqquH15br1y3UaD3BqVAWctGZYdcy3NeNEEZlwSRFhI7y8zar9FTwxY8TvU/DVv1mL+ho/nt83ggf+uNf+NxscBN7//lkbdL8Ef20Uyd6d6P/Bp5GdOExFiGSsXdoEv09B33gCh8amhb+e14VIo6GFEnPYnklmctpDJs9dM4DD1oymSDi79doZkJgxOkyKiFwktblqJ8Yhu1MzfGUyrTXDQRGp9NZMpSsiALC0tQ63v+84AMBXn9qF13vHrX+T7m5g40Zkfv97LFd8ePVdV6D9rz8HXzCM6T0vYODR25CbjlERIiF1NQGsWRQF4I5PhLUAOz0qRAJ+n5b54TTmnRUyisLf7xZ24MVLcHr4ZF9vJTJixugwKSJyoaeqcgiXsegR4S2T8TCr6oFmYivmVkEx71q8ewVOzBTjsvWduGBNB9JZFZ967GXzUuzgIPC3fwusWYNUdzci9U1I/s0X0XTGFVAUHyZefhKDP/si1HSSihCJWedSe0ZVVa0Q8UoRAfgZVplRNRIKcIlGMOJEEeHVjtfNqubvOcYHZSpEJINH9r7t1gxnRYTH+K5bUzOi9s3MJ0UEyI8Lbv7QCWhrqMFbA5P4zOOvlf/3LSgg6OpC5tFHcUYigejSE9B69TdQu/wk5FIJDP3qboz89t/1DbpUhEjL+uXuJKyOxFOIp7JQlPzmXa/gZVgVEe/OcJJgzasdr5tVrSgi+r3Oi6moclT9XB6P7H3bZlWOC+8AIKwFmtlflKXliIg2q2obeMWYVSt5amY2rQ0hfPnDJ+Kah17AT7YfxPb9o/jSB4/H6avagMWLgb6+GZ+fArAKwGDbUkRO/SA6jjsXis+P1OH9OPyLzcgMHwQAdNXWYt/YGC2vk5j1y/IJqzv78wvwRK1e6BnNK4kdkbCnExW8Qs1EGVUBoDZovwXOa9s6+zeyoojwzq3iiXwnchm+ZlW7gWacFBEt4t1+EqOWrCraI8ICzTgqIvFkRruALW6aX0/4563uwLevWof2SAjdQ3Fc8R9/wb/8+BWMjk7ihwAUw69o53FIfvhzWHTNfWg44T1QfH5M7ngK/d+7GZnhg/ADeFdjI/ZSESI9CxvdWYDntVGVwSvUTBvdFfBAVVuTv97b8oiwaz43s6oFRYQVQZIZVQFSRPQlQAEHZlW7u2Y4OagZumRoXxGZdNmsOpXKYjqVnbHDwS59BX9IJBwQ8iTkNRcetxCnrWrFvz25Ew9vPYAfbz+IJ//xexgZG8JiRYECAP4gApFWAICq5jD11nOI/eWnSPW9BSD/5DH15S+j5uabAQoqqwjWL2vGwdFpvLBvFGcevUDIa3i5dddII6d9M0JbM0H7LXAt0Mzhw2fIhiIi6+guQIUIl6Q5pmikbLZmeCkiTpfeqapqiHgX+9ZoCAVQ4/chlc1hOJ5EZ43zC+ChsXxbZvE8asvMJhoO4s5LT8Cla5dg089exduDkwg2L5rxOWomhcnXfofY8z9HZrRX+3gEwNDEBGoaGlw+NeGEdctb8PjLvUINqz0eh5kxGjmZVfXWjAhFxH5wJLdAMxs5IrynNHlChQiH7H2n47tOZTqGExMVOw+bMxetiLB9M/2xBEbiKXRy2PbZV8hamG9tmWKsX96CJ248E2/2xXDKhtMAqICa/8fLjPYhl9D3k9QC+NTSpfjirl3wkSm14pi9AC8gIJ5bntZMQRGZdmhW1Voz4hQRe60ZPh4RO75EmRUR+U7kMnoh4mR812GgGW9FxGYhwtZmK4ruNxEJa88Mc8oS6S0UIos8dP27SdDvw4mdTXjwrs8i1bsLqb63kOp7a0YR8q9LlmByyxZ8qbubipAKxY0FeNIUIsysyqs1I0ARcTKdmOQ0oGBHEUlymBAVhXwnchmerRnru2b4KiJO5tsBPUOkvob/7H0xeI/w9hYmZrwcP/SCK664AuoPfgAVyP/q6IC6cSPUJ5/E5gMH4KOdMRWN6AV46WxOK+K9LkQamVmVU2tGpEfEznQir/aIURFRVXNx+LwW7omg6q9OeqCZF8mqfBURJ4l/gDFV1Z03ahvnEV5NEZknGSKWuOKKfGtGVYH+fuC554ALL6QCZJ6wYUV+jPcv3fwLkb6xBHJq/jq2IBLi/v2twC1HRGRrxlHEO5/2iFFRMXvfofFdieE5vpvK5pCzsKyJd4XKFJFkxto5GG5NzDBaOI/wsgyRxVWmiBDzH1aIPN89YvoJ2CwHDBMzXgdd6WZV+5N/gGCzqqOld3yu+cavN6vE660ZUkSkI8nDrGp4U7BWjxl4V6jGEVg71bpbEzMM5hEZ4uARUVVVU0Tm89QMUZ2c2NmEcNCH4XgKuwcnuX5vWfwhgDHQLOWo4BI5vuss4p2PRyTgU8C652a9iUlOry0C+U7kMjynZgBrPhHeEe/GFo+dHxK3wswYeqiZ89bMSDyFZCYHRQE6Gr2VlwmCNzUBH05emveJbOXcnukZLSgizd4X8CzQLJ1VHe3MihUUFRGtmbCTQoRTZIOiKJbTVUkRkRimYAQdtGbsVKfGz+VVofp8iva97MiG7rdm8gUDj9YMa8u0NYSk/EEjCKdsWJEPqvvL3mGu3/eAJGFmQP5ayB4KnRhWJwqKiHStGY4hluwBOGFREeE1HMET+U7kMmkOioiiKLbmunnNlBtxEmpWya2ZQ1qGiPdPdQQhgg0rdcMqT5+IDFt3GYqi6FkiNg2rmWwO8UKRILQ1YydHhGOIZdji1nca35UYpoiEHIYE2ckS4b30DgDqHKT+TbLxXZemZnjum9HCzKpxYoaoCtZ2NaEm4MPhiSS6h+Jcvmcup6L7cP57LWut5/I9neI0XZUZVQGxishUOmu5IGTtpjoOKy201ozJe47uT5FPMaZChIMiAhhkMg89IvnvZX8hk66IuLOnhW3gnU5nHW0MBvQMEVJEiPlKOOjH2q4mAPzGePcOxTGRzCAc9GHVAjkKEaehZqwQqQ36HbXcS8GGAlTVemTDlPaw57xA0mIjSBGpfPgVIjZaMwIid5kiYif1bzLBChF3Kub6Gr/29+40XbWqM0SIqmEjyxPh5BPZcXAMAHD84kYh0fF2cBpqpk/MiGkx1xoeHK0qz/HCAxcPRSRkcfkemVUlJsnBrArYW8ssQipzsgdhMuWuWVVRFLRxSldlhUi1paoS1cWGlQXDKiefyI6D4wDy48Gy0ORwA29MM6qKUXYDfp+WO2XFi6eq+iQQT0XEtFlVgBWAF/KdyGV4mFUBo0fEnCKiqqpeoXJ8YzhJ/Yu7PDUDAC0NfEZ42dRMteyZIaqTk5c2I+hX0DeeQM/ItOPv90pBETmpq9Hx9+KF08V3+uiuuOuYNp1ocddLthA0ydMjYrY1I2I4ghdVX4jw2DUDWN83k8rm2LJUMVMzNjwXbk/NAPoIr5PWTCabw0CMeUSoNUPMX2pr/Jp6sbXbWXsmnc3hjd4YALkUEc2s6lARETExw7Cz6dyoUrMWuhPCNhURGt+VEN5mVdMpdwblhKdUxn5AbLVmCkYqNwsRHq2ZgYkkcioQ9Ctoq6cwM2J+o+2d2evMsLqrfwLJTA7RcADLW70f3WXoKqm9a4Ie7y6uENGmEy0oIuxBLxz0wc9hqWjI6vhumsyq0pLi5CS2uviO+UMUxbkaY4QVEexNb4XJZP5JwtXWDIcRXuYPWdgYdmVrMEF4ie4TcaaIsLbMiZ1Nnu+YMdLq8OFEX3gnsjVj/YFP84dwUEMA64qINhxB47vywSNZFbA+NWOsTnleBNjcfCxhpzXjviLCRniHHGzgpR0zRDWxblkz/D4FB0entSA/O+zoYUZVefwhgH5NsLuV243WjJ1QM21ihtNUouWId1JE5IVba4aZVU2PUokJl2FyJPthtIIe8e5excwj1Kx3jDJEiOqhIRTA8UvyxYOTMV6jIiITmiJi0zcmcvMuQ0+wNv/Ap2WIcFJE7NoByKwqIfzMqlZbM4UwM85vCjY7P2FREUlnc1pR5q4i4uyiAwB94yzenYyqRHWw0aFPZDqVxduFLb4yTcwAuiIykcxYikNg6K0ZN8yq5nOjeGaIAHYi3ml8V1q8CjRLCFpAxBSRCYuKiNFTUqkekUXUmiGqBLZ3xu7kzOu948jmVCyIhLAwKlcBHw0HEPTn29V2rgvuKiJWPCJ84xGsKiI0visxXk/N8FZENI/ItDVFhP3whgI+IbHIpWhl47vxpO2AJtaaoTAzolo4ZXkLgn4F+4en8NbAhOWvf6UQZHZSZ6NURlUgH3TY6mCs35XxXRsxCcyDx1sRMe0REZDkzQv5TuQy3FozmkfEmiLCWyZjcuRE0qIiknI/QwTQWzOJdM7WyDEA9BZaM4uoNUNUCZFwEGcdvQAA8OsdfZa/foek/hAGU0rtmNi1QsSN1owdRYTX1EyQ7Teb+wyiAjR5Id+JXIZbsqrl1owYmYyNrFn1iHiRqgrknw5YhW5Hhp1KZbQoaDKrEtXEe09cBAD49at9ltVEPdpdLn8Iw4l3jF37RI7v2slr0hQRTsMAVu45xgBN2r4rIZoi4nprRrRHJGPp4uRFmBlQ2DfjYISXtWUaQgGhT0AEIRvnr+lAjd+H3YOTeGtg0vTXjU+n0T0UByCvIsKuCcMWVz+oqqoXIiLHd4PWx3d5KyIhC4qIsVih1oxk5HIq0tn8zdqzqRnO1SmbmsnmVIvVujetGcCZYZUmZohqJRoO4qx3sPZMr+mve7WghnS11Go/e7JhN9RsKpXV9rmIfDCpY8GRlnJEmEeEl1nV/PZdZhngHaDJC/lO5CJMDQF45IhYXUAkxjhUG/Rr8cFW2jOTCfczRBhOZFiamCGqmUsK7Zn/stCekTU/xIgeambtmsD8IQGfInRMNVIoRCYtTCdOcc5pCltYtGq838hmTgaqvBBJGwoR58mqNqdmOCsiiqIY0lXN/5BMeuQRAXRFxE6kM4WZEdXMeavbURPwYe/hOHb2m5ueYUbVtTIXIlqombXWjLZ5tzYo9IbbULjGTlpYpcFbEdGnZsy3ZmQc3QWqvBBJGSpJbtt3LeaIiKjaozayRLxszdi96ADGeHdqzRDVRyQcxNnvsDY984qk0e5GNJXU4sMJu+aJzBAB9OukFdVZzxHhZVY1r4jIPLoLVHshou2ZURwvS7PsEdHeGPwrVDv7ZiY9Gt8FdBnWnkeEFBGiurEyPTMYS6A/loBPgRYTLyNOWzOijeu2FJGkKEXETCEiRoHnRXUXIhk+GSKAjV0zmlmV/z+BHmpmXRGpvNYMZYgQ1c15qzsQCvjQPRTHG32xsp/LgsyOam/w5GfdLLpZ1VrQod6aEfv/pnlELBQi+tQM74j3ue85ojyJvJDzVC7BCpEgh38cpmykzG7fZbn/AhSRqGGE1yzMrOqFItLWoF90rKCqqhZmRqmqRLXSEArgnGPaAczdnvnBX/YDyG/wlRm7QYdaaybkkiJiISZBzxHhHfFuXhGRMcwMqPZChFOqKmB/fFfEGyNipxBhmyE9UUQKrRmLMuzoVFr7e1xIHhGiijHTnvnDzkE8veswgn4FnzhzpZvHs0xdTUCLUbfSnokl3FFE2ANbJqeavuaLUkRS2Zw2slyKpMR7ZoBqL0Q4paoChtaMxUAzET07O1Mzcc6jZVZgMuxQPGVJhmVtmbaGkLQ/YAThBuce245w0If9w1N4vffI9kwqk8MXf/0GAOBj71qBlQsa3D6iZZgqMmRBKXXLI2IMJTP7wKdNzXBWRIC5lXiZN+8CVIgA4FSIBKzmiIhZegfoiYKWpmZS4jdWlqI9GoJPyf97HLYwOaNNzJA/hKhy6kMBnHtsvj1zxy9fP8K78L3n9mHv4TjaGmpw/blHeXFEy9gxrDKPSERwIeLzKZoqYsYnks7mtPsNb0UEmHuElxQRifG2NSMm4h2wt29GyxHh5Oi2QijgR1dLHQBg96D5qOqeUTa6S/4Qgrjh3KMRCQfwwv5RfOzB5zWVc2gyiXt+9zYA4F8uPKZiViG02Rjr1zfvir+OaYWIieus0efCa2rG71MQ9OenPRNzKPE0visxfBWR/PdIZXPIzdGvA8QGzNiZmtGTVb1x0h9VkIr3WChE3ioEOL2jQ36ZmSBEs3pRFN+/ZgMioQC27RvFxx7chngyg69seQsTiQyOXxLFX6/r8vqYprGTJaIvvBNfbDHDqplN58wfEvQrXO43DLNKfNWO7+7btw/XXHMNVqxYgdraWqxatQq33347UinrI5qiSPNURAz/wMbo+FKIVUSsm1W9DDQD8uOEgDVFZGd/vhd+zMKokDMRRKWxtqsJ3/94vhh5ft8IPvLAc3h02wEAwO3vO05b/1AJMBO7tdaMO4FmgDVFhHeGCIN5PuZSRKp2fHfnzp3I5XL49re/jddffx1f+9rX8K1vfQu33nqrqJe0TFKAIgKY84loyapCFBFrhUgqk9OMVE113si2q1ghcthcIZLLqdrG0WMWRoSdiyAqDWMx8tqhGFQ1v5PmlOUtXh/NEnbG+vXWjPjrWMRCqBnviRmGVUVE1kJEWNl40UUX4aKLLtJ+v3LlSuzatQv3338/7r77blEvawmerZmAT4FPAXIq68eV/0HQpTKBgWYmzapjU/knDp/ijqRZDKuKyIGRKUynswgFfFjeWifyaARRcaztasL3rjkVf/ed56EA2PRXq70+kmXsLMNkD1+uKiImChHeGSIMpqjPaVbVckTkbM24qsOPj4+jpaV0VZ5MJpFM6tVvLFY+JdApPM2qiqIgFPBjOp01uQ1RnEdEn5oxp4iMTuULlqa6GsdR93ZhhchALIlYIj1nQcQWfB3d0YCAhGutCcJr3rm0Gc/+67lIZnJoK0ygVBKthdbMkBWz6rQ747uAtX0zohQRpqgn5rjn6Aq8nNdK1061Z88e3Hvvvbj22mtLfs7mzZvR2Nio/erqEmus4pmsCljLEkkKXHpnlAznCroB9B0vzR61ZYD8haM9kr/wmDGsav6QDvKHEEQpIuFgRRYhgK6ImN1BlczoD4FutGas7JvhvXmXYXa1iDa+K6kiYvkueMcdd0BRlLK/XnjhhRlf09vbi4suugiXXXYZPv7xj5f83ps2bcL4+Lj2q6enx/r/kQWYWTXE6Yma9d+8XkJklCXNGKlGC60ZtvPFK6y0Z3YVFJHVi8gfQhDzEaaIjMRTpiYRjcqEG6b7iJXxXUGBkWYVEdnHdy3/a11//fW4/PLLy37O8uXLtf/u7e3FOeecg9NOOw0PPPBA2a8LhUIIhdyr3nl6RACDcWiON0U2p2ptIRFvjFDAj1DAh2Qmh1gijcY5lA5WiDTVeV+IPLtn2JRhlRUiZFQliPkJezDK5FTEEuk5r0/axEwo4Mp0kAyKSNikRyQhuSJi+W+lra0NbW1tpj730KFDOOecc7Bu3To8+OCD8Pnkqsb4FyLmWjPGPxc11x0JB5GcTJrqX44WpM8WCQoRYO7WzHQqi33DcQBUiBDEfKUm4EM0HEAskcHQZGrOQkTLEHGhLQMADSHzXjxRiojZh1/ZFRFhp+rt7cXZZ5+Nrq4u3H333Th8+DD6+/vR398v6iUtk+RoVgWMHpE53hSG1o2oN0bUwuTMSDz/Oc1et2YWmGvNvD04gZyaf2JaUKH9b4Ig5qZNi3mf27DKrnVuranQFZG5r7GiFZE5PSLVOr67ZcsW7N69G7t370ZnZ+eMP7Oy2Ewk3M2qJme6WfhMwKcIm/iIWJicYeO7XppVAT1L5MDIFBLpbEm1iE3MHLswAkWpnIAmgiCs0dpQg71DcVOGVbZnxq0IgoiF8V3hOSJzKiJVumvm6quvhqqqRX/JAs9kVcB8a0ZbeCewX6fvmzGhiLBCxGNFpD0SQiQUQE6F1nopBvlDCKI6aDFs5p6LCa8UESvJqpxNtOY9IrR9V1rEeUS8X8lsZd/MqDa+620hoiiKnrBapj2zy6CIEAQxf2m10ZpxzyNiJdBMUI5I4WHWdKBZtSkilUCKc9/MrEwmMsyMYWXfDAs0a6n3fiunblgtrYjQjhmCqA70DbxmFBHWmnFJEbEQaBYvtGa454hYfPgVsduMB3KeyiW0ZFXegWYmZTKRb4qIthnS/NSM14oIYMgSKTHCOzSZxNBkCopCW3cJYr6jKSIm9s3oC+9c8ogUrrHJTE57qC3FVMGsyn1qxqQiotkBSBGRj1Qm71cJcveImFzJLPBNoS++K9+aSWdzWrEiRSEyx+QMa8ssa6nj/nRBEIRcWNk3E9PGd925LtQb/B7xOR742J97poi48PDrBDlP5RI8d80AVloz4t8U2vjudPkfkFHjwjuXeqvlYIrI3sOTRePpd5JRlSCqBmZWHbZgVnVraibo92k+v7l8IqIUEeseETlv+XKeyiVShb6Z24Fm+gIi8YrIXDkio3F94Z0baYRz0dVSh5pCKuyh0ekj/nxnH/lDCKJasJQjMs0277r3QGU21GxKkEdEL0RKP/yqqkpmVZnhPjWjeURMtmbc8IjM8QOix7t7r4YAgN+nYGVbPQBg9+GJI/5810BhxwwpIgQx72ktKCKjU2lksuWvq4MTCQDu7syKmIx5Z+O79cJaM6Uffo0KPY3vSohXrRlR/UIjrM0ytyIiR7y7kVIjvNmcircGqDVDENVCU10NmFDL8o6Kkc7m0FNQUJe31blxNADGEd7S19lsTsV0muWIiGrNlL7nGO9HpIhISLpgVnW7NcNUCpHBO2YVkRFJFt4ZKWVYzSeu5hAO+rCstd6LoxEE4SJ+n6L7RMoYVg+NTiObUxEO+tARCbt1PFMjvNMG/4Y4RaRcIZJ/fUUBgn7v2+/FqOpChPv4rkkHsxsJgFGTUzNjEmWIMI4qoYgwf8jR7REp/CwEQYiHFSLlYt67C0nMy1vr4XPx2mBmAy9beKco/FsjTBEpFxmRNIzuyroSo7oLkQzn1oz2ppirEBFvqmKFSCJdfsad/XB7He9uxFiIGFcC7KREVYKoOlrr84bVoTKG1X1D+UJkWat7bRnAsG+mjCLCFt7V1wS4FwJmIt5lDzMDqrwQSQqLePe+NdNg+N7lVJHRKXnCzBgr2urhU/K5AIcNFx/aMUMQ1YeZLBFWiCxvc7dla0YR0T2B/P0ZZnyJepK3vLd7eU/mAmx8l1+gmTmzqr6uWpwi4vcp2l6Dcv1LGc2q4aAfXS35J5vdg5M4NDaNh7fux/P7RgAAx9LoLkFUDW0m0lX3DU8BAFa47B0z4xHRM0T4P3haUURELll1SlVHU6azedmf364Zsx4R8YoIkJ+ciaeyZSdnRqZYjog8HhEgb1jdPzyF637worYLB8j/HR+/hAoRgqgWWk2YVdm2bikVkZQ4RUSbmilnVq0ARaSqCxGvds24lQAYCQfQN16+Wh8rtGbcnL03w7GLIvjdzkGMTqXhU4B1y5px9jHt+KsTFkk14UMQhFhaGsqnq6azORxko7suKyJmPCJTgjJEAL24yOZUZLI5BIqo+7KHmQFVXIhkc6oWIc47R2SuBUhuKSJm9s3IaFYFgI+fsRI1fj9WLKjHWUe3UfFBEFUKM6uWSlc9WBjdrQ360RENuXk0a4oI5wwRYGa7JZHJoaFoIVIwq5IiIh/GYsH98V22rlqsIjLXvpl0NqedRSazKpAvjG48/2ivj0EQhMe0zaGIGCdm3B5P1SLeTYzvilBEjA/RyXRW86wY0TbvSuwRkbdEEoyxEOFmVg3OPTWTzua0gBu3FJFSHhGWIaIoQKMEC+8IgiBm06rtmyleiHQXCpEVLvtDAEOyahnVmY3vivCI+HyK9iBdyidSCYqIvCcTTCprLET4VNHa1EyZHBFjL7FBeCFS3tGt7ZmpDVJAGEEQUsLGdyeTmaLTIcyo6kXaspldM2zhnYipGQAIB8pPzmgeEcoRkQ+jUZWXnGemNcOKgtqgn5sSU4q59s2w0V3Z2jIEQRCMSCigPSwWS1fVRndd3DHDaDATaMbMqgI8IsDcQZpubHt3SvUWIqxK5FgMsEIklc0hl1OLfk7MhXh3hllFRDajKkEQBENRFCxszO+Pmb32ATCEmXmgiDBVO57KasMPs5nSxncFKSIsS6SEJUAb3yVFRD5SnFNVAb0yBWa2foy4NTGTf43yUzMj8fzHmyXLECEIgjBy6vJWAMBze4dnfDyVyeHgKFNEvPOIAPp0zGz0iHdBisgcloBKGN+t+kKEZ3vEaAYq9aaYcCFVlTHX1IyM8e4EQRCzOX1VvhB5ds/MQuTg6BRyat4IuiDi7ugukL/ms7ZRqfYMm5qpE+URmUMRYa0ZMqtKCO8wMwAI+BQwz2epyRk3FRFtA2+yvEdEtjAzgiAII6cVCpFXD47N8LwZjapebJZVFEX3iZQwrBqX3okgHCi/gVc3q5IiIh0iWjOKosy5bybmUqoqMLdHZIRNzZAiQhCExCxuqsWKtnrkVOD5vSPax7uHvDOqMuYKNZsSGGgGGGMjaHy34tAUEc6TK3NlibiqiLCpmenyOSIt9eQRIQhCbpgq8uc9Q9rHvDSqMlioWenWjDuKyJzju1SIyIcIRQTQ/7ETc3pE3J2aUdUjHd0jNL5LEESFwHwizxl8ItqyOw8LkcicrRlxS+8APTG1lCKije9Sa0Y+tEKEtyKitWbmUkTcaM3kXyOTU4sWRmM0vksQRIVw2sp8IbKzfwJDhb0zXm3dNaK1ZuZSRASZVUNmA81IEZGPtACzKmAINSupiLjXmqmv8Wvm2WKhZqSIEARRKbQ2hHDswggAYOveYaQyORxiW3e99IgUCoxi+2ZUVdUUEWHju0HWminhEUmTWVVahLVm5jAOxVwc31UUpWSWSCabQ0xbeEceEYIg5Of0VW0A8mO8PYXR3foaPxY0uD+6yyiniCQzObCcM1Hju3qid4nxXTKryktSlFnVdGvGncXH7HVis35IxqZp4R1BEJWF0Seib931ZnSXoXtEjlSd4waVpFaQIhE2qYiQR0RCRJtVSykibppVAX1MePbkDMsQaawNIiB45w1BEAQPTl3ZAp+S37jLws28SFQ1Ui5HZKqQIVIb9AtbLKoFmpX0iJAiIi3MI8J78ZxZj4gbOSJA6SwR8ocQBFFpRMNBnNDZBAD46YsHAXjrDwH01kyxvCbNHyIoQwTAnNlVZFaVGHGKiGytGeYRmflDMjpFe2YIgqg8WHuG5SAt83B0FyiviLDNu6IW3gFzKyIJas3IS0pQlVjOrJrO5jBdeLO4YVYFgGgt84jMas3QnhmCICoQVogwvG7NRMqYVacEZ4gAZhQRas1Ii4hdM0B5j4jxjeq2R2T21MwoZYgQBFGBrF/WMmPIwMswM8CQrFpGERGVIQKY8YjQ+K606Nt3+RqIQmUWELH2SDjo4+5NKUUpjwgtvCMIohKprfHjnUubAOTbIm0N3l7DynlE3FBEtGTVIr7EXE7V7nVhUkTkQ981w/cNUk4RcTNDhFFqamYknv99E3lECIKoMFieyLLWOk9Hd4E5PCKCN+8C5XNE2H0OIEVESkQHmhWTyfSJGXfaMkBpRYTFu7eQR4QgiArjw+uWYPWiKK7YsNTro+gekeSRO72mkmI37wLlc0SMKonMHhH37oiSIaoQYeFgY0U23k54oIiUmpoZIY8IQRAVSmdzHX5z45leHwOArohkCzu9ag1tGDcUkXCZje8sVdWnAAFBOSY8kLdEEoyoQqQjGgYADMaSR/yZ26O7QJmpGcoRIQiCcExdjR+sOzQxK13VDUWE+RLLKSLhoN/zFlY5qrcQ0TwifP9xFkTyOw8GJhJH/BlTRNwKMwPmzhFpqSePCEEQhF0URdF9IrOus24qIokiikgljO4CVVyIiNq+K5siou+a0Sv1TDaH8WlmViVFhCAIwgmREoZVV3NEiikiWl6WvEZVoIoLEfYPxHtqpr2giEwmMzMWHgH6mmhXWzNhfca9Z2QKALQiBACaaOEdQRCEI0pt4HUjRyRkUERmm2XZ0ARTTWRF7tMJRJRHpCEU0KrfwYmZqogXZtW2hhqc1NkIVQU+9t1tGJ9Oa2FmtPCOIAjCOaw1M+GBIsKmZlQVSGdnFiKkiEiOqEJEURRDe2amTyQ27b4ioigKvn3VeiyMhrF7cBL/38PbtbYR7ZkhCIJwTgNTnj3wiBj9H7N9IppHhBQROUllxSSrAkbD6kxFxItAMwBY2BjGd64+BfU1fjy7Zxifefw1ADS6SxAEwYOSHhEXpmZq/D5tamd2fhWbpCGzqqQws6qIf6BSiogXZlXGmsVRfPPKk+H3Kdg7FAdAYWYEQRA8KJWuOuWCIqIoip6uOsuwyhQRmTfvAlVciKQEmVUB3bBa2iPiTY7c2ce04wsfOE77PU3MEARBOKfUvpl4wSNSL1ARAQz7Zma3ZkgRkRtRHhEA6IgWWjMlFBE3c0Rmc+WGZfjk2asAAGu7Gj07B0EQxHxBV0RmB5rlC4M6gYoIAIRLhJpVilmVIt6FtmZmKyLetWaM3HLRsfiHM1agrSHk6TkIgiDmA5Ei47upTE7zIopszQC6GXW2IsI8I2RWlZSkK2ZVXRFJZ3OYLrwp3DarFoOKEIIgCD4U84hMp/SioFbg+C5Q+YpIVRYiqqoKS1YFdEXksEERMVbKXisiBEEQBD+KeUSYP6TG7xNynzFSavEdRbxLTCanggXQhQSaVScM6arsDRoO+hCkEDGCIIh5QzFFRAszE2xUBUovvtPGd6k1Ix/MHwKIUUSKpat6lSFCEARBiEXziBgKES3eXbA/BDDEvKeLKyJhas3Ih+hCpFi6qixGVYIgCIIvDaEjk1XjLsS7M7TFd5lZHhFSROSF+UN8CuD38TerAkemq3qxZ4YgCIIQj+YRMbZm2OiuwIV3jHAJRSRRIWbVqnw8Twoc3WWUUkSipIgQBEHMK5hHJJXJIZnJ4sX9Y7jnd28DAOpdUET0QLPZikhlbN+tyrsim+2uEWganZ2u6nWqKkEQBCGGBoPq8Xf/93n8pXsEQL4tc/mpS4W/PpuKOdIjQoqItOhhZuL+cWanq2oekRC1ZgiCIOYTfp+Cuho/plJZ/KV7BAGfgis2LMUN5x6ttelFwhSRI3NEKmN8t7oLEQFhZozZ6aqsd0iKCEEQxPzjqPYG7Dg4jvedtBj/9J53YHlbvWuvrS29OyJZNX+vk33pXVXeFUWGmTFmp6uSWZUgCGL+8v1rNiA2nUZXS53rr11aEamMpXdVWYiI3DPDmJ2uGmNm1dqq/CsnCIKY1zTWBtFY682DJiWrViBJFxQRVohMJDOYSmUMOSKkiBAEQRD80HJEZisiWo4ItWak451dTfjR/z5NaJXI0lWnUlkMxpI0NUMQBEEIoVSOiJasSuO78tFUV4NTV7QIf52OaBjdQ3EMxBKITVMhQhAEQfBnzmRVycd3XSmTkskk1q5dC0VR8PLLL7vxklJgTFfVA82oNUMQBEHwo7QiUhlmVVdOd8stt2Dx4sVuvJRUGNNVadcMQRAEIYJQkWTVbE7VwjtlH98VXoj85je/wZYtW3D33XeLfinp6CgoIr1jCUwXKlUyqxIEQRA8KZasalzuKrsiIvTxfGBgAJ/4xCfw+OOPo65u7tnqZDKJZDKp/T4Wi4k8nnDaC+mqew5Pah8jRYQgCILgiZYjYhjfNY7yyl6ICDudqqq4+uqrce2112L9+vWmvmbz5s1obGzUfnV1dYk6niuw1szuwXwhEg76EBS434YgCIKoPsJFxndZuJnfpyAg+X3H8unuuOMOKIpS9tcLL7yAe++9F7FYDJs2bTL9vTdt2oTx8XHtV09Pj9XjSQUzqx4amwZAbRmCIAiCP6EiZlVtdFdyNQSw0Zq5/vrrcfnll5f9nOXLl+POO+/E1q1bEQrNXPizfv16XHnllXjooYeO+LpQKHTE51cyTBFhUFuGIAiC4I3emtEVEW1iRnKjKmCjEGlra0NbW9ucn/eNb3wDd955p/b73t5eXHjhhXjsscewYcMGqy9bkRxZiJAiQhAEQfCFeUBSmRxUVYWiKJo6Irs/BBBoVl26dOmM3zc0NAAAVq1ahc7OTlEvKxXGdFUAiJIiQhAEQXDGOJ47kcwgGg5qiojso7tAle6acROjKkKtGYIgCII3dUE/ljTVAgC++fvdAIypqvLf5l074fLly6GqKtauXevWS0oBM6wCQCRErRmCIAiCLz6fgi984DgAwH/8aS9e6RmrmM27ACkiwiFFhCAIghDNeas78IG1i5FTgVt+skNL85Z9zwxAhYhwOoyKCJlVCYIgCEHc/r7j0Fpfg10DE/jmH/ItmpDkm3cBKkSEw9JVAVJECIIgCHG01Nfg84UWzduFIE1SRAhqzRAEQRCu8d4TFuGCNR3a70kRIWaaVak1QxAEQQhEURTceenxWlxEmBQRwqiIUI4IQRAEIZr2aBj/50MnoL7Gj9NWtXp9nDmhO6NgZrZmSBEhCIIgxHPJiYtx8fGL4PcpXh9lTkgREUxDKIDW+hoAM9s0BEEQBCGSSihCAFJEXOFbV63DYCyJhY3huT+ZIAiCIKoIKkRc4JTlLV4fgSAIgiCkhFozBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4BhUiBEEQBEF4htTbd1VVBQDEYjGPT0IQBEEQhFnYfZvdx8shdSEyMTEBAOjq6vL4JARBEARBWGViYgKNjY1lP0dRzZQrHpHL5dDb24tIJAJFUbw+jufEYjF0dXWhp6cH0WjU6+PMW+jv2R3o79kd6O/ZPejvWkdVVUxMTGDx4sXw+cq7QKRWRHw+Hzo7O70+hnREo9Gqf5O7Af09uwP9PbsD/T27B/1d55lLCWGQWZUgCIIgCM+gQoQgCIIgCM+gQqSCCIVCuP322xEKhbw+yryG/p7dgf6e3YH+nt2D/q7tIbVZlSAIgiCI+Q0pIgRBEARBeAYVIgRBEARBeAYVIgRBEARBeAYVIgRBEARBeAYVIhVOMpnE2rVroSgKXn75Za+PM6/Yt28frrnmGqxYsQK1tbVYtWoVbr/9dqRSKa+PNi+47777sGLFCoTDYaxbtw5/+tOfvD7SvGLz5s045ZRTEIlE0N7ejksvvRS7du3y+ljzns2bN0NRFNx0001eH6VioEKkwrnllluwePFir48xL9m5cydyuRy+/e1v4/XXX8fXvvY1fOtb38Ktt97q9dEqnsceeww33XQTbrvtNrz00ks488wzcfHFF+PAgQNeH23e8Mwzz+C6667D1q1b8dRTTyGTyeCCCy5APB73+mjzlm3btuGBBx7AiSee6PVRKgoa361gfvOb3+Dmm2/GT3/6Uxx33HF46aWXsHbtWq+PNa+56667cP/992Pv3r1eH6Wi2bBhA04++WTcf//92sdWr16NSy+9FJs3b/bwZPOXw4cPo729Hc888wzOOussr48z75icnMTJJ5+M++67D3feeSfWrl2Lr3/9614fqyIgRaRCGRgYwCc+8Ql8//vfR11dndfHqRrGx8fR0tLi9TEqmlQqhe3bt+OCCy6Y8fELLrgAzz77rEenmv+Mj48DAL1/BXHdddfhve99L84//3yvj1JxSL30jiiOqqq4+uqrce2112L9+vXYt2+f10eqCvbs2YN7770XX/nKV7w+SkUzNDSEbDaLjo6OGR/v6OhAf3+/R6ea36iqiptvvhlnnHEGjj/+eK+PM+949NFH8eKLL2Lbtm1eH6UiIUVEIu644w4oilL21wsvvIB7770XsVgMmzZt8vrIFYnZv2cjvb29uOiii3DZZZfh4x//uEcnn18oijLj96qqHvExgg/XX389duzYgUceecTro8w7enp6cOONN+Lhhx9GOBz2+jgVCXlEJGJoaAhDQ0NlP2f58uW4/PLL8atf/WrGRTubzcLv9+PKK6/EQw89JPqoFY3Zv2d2Uent7cU555yDDRs24Lvf/S58PqrfnZBKpVBXV4cf//jH+OAHP6h9/MYbb8TLL7+MZ555xsPTzT9uuOEGPP744/jjH/+IFStWeH2cecfjjz+OD37wg/D7/drHstksFEWBz+dDMpmc8WfEkVAhUoEcOHAAsVhM+31vby8uvPBC/OQnP8GGDRvQ2dnp4enmF4cOHcI555yDdevW4eGHH6YLCic2bNiAdevW4b777tM+tmbNGnzgAx8gsyonVFXFDTfcgJ///Od4+umncfTRR3t9pHnJxMQE9u/fP+NjH/vYx3Dsscfi05/+NLXCTEAekQpk6dKlM37f0NAAAFi1ahUVIRzp7e3F2WefjaVLl+Luu+/G4cOHtT9buHChhyerfG6++WZcddVVWL9+PU477TQ88MADOHDgAK699lqvjzZvuO666/DDH/4Qv/jFLxCJRDT/TWNjI2praz0+3fwhEokcUWzU19ejtbWVihCTUCFCECXYsmULdu/ejd27dx9R4JGQ6IyPfOQjGB4exhe+8AX09fXh+OOPxxNPPIFly5Z5fbR5AxuNPvvss2d8/MEHH8TVV1/t/oEIogTUmiEIgiAIwjPIdUcQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGdQIUIQBEEQhGf8P6ZyOHzGZunmAAAAAElFTkSuQmCC\n",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"filenames": {
|
|
"image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/exercisesweek41_41_2.png"
|
|
}
|
|
},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"backend = np\n",
|
|
"\n",
|
|
"def function(x):\n",
|
|
" return x*backend.sin(x**2 + 1)\n",
|
|
"\n",
|
|
"def analytical_gradient(x):\n",
|
|
" return backend.sin(x**2 + 1) + 2*x**2*backend.cos(x**2 + 1)\n",
|
|
"\n",
|
|
"\n",
|
|
"x = np.linspace(-5, 5, 100)\n",
|
|
"plt.plot(x, function(x), label=\"f(x)\")\n",
|
|
"\n",
|
|
"descent_x, descent_y = gradient_descent(1, 0.01, 300, solver=\"analytical\")\n",
|
|
"\n",
|
|
"# Change the backend to JAX\n",
|
|
"backend = jnp\n",
|
|
"jax_descend_x, jax_descend_y = gradient_descent(1, 0.01, 300, solver=\"jax\")\n",
|
|
"\n",
|
|
"plt.scatter(descent_x, descent_y, label=\"Gradient descent\", marker=\"v\", s=10, color=\"red\") \n",
|
|
"plt.scatter(jax_descend_x, jax_descend_y, label=\"JAX\", marker=\"x\", s=5, color=\"black\")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |