Morten Hjorth-Jensen
3044 lines
157 KiB
Plaintext
3044 lines
157 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d2db899",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html chapter13.do.txt -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4908a97",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Recurrent neural networks: Overarching view\n",
|
|
"\n",
|
|
"Till now our focus has been, including convolutional neural networks\n",
|
|
"as well, on feedforward neural networks. The output or the activations\n",
|
|
"flow only in one direction, from the input layer to the output layer.\n",
|
|
"\n",
|
|
"A recurrent neural network (RNN) looks very much like a feedforward\n",
|
|
"neural network, except that it also has connections pointing\n",
|
|
"backward. \n",
|
|
"\n",
|
|
"RNNs are used to analyze time series data such as stock prices, and\n",
|
|
"tell you when to buy or sell. In autonomous driving systems, they can\n",
|
|
"anticipate car trajectories and help avoid accidents. More generally,\n",
|
|
"they can work on sequences of arbitrary lengths, rather than on\n",
|
|
"fixed-sized inputs like all the nets we have discussed so far. For\n",
|
|
"example, they can take sentences, documents, or audio samples as\n",
|
|
"input, making them extremely useful for natural language processing\n",
|
|
"systems such as automatic translation and speech-to-text.\n",
|
|
"\n",
|
|
"More to text to be added"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43cb4913",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A simple example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "cf6b9dab",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAACefklEQVR4nO2dd5wV1fn/P3PvVmB3kbKFDoKAoIiggqKgKCrGkhij0aAmxgS7EqNfNEVNDCYxhvizEBMUDbYkWKMiqDSlSVOkCUpZytLZpW678/tj9957Zu6ZmXOm33uf9+uF7s6cmTk7c+acZ56qqKqqgiAIgiAIIk2IBN0BgiAIgiAIGUh4IQiCIAgirSDhhSAIgiCItIKEF4IgCIIg0goSXgiCIAiCSCtIeCEIgiAIIq0g4YUgCIIgiLSChBeCIAiCINKKnKA74DaxWAzbt29HUVERFEUJujsEQRAEQQigqioOHjyIDh06IBIx161knPCyfft2dO7cOehuEARBEARhg8rKSnTq1Mm0TcYJL0VFRQCa/vji4uKAe0MQBEEQhAg1NTXo3LlzYh03I+OEl7ipqLi4mIQXgiAIgkgzRFw+yGGXIAiCIIi0goQXgiAIgiDSChJeCIIgCIJIK0h4IQiCIAgirSDhhSAIgiCItIKEF4IgCIIg0goSXgiCIAiCSCtIeCEIgiAIIq0g4YUgCIIgiLTCU+Hl2Wefxcknn5zIdjt06FB88MEHpsfMmTMHgwYNQkFBAXr06IFJkyZ52UWCIAiCINIMT4WXTp064bHHHsOSJUuwZMkSnHfeebj88suxatUqbvuNGzdi9OjROPvss7F8+XI88MADuPPOOzFt2jQvu0kQBEEQRBqhqKqq+nnBNm3a4M9//jNuuummlH33338/3nnnHaxZsyaxbezYsfjiiy+wYMECofPX1NSgpKQE1dXVVNuIIAiCINIEmfXbN5+XxsZGvPbaazh8+DCGDh3KbbNgwQKMGjVKs+3CCy/EkiVLUF9fzz2mtrYWNTU1mn/Zwvxv9uD1z7cE3Q2CIAiC8BXPq0qvXLkSQ4cOxbFjx9CqVSu8+eabOPHEE7ltq6qqUFZWptlWVlaGhoYG7NmzBxUVFSnHTJgwAQ8//LAnfQ871/5jEQCgZ2krDOraJuDeEARBEIQ/eK556d27N1asWIGFCxfilltuwQ033IDVq1cbtteXwo5btYxKZI8fPx7V1dWJf5WVle51Pk1YV3Uo6C4QBEEQhG94rnnJy8tDz549AQCDBw/G559/jr/97W/4+9//ntK2vLwcVVVVmm27du1CTk4O2rZtyz1/fn4+8vPz3e94GnHwGN+kRhAEQRCZiO95XlRVRW1tLXff0KFDMXPmTM22GTNmYPDgwcjNzfWje2lJDQkvBMOBI3WIxXz1wycIgvAVT4WXBx54APPmzcOmTZuwcuVKPPjgg5g9ezauu+46AE0mn+uvvz7RfuzYsdi8eTPGjRuHNWvW4Pnnn8fkyZNx7733etnNtKfmaEPQXSBCwpdbD+CUR2bi51OXBt0VgiAIz/BUeNm5cyfGjBmD3r17Y+TIkVi0aBGmT5+OCy64AACwY8cObNmSjJbp3r073n//fcyePRunnHIKfve73+HJJ5/ElVde6WU3Q0vNsXq8v3IHjtU3mrYjsxER55/zNgIAZq7eqdm+ZkcNzn9iDqZ/tSOIbhEEQbiKpz4vkydPNt0/ZcqUlG3Dhw/HsmXLPOpRejH2X0sx/5u9uHpwZ/zx+ycbtjt4TKt5UVUVW/cfRafjCg0dnYnMpL4xxt1+x6vLsWHXIYydugybHrtEs+9oXSPGTF6EEb3b4/bzevnRTYIgCEdQbaMQM/+bvQCAfy81j6CqPqrVvDz87mqc/adZmPzpRs/6RoQTI+HlaJ2x9u7fSyqxZPN+PD7ja6+6RRAE4SokvKQBVjmQdx3UOkBPmb8JAPDH6Ws96hERVuoa+YMlGjHWwB21MEsSBEGEDRJe0hT2S3pnzTFuG38LPxBhoL6Br3nJMRFeaJwQPFZUHsBtryzD1v1Hgu4KQaTgeZ4Xwj2qj9SjtqERR+sbMfzPsxPbaw0WLFqTsg8js1HETHihkZK1qKpq6Bd3xdOfAQC27j+Kt287y89uEYQlpHlJIwY8MgOn/+Fj/H3utyn7eBFJPtfcJEKAkfASNXHcpmGSnfx9zjcYOuETVO5r0qxUVR/Df5duRW2Ddi7ZuPsQPl6zE8/N/cb0fP9asAl3v7YcjZRjKGP5YOUOPD1rQyjWFtK8pAmscFJVnWomOlrXiILcqGZb8MOL8BsjnxdTzUsIJiLCP1RVxY7qY5jwQZNP3IQP1uCZ6wZh9JPzsO9wHTbvPYxfjOrNtAduenEJAODkTq0xpAc/2/mv314FABjZtwyXDujg8V9BBMEtLzdFAp/RvQ0Gdwu2nh5pXtKEA0eSEUUt8qIp+2OcBYjWpOyjroHvfEs+L0ScJz/egDMf+yTxe6xZWbfvcB0AYNa6XZr27NwS19KYcaiWkmZmOnsO8bPk+wkJL2nCgaN1iZ95aw2tPwQA1NuINmKPuPc/X+AHf19Aqv8M5q8fyYXEsyOhrtksuXHPYXykS4RIZA9hyB9GwkuawGpeeJIKT/NCZB8NRj4vJsILO3b+u3QrFm/chy+2HnC7a0RIsXLYZsdHXXNwwLmPz8ZPX1qC+d/s8bRvRDh55N3V2HbgaKB9IOElTWCFF95kQ7ILAWh9XlgHS3LYJezCKuHqdJGNX1RW+9wbIgxsO3AUNzf7QQUFCS9pQjVrNiLNC2EAG230h/fX4uudBwEAEZM3nTdyzIQdIrtQOZoXgli9oybQ65PwkiYcqk06YvKFFx87Q4QWfah0vO5Vjon0wos2ipDwQjSj0bwYmCUJwm9IeEkTYswMwjMbxUh6yTo+Xb8H3+w+pNmmF17yok2vuKnDLmfokOySvejHA8/nReZ4IjMI2xpDeV7ShEZmRqDJgVizowY/mrwIADRVovXRRrk5TVKIebRR6oAya09kF+x8Y5TNm8h8GnULT9AfOKR5SRPY0FWeAEw+L9nFuqqDQu3i5iIzMxBvPLHtl23Zj/kbKKqEEBNe/vlpagZwIv3Rp08I2rRMwkuaoFXZccxGJLtkFaLzRjw5nWySunhzVVXxvWfm49p/LkokMSOyFxGz0be7D2PTnsOabYs37sOagB08CWfoP5CDVs6S8JImsCq7uBMmC2lesgtRs058VLDt//fldvzon4sSWTJ5ZqN4Eip2WO07HHxWTcJ9ZKYOUYdddo5aV3UQP/j7Alz8t3myXSNChF7zEnSiOhJe0gRW87Jo476U/VSfJruQDWVmhZfbX1mOTzfswR+ba9uYal6YbTTECKPyE3riAvHbK7bhwolzvewS4RMxndwadDoFEl7SBL2zlB4yG2UXol89caGWp6k5cLQp8aFZFAEJxQSLbJ6XB9/8yqOeEH6jX4OCNhtRtFGaYKWtJbNRdiE6ccRHhZlznVmtLHNPKyITkHmuotFGCihaLZPYe6gWs9bt1mwL2mGXhJc0wUo4IdkluxD2eWkeFzyHXUXXhnccu6++MYYnZqzDOSe0x+BubSR6SxBEOjP6yXnYWaP1eaNQaUIIqyq/pHnJLtivHlVVcaSuAYdqUx25E+15wkvzJv7YUZv/m9w3deFmPPnJBnx/0gIAQPXRehyrF/OBIPznsw178Pc531ia/khHQlihF1wA/pziJ6R5SRNWbjUvgEayS3bBThz1jSpO/M2HBi2bBgZf82IdPs2Oqw27ktl8j9Q14Jw/zUJuVMEHd52D9kX54p0nfOG6fzYlMezeriVG9SsPuDdEpkEOu4QQizelRhixmGle/rVwM578eL3bXSIChJ04jtQZa1ziw8K8PIC85LurphbVR+ux51AdPl6zU/p4wj826EpI6PHiuyeusSOH78yFQqUJVzCzKv36ra/wxMyvsXHPYaiqilXbqxPq/r2HaqlSbBrCyiJmJkVenpc4SbOR8XFGa4+m3g0V6ws1x+qDez764UPCTPipb4xhw66Dls8q6GgjEl4yBBGfl8O1DXhrxTZc8uSnuPYfC1G57wgG/f4jysOQhrBfPVZh9IB8baOE2YjZx16GFXis/LGIYKmV9EvyUr4g2SX83PzSEpz/xFy8tWKbabugo41IeMkQjCYFvfT86qJKAMCyLQcwY3WTun+jLpU3EX5YYcTMJBh//LyJJr6JG22UUPsbnjnxEwkv4eaopPCyekcNdtYc4+5zul7RSAk/s5tDoid/utG0HWleCFcwUvGZlbcnFW76wi4iUxduMWyncqKGEueAgvnf7MGnnKKLSc0Le64kpHlJH+Im4rqGGLYfOCp0zIg/z3Z0zbgzuH6KWbmtGv9eUklzTxpg5U5A0UaEKxitH3pzEk0Z2YXZGlHbEMO1/1hkcTz/BOy4EjFbEcER93m57KlPsVawGrmstkbPn2esw0s/OT1l+xVPfwYAKC3Kx4jepY6uQXiLpfBCZiPCDYx8XvRCDX3xZCmcx15rUqeGp3lhpyq2zolZeQEieOKaF1HBxQ3mfr3bdK5Zv9M8AooIHmvhxaeOGF0/2MsTbmEkvOjNBew6Q3JM+iL67HhCiNBxFj4vGs0LBRuFmmMBRROqKt9cCRhvJ8KDVRRhRmteJkyYgNNOOw1FRUUoLS3FFVdcgXXr1pkeM3v2bCiKkvJv7dq1XnY1UOoaYpi5eidqjtXbPoexw67+d8bnhSaQtEVUg2aWb8MsT0OiucFlKvcdSfxMZqNwY5UFeebqnabZme1CoyK9sapjldHlAebMmYPbbrsNCxcuxMyZM9HQ0IBRo0bh8GHr6JZ169Zhx44diX+9evXysqueo6oq/jnvWyz6dm/KvsdnrMPNLy3Bjc8vtn1+Y7ORseaFSF9EHyMvU25yn/VZjATcW15elvi5MUaqlzAjEir98sLNrl9XVVXS7qYxVmYj0fpqXuGpw+706dM1v7/wwgsoLS3F0qVLcc4555geW1paitatW3vYO3+ZsXonfv/eGgDApscu0ez779KtAJrCl//fx+txx0h5Qc1Yva9rZ5C3g0gvvH52PKHH6JJkNgo3Is639TYfotnXN00v6Y2V5iWjzUZ6qqub6vO0aWNdkXbgwIGoqKjAyJEjMWvWLMN2tbW1qKmp0fwLI9/uTmqbZq42Tqf+l5lf2zq/oc+LXvPCjEeaXNIXUZPfu19sb26fikhiQ5GrUFHQcONlKLvZo6dhkdlkTXkAVVUxbtw4DBs2DP379zdsV1FRgeeeew7Tpk3DG2+8gd69e2PkyJGYO5efBXbChAkoKSlJ/OvcubNXf4Ij2An+5peWYIlJrSI7EUHimhfrY4g0QPDZ/X3utzhW32hgNjI7fdNOEcGE8rwQPMinLrMJOtrItzwvt99+O7788kt8+umnpu169+6N3r17J34fOnQoKisr8fjjj3NNTePHj8e4ceMSv9fU1IRSgNELJGuqDmJwN74GKqYCUcmBIap5oVDp7KOuMcZdSMwEE67ZyKA9CS/ZBTsOTM1GKml3M5msMBvdcccdeOeddzBr1ix06tRJ+vghQ4Zg/Xp+CvT8/HwUFxdr/oUR/fxuJrU22HCANE5Sp/1d68NAU0u6IvPkVIPhZCZzJIONko2M2pPwkj0oUHQCrcUBNDQylqAz7HoqvKiqittvvx1vvPEGPvnkE3Tv3t3WeZYvX46KigqXe+cv+q9cM6nVTvCGsOaFHHYzAplnFzOI+jDTwiX2iTjs0kAKnKrqY7junwsxY1VVyj63nw7vfLyxRMMis8los9Ftt92GV155BW+//TaKiopQVdX0YpWUlKCwsBBAk9ln27ZteOmllwAAEydORLdu3dCvXz/U1dVh6tSpmDZtGqZNm+ZlVz1j/c6D+M3bq1K2W2teolLXMU7jzrahUOlMQUZrJpp9mX8d5mej89CgCpxfv/0VPtuwF59t2JsSzfjt7sM46CCHFEtdQww/fG5h4nez4p4y5zxa14iSFrkOe0f4SdBmI0+Fl2effRYAMGLECM32F154ATfeeCMAYMeOHdiyJVlYrq6uDvfeey+2bduGwsJC9OvXD++99x5Gjx7tZVc94ycvfo7KfanF0Mw8te2o4UWqSqtQyeclQ5B5jCqMvoxNNC+c6xg1J7NR8Ow6WGu6/45Xl0ufU1FSn/liXaCBWQZnUQH7gr/Owea9R7DogZEoKy6Q7ifhLhFF7MMmozUvIgvllClTNL/fd999uO+++zzqkb+8/vkWruACAFGXhRcRn5eYKuaASYQfmSdnpBkx9XlJLEpa4ZcHCS/BY/Uuz163W/qcUUVBg2gmZwOzkWF5AGbz5r1N2ZrnrNuNH5wWvmCLbEPhSa1G7QKEaht5yP3TVhrui5jceXvCi8EkoXG4VHVmAOnLEGlIzCDqw3zBU5vbMOcx8MUin5fgYeeMlxe5ky1XxCyQMBtx9tGoSE9ENSpmH+B+QMJLQJhNDHYWAxG/hiafF/ZLOpVXFm3BsD9+gm93U9XXMCOjNWs0cNgV07wInJ80L4HDPoIH3/wKB47USR3P/YoWWJuclp/QtCdxJxQoIg8eGV7biDDGTOXW0Kg2txE/n2GSOmZWE6k18sCbK7F1/1H89p1UJ2MiPMiajWTzvCSuYyHsip6H8Ba9afBYvVzIIk/QkPFp4I2vJl8rqW4QIUB03QnaYZeEl4AwmxjiX7Iyg8M4VJptIx4qHRegiJAiGSrN3259ehEfKXasbNpzGC8t2ITaBut6OoR7eCFAighAVknqZCBBJxyILjtBa158y7BLaDF12G1+i2XGhmG0kc7nRVvbyHi2yJFN8Uv4ioyKvTFmJ8+LeF/iC2dDYwwjHp8NANh/uB53nZ/eleDTiaD8jszMRjDwtTI8lxsdIoSpbWhEfk5qSg5Rs1HQkOYlIERCpWUkW1GfFxb299XbtQUtc4KOgyNMkU5SJ3mOuGCj1dwZRxt9uKoKfX6drCL/uUntLsJ9gtJavDh/E15asIm7z0zAJkElWD5ZuxO9fzUd/5z3bco+0amfzEZZimmSOhsmG+PCjFqfF6MFaPST87Do272J33OiNDTCjMxi1RjjtzetbRQ/VjN+DM6vAj//11I0MJJy0CrlbCMop+mqmmP4zdursO9wqoNwTJXrF5mN/OOe178AAPz+vTUp+0RDoIN+x2mFCiGxhNnIDZ8X1mykr22k5WomcyZpXjKHpgWE71BpRHycXDVpvmV7yrAbPEFHfPGS5P17SaXUOSjayD/MPlyCFkpEIeElIMzmmsQXrJTZiL9dX4hRM2hNBnCUhJdQIxVtZGhStD7LnkPJL2qj9rxCokEnsMo2go742sMRXt5cti2AnhAimA2XdHlzSXgJCDNnycbmxUBGfhDxedEnKzOb7nLJbBRqZHJoGBdmNDk/b3QYhuOnbkuXCTBT0L//fsuOOw8eS9kmq0khs5F/mM0foh8eQT8vWqECwuy5NzYvBjJmI+PCjNpoI9FFb8+hWqzcWi18fcJfZOYNo2gj0691cdnFMtKl+kg96httlEonhAn69u6s5ggvsqHSLvWFsMbsnRUVfIN+XiS8BITZwhFXw0slqRO4jj5JndnkMm/9Hlz61KfYsOugeCcI35CPNko9QMRhV6Q9z98iPnZ31hzDgEdm4JIn5wn1lbCH0zplTs18h2pdyOsT9Kd8FmHmtiAaRRR0bTwSXgIiPnh4AyARKi1zPoPRqBdWtGYj68G3opK0L35w4Egd7nl9BeZv2CN4hEyeF4MzmIZKi7fnCTXxsfvxml0AgK93UrkJLwm6vhRp1tIMF3xegpY1SXgJCFVV8fynG3H6Hz5OCTNM5nmRiTYyuo62jVYTI95fwlt+9781eHP5Nlz7z0VC7eVCpeV9XrjXNJjxeKH98bGbl0NTjB/oP17ks9s6mwx4TttEeJGJNlpbVcNtF3R0GGXYDYiYquKR/63m7rOleRH0eaGw1nDipXlONUhSZ2424piZDNYnM81LLmVq9gX9a+33wlLXIBeKz4PXfsOuQ2iVn4PykgJb/SL4mD0b9qNZVVVcNJFv8g3645c+iwLCtK6QjQy77PnY41RdG7M8L0RwSBfSk2hrXFXaRHiR1OwYkc9oXoLORZLJ6J+l3wsLz2wkXVVa13z3wVqc/8QcDJnwsZOuERxMNS+adtp9rCaVhJcsxWwej9kyGyVPqBhs13+BBz34iCRH6+UcHt0QLkQKM2qvKe+wy4bcU7FG79A/A79f7boG2SrW1m2+2U1+Ul5hdv9Zh11Z7ayfkPASEObRRvKaF3bu0qv92DYaYYZ0L6FBWniReHZG0UYitY201zQ+fyqpPi+y2iVCnNS6Zf6+22447Or7TAbHYGDXHb1QbKaV8RvyeQkIs8klWR5A4nzM0mI0wIySlRHBc6zOS80L+HlbHKr1E9s523iC9zFJAY0QRx9t5Pd7XsczG0meQ9+esjS7Tyym4uVFm03bRDQfvyYNyWyUnZiWB3CxMKPex0XzBU6CTGiQ17yI0xiz47DL22Ydjp96jeTPJLx4R9D+RFzNi8MukeziPv9buQO/fnuVcHuzEPygNfckvASE2cJhK1TaoKJvalXp5D6SXcJDg4eLj2qQWdn0ihKh1byxHC9twY5LMhv5h6zm5eVFWxxdr97GB5cefZ9JdnGfdQZhzywRRiowKzsRtBafhBeP2LjnsOl+swffqKrYduAot8y8EUY+L/pQaXZRorDp9EXG5GP09WT2/FWoeOyDtbptRn1J3RYvbcGOv2PksOsbsoUad3DS+8vAjTaSPEeq2YjZF/RKmSHkRKyXfLYsjVn+oKCfCAkvHnHu47NN95u9jA0xFTe/uETqeuxkFTGQjlVVflIjvKWhMYZbpi719BpGZiOrDLuT5nyj2yY/dho1mhcSXvzC7/ecF20k71Nl5gco3SWCg0jeJa3m3mwfmY2yErOXsbExhtU7rNV7LOyLH2VG2BrmPDJVpQl/+HbPYXzwVZX0cdK1jTjtD9Y2SJ1fqjxA3GzE7KqVDKcl7OP3Ys/TvGzae8ThWflRk4R9cqIimpckZr5UQT8SEl4CwtTnxcagYA9hvcV//94azTWpPEC4sOOcDcg5yy3euE/6K0nGwVc05LqWNC++4X+otOz4sm5vpgEg7JETsda8aD5wJZ36/YRCpT3gkMkXbRzTBGEO1fMRgwGqryotAjnNeYtdj32Z5/jq4krp8/NMPIY+L5w9cQGa9bfx0imZ0OL3rfYiSZ1Rsk3CPrkCmhd9PTyhhgFAmhcP2Flj7fwmkudFBlZtayRcpySyElg4acrwFrvvv9fzxoGj9cLX5E5wHLMR4R9+h7HKJqnjRr+l+FcI5hshhBEJYGXHjt7ZX+PMG/AzIc2LBxwW0LyYRhvZcA1g1bZRA+klpXgbTQiEAdVHUiPd7JiN2GgFGm/+4XeRZ16SOjNeXVyJQ7WNuGXE8YZtSPPiPiLmPY3mxSIiMUhIePEAkYRRZi+jnReVdYY0yg9DE0D4sPtMvH6S1RzNizEch93m/2vLURB+4fe7Lqt52XbgKCbN+QbHtchNbNMvhmGKbMkUGgwKaK7aXoOubVugqCBXZzYih92M47m53+D65xdzi82JvGhu+7ywNueogfCiPytNCMFj32zk7bPjZvyVMBvFBehGjeaFxptf+H2r7SapO8yUxUhNUhceE0W6otec8PzO5q7fg+/8v08x6q9zU4+3SKcQJCS82OQP76/F3K93481l21L2iXyEmGte5PtTJ+Tzoj2xXeFlz6FavPvFdmknPSIVu++/1/MGT3tobDYy1rwEPcFlK35/mNgtT8BOVfozUJI6Zxw4UoczJnyMB99cmdjGm7M/WLkDQDJRIXuvTUOl3eqoTUh4cQjvC1XkRTZ7Ge1MBPXMoDSKNtJL4VMXWqcE553p8qc+wx2vLsfTszZI9ZFIxfYi4/HMwfuSNo42SiWZ54V8XoIg07SqpHmR55XFW7D7YK2m9EODgDOUWah0mARKT4WXCRMm4LTTTkNRURFKS0txxRVXYN26dZbHzZkzB4MGDUJBQQF69OiBSZMmedlN1xGZOERzY4ii1byImY3ssu3AUQDAh6vkk6sRWuy+/3I+KfLwBGjDaCOTlcWssBvhHely11VB4TbThDE/4N0ykbxS7HGmhRkz2Ww0Z84c3HbbbVi4cCFmzpyJhoYGjBo1CocPG9f92bhxI0aPHo2zzz4by5cvxwMPPIA777wT06ZN87KrtuGJCWIOu/b2GSFiwrFzXrNDqGS9PEs378P/TfuSqVsl/1CO1jXi0ffXWDd0AM8B09BsxNmWdNhl2yV/mTTnG4z8y2zsOVTroJeEEUF/FYui+co3eRdIeHEHkagw9jmYKWoyOtpo+vTpmt9feOEFlJaWYunSpTjnnHO4x0yaNAldunTBxIkTAQB9+/bFkiVL8Pjjj+PKK6/0sru24C3gIi9ao8mosPO1ygovRoPK7Qktbp1a+O1ePPnxejxyeX/0LG3l6jUyjSufXQCgKQncby7th+0H5AviWRX9dAOu5sWosYnDLi9UelfNsUTRx6c+2YCHLuvnpKsEh3QxswhHs6TJ3xMmeBpRnuZFv4SlS7SRr6HS1dXVAIA2bdoYtlmwYAFGjRql2XbhhRdi8uTJqK+vR25urmZfbW0tamuTX281NXI1gbxARHipN5ld7HxlHGtoxN5DtWjbKl+qBo0T4oP+mucWAgB+/q8l+PgXI1y9Rqaycc9hnPq7mdLHqaqKldsOuN8hHdxsuBLj6s3l27D9wFGc37cseXhzs4v/Ni+xTTbElhAjXSrGm0azsBqA9PhzQgXvlvHfN630wh6nf7fNHKz9xjeHXVVVMW7cOAwbNgz9+/c3bFdVVYWysjLNtrKyMjQ0NGDPnj0p7SdMmICSkpLEv86dO7vedzN4lhOR+bjexMxjJGNc3L/c8Jgvt1Zj0O8/wpdbDxge77WkvKuGTABuUN8Yw60vL8WL8zel7HtpwWbcP21l6kEuw80HYaTRMzjHoo37MHPNzpR2ew+nJsAj3CVdFnszh25RDQDBh3fPrELaj9U3an1eXP7IdhPfhJfbb78dX375JV599VXLtnpTTNzcwTPRjB8/HtXV1Yl/lZXydVycYNfnxazOi9Hxj1zeH1NvOsP0vC+bRBDZmdDMvFoUqnxkHxN/oXdWbMf7K6vw23dWpez7x7xvvexVAt74tCMUU56XYEiXe22ewZVplyZ/T5jgOuxaRBvd8PxiQFTjlQ1mozvuuAPvvPMO5s6di06dOpm2LS8vR1WVNopl165dyMnJQdu2bVPa5+fnIz8/39X+mlFVfQyFudHkBps+L2bqcqPj83IiGNarnel5W7fINZy4vDIbxaHpxR14dYXiGJV+cBuebdxo/JiNK72aOV0W1XQnXe6ymZZaNBKJ4CNqNmLn8UUb96Fty7zkObK1qrSqqrjjjjvw5ptvYvbs2ejevbvlMUOHDsW7776r2TZjxgwMHjw4xd/Fbw4cqcOQCR9bthPRvJgKLwbHiwT3lLTIdXVQmUYb6dvSDCOM2aM0c+b2TXjh9MHo6ZpHpGl/f2z6Wtt9IsRJF02F1mzU9PMzszfgw1U7ce+oE7jtCDFWbj2Q+HltVQ2Ob98qYTZqkRfFkbpGdGvbIuU49k6bJqnLZLPRbbfdhqlTp+KVV15BUVERqqqqUFVVhaNHjybajB8/Htdff33i97Fjx2Lz5s0YN24c1qxZg+effx6TJ0/Gvffe62VXhVhXdTBlG28pEdO8GLcxMikZ5W9haV2YZysfhy0oVNo2Zk/CaGxM+Wwjvt3tfaQRIGc2MvtjNKZFFfj7HH/MXtlOuvi88LQrf5q+Dl9UHsCL8zcn9qXL3xMWtu4/glnrdid+v2jiPNwydVnCl+2yAR0A8F9d9pmYO1QHi6fCy7PPPovq6mqMGDECFRUViX+vv/56os2OHTuwZUvST6N79+54//33MXv2bJxyyin43e9+hyeffDIUYdK8r16+w64zzYtRIiERUSEvJ2LoWOm67OLu6bILGxmWH3p3tVe9Ee4DD1NhXSO7BD3dZQ/poqlg00Loe8zmJEmXvycsrKg8kLLtozU7Ex8ludGmpZ93X9lXPyXaiFnwgn4mnpuNrJgyZUrKtuHDh2PZsmUe9MgZvLT7PKdVp8JLvYHZQETR4Xa1alOHXZJebPPF1mrDfbxIH7+RKbQnalqk9cc/glbpi2I2VbLTbbr8PWHhwBG+31z8NsY/xHlLjVbzYmY2st8/N6DaRhKImG0AMSHBLE2zsebF+vqqqhpHhVgeLUeKz4vL589WzCLRfOuDhABl6rCrtRoRPnHwWIOj43N88q3iJTGMw86366oO4by/zMbbK1IL4RKpGJUPiWs/WSuC2Tw+ZvJiw2uQ8JJGRAWFF6E8LyYL1DtfbOduF7l8Y8zEsdL1aCNSvXiB3Qq9QfXB3GqkCLUj3OWu11Y4Ot4vx3CNw65u5mJ7MO7fK/Dt7sOO/65swUh4iWtaEpoX3kspZgUOHBJeJOCt1VyfFxGHXYFaRHaImWhegrZREmKEQvPiUh8U8nlJS+I+EV6j/9Cbtz7pZMqOnVqP5stM5cARfiJIveaFtyaIpnYJej3xtTxAusN12OW0E4nqESlNnnItAbG3SbvCv76dsSYXKi1/fiKVMGhe3PK7CXqCI+yRE/XnG5vVBq/eXoOJH61n9obpOz+9OFLXyN0en1pyEsJLahszDX2YfNhI8yIBz+fFbrSRnS9bEZ+XxpiZ5kX6kub9obnFECcmOjuCrdu4pXlh/5SgJ7tMYtOew6hriOHDVVWaWlFu4ZfPC6ulXr1DW5eO5hf7GPlnxuel+H7ePGX2mrJJUoPWpJLmRQLR91msqrQN4UUo2shdnxfzaCN9QS9anQBg2Zb9+NlLS/CrS07EFQM7Sh/Pjo1YTOVGuXmNW8KLWSgsYY+P1+zETS8uwWndjsPnm/Z7cg3/fF6SP2/df1SzL4BhnzEY3Tt9tJGqmleV1vPY905GJKLgvS93BP4xQpoXCXjPyn6otLuCRJwmnxf+ud02R9DcwudnLy3BnkN1uPv1FbaOZ8dGUGYXt6o9a4SgoGe7DOGVRU15sbwSXAAgJ+LP0mBeWoJmGLsYaV7i99vc58X4mZS0yMXt5/ZsbhcsJLxIwH3RbDrs2vEpEInuianGQ0+kX4RzjJwLjxrYofWwQmZQz8wtQVcTCuvKGQk/8Mvnxcw/0Cf5KSMx0tbG73bU1OfF/NzxZSjo3Ds0PCQQlF2EHHZtmY0E2sRU48H3wmebpK9pBtmk+XCruTbGMODhGULHN2jMRm71qom8HLFX3i2zEXsekp3TB998XkzGGWle7GP0+OK3O572I6aqKffZ6jWNtw/6fSbhRQLRhyWW58WbaCM/I1VocuHDewZ7DtVp0p2bH59s57bmZWDn1kLtXIs2shiPJM+I8czsDRj/xkqoqurLR4N/ZiNfLpN1GM7NcYfduHTD8XmxeikVRaiZ55DwIgHPIMMz5YgsOI12fF4EZi3VxOfFbWQcvbIJ3vOvk8hTUVvPCC8uz+5FBWI++m5dlo2cMkqcRVjzp+nr8OriLaZlJdzEymH3mtM6AwBGnVjm6Dpmc9V7K3c4Onc2EvdVs6pvl2Pi82L1kRUfGWQ2SiNEn5VYnhdvHrxZtJEd1u08iDeWbcXW/UdS9qUILy5eN53hvdSH68TTte89nEww5XYl8NEnVbh6PivY7j8x8+uU/QqAR95djb9y9hGpHJEYR06w8nn55YW9Me++c/HsjwY5uo4dzeLaqhrrRlnIH6evRb/ffogNuw4atokLNRETnxcr4h/RQc/3JLw4hPeKCznseiS8NLosvTw391uM+/cXGPbHWSn7yGzEh6ctOVwrvujsPlibPJfLXze50QjuOf8EV89phpXmqHL/UTz/2Ub87eP1rgtqhH2sfF6iEQWd27RANKI4Cmm2o1m8aOI816LhMolnZ3+DuoYY/jLja+NcX823zUzzYkX8ozXo95WEFwl4D5pnyRF5qF69fKpJtJHbkMMuH97jP2RTePFigvDzuVktTsfqxSKwiGZ8erlzLMoDsCZsJzXO7MrmMmbYbENRTHJ9Nf8/qth3uk2YjeQPdRUSXiQQd9j1JtpIhKZoo6CHFaHncK34Is3anN3WvCiKv/l5pAo8etiPTMGve2SleWF3O9G8UPkI91EUxXCtUnUOuypU6flACYnHLgkvEnCT1NnN8+KV2chXzQs5vYgiYzZicT+xoOKr5sVqnGtrpdAA4sHeF79ukZXDLrvfieYlDHW8Mg2jBHUAm6Qu/rv8+cOieaHyABLwJlee34eQw65HZiOzqtJuY/SKxO+Tk0kt05AxG7G4nefF70ci82Ud9GQYVtjpJCyaCnaBDELzEo67EA5iMRXX/nNh4ncFxtFGyfIATdKLnftvVhfJT0jzIoGbmhevPjhiMTXwGkM3vPA5rnj6s8Aduvxmy94jhqXobWteQrJY2cVqcmT3pvmf6hnsPQxMq5qyn/nZgSHS7hQR9MIZJlbvqMHCb/clfleapBcuCc2LE5+XuMNuwI+ANC8SiL4wQTrCm2XYdRve/KaqKuZ+vRsA8M3uQ+hVVuRPZwJm24GjOOfPqRFZceyaCd1Qq59zQvvEM/FbF5ZtAqwXBGFasRon0YA1LzSskujvYURRDIXcRIZdRm1h91YG/ZFMmhcJ3Mzz4hVeqpX1pi69bVWFqplUsslqtHzLftP9dr8U3Xie3dq2SPzs9zOxrJPCtiVjABf2HgaVgFKP1mzkQPNiW/Vi+5IZh35IbN572HCcxKO0okwGZdkhlaxtJHec25DwIgHfbNQcLx9TUXOsKYNokKr+JrORN1z19wWa31OySmdxpJPVn213jnbjq1u7uBhHIniBlM9Ldg4dS4IwG1kJJBqzkZM8LwEK9ZmC/l4s23IAa6v4ieriGa9ZzYvsvYwoCnKjCnItwum9hsxGEpg94x9P+Rxzvt6NmfecE7DmBZ7NcMu3HLBsk61TitUEYHeydUN4sYoc8RKr7mfreJFBM3ZUwA/jn9UV3MrzYtchnYSXJLw7YSS81DfGo42SgofsFNOhdSHWPzpa7iAPIM2LBPxooybmNPsUvLq4MljNi+qfwy7f54X/c7Zj91a4MUlrw1r9FRhI8yKOkaAa08gu4TAbsZDPS7D8+IXPhdvuay49ElW0Btt0hIQXCUSjjYJ8sfwMleZ9n2Wr30KYNS+sCcBvHQz5vIgxdeFm9P3NdCz8dm/KPlaTG+S7bYQjnxfbodLZO1ZYYjHVVsFTtmi46CP41SV9pa/jJSS8SMAtD8BbwAP1efEznDJ1W7Z+PVv93XbvixualxyXEorZgTQvYvzqra9Q1xDD7a8sT9mn8Xnx6R6ZaVOe/OFAze+OzEa2Q6VtXzKjsKvlZzUvou/ogM6tbV3LK0h4kUHgGStKsC9Wk+bFJ9WyL1dJD4wm4Q27DkFVVdt+UG6E3UciWs1LmBx2VYOfs4mDx8y/nNmh45dJ2kweKcjRLhtO5GH7ZqNsHS1a7Gpm2arhoqcI23xPwosEvGfMU1/aVWn+8PTOto5j8bM8AKBLXQ6dz4uP/QgaI4Hx/Cfm4NXFlY6ijcZMXoQbX1hsu29RTki7X5BvgjkzVlXhpIdmmLZhx5ZfwQBmief0mhZHPi82/x4aV03YFeJYU5/oKcKW+oKEFwnE87zYO/+ATq3tHcjQFK7s+DRCRBRFI/mrPjoLhw2zv3rSnG9sTzLbDxzFvPV7MHvdbnsdgzYsMmwTULbXNnronVWWbfzWvPQpN08sqRdWnPi82P17snGs8LCreWGd+MXn7HBNHiS8SMDVsug2mdWV8AM/s3EqSuoXULbOKWaTqZMv02MN4tWoDa+vizYKE9luNhL5m9kF3o/3+907hpnu1wsrzpLU2TsuW+cZPXaHQw7jsSs6psI2d5DwIgHvheE9dq/nl/wc48fmpy24SXhJXs8sLXWmY3bbFUWx/VyO1Tt3etE47Ibs64mFFiQ+rGnFj/fbMvmYi0MoyMzTmYBds1skAuQ2+73Es+5aEbaZg4QXCXgvjP7ls+uwG+FoMYyYde8Iw32+Ci/Qmo0iiqL1gcmi+cXsT1Vg/7kcq3dB86L3eQnrcwlrvzxE5FmwbRoag3fYTdG8OFhF7JuN7F8zk7B7/yKKgrxmIVVYeAmZ6sVT4WXu3Lm49NJL0aFDByiKgrfeesu0/ezZs6EoSsq/tWvXetlNYUSHiZ2viYii4JKTK4TadmhdiB7tWnL3+V3EjX15/E6AFia8Kg9wtM658KLJsBuyZ8ROh/sMKnJnO6zgGwaNg6s+L5Rh1xF2NS8KgLxmDX6d4EMIl+jisfBy+PBhDBgwAE899ZTUcevWrcOOHTsS/3r16uVRDyURddi1KbyUFObi+qFdxQ4wGEk84eW3l54o3R/RPsRSNC/eXCrsmD5zxb56/HBdg80eJYnqQqXDBHtXzn18dlDdCIxU/7jUccKOrcZY8L4HetOjE+HFvtnI9iUzCruaF0VRksKLoObFyXP2Ak9rG1188cW4+OKLpY8rLS1F69at3e+QQ0QcdpvayRMfF6LFrowGEk94ad0i10aPrFF014s0eStnJZZmI5tfmIdrnQsvmgy7IZuAsh2RtSeIPC9m6DUvTkaU7Qy7IbgPYcCupj2i2NC8hGzqCKXPy8CBA1FRUYGRI0di1qxZpm1ra2tRU1Oj+ecVfEGF96Ukf+74AiMamWLU7hBnsfNKYo4oSspkqq2Amz0TzGfr9xjuUxTF9r04XOvcbJQTYGFGO2zdfwR//nAtdtUcC7orniMyKjRmI7/yvJgMGb0ArG+bJ1Ft2O7iqwLYe6gWf3h/DTbs4hchzAbsynCKDZ+XsBEq4aWiogLPPfccpk2bhjfeeAO9e/fGyJEjMXfuXMNjJkyYgJKSksS/zp2dJ3ozQvQ9s+fz0vR/0S9jVnXL+r98vml/alsXhRdV5+PCahRUZKfipfpIPaavqjLc3+Swa+/cPGFUFn2G3TDZ9ngj87p/LsLTs77Bz/611Pf++I2Y5sW/UOn/jB0KwCpJnfZ39uPopI4lKC4U1/Q6KZvx23dW4bm53+L8J4zXh0zHmeYlCkDGYdfWpTwjVMJL7969cfPNN+PUU0/F0KFD8cwzz+CSSy7B448/bnjM+PHjUV1dnfhXWVnpWf94QklKnhebfh/xCUB0gLDt/jN2KM7rU2pybvn+GKF/V/TOhBrhJnQeFt5w4Ki5o2nEQai0G2ajqMZspBUwbzyzm+PzO2E/x0l3894jAIAVlQd87k04YT8QvHZUjQpMFtZ5XsT7aNcMFosBa6uyV+MSx7bPC5I+L/XCDrvhms9DJbzwGDJkCNavX2+4Pz8/H8XFxZp/XsEbJqKmJCvi778dE0/bVvm4bEAH43O7OOi0wgl0GXb1ScfC84XvJVbPzEm9Kzc0L1GTPC9n92rn+PxO+HrnoUCvHzzWA4MVWBo81ryIzD8pPi+K9meZse6kqnRRgacum6Gm5lg96hpi9qONFCC/2Wy0qfljQeSYMBF64WX58uWoqBALIfYaEedcu86ZcdW+6PhIqS9i8sXkleZFn3xNha62UXbILkLY1ry4EG3EFmEzU/kT/iMyLB5866vEz16bjZSUHzhtUoQXXeFPies5ybBbXOBNIELY2Xe4Dic/NAPn/WW2g2ijpMOuzDFhwlPR9dChQ9iwYUPi940bN2LFihVo06YNunTpgvHjx2Pbtm146aWXAAATJ05Et27d0K9fP9TV1WHq1KmYNm0apk2b5mU3JRAbKHY0DkmHXbERkpprwbitmz4v+kVYM5mq2r89k4UXVVWx+1AtSosKhNrbXXM8d9gN2YT0YwcFKNOFxRv34a8zv8bDl/cTav8FYz5b+O1ezDNxDneKyFRhWphRV+/MCidVpWV8azKJRd/uBQBs3X/Ugc+LIi+8hGyy8FR4WbJkCc4999zE7+PGjQMA3HDDDZgyZQp27NiBLVu2JPbX1dXh3nvvxbZt21BYWIh+/frhvffew+jRo73spjBczQtno71oI+3/rduL51pwU2Jm/9w3l2/D907tmNwHrd0okxNJ/eI/X+CNZdsw6UeD0L+juanSSXkAd5LUJScpBdpnGDbNyyyLApTH6htRkBv1qTfe8IO/LwAA/PiFz6U/c7wUXACxBcps7mnSPPvg86Iia81G7O23H20kFxWmv24Y8PTpjxgxwjTyZsqUKZrf77vvPtx3331edskRvHcyZZPNhGSJrxnRaCMJ9b+bC9Scr3dpfn98xteJn/U+L5nMG8u2AQD+3yfrMelHg0zbNkX42LvOURfKA7Bmo6YMu1q/pXRhwvtr8Pe53+I/Y4fitG5tgu6OY3bWHEvRHgQt7wtpXkyOURQ5gcTu37tt/1GN2Ygn1G7dfwQrt1bjwn7lpmb19CP5t8z52l61eXual3ARep+XMME1B3G1MfLnltW86JuZHRfRTS5OGDt1meb3BsZTvSnaKLkv6InYD0SyCusLWPqNWWHGsGlezPj73G8BNAkxmUCTj1i4XhIxs5H+d63mxY8SJbe9sgwt85LCCi8qb9gfZ+GWl5fhnS+2e96foPjzh+tsHZcJPi8kvEggnqTOvs+LsF1RN5LMQhzZBcrtxUrn8qL1eckCPUxEQDBxEirtBjkmlfOshOXrzujicm+ckymp4WNq+N6Q+PxjNixSBWBmX0Bj3WxMLGz2ESGSsKHSMkeFCRJeJBB9Je28uk4z7OqFl7gfRo92LTVjLuqy8KKvIs3OW5myyJihcLIMp7YJ9l6YRRud1r0NTihrhe4GhT5vPrsHpt99tpfdkyZThpX+fQkDdjQvKT4vPv1NomkZMstk5I4GJKIA+aR5yR5EktQB9l7eRJ4XwRdNr0HRf11fenIHLBw/Eh/cfbZW8+LyE9d/ZWkmlLDNzB4QjSiWf2eTk2xw90JfmJHtSm40gg/vPgf/uJ7vtxONKOhT7l3uJFukybiqOVaPbQeOmrYJ2zuSdL0TX6l0wUa+VbZXtdILAODgsXpMXbgZuw/WJna5/cGWCSiKglb5ci6vYbuL2emubRPhIowOzEaipPi86ISSaERBeUlB87mZ7S6/yPqJSqOJcfVK4UToS1NRAtW85LLRRoqSmptIUTQRSdp9HnbMJukyrgY8PAOqCiwYfx4qSgq5bcL2t8RNQjIRQ0H5TWlN1E088OZXePeL7Xjt82QUq0jW4HTCjb8moshHa4WtqCtpXiQQqSqtQHyh+sHgTsnjEg679qKN9JoXo6yqqWGOQpczJKVcAFvrKGwzswdEBPJa1DfE8MnaXaZtvERk8jbKBRNGh950GVfxfi7bfCDQfsgQHwbHTKLc9EOinnnp/cwFwgsOeO/LJufcr7YlC/SGcQw7wQ0hQoGCIskkf2G7iyS8SCA6aYq64bXIS0q+0rWNdENJH7LPLljsuqQ3S+VIxvrrSUlap/GBSZNVxgEikUSrd8hXOnfzazHXxOcljpG5MowTf0blDwrZnxJ/3McaxEP0l285wJzA3f6YwfN54X1HOJziQocbt1iJ2NG8uHBhF8mwx+otItFG+krLZrB5CZKaF37bdq3yU67Dolf7a5zomJ/1i2Kuw0VSr3XQlwvINBp0RcxEQqVZhp/QXqgdK3A4Re/zwsNQ8xLCGSKTZBe9kBD8n9Y0DsySI5pWnHa9P8bEdHXVjMg0h103iCgKCiWTPYYtw24Ip6bwwns/9C9NzdF67DlUy2mZCjt4rMoDnNqlteZ3fTu9L0uOkeZF386p5sXM5yX4mdhVVm2vxoCHZ+DpWcmSF5GInCbANFU/Q66Ln4s5Gp8X/nMx0vSEUfOSbsPKTBNb32j8/gRBQvNSb/wFZjYk/BwurLnK7K6xY7iq+hiufHY+3l6xzcOeeYsb91gB5XnJKrjRRrrfX160BbsOigkvLfJY4cW43Xl9SjHheydptl1zemcAwIDOrQGkLj7s1waredF/0Tv9wteHCWvyvmSY9PLIu6txuK5RkxhKxOeFRfQrUDZ1txmaDLtQuItpevm8ZNa4ChPx511rYjYyGxJmOYXcprGR/VAyHhPsh92j76/B0s37cddrK7zsmqe4EyqtUHmAbEJ2zmzXKg97DtUZ7i9ghJe4So63WPzlqgE4rmWeZttlAzqgZ2krHN++FYBU4YV9Ydld+i96p5NNisMuc5MyIc/L9gNHMXXhZowZ2lUjbMZRJCOJRKO93NW8WF/T2OfFtW5kLWFTt5sR76mp5sXk78lx0dxpRYMNs9GhY/VediltUBSgV1mR5DHhGsekeZGAXx7A+K2Z8L2TTc/XgjEbNTSrQHnjg3cFRVHQr0NJwm9GL7xoE5MZa15kJ5vWLfS1WLS9e2VRMkTxphc/lzp3GLnh+cV4ZvY3+PELn6MFJy9CRJHTBIg64ubmuDdRiJgGjX1ewjVhAennsCuTR3f/kXos2bTPw94Ax7dviSE9krWhiphxnTQbhV/z0iDoXPjkx+tx3T8XojGmplXYdFX1MYx/40usrdI6/LshDCsK0KZlHs7vWyZ+jOOrugsJLxII53lpxuqLt5D5ko/bvnmaF5HFMcVspHHYTW5P1bzIDUl9vgq9yeSlBZsTPx9xoSJy0KzfdQgAsLbqoKaWShxZs5Gw8OLiIsBe02jhTyufl/SSXaT5/qQFnp375E4l+PgXI3Bhv/LEthdvOj3xc3xhNBNezHDT0dyKhkYxzQsAfLZhL77YeiCthJeXFmzCq4srcdHEea6bSuPP+fhSfmZtXvbdsE0FJLxIIOKwy2L11coOkPrmKBb9IYO6Hoc2OpMRD70Qog2VTv6sd9KSddjVR9tkgmlIlJYGmhcps1EADrvsgtLQqHIHspE5K4xzvb77sZiKD1buEHaUz2YqmhNXskQ5Hzq1DWZmI2OcBgDIoDEbCWi3GmNqWgm+7MffQU7hSSfE32sjLQ5vOgib+ZOEFwlkB76VVoMVbuLCCztqHhzdF/8dO1TI1qgXlIzyvOgXxbN7tbM8N0tKaHQGSy/6v7VlXqrwoijW5QFYgjAbsddsjPGn+fTSvGj/gic/WY9bXl6Ge15fEUyH0oCbhnXH0B5t8cjl/QFoBRBNKH3zjw0m77V+SPz+iv6Jn52mXpChUdDnJc72A0cxY/XOxO9hrzbN5mFRWVnSjWgji1p6PEElbFMBCS8S8DPsGr81VhM/u7+uIVXzEo0owk5SKZoXTbE0Y5+X7w/qBBnqdXZmq6KE6RoZsmHXQQx4eIZmW4t8ntnI+h6wBOOwmzyXkZ+A0TgLpfCi+33iR+sBAPPW7/G/M2nCRf3L8erPhqCsOFXzwj7i+Dg48/i2wucefVJF4mcZHymnEXVazYs1L3y2SfP7na8ud3R9tzCaI9l3Mj7HzFu/Gz9+wbkvYfwxGb3ffM1LuCDhRQLex4ipz4uF/Zd9z+uaNS/aVP7ifUvJ+8L5mgJSF8VoREFvCa/zBl1eCiu/ljSVXfD799bgkE5Vm5+TKrxEI3LRRqKTu5vCi4jPixGhNBsZ/Amy/lvZjFHiyvhPT/5wIH55YW+jozW/iSRB5CFb1VgPa8IW+Ujyq2CkDPsO12HohE/w+/+tNm0Xf2/HTF7synWtNC9coSZkrxcJLzIYVJU2enGsTATsAEk67IofzyLq86KfMGTtmPqkWlaEb7oQQ3SNb6plJWM2EmvnZp4XFr3waUUoNS8G99tNgS/T0D9F9rHynPvbtcrHbef25J9LdzKR8hM8cp0KL5Kal/pGYz+e6qP1qKo+5qg/dnhx/iZU1RzDPz/dmLqTTTvhkeBlpHHlbSWflzTGaPgYjSuriZ/dHf8qYI+RUcGa+rwwT1k/wcuuTY2itQ+akVnYVVXFQ++swjOzN1g3DgDeotlUDkImw67YK+dV1EaT06J4f63GR6/SVg57JE+6CsRhRTPn2BBW7UbwyNbW0SPr82IWQXXhX+diyISPsavGXwHG7F1kpxUZ07QMhs+bp3gJl+xCwosMRqHSRgPQ6p22Us2J+kcAqZoXzdeUxudFr3mRQ/bLXeadW1t1EFPmb8Kfpq/Dff/9Ah+v2Wl9kI/w/paIZJI60cXBKy1CoyoXcWHlc/XuHcNwbm+xek1uYdT/sE2u6QI7dRjdwhwT0xAb1i/6df6DwZ1wWrc21g1N0DoVWw9qswiqqmahZc7Xux31yU1YH0uvLF4yZqOwvV4kvEjALQ+gGgfpWb3IEUVJmXDtfgWZJanTRxtd3D+Z40E2a6LeYdcKmQRdB48lfUz+vWQrbnpxidS1vIanRWoKlZbQvAhqVJyq1I1w2+5fkBvFuX1KXT2nFfExpfdJChPp5KgeFZBe8nMiOL9vGU7v3gbd2mpzg8gmMhx9Ujn+9P0BjgV0rc+LdXsjzQtrTtp+wH/TkRHs3+SV2cjo2XG/q0P2dUDCiwRG48do8RJ51h10Sd80RRQlJgXzJHXJn/NyFI0jnux49FLzYmaT9htuTh/OtoiiSE0sogJpSWGudSMbOBVeeIn6/MyqCjRVbX950Wb0/+2HeG3xFusDAiDssgs7Co20tCzRiIJ/3jAY//75UMdZl+PzkVPTqKzPi1HJAzboYOfBEAkvzM9eZZWWmf/DJbqQ8CKF0fBxosYe1PU4w2NkPkz0JiazaCNFM1nJYZb/gYfMOyd7br/ZvPdwyjbp2kaCz7RNC+vEhDKMPqkc7YvyMYrJrGoH3tdXEFE+D775FQDg/95YmdgWpslVY9BwOKy9zlobMZgrWESTz4nMefEmToVeaZ8Xg2KTR0OaCZwVWLyKlDL6mOKajcL0goEKM0rBNxsZtxf5yn7k8n5ojKm4anCnlGNkzEb6r6Ecg2ij3GhEa+P2eETKmI302XuDhPesX11cmbJN1mwk6sdklVW5Xat8qYyyT197KmJqk1DrZBrk9d7PYnxAephk0qGPcYy0MCxuCqgJzYvDRIwas5HAqDZ6JEdtlkLwHNZs5LvPS+o2ijbKMFSohouXSLRR6xZ5ePq6UzGid6rfgFthqnqfF62A5MolDJF56WTDsMNAk8OuTKi02CtnJbw89r2T8On95wpfV1GUhDbOqLuv3HwGzrPyX+GMF7/rxbDd91MQl8FNLaIncpDBR5KIw67gaY3bNP/faf0u0arSVhypS/pO+S1zml3OD7OR8RpjHkwSBkh4kYAbbaS6G/3ADiYniwL7YmtqG0W1TsJeS9MyX6CiVWLDRCQiqXkRfOPYop28nC/RiIJOx7UQvq4IZx7fDs/feBoe/W5/wza80eJ3fhX2dhfmpvrgBM3/+3g9+v/2Q9fO58ay1cskEaU2wy6/TdSmdu3P3z858XNpUT4A4LozugBwrrGTNRsZoTUbhecDip0743OMX/I532HXn2uLQsKLBEaqSWPNC/DXqwfgdImQQKc5F+IY5WPJy9H5vHg8IGWmAllnYDcwEq6Ek9QpCmRkLlHNC/vsRQpzukkBJ5NwnDD4vLDvWyHHgTho/jLza5c1L87OtezXF5g6gIvMAeL+KcmTHdciFxcykY2v3HwGFow/D2f0aCo94DjaSLIwoxFWWcK94N9LKvHkx+tTtv936VZ8uKoKgC7PS/Mvbr9phuUBBLcFCQkvEhiVBzB+bRR8d2An3H1BL+FraDNfSnROB2uCYf1hcnQ+L16jSizsfjvs1jfGcPHf5uGWqUttn0O2tpHoQh9RgF9ccAJO7dIaPzy9S8p+o8n64cv64T9jh5qe22qi71Mh9pUex3efF+ZnVngJ2+TqFk7fCivhl9W+Gr2CMlrgbm2bNIKjTizXFYCMoIKJrnRqFpcNlTaCFV78Mhvd998v8cTMr7Fmx8HEth3VR3Hvf77Az/+1NKUvXvXLsDAjZ3vYsm2Tw64EsmYjq/edt9tueQA9bAE29jQlhbm2s/jaIcwOu8s278faqoNYW3UwZZ9ovyOSVaV5kSMdSgqwXZeaXFGAO0b2wh0je+F5TupwI23PDWd2s+yDVXf7dSjBpB8NQkUJp4gfp73fodJhNxu5jRcLF/sc2TXJaCzL+Lz8++dD8dGaXbhiYAeNMOT2TOPWx87R+uB8XmqO1id+3n+4XrOPnYPYDOxu+r8Yzf/pEG1EmhcJuFWlYZxu3cqBkL/buWDRu6wI3dslE0mxX1Zlxfk6nxdvkcrzYjIZbTtw1DS9d1DIZtjlJZ/7zoAOKdvYscPTbHg9x17UvxwDOrdO2R4GsxH712uEl5BNrkFyfPuW1o2aYW+b0biS0a6VFhfg2jO6oEVejukjcWLqAdzzeTHK/+I3eiGS/ZsaPfJ5kXFyp2ijNMboBbGrebE6RqY8AMslJ1dofmfPWVpUkFKIzUu/gemrqoSTuPE0L2t21OCypz7FWY99gvMen+1q38yEQ3GfF7kcDKIRFla+T0GF4vJDpYPTvBRkuObFznP+yVnd0avUvFI8O6Ra5CUV8EaaLFFfLbPr6HE6hN3yeWE1GU4FKrfQF/yN/+y2ACFVHiBcsou3wsvcuXNx6aWXokOHDlAUBW+99ZblMXPmzMGgQYNQUFCAHj16YNKkSV520TGqavy1YsdG6IbDrt7cxK6tpcX5WpUxFDx+1QBb1xFh/Bsr8fqS1PwoPHhCwHf+36f4cms1AKSYVpxi+lUoKrxAzmwk+gXLPkKeZiOoKZY3JP0OlTZy2A3Z3OoKdhb4/NyI1DMpzIvihR+fhudvHIyW+XxPAlHtWmuHjsEy6H1e7Ar07LTj+zeBQaRXTFd2JvGnunAP5913buLnsPmxyOCp8HL48GEMGDAATz31lFD7jRs3YvTo0Tj77LOxfPlyPPDAA7jzzjsxbdo0L7spjJEGwdIGKfFCGPmqyKAfjy3ykxN825b5KQO2Z2krT51456wTK3bGy/PiVWZJQMzWL4JMF0W1FFZ+ScElQUvti9cZYPUcrk2aD1sZLLZeMP2rHZi/YY9v1wPsCan5ORFLk7P+C/7c3qU4r0+ZYXsrYejP3z8Z55zQHmNHHG96HRanY1hfHsDOVPH5pn1aDYejHokh8ner0NU2SmhenNO5TTLFgozDbtjkHE/f/IsvvhgXX3yxcPtJkyahS5cumDhxIgCgb9++WLJkCR5//HFceeWVHvVSHN6Q09smWez4rLCRHnsO10kfD6TWSyouyMWbt56JgtwoohFFszDG5fswKEv9r22kjbJg12BR9bEKVdJsJO74GIdnPnQy7ztZNMKgealjxolXNaD0bD9wFGOnLgMAbHrsEqlj43d73+E6/PnDtWLHqCoUSWfwOAW5UfQpL8K7X0gfakhni5xCVw3ujKsGd07Z7qXZiH3v9h+us+XIetWkBXj4sn7OOiKJYV4wzXykTX4a/3D2y+eFX1U6XNJLqKKNFixYgFGjRmm2XXjhhZg8eTLq6+uRm+vPRGWEcbSRgcOujWvkRiMYM6QrPl6zE+f0aid17HNjBmHxxn24lOMAOrDLccl+MR//8a6HIZu5G9EDqqpibdVBHN++FfIsKjOz72djTLW1CDc0qpi1bpdwe73mxeiaWs1L6v6gykDxeut3kjoWv27D7oPWpRjqG2OmC+hjH6zBv5dsFbqeqjaNT7ual2vP6IL6xhjO5WTuluHFn5yOfy+pxIOX9HV0Hh5Onx07X/x4yueYePUpts6j8XnxcEDtPliL+d/swagTk7lvtM7S2n6wXYn/qVYCRDSiSH1MGeZ5yXbNiyxVVVUoK9OqLsvKytDQ0IA9e/agoqIi5Zja2lrU1iYnlpqaGs/6Z/Q1btfnpaSQn3/hd1f0xyOX95NOdz6qX7lQ4b2QjcEEboRK/2fJVtw37UsMP6E9XvzJ6aZt2fugX3REJ7F/Ldws1T+9z8uMe87B65/zaiYxTtWcJ2alGbrkpAq8t3IHTut2XMq+U7sehxcXyPU70ReleVFlLu+35oWF7YeX5QFEhsNVkxbg292HUrbHe1W576jw9WKqiggUez4vOVHk50Rx9/knGLYRvVXDT2iP4Se0l++EAI4ddnXzxe/fW+O4H1467F757Hxs2XcEt+hMa0bwoo2sXjV54YW/nbc5bOtGqIQXIHUCSnhZG7xtEyZMwMMPP+x5v5r6wtlmsB1gJghd13OjCn55YW/0LG1leC0vJ2K/a8CIXs4Nzcvzn20EAMz52trPhhUQvKodokfv+Hh8e/4Y0NbsSd1v1d0/fv9kjOjdHhecmOrHcOnJHRBTVQzo1NqquynwBCmnNWqckbwRXg5rEfPNisoD/GOb/9+qQHy63bD7EPqUF9taTPMtNI4AcGJFsfR5naIfO04FBbe0j369+1v2HQEATP+qKrHNqFRLU59U3e/Wc3dORIGMs4Gx5oUXbRQu8SVUodLl5eWoqqrSbNu1axdycnLQtm1b7jHjx49HdXV14l9lpVhkix34Pi/Gg9/oWV9xSkf87Bwx6dtrQmAtSuCGz4vMRKT17tfu8+q+yJQd4P0cx+rvbJWfg6sGd0brFqnavUikKfNzDwPBybxfqV9gduveuEEQ5k5ZP5R4+yIJ4eWiifOw7cBR29FGVgzo3BpTfnwaPho3XP4CLnGaRNkUEVxZW30YT0IOu6o2EaVougnZ9BpGzdNB8xIq4WXo0KGYOXOmZtuMGTMwePBgQ3+X/Px8FBcXa/55BW/QqSbfDwlJWtcgaAFWY2cNg7NLM3UNzoUXqz9n/+E6XDRxLibN+UbzpeNlVBOLWep9Fo3mxaO+2IHr8+Kh2eidL7ab7vdr+LKX0aZtV/H2im1YW2Vsro63Ly6Q89lb2ZwiQJZubcUS1I3oXWqq/XUbvcn0rJ5yPn1eoc3z4i/sHJQaKp3sTaOgw67sh4SRNiXrfV4OHTqEDRs2JH7fuHEjVqxYgTZt2qBLly4YP348tm3bhpdeegkAMHbsWDz11FMYN24cbr75ZixYsACTJ0/Gq6++6mU3hTFOUsffEaArgClhG4T/WVKJGat3cqsny2KlkfjHvG+xtuogHvtgLf53x7DE9pRn6MEs9tZtZ6G0KDXlPg99IkE9vD8zXq3XS3iTnZc+LxM/+hqXDehgKBzwPh22HTiKLyoP4KJ+5Z6Uv2CvOOfr3bjrtRWm7eNjUjasW+9bJMKfv38y+ncskTvIQwpyo7jxzG44WteIDq0LU/Z3bF2IbQfEfYHMEHGq5qEXRv3EaC5ODZVubm9xPtls18ah0skd024ZitxoJHRmI0+FlyVLluDcc89N/D5u3DgAwA033IApU6Zgx44d2LJlS2J/9+7d8f777+Oee+7B008/jQ4dOuDJJ58MRZg0YFAeQDXLsMt/2E4jAJxSmBtF93YtcbSuEeXFYoupl/zyv1+6di6ruceoiKJe8+KF4168YJ0IVg67eiHtilM64PdX9LffOQd4mWE3/mdeNHGe6X4gObGf9dgnAIC/XDUAVw7q5EGf1MTVVm23DhBQVeDpWRvw1KwNlm1Z9hyqlR6HvHDloHnIJBQ5DOuhU6Xr4x+uQ0QBxo3qLdTeSIun0YjH9NFGYj4vsknn2PbHtcjF/iP1KX0Z1NVd855beCq8jBgxwlSSnTJlSsq24cOHY9myZR72yj62HXYZXv7pGTjzeL7/jl8oioKPxg2HqqrchadtyzzstZljhn89105liZXmxUi744fVSEYLoHHkE9C8dG7TwtMvowGdSvDF1mpcOagTnp61QdMBL2sbWX0Js3v1f//8b/a6JrzwvoJFiakq/vzhOulrPvjmVzhwpN66YRoTDuHFvtlo/+G6hFB68zk9UCRgGhRR7qjQ5nnZ1zwfi0QbycA2H35Ce7y1Ynvz9hA8GAtC5fMSdrhjTjX2euEtJmf1bBcK9Vs0ohh+MbtdL8ZPTazVpdicJLwMlrx9bhF3phOZX7RlIlL367vntf/Av356Bl648TTccV7PFD2QTNE+WZo+Dowfhtlzcvc1Yxc4SYddB1e1I/SkE2FLfCb73rNBBkaV3lOuYbheJH+OaYON8Ku3vmpuY36/FMi5K7DnYz+uQrBEWULCiwRGX/VGX2LpMAB4eN1vVVVdyenCw0rzohFeOKGIXhIXSESSummjjzl5Xpr7+/mD5+O/Y4diSA9vtXnFBbk4t08pcqMRXH1ak2liWLPAlONhqHRMVXGotsFwv5kg4ZVCSHqohMcnPnSEYY4UjeRxC21uIqM29g3XMu8j+5FktxBwUIQuz0uoMTQbGTnsKkaHhRqvVYZ3vLoc89bvwdxfnoviQneHoNXCwtbh0SSBSvF5kaNLmxbYfbAWR+sbDdvE55TcaAS1FpFVog677Yvy0b4oX7K3zvj1d07E8BPaY2iz+dNLh91YzMJMY6Z58eirXlZ4Wbp5vyvXjSjBZVb2ijCYJ1SDnz27nmHgh7YfvHVFSGsbAWA8DRmej30WJ5QVYW3VQbGTBAQJLxLwxlxTnhd+++BfS3t4HSX1vy93AAD+s7QSryzeYtFaDquFhS0ZYOQ4Z4dzTmiH91dWmQov8S8bkflaMfg5Tt8AkozFKciNCmVydg2TZ6PxeQHwu/+tTv7uleZFcokTrapuRcv8HBw8ZqyFSkfCMEdqywMEp4WpOZb0b2qqbcQ7wvqONWlexDTbRgVgbzu3JypaF2hKGYQNMhtJYJTnxWh2DcNXhR28CC/l8dGanfh292FXz2k1+bAqVXbSStG8SE5iEcX6Oz+uoRAZF1qHXW37KT8+DSd1Ck84LAC8eeuZnpxXNfEpi++Ps/dwHSZ/ujHxu5u+ZU4cdt3CzwrafhGGKVKv8ZA61um1mTNc+ewCzYl557a6X4qiSPq8JH9mrdkt8qIYf3FfDOp6nPjJfIaEFwlE1H0s8YHBy28QZtwWuoxOJ+rgJoPVwsKajeoZ002Kw67kdRWITSwQaMc7N8uIgEPtebCFP91EhfkzNXtObg5jrZYuGOmlZQYKL1ZzjWgdICe49jxN/hTVQLtjdOmYyu+XyJCWSV1g5PMSBqHSisx7GzxE2mzUPAC6t2uJST8ahHat+IUYw4as4uWmYd3x/sod2FF9zJsOSWCl0mfNRrUmwossTYKJ2I2TbVVcGGw19SCJqSrmrTeuU2UabeTw2vWNMfxp+lqc1bOdRnAIyu0kE4UXS4Hfhz40OlG9CGJUt83wgxgqd5+IUCHjg2ZkNgpDRKwVpHmRgLfANRmNrM1GF/Uvx2CXa3l4hazm5RejjKvX+o215sVIeNG2k5VlIooi/LUiOzHwKkNnCzEVphlszR6TUw3i659X4h/zNuLGFz7Xmha8CZSzpCgThRcJ8WRgl9YoLsjBo991Nxnj07O+SfzsVVVptvQJewWjj6YXPtuEVdurU7aL3C+ZvEvsK6JNjBl+SHiRgJukTjUxG3nbHc+QXVwVKIEUyONhpgL+84drNT4RtQ1J51qntY1ktFUibfU+L3ec11O+UxmA1bgye95OXbcq9x/hXzMg3UvLfHfzL4UBa1Nr8ucRJ5Tii9+OwpWnup81WZR1VQdx56vL8e3uQ6k7TYaFRngRUPQ8N/dbbNqbOv5EpmYzof2GoV0NzxdNszwvmSfKB4CR9Jy2DruyPhkh+jON1rLKfUc0X1gAUFvvns9LJCLzDSl/w9JBjesN5k/C3OfF4T0z8UcIgsw0G1klXdMuqIqiID8ngmhE8aSYqpWw/L1nPsPhukYsr9yPefedpz2WGTCV+47gf1/uwHVDuqC4IBd1TF6rhhgryMgHBlhhlDRyzJCu+M2l2lIN2mSY/J/DSua9DR5iWFXawucl3ZDN26Eo5l+jfmbRNBIkeTZnjdnIoSmgaWIVb2vZxuL3bMFyfTLzeXHTYTfAcFqg6W9pmZd507XVI+JVV1cUBS3zoqjxIGzc6tEermvS1lbuSy0myR77nf/3KaqP1mNH9VE8cnl/TSZe9qPJi5FkNH/3qShK2adx2GXsMOkw35DZSAJuUT8zs1GaSi92zEZhwWgyyOV8jbBmoxShx1aotHsOu/pnkKZDyTFWgoKZo7Wb49LvRGZ6FAAtMtBsZJUMlnUiZX/2SgslXfrBoHn10aacLcu3HEhpV9to7GvnBjKZco2S1IVoSjeEhBcJGhrlHHbTiR7tWgIA8nMi0majiBIeAcYo1bfeZARoNS96wdSLUOlEWxu3Kiz312+sJndT4cVFqxF7HT9KSehp0jZknubFyjxhtBC3yAufIMcbFeUlBU37mJ1a/xfxsaSqqqWwpyhazcvvLjeu6N3Unh9hlA7zDQkvEtQZ1OMJi7OqE56/8TRcfkoHvH37WdL2TkVRbAlwXgh9Rmd8lZPJV6O+dfgQRZLUsW2t0Lf47sCOALIv8sha82K8z6nDrmE+jgDedwUZ6vNisT+XSW3AvjZuF4+N42QaiI+XY0yW7Yq48GIwaGSup6rWQoWqan1eTu7U2rQ9+45QnpcMpp6neVHVQL7E3KZbu5b42zUDAdhw2LV5TS+ke5lHUdfIRhvZPw/QrH0SfOPt/NVd2rbAlw+NysivbzOsnoNZUT2nZlujquOBmI0UoGUItQ2OMXlGZ3Rvg3N7l+KxD9YC8MehVPbZskJJ/Ce2kGg8K7JZPheZvgnleZEyG7H3lL89rJDmRQJeJWRVDS5plVdI+7wo4dE+cZ2qBaqBu5Okzr22PKe74oJcT4sghpGDJhWlASufF2dolC26sVJ9tB7Pf7oRu2r8ScyoQEGLLNK8DD+hPV7/+VBNUkm2rR8lTLbsPYIH3lyJjXvESpjExwg7VuKBAkajVCZQQFVVS6FCbzaymmryDDRb6TDLZN7b4CH1POEF4Vm43UI+VNoqWYP9vsjC+xA3TMHNNNZ/wcuatGSS1Jlx9eDOOFzXgE7HpVdJiaCYu36P8U7HZqPkzzGdCen+/36J6auqnF1ABgU4q7mKt54zj2+L+d/s9a8vLmI11+QYLMReyS7sM//Ji59jw65D+Gj1Tix+8HzL9vE5g5076htjaIypJh9Q7mte2PptVtpt1ndISTOzEWleJKjjmI2A4GqdeIVfX/de+LwYZUG2auvU6z8iESpt5nT3x++fjKeuPTVtI9X8xizXh1PVt8YkoBNkPlqz09G5ZVEAtG2Vz3XATGdtnNU416SsZxZimWc7oHNriR4lH/SGXU2J6HYdrBVoDXxddQivLNqi0aas2laDkx/6EJPmpAYMyKKqYn83O7+wzXnLVGFulNuWHHYzDLaQX5xMNBu5bu/08QbxsyBbm41Soo1kfV4iMqHS4Z8YMgF2Ta/cdwTnPT4b/1q42da59JoXv30C4pdr3SK1Plo6C7pGclf8T3KqeRl1Yhl+MFg8I6/se8/OLT+avAgPvLkS/11amdi2eNM+HK5rxL+XbHV8PRWqkD9LjlVIEkMBo3lpNPg4DyskvEjQwDFQqlDxwmcbA+iNd9iZDO0Me08cdjk9Meob+9UeX5xUVcVjH6zFqu010tcW1ryk71qTVrDj608frsO3ew7j1299JXy8VtuiO7fPzzD+t/Cum87jyWoO0NTbMfjZ9Pwe3Zv4eXnCxxdbq4XPI+Wwq4qVU5DRxLGal2NM3qt0gIQXCXhmI1UF3l/po+3bB9J5MuRZEYzsyhqzUfOBc9fvsaXilQmVTucv5XSCHcfss174rbx/iD7PS1CaF951ZaJLwoZV19mFmNVyiP7NCsQ1ooD2Qycvarw8mj1/GW2KrLla5NwyDrtsodpjmsy/4dfCkPAiAc9slIl4lQCKzX8AeJTnhRttxG/L83lZv/OgrevKCHzpu9Q4x9dwX2bmLshJXvea5xZi3+E6qVNpHDNV/wV8Rfd/zb6sEV6S20UtIzJlO5qukbyIUY0gIPkc+HOLXOI58bZiDr5RAz8hK/Tzc9gh4UUCXrRRJvLgJSdKH2P6TinAvxZsQp9fT8dHq711dJT5ktH4vDT/stNm6GtTtJHgRJG+a41j/vC9k3y7FnubC3K1U92eQ8ZOmHGM6hnFVNWXUF2W+NjijTETBUHosVpc2YWYXbhFNV+KYv91MzO/JMxGPDO1lB+LTFuxnGJGfkJW1BpUvg4raTzs/YdX3E9VVfStKA6gN97RsXUhyosLXD3nr99eBQC4/dVlrp5XBJFQ6fjitLPGelHjITNJZqrs8rNzeli2yQ1opbWTkZUdNuyrf95f5uCgB0UBzUhoXjiDp7gg19e+uImVBiVHI7wkt4v7vMilMDAyG839erf2vM1PhDe3yIQ/myVZ1HO4thHf7DbPOaNAsS1Y5zM5X1oVhD+LCgkvEtTxoo0C6IcfuGnS2c2EGpYUJidav6R7o7/l7RXbEz83qs41L6JSSTqr+c0Yf3EfvHrzENM2Zn4EbsNqSwp1wovIEzDK8xIIJj4vXqXK9wMjzUt8K/v3ajUvoueXM52wj5k1G/3yv19wO8gbFTIjRUZT/Mfpay3b/GhIF9ual5iqYvEDI7H4wZGBfWTIEP4ehggjsxFvO7tIpyM9S1u5dq7FG/clfmbvix9r+LH6RsMJ4mg9W1W66f9mOR3MiFjILvdd1FvTNhNRFAVDj2+LJb86H/eOOoHbxs+cJOxz15uNRGCF3qCFFyOfl46t0zuZoecOu5J2I/YpsyHHegEx3i0ZHzv+9cQbLzBJRPjwZf0w7ZYz8dNhPWz7vNTWx1BaXIDSIne17l5BwosE3Ay7aur2ooIc/PvnQ/3qlic88YNTcOWp4vkRRL83vFJx7z5Yi7eWb9Ns++P0tejz6+lYsmmfwVFJYg59XhQLn5fLT+mYbJuxhqMm2rXKR1Qi14RXsAJHfo5O8yLwCIJWtrDEx5b+tpaXFKRFNlQjLJPUaSLGxI9LtIN9My2bOr9AP37iZiPOcVJ+LBKNzRyI83MiGNT1OEQiirTm5a6RvdC6RS7uHNlLvDMhIPyGrRDRwAuVhpoShTTlx6ehd3mRX93yhLLiAvzlBwMwbRk/uZJdvNJIfe/Zz1C576hm27Ozm0KeH31vjeXxMVVFQ2MMR+rsedxbaV7Yfem82IjSpmXwmseYgQnA3rkC1rw0d18v+BbmRtG/Y0kAPXIHK0UcK6SweZlEFXgyBVMBXbQRcxG95s4sz4tsBJEoolpLrebFmnsuOAF3jezluxO6U0h4kaCOo3mZunAL2rXK12zLi6avDdprij3yedELLixmKeTZNnazrwLWjoHZILCw9GhvYHb08T5oo4VsHM/8LFNAzwvivh/6cVSQG8H3T+2E2vpGDO7WJoCeOUNmOKganxcJh13JPsXJYfw+8g19ppxNYjKCzrcmzrrs7bBjmk03wQUgs5EURj4v+rDLfBv29WyhVQCVcesFVh5VBaqq7VcIbkpSZ5YXglXlatvdNKw7igtycOOZ3WxfP2z0aNcy6C7oooX0i4T1ZB0mh91ktJG23/m5UUQiCsYM7ZaWUY8yWhH2G0Q4z4tkf1hyo6zmRe/z4k60kVujip1fNIUZ008mEYY0L4I0xlRhz3A/IyrSjSBepvoGAc2LqmryHMhiVZiR/bDRf+SUFRdg+W9GpXWBPT1tW+WjR/uWpl+LXhPjlH+Qw5nmxk2SGXa12/W+GOmGcW2j1B128rzA4r3Uwz5nNuKmIEc3p5tFG0nlmnJ/YGnvTebMKXpolRVEJkFdnn6gZwFO1fLi15E/ipefR0/MsfBiZbw3b5tJgkucCd+VT0g3pId7po+Yyv/Z3rn8k15+fFY3ztZms5FuMbITRRUm7r2wNxQF+Plw6xxBGs2LTHkAqWgjI58XvtmI7/MicT0PhhXr36WpKu3+pQLFl5H/zDPPoHv37igoKMCgQYMwb948w7azZ89ORG6w/9autY5x95LcaARTbzoDI/uUWrbNz0LhxS/iL/v8DXtw73++QPXRestjGgXMRrGYiloHhckUC8dArdnI9mXSH5MZtHWLXLTV+Y85QV+PSBZNSQA3OiQIz7RqpHnR569JN/qUF2Pd7y7G+Iv7WrbV+ryInb/Jkd5enheN5iXFYTcebcQP4hDFboBACgI+L+lcA4uH52aj119/HXfffTeeeeYZnHXWWfj73/+Oiy++GKtXr0aXLl0Mj1u3bh2Ki5M23Pbt23vdVVOiEQXDerXDxj2H8PHaXaZts1LzYuMYEY2I0XWu/eciAE2C4qMWX/himhd+EkJRrBwD2Xkjs6YQY/TCXJGAv5Ob98axw25APi88rYKRz0s6J6iLIzpf2i4PYHNQsUKIPtQ+ogCvLt6CyZ9u5PRT/BqHat3P1MwKKQqAn5/TA/PW78F3B3Y0PigN8XyVfeKJJ3DTTTfhpz/9Kfr27YuJEyeic+fOePbZZ02PKy0tRXl5eeJfNI0iePQDPROoKHEncRG7BnxRecDG8dqZYet+4yijOLwQdz2NMa99XljpJVvElyT/uul0zP7lCF+vqTEb6VYUoTwvmiR1bvXKGq7w0rwpLyezzEYsf716gOl+e3le5N41dnphcxXptemKomD8GyuxYdchzjmCNdBoq0orGD+6L96/62wU+lkU1Qc8Hfl1dXVYunQpRo0apdk+atQozJ8/3/TYgQMHoqKiAiNHjsSsWbMM29XW1qKmpkbzz0tEhmWuw5wSmUyjwxc7JWZE4FaLhEp77fOSHS50xpzevU2TSUgir4dTeFXD7eLngsTz948vwvoPo0zQvMT57sBkUkxePij2eYrGRFiZc/WwAmt5cdKEGZWY04OQXdje5UjmeUlXPBVe9uzZg8bGRpSVlWm2l5WVoaqqintMRUUFnnvuOUybNg1vvPEGevfujZEjR2Lu3Lnc9hMmTEBJSUniX+fOnV3/O2TJ1No1biBTiIwHb2J478sdpsfw8vPoiakq6hz7vJjvj5OBvrlctKYy0S9l99BUDdcNHKvrfLZhD/69JJmg0em4lSGHszLH76VeA6DPP5Lu/O2aU3B6tzYYP7pPyr5YzKbZyGZfNENG9/jNzhm0Y2xEMsNuuuJLqLR+MVdV1XCB7927N3r3TtaBGTp0KCorK/H444/jnHPOSWk/fvx4jBs3LvF7TU2NpwJM0CGT6Y6IFsQMnjPcba84r1QdU+FY82I3z0s2IBzZ6uKt0fq8aMeN1Si8rtmnKo6fZqMcjnQb36L3DzmxIr0zeeu5/JSOmlIaLL5UlTZw0tb7PJmdM+icQLzxk4l4qnlp164dotFoipZl165dKdoYM4YMGYL169dz9+Xn56O4uFjzj/AfUbW6c/W9s+ONaIypqK13KLyYaV6YNy07phZ5U5ki2E4Us2gj2XHk53LEixaJL9as2Wj0SeUY1DX9suraxW5VaRk0yhaTUPuwfYCw/bFbmDHd8FR4ycvLw6BBgzBz5kzN9pkzZ+LMM88UPs/y5ctRUVHhdvdsEbQzVlhoadP5K6z3T1VVIfOSEVbqac1CnrnziSEyX8puYbb4yIojbo3bYT3bWbbhmY3isGajS0/u4Eqf0gVbeV4kQ6Wh0bwkfzla34jKfUeS5zU7hU9THBtEwfYnSmYjdxg3bhzGjBmDwYMHY+jQoXjuueewZcsWjB07FkCT2Wfbtm146aWXAAATJ05Et27d0K9fP9TV1WHq1KmYNm0apk2b5nVXCQku6l+hKdoo+r46dtjVHT573W5H54uzYdchbuSAKBEFpjMFuyhn8teQEaJ/sVc+L3rNS+X+o+hZKm5yccsUIJKMkGs2ivu8MNFFToTtdIT1eRFN6iibpE4D88hfWbQFryzakjyvyTn9+kAzMsFni9nIc+Hl6quvxt69e/HII49gx44d6N+/P95//3107doVALBjxw5s2ZIcFHV1dbj33nuxbds2FBYWol+/fnjvvfcwevRor7sqRDj1Bv6gKEnhQSYRE4tjs5FHT+CtFdsdHW+Z50XT1tGl0gZb0eGe+bxo9/34hc+x6bFLhM/lls+LU+GFLT3ixEcrHWEFSBkfKpkhxc4vZo/c7G33a40wEqijooWf0hxfHHZvvfVW3Hrrrdx9U6ZM0fx+33334b777vOhV4QsCpy/mF5EG4UBK58XVs2dLcILS1zzdObxbc3buSi9sJO7U0dxtzQvIuaOHE5Ybvy+sCYlJ0kV0xH2CYiajazey5RrsGYjk2duJoP6NUexY1pbVRrc7ZlGdohoLhLWxdNr+lYUm04Y7J5TOrc2bOd0EfEKp7WFIlY+L5pQ6QyeUSzIz4ninvNPMNzv5q0xMxuZHscZo26991Yq/Y6tC7lfzrz7ki3CS7xcwrm9k6VZ5F5Xe4PKVPNiMlD9ijYqLeInDo1qqkpn7lxDwgthygd3nY3rh3bFn75/sunC0iIvqcR767azMPwEfjkHt5PUuYVTO3GkuQaXCBk8n+jg/6Fmt9qraCPesDtwpA5rq1KTWh6qS03Z7pYfg1Wys/fuHIZck1BplmzxeZn9yxF4+adnYPRJ5YltXpUHMIo2ksFr2aVfh2Kc1bMtnr7uVO7+bElS54vZKJMwehF6lrbCSR1LMq5+RN+KYjxyeX8AcdU1/83UF5Mzuk9OFwGvnOHiGqHLT+mAt234vygKsO9wnen+xM8ZPaVYI5rMzykJ/yxV5X4Nn/GHj1HbEMPbt52FAYy28BinWJ5bCkOrRTcSUUxDpVmyRfPSrlU+2vXUFuyMyDjsSlxL4ydldl4zs5HE9ewwpEdb/Po7Jxr2h5LUEVyM1s52rfLw16tP8bUvftOhdQE27T3C3dcyn18yXs9Ha8yLWlohMzG0ys8RLnwW1wjZfdcVRcHGPYeN92dhVWnZv9NtFXdMVfHOF9vx8Dur0L4otVp13OF17te7tcILJ9+Pez4v5vsVaKsZJ7ZzjqvPEs0LD+E8L4pseQDmZ1OfFxOHXY9VL1Z/TbZEG5HZyCV4E06m8c8bBuOsnm3x758PTZEiWgpUDHYDmXlB5h2On9euP4rVtdj9mWyHFsHsGbrtsHvnq8ux93Ad1lYdNGyn/4o/xikT4ZfmRVEMNC+cttmieeFhdh9ZLbATOcKu5sVreNduXZiX+JmS1BFcjAZ0XhYILz1Li/DyT4fg9O6pWT31ZiPPkJiMbC04Nt91kUXJ4SXSjlwbIZteOeyaoX92x+pThRe3vqat/jwFRqHSqds6tWnhSp/SEaMPgOfGDML/7hiW+D2mqpJmI/YXk+ubnMNrh112vP75+yfjxjO7YUTvpI9hDpmNCBmyQfNihl+al92HjqHmWL1QWzuRTXa/VKwmiWzM89KvQzFG9ilFRWttVITXPi9xc6GowBGf6zfuOYz2Rfnc/CluRcnxFt1RJ5ZhxuqdzfsNCjMyP//750Px6YY9uOa04IvQBkXUYKCM6leOw4ypuKmOnvh5zeoZsZhpTz0PNmIufdXgzrhKtztCDrsED6MJMTcnu4QX/V3wS/Ny/hP86uI87EQ22TUXW2temJ/tXSLtiEQUTL7xNMmjnN+dQV2Pw5yvdwsvIhFFwVfbqvGd//cp+ncsxv0XpVY0dholl7xW6ra+FcVJ4QVGDrvJn0/v3oar/cwmzN5T9v41SgovLObmTZPj7F1OGKsPLI3mLoMnm+xacV2gR/uW3O25FiGQmc75fZsKbYbpPtjSvPhhNsoW1YsN3Lg18TEoqr5XFGDqws0AgK+21fAddl3TvKRuK8iNavbzq0rTmGExizZi38WYKlvbiI02MtO8mJ3Ca7OR+X6nOavSBRJeJDm3dykevqxfyvZs8HkxY1ivdnjtZ0Pw6f3nAQjHAm1HePHKYddu22zDjVuT0+xrI+PzsoZx6OX5vLgV2MMbX/k6rS03wy6NGQ1m94MV/lQVUoPqi63VWLZlf/JYw+ubmI3EL2cLq7HAmtQyWejN7hXXBoqi4IYzu+HkTiWa7dnu8wI05R8oK+ZnfUwX7C4Scsdl7oQSBqI2NC9fVB5I/M4TXtxywuQtemyxxSbNC80lVph9ZLBamVhMzmEXAG6YvBiAAyHEY+nF0mzECL+ZLPTSW2KThkbtCCXhJVOw67ArflwmTyhOcOu2xL+8ReWNHdXHEj+3aZmHYxyHXfeEl9RtBTmM2QgKV/NCaDFy2NUTU1VpLfDBZodfuz4v3kcbme/XlAfwtCfBQiuuTfQmidycTB4m2YNXDrssNFK8JWk2EltE2GRvuVEFtVyzkXcOu6mal9RGDSGtCRYUvNfttnOPT9nW5PNiF5tJ6mxfTxCLuUZUsEt3SHixyYX9yjS/52eZ5sVrp7SgsPveyzgqZ3NhRivcWKPji7+o8MI648ZUcEOlvawqna/RvPAdLt1yGM4UePfxlxemRok5mafMfV7sHecGlrmC2GCjDJ5rsmvFdZHbz+ulGSRuhVJmCum6QNt1cNM7XZpRkEuvnRFuLNI5CZ8Xsfbsu6uqKt/nxSWHXd7oYseOoihcEzRpXrSIakhjNkOlN+w6iI/X2itlYhal5AZWc2s0S/K80Cxqk7ycCDqUFCZ+j4cKE02ka0SN3X7nRaPWjZq5/+I+OL59S27UWjZgJue78REQX/xFv7rZSKKYyk+776XDriZUGnzNi1tmq0xBtDCjdKh0M3e/vsJ0f5BJ6iyjjbIkwy4JLw6447yeAICfnNUdA7scF3Bv/MXq/UzXXAN21ay5OQpe/MnpQm0rSgrx8S9G4IYzu9m6VibjxiIdjchpXrRmIxX1jakHelnbSKt54fu8kPCiRVSzazdJXc1R84KuZtNb0GYj9t5kcqg0Zdh1wNWndcbgbm3QvR0/cV02k65mI7vkRSMYfkJ764aE6WIiukhHFGOBIm42Eta8MO1iMZVbrdnLaKO2rfKY/QpXgCazkRbR+UWVrG0Ux+r0q7bXmF7TS6y0Tun64SgLaV4coCgKepa2yprBwvK9UzsBAHqXFXH3p6vsYlfoysuy8hBe8Pcxg4TNRmbPKVcySR2reVFVoIHj4OKew27qtk7HtcBvvnMi/vT9kwEYmY2yt4I0D2Gflxh8d/wIWs7URBul6TwsAs24hC2Gn9AeH407B2/ffhZ3f7oKdHaFLhJe7PHI5U1+P7+6pC8Gd2sj7LBrJrxEI3KaF1YwaVRV1DX4azYCgJ8M644fDG4qtNgiN4qC3IhmgSbNixbRj4ymqtL+zkVeO+xaFoFV+D9nGmQ2ImzTs5SvdQHSN9eA3V5ne3kIu1w/tBsuH9ARJS1yAUj4dpg8qFzpaKPkzzFV9UzzcnavdkIDLBJRsOI3oxBTVZz4mw+brk/CiwbR6SWm2lvAncxeXvu8WAlu2ZLkkGZcwhPSNb+AaBSDnnT9e4PgVJ1ze1xwASRS+pvsy4nKJanT53nh+ry4IDw8eElfYY1BQW4ULfKS35akebGHfZ8XL1LbuYNVz9gPx0wWeknzQnhCuioiSATxnjN7tsPzNw7G8e1bpewTd9g1flI5ktFGrLCiGkYbOV8EFCi2Q/F5EUjZjKgw12ijPIBT3HbY1TunW5uNGOElc2UX0rwQ3pC20UYudrt/x2L3TpZhnNenDF3bpkbpceQGLmZruazPS4OA5sWNqtKKYj90tagg17pRFjH6pApc3L/csl0Qi7fbZiN9oU6rubUwL5k3KJNNSCS8EJ6QrmYUN4Wu687o6tq5sgU3HHZlzUassBJT1ZSiq4A7X9MK7CdBbJkvngQxGygpzMWzPxpk2U61mefFySzgdmFG2SLjrfJz8Oh3++N3V/RHcQYLvWQ2IjyBzEaEHUTNRmYLkqzZiM2oq6pAnY08L3nRCPc4FkWxL9S3yqep2g4xmz4vYZoI9IK6yBjKhg+nNF1iiLCTrtFGaWvuyhCE87yYqDBkCzPqzUTxwox/uWpAYpuVICRWmFORKuDJ0qqAhBc7xGL+hwu7banSz6Xk/tQECS+EJ6Sr2ciq2zIFGAl5RM1GZo8pWdtI7Jp6B93a5sKMbVrmJRYKK0EoR0DVqCjgFl0UoWUeCS92aHpu8nORI7ORy442ekE9PWdW96GZmPCEtE1SZ7G/bcs8ixZJqNC4PKJRJG4mqdMXYoybf3KjkYQQbiW8iAglCsSEHJYOJQUAgMtP6Sh1HNGEajPPi6Nrunw+/Vyarh+GbkPiPOEJdmUXs5o1fmA1MRzXMg/bq4/51JvsQzjPi4tJ6lLMRvVNv+dEk7FBVr44IuYgRZE3G/3vzrOxtqoGQ3u0lTqOaMK2z4sTXJ6/9HMpyS5NkPBCeIJd35FoREFMNF7WA6y63YbRvJzbuz06tC7E6JMqPO5V9iDusJt8UIO6Hoelm/cnfs+JyEUb6R1ttZoXNJ/L/Bwi411BatirFW1a5uHM49tJHUMkidnM8+JEu+F6tJENh91swBez0TPPPIPu3bujoKAAgwYNwrx580zbz5kzB4MGDUJBQQF69OiBSZMm+dFNwkXsZqoN2mHWKg8HazbKy4ng0e+ehLN6JhcXNpmY1zVOMhGZqtK8n4GkACru86LXvDT5vORGlcR4cCVUWhF17CXcolFNfx8R/ZSY7n+PW3guvLz++uu4++678eCDD2L58uU4++yzcfHFF2PLli3c9hs3bsTo0aNx9tlnY/ny5XjggQdw5513Ytq0aV53lXARu9FGQfvKWF2+kHGcrG1IDY1lCzSOOrEpidYpnVu70rdsQPSrlRVyjQRO8WgjbTtW8wJBzYsIChTbDrsEn65tWwAAjm+fmvAQsC90OhFWPS8PQNILAB/MRk888QRuuukm/PSnPwUATJw4ER9++CGeffZZTJgwIaX9pEmT0KVLF0ycOBEA0LdvXyxZsgSPP/44rrzySq+7S7iEXRkk6BBrq8uzfxdvfmtanJq+3NsX5WP1IxeiIIcSjIlipzyA/pnlyiapa+CHSucyPi+sWYqHyLBVlMzOeBoEU286A5M/3YibhnXn7rfrsOtE0ea22Uh/uqC102HB08+Auro6LF26FKNGjdJsHzVqFObPn889ZsGCBSntL7zwQixZsgT19fUp7Wtra1FTU6P5RwSPXbusXXOT4fkkT2fVb3bi4E1SebpQ6hZ5Oa7/TZmMcFFp5pay2rqWeVG0a5Uvda56XRXp+GNlfV7cgmoUuUvnNi3w0GX90LlNC+7+XqWtbJVkcCKAeB1lSCOoCU+Flz179qCxsRFlZWWa7WVlZaiqquIeU1VVxW3f0NCAPXv2pLSfMGECSkpKEv86d+7s3h9A2Mbu14Hbk7usg6RTzUsemQUcYUfzwv5823k9E+nURVX/MYPEuDnRCI7Vu1DUqBlFkR+PhD3evX0YbjyzG35z6Ym2BFDRZIk83BZe9KcjxUsTvrxJ+q9Z1cIDnNeetx0Axo8fj+rq6sS/yspKF3pMOMXuGu665kWyH1ZfaexEwvs6I4dMZ9hx2NVPC3FhRlTzYrRQua0kURQFURofvnBSpxI8dFk/tG4hnpeJxUigFcFrR32KNmrCU5+Xdu3aIRqNpmhZdu3alaJdiVNeXs5tn5OTg7ZtU3Md5OfnIz8/371OE65gVwhx2+dF9nwyzY19Xgi7iOd5Ubg/RxRFOCtuHCOByW3fgqZQaVp4/Maez0uINC+689EIasLTmTYvLw+DBg3CzJkzNdtnzpyJM888k3vM0KFDU9rPmDEDgwcPRm5u5lbIzDTsTvxuz+2yQpRVc3Yi4WteSHhxgnCeF+ZnjRYGSWHG6SLi9iJBZqNgEPF50b/3jsxGto8kZPD8TRo3bhz++c9/4vnnn8eaNWtwzz33YMuWLRg7diyAJrPP9ddfn2g/duxYbN68GePGjcOaNWvw/PPPY/Lkybj33nu97irhInY1KG6rRGW/dK3NRsmpiat5odpHjrCTYVcfeZQ0GzlbRtwei4qDwoyEfUQeoz5Fg5PQeDdyAunO6PL5MgPPQ6Wvvvpq7N27F4888gh27NiB/v374/3330fXrk0lu3fs2KHJ+dK9e3e8//77uOeee/D000+jQ4cOePLJJylMOs2QmfePa5GL/UeaIsncDiWVzRtj1W8rzUseLU6OsFPbSKt5SZqNHGteXPd5CT6PUTYi8hybxlNywDgprmh2JDvXeXH+bMKX8gC33norbr31Vu6+KVOmpGwbPnw4li1b5nGvCC+RMRuxE7rrPi/Swot4ezIbuQ9rNjq7VzvMW58aYQjotS0Kd59TzYvMGGabPvrd/njwza9S24DMRmFFP094FW00sMtx+GTtLmfnI+kFAFWVJjxCRmjQfEUHHSot0Zb3cfbI5f3RukUu/u/iPlLXJZp4YHRfAMCNZ3aziEhM/qyPPErWI3JoNnJ0NP+E+vfiOydTXSyvEfF50X80HXCoHTFCdHrrWdoq8TPJKnyoMCPhCTJCiJeaF9kPXUuHXfZnzuLYs7QVlv3qAkpMZ5Mfnt4Fw09oj4qSAvxkyueG7YzyvCiKIh0qLXINGYwWy4iuqvSK31yAkkIKQvAaIZ8Xn8y9oprd9q3ysWHXIQBe+NBkBqR5ITxBZu32UvMiHypt4bCr8XnhtyHBxRkdWhdqhBAA6F1WpGnDCqXaOkdsYcZgwo2MhpACraCelxOhnB0+IHKH/SpLIjo1aAIDTPZlMyS8EJ5wwYnaPD5XD9ZmPs5nonI0mheXR6SsIGE9h7GTCk0iXsI+u3su6KXdZ1DbSBtt5Oz6dnP+GB2mKFSYMQjsRBt5BTlsuwe9SYQnlBYV4PuDOiV+//7gTpr9BbnJYoVemo2kQ6VlHHbdyxxPcDAbC0ZJ6ljNi58Ouyx6YSrxM7TRdKI5bQiniOR5CZfZiB26ZDXiQ8IL4Rkt85ICiv4FZDUvrHzB05Tcd1Fv232QnZSsWluFShPuof1K1UcU8X9mzU1BJaljfV70GqLjmHT1+VRt3BfCpHlx4yo07TRBDruEL+j9D/Jzk8ILu4f3tX3riJ749+eV2LT3iPR1ZfPGkAtCeDB7FuwuoyR1TnHjPFFFQWPzCFegoCA3ik9+MRxAagVywhuEfF58El5Ex5RVYABBmhfCQ8xeOY3tn2lo5KNi17ExKhluFFEU/PXqATixohh9yotS9pPmxT/MFhRFJ7AkfoZ7JSZkhpxioCTibe/RvhV6tG8Fwh9E5o4WecZasB+f1Q1//v7JrvRFeGyaTC006zRBwgvhC/oXLi/K17y4XbhONgJSAfDdgZ3w/l1no0ubFin7WSddclnwFiOn3KZ9/HZQFNcieOyexkwrRISLH57eBX3Ki0zzMl1yUgUGdG7tyvVEAwjMoo2IJkh4ITzDTDHBal5YDUa+gSrd7rzvpDyA1WJDmhdvaZmf/BrWPwptJWlo2rmmebE56ti+sWZLkl2Cwey+/2hIF0y/+xyUlxQYH6+4N6ZEzEZXDepkup+mnSZIeCF8Qf/CsUIFu++KgR3Rv2Nx6glsTh7SDrsGyc/iaP4OmkQ85e7zT8CJFcV4+LJ+KfuMNC/u+rzYO449jB3nlNMlGMx9pxTN/41auSV6Wo2pscOPx++/21+7keYZLiS8EJ5hlgeFnVBYDUZhbhT/u+NsFOVrfcntTh3SDrvsz6R5CZR2rfLx/l1n44Yzu6XsM8zzAsU14cUNYYM1g5LoEgwiGjRTAcdHzcv5fUuRnxPVhkrr2lB+qSZIeCE8Q/sCal84xaBdYsHQveN2FxInmhfepMf+FU6KtxHO0D5XrSDjloLDtubFRCtE+I+VYGJ5PNzTmlmdJ57/imYWa0h4ITyjbwVj/tG9jeykztNgpPg42OyDrAOwvsifHk20ESWpCwytL1PyoTT5vASreWEP02peSHoJGwbfSro2imuaF6sEy3FfwCsGdgQA9CkvolBpAyjPC+EZ15zWGcfqGzGkR1vUHNVWaWUneF5qdf3CYXc9kg2V1jrs0mITFvTPgi1wyAqRbmpe7MIKKWzBv6D7la0I+byYtnFP8LQSrONC+XWnd0HP9q3Qr2Mxhv7hY00bkmWaIM0L4Rk50Qh+enYP9O9YkqIGZScDVvOS+BLSm41sTh6ypWS02VFT92tDpWkWCQp2EWDNd275vMh+aSvgCyl+FfwjjDH7CBEyG7lqijQ/UVxTF4koGHp8WxQX5HJ8XgiAhBciILQOu6nbU0Nj7V3HUai0RVsSXoKDdcSO6VR3bqj43dK6aaONXDklIYlVHJFVKwWKa8/O6jxUuFEcEl4IX9Cv84WajJbaL2fAvcXD7VBp9rOHktT5hz6VPmsO1Jsd3Rg7bq0hOUw/yeclGES1K2b73PKjstLE8aIj6RuJDwkvhC/oo41aGBVtNNS82Js8nDjs8tYabc0RW10ibDCsZzuM6N0+8Tv7XLVmx2a1u8O1xomgoRGASfOSFlg9GtdyB1kMTJ7mJSU0miYeACS8ED7RUpe3pTA3+btm8Yn/P8XnxR6i6biT1xGPDqEoAP+IRhRM+fHpid+10WrJdgpnvx2cHM4eSnlegsfsPRZ5zhHFP7NRDifAoK6Bwhp5kPBC+MLAzq3RuyxZ6JDVvGh9XhLii+Z4u5OHiObFKDya67CrksNuGDBKchjf7rfw8qMhXQAAZx7fVjuGKMNu4IiZjcwFHL8cdnlmJb15mmadJkh4IXxBURQ8cnkyzbvWbCSgefHQYZedUKzyvLDQJBIc7KOJxVKFFzvjpSUzJo/Vy33t/nRYD0y7ZSgm33CaRqtPmpfgEbnvpk69ipuh0ub7o7KVZLMYEl4I32C/QlmHXY3Li5HPi83Jw+pL5707hxlma+XWNmJ+jpHHbmAYal4Q93mRHy/Fhbm2+xOJKBjUtQ0K86Ka/lC0UQhwwf/JCwdu/n4aJKKQ8EL4RmtmcTBy2DVKGmV34rdaxPp1KDGsJG2VYZesRsHBCrONuiR1gD2H3RaaCDj7NDJCLWsGILNRMLhR28itZ2clnIhoimneaYKEF8I3jMrOc5PU6X1ePOuVdsKIWCw2Gs0LzSKBoc3QnPoc3IoOsQOrkKO8HcEjVr/IvJFr5QEszEIiSQ0pUKAJEl4I3ygqSGpeqplyAfzyANpjZaOG4jQImHY0Aguz3eqKNIWEA16otFvS7gUnlsn3J8Y3GxHBIOTzYtJIVd3zebESTuzOc9kICS9E4GjyGBisPXbTrNc2NFq2MTIVWX29k+YlONhH0+hBqHSc58YMwl+uGiB1TKOBzwsRRgQ0HVChuLRSujEe2rTKd6En6Q8JL4Sv/PbSE3FSxxL85KzuiW0xrs+L9iW3+0VSKxA1oo02Mnew1IRKU/qFAEk+HJUbKi1/Rp4oqigKWhXI1a9lfV5IdgkeEe2rGU2aF3dwKlRfd0YXXHJShUu9SW9IeCF85cdndce7dwxDW+brgbf46LGreTlWb615Mcqqy7si+byED7eijb43sCN3u+y5aFyEi/pGs6+Mpmdl9YhdKw/gQJpt3SIXj373JNLmNUPCCxE47IdRfN7XzxV2X9ijAsJL1CAXBzfCQOX+SPiMJlSaE21kJzrkvD58/xbZocf6vJAcEzytW+Q5Ol5VXUxS50DwyIvScs1Cd4MInF6lrRI/x7UwbjnsiggvRsUYrSYs+sIODk2SOk6SQzvDJRpRkMuJBpH96mZ9cFjHdCIYWuXnYPrdZ+PjXwzn7OWbqVlUqL4VZjRDX5w026G7QQTG/+4Yhu8P6oQnfzgwsS0+7+u9++0mnjxaJ2c2knHYJdklOKzKA+w6WCt9zmiEH1Uiu96wmhc7/SDcp095MY5v38pwv9kjdlPzwipPTupYgtKifE1mZzNIeNHi6d3Yv38/xowZg5KSEpSUlGDMmDE4cOCA6TE33ngjFEXR/BsyZIiX3SQCon/HEjx+1QB0aF2Y2Oa22ahWoKiZNlSab0KKo0JNVDf+4emdbfWJcA77nLT+mPZXGaUpD3wK8pqXZId21hyz3R/CfT68+xw8fFm/lO1Wj9i1UGkmw+7okyqw+MHzMahbG9Nj2hc1+QfaCdvPZOTc6CW59tprsXXrVkyfPh0A8LOf/QxjxozBu+++a3rcRRddhBdeeCHxe16eM5slEW5YuSQeNp0SKm3XbCSkeWHNRsx2Xnl6FXj62lMx/5u9OLtXO1t9IpxjpXmxg5GQIi28MNKUiPBM+Efv8iJ0bdsCv31nldRxriWp46gLrE797u3DMPfr3bjslA7udCJD8Ex4WbNmDaZPn46FCxfijDPOAAD84x//wNChQ7Fu3Tr07t3b8Nj8/HyUl5d71TUiZLCLQ1LzojMbCcweuVEF9Y1aW46Iz4um3IhF5lZVBVrm59BXUMC0zE9OXbGYimE92+Gr7dUY1tO+QBnhK16kFy523Nx27vF4etY3uO3c4233i/AeK82KW+UBeIKw1anLSwrwg9NIy6vHM7PRggULUFJSkhBcAGDIkCEoKSnB/PnzTY+dPXs2SktLccIJJ+Dmm2/Grl27DNvW1taipqZG849IL9gXOv4VrX+fRb5+2aJnJ1YUAwBGCQgZRmYjvSBEBM+j3+2PM7q3wdjhSWEgpgL/uul0fP7g+QmhZkDn1tLnjigKdyGRXbjYyNxfXNAbH959Dn5xgfHHGhE8Vhl23dO8iAcEEOZ4JrxUVVWhtLQ0ZXtpaSmqqqoMj7v44ovx8ssv45NPPsFf/vIXfP755zjvvPNQW8t3fJswYULCp6akpASdO5OEmm5ozUbN2PB5YYue/eum0/HY907Cbzn27dTr881GdZz8ECoFSAfKdWd0xes/H4oSpshnTFWhKApyGZ38+Iv7SJ87ElG4X+CyCxfr8xKJKOhdXkRp30OErNCgQnVN8+Ik2ojQIi28PPTQQykOtfp/S5YsAWBQ2E41HwhXX301LrnkEvTv3x+XXnopPvjgA3z99dd47733uO3Hjx+P6urqxL/KykrZP4kIGI1WJW420rUReelzmJCktq3ycc3pXdAq39oyqi0PwGheyF8hLeAlUC3Mla8QHVH4CxsreEy96YzUBjqocF644UaU+XRtEmLdQ9rn5fbbb8c111xj2qZbt2748ssvsXPnzpR9u3fvRlmZuL9ARUUFunbtivXr13P35+fnIz+faj2kM1rZJZ7nRfuSi7z00YicLH7D0K5NxxloXniZOWldCh88YaFQIPz0x2d1wwufbUr8buywm/xZpFRAo8N09IR/JB65yfTCFpR1Cu8jjMQZe0gLL+3atUO7dtZOcUOHDkV1dTUWL16M008/HQCwaNEiVFdX48wzzxS+3t69e1FZWYmKCqrnkKmwgorqRPMi8VVzcf9yPHx5fwDGielYn5efn9MDz837Fr8YRb4LYYMnLBTkWAsv+bo2BpHSmvEpMsIaScINNVy/JoMne2JFMbq3a+natXnmb7dMUtmGZz4vffv2xUUXXYSbb74ZCxcuxMKFC3HzzTfjO9/5jibSqE+fPnjzzTcBAIcOHcK9996LBQsWYNOmTZg9ezYuvfRStGvXDt/97ne96ioRIozyvLCaFyMhJUcikx37Za69VvIX1udl/Oi++Pr3F6N3eZHwNQh/4GU6LsizntpScgk1m731yGRdBoBzejXlAeJl6yWCh30qVnLmz87p4eq12XmMRoczPM3z8vLLL+POO+/EqFGjAACXXXYZnnrqKU2bdevWobq6GgAQjUaxcuVKvPTSSzhw4AAqKipw7rnn4vXXX0dRES0a2YBhhl1mLcqJKtxKsbkytT+Yw9nFif0yatCZjaTOT/gGbwES8XnhRbRZhUqLJCs7q2c7/GfsUFe/2An34Amofik/ZLTDhDmeCi9t2rTB1KlTTduw9urCwkJ8+OGHXnaJCDlGtY3YMOjcSATHkOqPIjMxsOsd6yqTz6Tg5kUbEeGDZ6YpEBFe9No9gSR1oovcaRZZU4lwkCjk6dP1eGPsnF7t8MnaXZq5h7DGU+GFIGQR8RaIGqjjjcKpvzewI95Yvs34fMyEwk4g9Q3ku5AO8MxGIloyvRZFiYC7immj0WR7R4QN9hG2sag47XZqBN4c9aMhXXFcyzwSeCUh4YUIFUnNi/EqYbQwGfm8PHH1KSnCi6pJKZ88ji1+Vh8jzUs6YPSYjm/fEt/sPmx4nC3NC3kqpD2RiII3bj0Tx+obcVzLJuHFL6dZ3tSVE43g8lM6+nL9TIL0VESoMIo2Yr+ujSKP+pQXC1+HdZlhT8dGoPBCpYnwYZRX5dHvnpSy7elrT0VORMGz152aMsbyohEDnxfKipppnNrlOJx5fDJq1uixuh04RmPJPUjzQoSKmEG0ESu8GNUrumlYd1SUFCSqPovCXio/l8xG6UZZSQF3Oy/a55KTK3BhvzLkRCNYU3UwpT0/2ij5My04mYnRc3VbeLFbYJZIhTQvRKiI25hvGNoNAHBWz7ZN25lJ5HBtA/fYvJwIfjGqNwZ1tbYds3MSO6HkMXpd0ryEm9d+NgTn9m6Pv109kLvf2LzYtF2/jCgCtY3IbJRduP35IluhnDCGNC9EqIgLKVcN7oT+HUtwfGlTuClr5uGFSQNyEQOsqYEVXijaKH0Y0qMthvRoa7jfymmXn6wsFfZjmT6cMxMnQunkGwbjpheXCLUlzYt7kOaFCAVFzTWITu/epDVRFAUndihO+KCI1IuRcboz0rxQtsvMwSpJHLfGDTcHCPkpZCsi887IvmW447yeQufTzDWkxXMECS9EKFj84PlY/OBIlBXz/RdEbM92pwKjmkj/74cD0bZlHv5y1QCbZyaCJMei1pUdzQvlRc1MjEKi2a3/u2MY7jn/BG470VFBmhf3ILMREQoK86KmxfRE8i1IfRUzpzP6QB/Y5Tgs+dX5pI1JU9q2Ms/hIfpUKUKEAID+HUugKMBfP/o6ZZ/oHCFSo40QgzQvRFrAurm89rMh3DYyalhWGDKrRk2CS/pSVJCLN241LgLLc53iO+wyP3POQ0OEEB0DpHlxDxJeiLSANRsN6dGWq741m0Cm3TIU/Tsm88Cw56OSRZnLqV2OMxwXjYJJCFnNC0WLZCaGZmnddqN2rQtzha4TIeHFNWjaJtICveMcr56NGYO6tsH/7jibu4++hjIbI4GDP4bkq0p3b0sFGNMdpyHR15zeBaNOLLNsFyUTpGuQ8EKkBfr6NTGOzl9mMtBqXug1yGSM/Ax4kfC8pkZVpS/sV4aL+pVj8o2nOe0iEVJEaxsV5Ebx3PWD0busyLQdCSzuQbM2kRboZRVeMT65UOnk8fHwbCIzMZJNeWYjvk8L/2v5pI4lmDRmELq3I81LumMUEi2bYTemqZkGnN2rneF+whkUbUSkBfpXnqfyl0tSl/z52tO7IKIAZ3Q3TnhGpC8ymhceRlZFcubOHEoKc6EoqcKKrKjBzkuFuVH866Yz0O3/3ktsM8ivSdiAhBciLXDbbMQSjSi47oyu9g4mQo+RkyRfe8c5noSUjCcnGsHqhy8CAPT9zXTb52HnpUbOHFVhkMeKkIfMRkR6oJsHvndqp5QmcqHSRLZg5JDNW1x4Y4iEl+yAl2sqRRNjMXGwmhf9+Hro0hM1gnRRAekOnEDCC5EW6B3n+lYU4/MHz7dd8ZdMz9mDUY0jXo0sbp4X5nAaN9mFft6xcuBl3aj0pu14QdDfXd4PF/Yrw3cHpn6AEeKQ8EKkBSdWFKdsa1+Un5gQANnE7bQKZQstDTI3c02PnHas5kU0+oTIDGQz4j525UmJn/WCblwDOGZoN/x9zGDk5dDy6wTSWxFpwU/P7gFVBc7tU6rZbrfsDH1BZw8t8vjTHPtlfHo344gzVrtH4ya7uPyUjlLtz+7V3nAfpZNyFxJeiLSgIDeKO0b2StmuTd0uPjucUG6ej4HIHFrm8zUvrE/Cc9cPAsCPICKfl+zirdvOwmcb9uBn5/QwNDnagcaRu5DwQqQ1rMAiMje8fdtZ+HBVFW4XLGFPpD8t8w00L4zw0rqFcRFHdlyR4iXzOaVza5zSuTV3nxPNGwkv7kLCC5HWWBXN0zOgc2sMMJiYiMykpYDZKI5VqLRRMjMiO3Dy9CmRt7vQ7SQyBkoaRvBoIeGwy4MdVSS6EHYhzYu7kPBCZAw0NRA8jMxGoqHSbJ6Ydq3yceWpndAqPwdXn9bZtT4SmQ8VgHUXMhsRGQN92BA8epW14m7nh0qnDiJFUfDRuHNQ2xBDSWEu/vKDAfhj40maMH2CiPPCjadh3L9X4PGrBmi2y4ZdE+aQ8EKkNdpCaDQ5EKlcc1oXbN1/FEN6aGtXifq8AEDPUm10Ggku2YmIz9O5fUqx7NcXpMxHPUv5QjRhDxJeiLTmWH0ypWWRgXmAyG6iEQX3X9QnZTu/PABBOIcVXD68+xzsrDmGXmWUnsFNaLYnMgajAnwEwYNXmJEgZOhQYl1osXd5EXpTXinXIeGFyAjIYkTIwtW80EAiTGBHzP/uGIaubVsE1pdshwy3REZQmMsPhyUII9gienHO7d1UfqK0KN/n3hDpAKus69+xBEUFucF1JsvxVHh59NFHceaZZ6JFixZo3bq10DGqquKhhx5Chw4dUFhYiBEjRmDVqlVedpPIAI7UNQbdBSLNyM9Nnf7uu6g3JnzvJLx9+1kB9IggCFE8FV7q6upw1VVX4ZZbbhE+5k9/+hOeeOIJPPXUU/j8889RXl6OCy64AAcPHvSwp0S68vS1pwIAfv2dEwPuCZFuPHJ5f/QsbaUJaS3IjeKHp3dBRUlhgD0jwkpZMWnkwoKi+pDvesqUKbj77rtx4MAB03aqqqJDhw64++67cf/99wMAamtrUVZWhj/+8Y/4+c9/bnmtmpoalJSUoLq6GsXFxW50nwg5B47Uobgglxx2CYLwnOlfVaF1i9yU0HvCOTLrd6h8XjZu3IiqqiqMGjUqsS0/Px/Dhw/H/PnzucfU1taipqZG84/ILlq3yCPBhSAIX7iofzkJLiEgVMJLVVUVAKCsrEyzvaysLLFPz4QJE1BSUpL417kzpewmCIIgiExGWnh56KGHoCiK6b8lS5Y46pQ+XFFVVcMQxvHjx6O6ujrxr7Ky0tG1CYIgCIIIN9J5Xm6//XZcc801pm26detmqzPl5eUAmjQwFRUVie27du1K0cbEyc/PR34+OVERBEEQRLYgLby0a9cO7dq186Iv6N69O8rLyzFz5kwMHDgQQFPE0pw5c/DHP/7Rk2sSBEEQBJFeeOrzsmXLFqxYsQJbtmxBY2MjVqxYgRUrVuDQoUOJNn369MGbb74JoMlcdPfdd+MPf/gD3nzzTXz11Ve48cYb0aJFC1x77bVedpUgCIIgiDTB0/IAv/nNb/Diiy8mfo9rU2bNmoURI0YAANatW4fq6upEm/vuuw9Hjx7Frbfeiv379+OMM87AjBkzUFREtSEIgiAIgvApz4ufUJ4XgiAIgkg/0jbPC0EQBEEQhBUkvBAEQRAEkVaQ8EIQBEEQRFpBwgtBEARBEGkFCS8EQRAEQaQVJLwQBEEQBJFWeJrnJQjikd9UXZogCIIg0of4ui2SwSXjhJeDBw8CAFWXJgiCIIg05ODBgygpKTFtk3FJ6mKxGLZv346ioiLDStR2qampQefOnVFZWUkJ8DyE7rN/0L32B7rP/kD32R+8us+qquLgwYPo0KEDIhFzr5aM07xEIhF06tTJ02sUFxfTi+EDdJ/9g+61P9B99ge6z/7gxX220rjEIYddgiAIgiDSChJeCIIgCIJIK0h4kSA/Px+//e1vkZ+fH3RXMhq6z/5B99of6D77A91nfwjDfc44h12CIAiCIDIb0rwQBEEQBJFWkPBCEARBEERaQcILQRAEQRBpBQkvBEEQBEGkFSS8CPLMM8+ge/fuKCgowKBBgzBv3rygu5RWTJgwAaeddhqKiopQWlqKK664AuvWrdO0UVUVDz30EDp06IDCwkKMGDECq1at0rSpra3FHXfcgXbt2qFly5a47LLLsHXrVj//lLRiwoQJUBQFd999d2Ib3Wf32LZtG370ox+hbdu2aNGiBU455RQsXbo0sZ/utXMaGhrwq1/9Ct27d0dhYSF69OiBRx55BLFYLNGG7rM8c+fOxaWXXooOHTpAURS89dZbmv1u3dP9+/djzJgxKCkpQUlJCcaMGYMDBw44/wNUwpLXXntNzc3NVf/xj3+oq1evVu+66y61ZcuW6ubNm4PuWtpw4YUXqi+88IL61VdfqStWrFAvueQStUuXLuqhQ4cSbR577DG1qKhInTZtmrpy5Ur16quvVisqKtSamppEm7Fjx6odO3ZUZ86cqS5btkw999xz1QEDBqgNDQ1B/FmhZvHixWq3bt3Uk08+Wb3rrrsS2+k+u8O+ffvUrl27qjfeeKO6aNEidePGjepHH32kbtiwIdGG7rVzfv/736tt27ZV//e//6kbN25U//Of/6itWrVSJ06cmGhD91me999/X33wwQfVadOmqQDUN998U7PfrXt60UUXqf3791fnz5+vzp8/X+3fv7/6ne98x3H/SXgR4PTTT1fHjh2r2danTx/1//7v/wLqUfqza9cuFYA6Z84cVVVVNRaLqeXl5epjjz2WaHPs2DG1pKREnTRpkqqqqnrgwAE1NzdXfe211xJttm3bpkYiEXX69On+/gEh5+DBg2qvXr3UmTNnqsOHD08IL3Sf3eP+++9Xhw0bZrif7rU7XHLJJepPfvITzbbvfe976o9+9CNVVek+u4FeeHHrnq5evVoFoC5cuDDRZsGCBSoAde3atY76TGYjC+rq6rB06VKMGjVKs33UqFGYP39+QL1Kf6qrqwEAbdq0AQBs3LgRVVVVmvucn5+P4cOHJ+7z0qVLUV9fr2nToUMH9O/fn56Fjttuuw2XXHIJzj//fM12us/u8c4772Dw4MG46qqrUFpaioEDB+If//hHYj/da3cYNmwYPv74Y3z99dcAgC+++AKffvopRo8eDYDusxe4dU8XLFiAkpISnHHGGYk2Q4YMQUlJieP7nnGFGd1mz549aGxsRFlZmWZ7WVkZqqqqAupVeqOqKsaNG4dhw4ahf//+AJC4l7z7vHnz5kSbvLw8HHfccSlt6Fkkee2117Bs2TJ8/vnnKfvoPrvHt99+i2effRbjxo3DAw88gMWLF+POO+9Efn4+rr/+errXLnH//fejuroaffr0QTQaRWNjIx599FH88Ic/BEBj2gvcuqdVVVUoLS1NOX9paanj+07CiyCKomh+V1U1ZRshxu23344vv/wSn376aco+O/eZnkWSyspK3HXXXZgxYwYKCgoM29F9dk4sFsPgwYPxhz/8AQAwcOBArFq1Cs8++yyuv/76RDu61854/fXXMXXqVLzyyivo168fVqxYgbvvvhsdOnTADTfckGhH99l93LinvPZu3HcyG1nQrl07RKPRFClx165dKVIpYc0dd9yBd955B7NmzUKnTp0S28vLywHA9D6Xl5ejrq4O+/fvN2yT7SxduhS7du3CoEGDkJOTg5ycHMyZMwdPPvkkcnJyEveJ7rNzKioqcOKJJ2q29e3bF1u2bAFAY9otfvnLX+L//u//cM011+Ckk07CmDFjcM8992DChAkA6D57gVv3tLy8HDt37kw5/+7dux3fdxJeLMjLy8OgQYMwc+ZMzfaZM2fizDPPDKhX6Yeqqrj99tvxxhtv4JNPPkH37t01+7t3747y8nLNfa6rq8OcOXMS93nQoEHIzc3VtNmxYwe++uorehbNjBw5EitXrsSKFSsS/wYPHozrrrsOK1asQI8ePeg+u8RZZ52VEu7/9ddfo2vXrgBoTLvFkSNHEIlol6poNJoIlab77D5u3dOhQ4eiuroaixcvTrRZtGgRqqurnd93R+6+WUI8VHry5Mnq6tWr1bvvvltt2bKlumnTpqC7ljbccsstaklJiTp79mx1x44diX9HjhxJtHnsscfUkpIS9Y033lBXrlyp/vCHP+SG5nXq1En96KOP1GXLlqnnnXdeVoc7isBGG6kq3We3WLx4sZqTk6M++uij6vr169WXX35ZbdGihTp16tREG7rXzrnhhhvUjh07JkKl33jjDbVdu3bqfffdl2hD91megwcPqsuXL1eXL1+uAlCfeOIJdfny5YkUIG7d04suukg9+eST1QULFqgLFixQTzrpJAqV9pOnn35a7dq1q5qXl6eeeuqpiRBfQgwA3H8vvPBCok0sFlN/+9vfquXl5Wp+fr56zjnnqCtXrtSc5+jRo+rtt9+utmnTRi0sLFS/853vqFu2bPH5r0kv9MIL3Wf3ePfdd9X+/fur+fn5ap8+fdTnnntOs5/utXNqamrUu+66S+3SpYtaUFCg9ujRQ33wwQfV2traRBu6z/LMmjWLOyffcMMNqqq6d0/37t2rXnfddWpRUZFaVFSkXnfdder+/fsd919RVVV1prshCIIgCILwD/J5IQiCIAgirSDhhSAIgiCItIKEF4IgCIIg0goSXgiCIAiCSCtIeCEIgiAIIq0g4YUgCIIgiLSChBeCIAiCINIKEl4IgiAIgkgrSHghCIIgCCKtIOGFIAiCIIi0goQXgiAIgiDSChJeCIIgCIJIK/4/rHvb7I/xE9MAAAAASUVORK5CYII=",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Model: \"sequential\"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Layer (type) Output Shape Param # \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" simple_rnn (SimpleRNN) (None, 32) 1184 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense (Dense) (None, 8) 264 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense_1 (Dense) (None, 1) 9 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Total params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Trainable params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Non-trainable params: 0\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 1/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2025-08-05 16:39:13.109252: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 2s - loss: 0.4905 - 2s/epoch - 50ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 2/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4094 - 315ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 3/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4041 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 4/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4015 - 309ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 5/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3980 - 312ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 6/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3977 - 335ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 7/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3970 - 330ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 8/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3970 - 325ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 9/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3968 - 338ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 10/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3969 - 327ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 11/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3949 - 327ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 12/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3970 - 307ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 13/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3923 - 311ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 14/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3941 - 318ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 15/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3932 - 340ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 16/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3933 - 327ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 17/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3914 - 321ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 18/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3924 - 313ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 19/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3929 - 407ms/epoch - 8ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 20/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3914 - 331ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 21/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3907 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 22/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3894 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 23/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3908 - 362ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 24/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3899 - 452ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 25/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3909 - 431ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 26/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - loss: 0.3904 - 521ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 27/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3883 - 483ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 28/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3890 - 411ms/epoch - 8ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 29/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3897 - 351ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 30/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3866 - 435ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 31/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3895 - 341ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 32/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3880 - 412ms/epoch - 8ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 33/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3897 - 366ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 34/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3885 - 412ms/epoch - 8ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 35/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3887 - 336ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 36/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3871 - 340ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 37/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3862 - 324ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 38/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3880 - 343ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 39/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3857 - 351ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 40/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3879 - 350ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 41/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3877 - 346ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 42/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3889 - 351ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 43/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3882 - 314ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 44/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3886 - 319ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 45/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3843 - 323ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 46/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3859 - 317ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 47/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3864 - 334ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 48/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3861 - 351ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 49/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3841 - 350ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 50/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3855 - 349ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 51/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3877 - 348ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 52/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3849 - 345ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 53/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3857 - 335ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 54/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3868 - 324ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 55/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3856 - 326ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 56/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3856 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 57/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3863 - 295ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 58/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3852 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 59/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3853 - 298ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 60/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3863 - 339ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 61/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3828 - 316ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 62/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3854 - 299ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 63/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3846 - 307ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 64/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3858 - 300ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 65/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3842 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 66/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3855 - 302ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 67/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3837 - 301ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 68/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3855 - 300ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 69/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3831 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 70/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3824 - 315ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 71/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3858 - 309ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 72/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3837 - 301ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 73/100\n"
|
|
]
|
|
},
|
|
{
|
|
"ename": "KeyboardInterrupt",
|
|
"evalue": "",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
|
|
"Cell \u001b[0;32mIn[1], line 58\u001b[0m\n\u001b[1;32m 55\u001b[0m model\u001b[38;5;241m.\u001b[39mcompile(loss\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmean_squared_error\u001b[39m\u001b[38;5;124m'\u001b[39m, optimizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrmsprop\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 56\u001b[0m model\u001b[38;5;241m.\u001b[39msummary()\n\u001b[0;32m---> 58\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrainX\u001b[49m\u001b[43m,\u001b[49m\u001b[43mtrainY\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m16\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 59\u001b[0m trainPredict \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(trainX)\n\u001b[1;32m 60\u001b[0m testPredict\u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(testX)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/utils/traceback_utils.py:65\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 63\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 64\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 65\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 66\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 67\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/engine/training.py:1685\u001b[0m, in \u001b[0;36mModel.fit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 1677\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m tf\u001b[38;5;241m.\u001b[39mprofiler\u001b[38;5;241m.\u001b[39mexperimental\u001b[38;5;241m.\u001b[39mTrace(\n\u001b[1;32m 1678\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtrain\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m 1679\u001b[0m epoch_num\u001b[38;5;241m=\u001b[39mepoch,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1682\u001b[0m _r\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m,\n\u001b[1;32m 1683\u001b[0m ):\n\u001b[1;32m 1684\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_begin(step)\n\u001b[0;32m-> 1685\u001b[0m tmp_logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1686\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m data_handler\u001b[38;5;241m.\u001b[39mshould_sync:\n\u001b[1;32m 1687\u001b[0m context\u001b[38;5;241m.\u001b[39masync_wait()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 150\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 152\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:894\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 891\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 893\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[0;32m--> 894\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 896\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[1;32m 897\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:926\u001b[0m, in \u001b[0;36mFunction._call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 923\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m 924\u001b[0m \u001b[38;5;66;03m# In this case we have created variables on the first call, so we run the\u001b[39;00m\n\u001b[1;32m 925\u001b[0m \u001b[38;5;66;03m# defunned version which is guaranteed to never create variables.\u001b[39;00m\n\u001b[0;32m--> 926\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_no_variable_creation_fn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# pylint: disable=not-callable\u001b[39;00m\n\u001b[1;32m 927\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_variable_creation_fn \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 928\u001b[0m \u001b[38;5;66;03m# Release the lock early so that multiple threads can perform the call\u001b[39;00m\n\u001b[1;32m 929\u001b[0m \u001b[38;5;66;03m# in parallel.\u001b[39;00m\n\u001b[1;32m 930\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py:143\u001b[0m, in \u001b[0;36mTracingCompiler.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 140\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock:\n\u001b[1;32m 141\u001b[0m (concrete_function,\n\u001b[1;32m 142\u001b[0m filtered_flat_args) \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_maybe_define_function(args, kwargs)\n\u001b[0;32m--> 143\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mconcrete_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 144\u001b[0m \u001b[43m \u001b[49m\u001b[43mfiltered_flat_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mconcrete_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/monomorphic_function.py:1757\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1753\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[1;32m 1754\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[1;32m 1755\u001b[0m \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[1;32m 1756\u001b[0m \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[0;32m-> 1757\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_build_call_outputs(\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1758\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcancellation_manager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcancellation_manager\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m 1759\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[1;32m 1760\u001b[0m args,\n\u001b[1;32m 1761\u001b[0m possible_gradient_type,\n\u001b[1;32m 1762\u001b[0m executing_eagerly)\n\u001b[1;32m 1763\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/monomorphic_function.py:381\u001b[0m, in \u001b[0;36m_EagerDefinedFunction.call\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 379\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m _InterpolateFunctionError(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m 380\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cancellation_manager \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 381\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 382\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mstr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msignature\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 383\u001b[0m \u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_num_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 384\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 385\u001b[0m \u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 386\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mctx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 387\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 388\u001b[0m outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[1;32m 389\u001b[0m \u001b[38;5;28mstr\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msignature\u001b[38;5;241m.\u001b[39mname),\n\u001b[1;32m 390\u001b[0m num_outputs\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_num_outputs,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 393\u001b[0m ctx\u001b[38;5;241m=\u001b[39mctx,\n\u001b[1;32m 394\u001b[0m cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_manager)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:52\u001b[0m, in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 50\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m 51\u001b[0m ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[0;32m---> 52\u001b[0m tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 53\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 54\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 55\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Start importing packages\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras.models import Model, Sequential \n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"from tensorflow.keras import optimizers \n",
|
|
"from tensorflow.keras import regularizers \n",
|
|
"from tensorflow.keras.utils import to_categorical \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"# convert into dataset matrix\n",
|
|
"def convertToMatrix(data, step):\n",
|
|
" X, Y =[], []\n",
|
|
" for i in range(len(data)-step):\n",
|
|
" d=i+step \n",
|
|
" X.append(data[i:d,])\n",
|
|
" Y.append(data[d,])\n",
|
|
" return np.array(X), np.array(Y)\n",
|
|
"\n",
|
|
"step = 4\n",
|
|
"N = 1000 \n",
|
|
"Tp = 800 \n",
|
|
"\n",
|
|
"t=np.arange(0,N)\n",
|
|
"x=np.sin(0.02*t)+2*np.random.rand(N)\n",
|
|
"df = pd.DataFrame(x)\n",
|
|
"df.head()\n",
|
|
"\n",
|
|
"plt.plot(df)\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"values=df.values\n",
|
|
"train,test = values[0:Tp,:], values[Tp:N,:]\n",
|
|
"\n",
|
|
"# add step elements into train and test\n",
|
|
"test = np.append(test,np.repeat(test[-1,],step))\n",
|
|
"train = np.append(train,np.repeat(train[-1,],step))\n",
|
|
" \n",
|
|
"trainX,trainY =convertToMatrix(train,step)\n",
|
|
"testX,testY =convertToMatrix(test,step)\n",
|
|
"trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
|
|
"testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(SimpleRNN(units=32, input_shape=(1,step), activation=\"relu\"))\n",
|
|
"model.add(Dense(8, activation=\"relu\")) \n",
|
|
"model.add(Dense(1))\n",
|
|
"model.compile(loss='mean_squared_error', optimizer='rmsprop')\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)\n",
|
|
"trainPredict = model.predict(trainX)\n",
|
|
"testPredict= model.predict(testX)\n",
|
|
"predicted=np.concatenate((trainPredict,testPredict),axis=0)\n",
|
|
"\n",
|
|
"trainScore = model.evaluate(trainX, trainY, verbose=0)\n",
|
|
"print(trainScore)\n",
|
|
"\n",
|
|
"index = df.index.values\n",
|
|
"plt.plot(index,df)\n",
|
|
"plt.plot(index,predicted)\n",
|
|
"plt.axvline(df.index[Tp], c=\"r\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "897a47c7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## An extrapolation example\n",
|
|
"\n",
|
|
"The following code provides an example of how recurrent neural\n",
|
|
"networks can be used to extrapolate to unknown values of physics data\n",
|
|
"sets. Specifically, the data sets used in this program come from\n",
|
|
"a quantum mechanical many-body calculation of energies as functions of the number of particles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6776ae2a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"# For matrices and calculations\n",
|
|
"import numpy as np\n",
|
|
"# For machine learning (backend for keras)\n",
|
|
"import tensorflow as tf\n",
|
|
"# User-friendly machine learning library\n",
|
|
"# Front end for TensorFlow\n",
|
|
"import tensorflow.keras\n",
|
|
"# Different methods from Keras needed to create an RNN\n",
|
|
"# This is not necessary but it shortened function calls \n",
|
|
"# that need to be used in the code.\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras import regularizers\n",
|
|
"from tensorflow.keras.models import Model, Sequential\n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"# For timing the code\n",
|
|
"from timeit import default_timer as timer\n",
|
|
"# For plotting\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"\n",
|
|
"# The data set\n",
|
|
"datatype='VaryDimension'\n",
|
|
"X_tot = np.arange(2, 42, 2)\n",
|
|
"y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,\n",
|
|
"\t-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, \n",
|
|
"\t-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35227d36",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The way the recurrent neural networks are trained in this program\n",
|
|
"differs from how machine learning algorithms are usually trained.\n",
|
|
"Typically a machine learning algorithm is trained by learning the\n",
|
|
"relationship between the x data and the y data. In this program, the\n",
|
|
"recurrent neural network will be trained to recognize the relationship\n",
|
|
"in a sequence of y values. This is type of data formatting is\n",
|
|
"typically used time series forcasting, but it can also be used in any\n",
|
|
"extrapolation (time series forecasting is just a specific type of\n",
|
|
"extrapolation along the time axis). This method of data formatting\n",
|
|
"does not use the x data and assumes that the y data are evenly spaced.\n",
|
|
"\n",
|
|
"For a standard machine learning algorithm, the training data has the\n",
|
|
"form of (x,y) so the machine learning algorithm learns to assiciate a\n",
|
|
"y value with a given x value. This is useful when the test data has x\n",
|
|
"values within the same range as the training data. However, for this\n",
|
|
"application, the x values of the test data are outside of the x values\n",
|
|
"of the training data and the traditional method of training a machine\n",
|
|
"learning algorithm does not work as well. For this reason, the\n",
|
|
"recurrent neural network is trained on sequences of y values of the\n",
|
|
"form ((y1, y2), y3), so that the network is concerned with learning\n",
|
|
"the pattern of the y data and not the relation between the x and y\n",
|
|
"data. As long as the pattern of y data outside of the training region\n",
|
|
"stays relatively stable compared to what was inside the training\n",
|
|
"region, this method of training can produce accurate extrapolations to\n",
|
|
"y values far removed from the training data set."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "7dc577d1",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# FORMAT_DATA\n",
|
|
"def format_data(data, length_of_sequence = 2): \n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" data(a numpy array): the data that will be the inputs to the recurrent neural\n",
|
|
" network\n",
|
|
" length_of_sequence (an int): the number of elements in one iteration of the\n",
|
|
" sequence patter. For a function approximator use length_of_sequence = 2.\n",
|
|
" Returns:\n",
|
|
" rnn_input (a 3D numpy array): the input data for the recurrent neural network. Its\n",
|
|
" dimensions are length of data - length of sequence, length of sequence, \n",
|
|
" dimnsion of data\n",
|
|
" rnn_output (a numpy array): the training data for the neural network\n",
|
|
" Formats data to be used in a recurrent neural network.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" X, Y = [], []\n",
|
|
" for i in range(len(data)-length_of_sequence):\n",
|
|
" # Get the next length_of_sequence elements\n",
|
|
" a = data[i:i+length_of_sequence]\n",
|
|
" # Get the element that immediately follows that\n",
|
|
" b = data[i+length_of_sequence]\n",
|
|
" # Reshape so that each data point is contained in its own array\n",
|
|
" a = np.reshape (a, (len(a), 1))\n",
|
|
" X.append(a)\n",
|
|
" Y.append(b)\n",
|
|
" rnn_input = np.array(X)\n",
|
|
" rnn_output = np.array(Y)\n",
|
|
"\n",
|
|
" return rnn_input, rnn_output\n",
|
|
"\n",
|
|
"\n",
|
|
"# ## Defining the Recurrent Neural Network Using Keras\n",
|
|
"# \n",
|
|
"# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.\n",
|
|
"\n",
|
|
"def rnn(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with one hidden layer and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer\n",
|
|
" hidden_neurons = 200\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to \n",
|
|
" # the network immediately after the input layer\n",
|
|
" rnn = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\")(inp)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1978b6a1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Predicting New Points With A Trained Recurrent Neural Network"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "d4ad417a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_rnn (x1, y_test, plot_min, plot_max):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" x1 (a list or numpy array): The complete x component of the data set\n",
|
|
" y_test (a list or numpy array): The complete y component of the data set\n",
|
|
" plot_min (an int or float): the smallest x value used in the training data\n",
|
|
" plot_max (an int or float): the largest x valye used in the training data\n",
|
|
" Returns:\n",
|
|
" None.\n",
|
|
" Uses a trained recurrent neural network model to predict future points in the \n",
|
|
" series. Computes the MSE of the predicted data set from the true data set, saves\n",
|
|
" the predicted data set to a csv file, and plots the predicted and true data sets w\n",
|
|
" while also displaying the data range used for training.\n",
|
|
" \"\"\"\n",
|
|
" # Add the training data as the first dim points in the predicted data array as these\n",
|
|
" # are known values.\n",
|
|
" y_pred = y_test[:dim].tolist()\n",
|
|
" # Generate the first input to the trained recurrent neural network using the last two \n",
|
|
" # points of the training data. Based on how the network was trained this means that it\n",
|
|
" # will predict the first point in the data set after the training data. All of the \n",
|
|
" # brackets are necessary for Tensorflow.\n",
|
|
" next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])\n",
|
|
" # Save the very last point in the training data set. This will be used later.\n",
|
|
" last = [y_test[dim-1]]\n",
|
|
"\n",
|
|
" # Iterate until the complete data set is created.\n",
|
|
" for i in range (dim, len(y_test)):\n",
|
|
" # Predict the next point in the data set using the previous two points.\n",
|
|
" next = model.predict(next_input)\n",
|
|
" # Append just the number of the predicted data set\n",
|
|
" y_pred.append(next[0][0])\n",
|
|
" # Create the input that will be used to predict the next data point in the data set.\n",
|
|
" next_input = np.array([[last, next[0]]], dtype=np.float64)\n",
|
|
" last = next\n",
|
|
"\n",
|
|
" # Print the mean squared error between the known data set and the predicted data set.\n",
|
|
" print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())\n",
|
|
" # Save the predicted data set as a csv file for later use\n",
|
|
" name = datatype + 'Predicted'+str(dim)+'.csv'\n",
|
|
" np.savetxt(name, y_pred, delimiter=',')\n",
|
|
" # Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
" # for the training data.\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.plot(x1, y_test, label=\"true\", linewidth=3)\n",
|
|
" ax.plot(x1, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
" ax.legend()\n",
|
|
" # Created a red region to represent the points used in the training data.\n",
|
|
" ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn(length_of_sequences = rnn_input.shape[1])\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2cad4fc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Changing the size of the recurrent neural network and its parameters\n",
|
|
"can drastically change the results you get from the model. The below\n",
|
|
"code takes the simple recurrent neural network from above and adds a\n",
|
|
"second hidden layer, changes the number of neurons in the hidden\n",
|
|
"layer, and explicitly declares the activation function of the hidden\n",
|
|
"layers to be a sigmoid function. The loss function and optimizer can\n",
|
|
"also be changed but are kept the same as the above network. These\n",
|
|
"parameters can be tuned to provide the optimal result from the\n",
|
|
"network. For some ideas on how to improve the performance of a\n",
|
|
"[recurrent neural network](https://danijar.com/tips-for-training-recurrent-neural-networks)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c39f1516",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer, increased from the first network\n",
|
|
" hidden_neurons = 500\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Create two hidden layers instead of one hidden layer. Explicitly set the activation\n",
|
|
" # function to be the sigmoid function (the default value is hyperbolic tangent)\n",
|
|
" rnn1 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=True, # This needs to be True if another hidden layer is to follow\n",
|
|
" stateful = stateful, activation = 'sigmoid',\n",
|
|
" name=\"RNN1\")(inp)\n",
|
|
" rnn2 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False, activation = 'sigmoid',\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN2\")(rnn1)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn2)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn_2layers(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "842c7602",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Other Types of Recurrent Neural Networks\n",
|
|
"\n",
|
|
"Besides a simple recurrent neural network layer, there are two other\n",
|
|
"commonly used types of recurrent neural network layers: Long Short\n",
|
|
"Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short\n",
|
|
"introduction to these layers see <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>\n",
|
|
"and <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>.\n",
|
|
"\n",
|
|
"The first network created below is similar to the previous network,\n",
|
|
"but it replaces the SimpleRNN layers with LSTM layers. The second\n",
|
|
"network below has two hidden layers made up of GRUs, which are\n",
|
|
"preceeded by two dense (feeddorward) neural network layers. These\n",
|
|
"dense layers \"preprocess\" the data before it reaches the recurrent\n",
|
|
"layers. This architecture has been shown to improve the performance\n",
|
|
"of recurrent neural networks (see the link above and also\n",
|
|
"<https://arxiv.org/pdf/1807.02857.pdf>."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "6f0e9b62",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons on the input/output layer and the number of neurons in the hidden layer\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input Layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)\n",
|
|
" rnn= LSTM(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True, activation='tanh')(inp)\n",
|
|
" rnn1 = LSTM(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True, activation='tanh')(rnn)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn1)\n",
|
|
" # Define the midel\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the model\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with four hidden layers (two dense followed by\n",
|
|
" two GRU layers) and returns the model.\n",
|
|
" \"\"\" \n",
|
|
" # Number of neurons on the input/output layers and hidden layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden Dense (feedforward) layers\n",
|
|
" dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)\n",
|
|
" dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)\n",
|
|
" # Hidden GRU layers\n",
|
|
" rnn1 = GRU(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True)(dnn1)\n",
|
|
" rnn = GRU(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True)(rnn1)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Define the model\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the mdoel\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Change the method name to reflect which network you want to use\n",
|
|
"model = dnn2_gru2(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)\n",
|
|
"\n",
|
|
"\n",
|
|
"# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)\n",
|
|
"# \n",
|
|
"# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"# Reshape the data for Keras specifications\n",
|
|
"X_train = X_train.reshape((dim, 1))\n",
|
|
"y_train = y_train.reshape((dim, 1))\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Set the sequence length to 1 for regular data formatting \n",
|
|
"model = rnn(length_of_sequences = 1)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(X_train, y_train, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict the remaining data points\n",
|
|
"X_pred = X_tot[dim:]\n",
|
|
"X_pred = X_pred.reshape((len(X_pred), 1))\n",
|
|
"y_model = model.predict(X_pred)\n",
|
|
"y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))\n",
|
|
"\n",
|
|
"# Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
"# for the training data.\n",
|
|
"fig, ax = plt.subplots()\n",
|
|
"ax.plot(X_tot, y_tot, label=\"true\", linewidth=3)\n",
|
|
"ax.plot(X_tot, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
"ax.legend()\n",
|
|
"# Created a red region to represent the points used in the training data.\n",
|
|
"ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0752ba7f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Generative Models\n",
|
|
"\n",
|
|
"**Generative models** describe a class of statistical models that are a contrast\n",
|
|
"to **discriminative models**. Informally we say that generative models can\n",
|
|
"generate new data instances while discriminative models discriminate between\n",
|
|
"different kinds of data instances. A generative model could generate new photos\n",
|
|
"of animals that look like 'real' animals while a discriminative model could tell\n",
|
|
"a dog from a cat. More formally, given a data set $x$ and a set of labels /\n",
|
|
"targets $y$. Generative models capture the joint probability $p(x, y)$, or\n",
|
|
"just $p(x)$ if there are no labels, while discriminative models capture the\n",
|
|
"conditional probability $p(y | x)$. Discriminative models generally try to draw\n",
|
|
"boundaries in the data space (often high dimensional), while generative models\n",
|
|
"try to model how data is placed throughout the space.\n",
|
|
"\n",
|
|
"**Note**: this material is thanks to Linus Ekstrøm."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "784138f8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Generative Adversarial Networks\n",
|
|
"\n",
|
|
"**Generative Adversarial Networks** are a type of unsupervised machine learning\n",
|
|
"algorithm proposed by [Goodfellow et. al](https://arxiv.org/pdf/1406.2661.pdf)\n",
|
|
"in 2014 (short and good article).\n",
|
|
"\n",
|
|
"The simplest formulation of\n",
|
|
"the model is based on a game theoretic approach, *zero sum game*, where we pit\n",
|
|
"two neural networks against one another. We define two rival networks, one\n",
|
|
"generator $g$, and one discriminator $d$. The generator directly produces\n",
|
|
"samples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a42f89ee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" x = g(z; \\theta^{(g)})\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abe7212f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The discriminator attempts to distinguish between samples drawn from the\n",
|
|
"training data and samples drawn from the generator. In other words, it tries to\n",
|
|
"tell the difference between the fake data produced by $g$ and the actual data\n",
|
|
"samples we want to do prediction on. The discriminator outputs a probability\n",
|
|
"value given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0821eaee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" d(x; \\theta^{(d)})\n",
|
|
"\\label{_auto2} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55e0ccf6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"indicating the probability that $x$ is a real training example rather than a\n",
|
|
"fake sample the generator has generated. The simplest way to formulate the\n",
|
|
"learning process in a generative adversarial network is a zero-sum game, in\n",
|
|
"which a function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f37ece14",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto3} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d6d6d5fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"determines the reward for the discriminator, while the generator gets the\n",
|
|
"conjugate reward"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c74b45c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" -v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto4} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "605ac8c9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"During learning both of the networks maximize their own reward function, so that\n",
|
|
"the generator gets better and better at tricking the discriminator, while the\n",
|
|
"discriminator gets better and better at telling the difference between the fake\n",
|
|
"and real data. The generator and discriminator alternate on which one trains at\n",
|
|
"one time (i.e. for one epoch). In other words, we keep the generator constant\n",
|
|
"and train the discriminator, then we keep the discriminator constant to train\n",
|
|
"the generator and repeat. It is this back and forth dynamic which lets GANs\n",
|
|
"tackle otherwise intractable generative problems. As the generator improves with\n",
|
|
" training, the discriminator's performance gets worse because it cannot easily\n",
|
|
" tell the difference between real and fake. If the generator ends up succeeding\n",
|
|
" perfectly, the the discriminator will do no better than random guessing i.e.\n",
|
|
" 50\\%. This progression in the training poses a problem for the convergence\n",
|
|
" criteria for GANs. The discriminator feedback gets less meaningful over time,\n",
|
|
" if we continue training after this point then the generator is effectively\n",
|
|
" training on junk data which can undo the learning up to that point. Therefore,\n",
|
|
" we stop training when the discriminator starts outputting $1/2$ everywhere.\n",
|
|
"\n",
|
|
"At convergence we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfec7462",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto5\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" g^* = \\underset{g}{\\mathrm{argmin}}\\hspace{2pt}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto5} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37c65d4c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The default choice for $v$ is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c868a092",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto6\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)}) = \\mathbb{E}_{x\\sim p_\\mathrm{data}}\\log d(x)\n",
|
|
" + \\mathbb{E}_{x\\sim p_\\mathrm{model}}\n",
|
|
" \\log (1 - d(x))\n",
|
|
"\\label{_auto6} \\tag{6}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ad465af3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The main motivation for the design of GANs is that the learning process requires\n",
|
|
"neither approximate inference (variational autoencoders for example) nor\n",
|
|
"approximation of a partition function. In the case where"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "27858a4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto7\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto7} \\tag{7}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86006023",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"is convex in $\\theta^{(g)} then the procedure is guaranteed to converge and is\n",
|
|
"asymptotically consistent\n",
|
|
"( [Seth Lloyd on QuGANs](https://arxiv.org/pdf/1804.09139.pdf) ).\n",
|
|
"\n",
|
|
"This is in\n",
|
|
"general not the case and it is possible to get situations where the training\n",
|
|
"process never converges because the generator and discriminator chase one\n",
|
|
"another around in the parameter space indefinitely. A much deeper discussion on\n",
|
|
"the currently open research problem of GAN convergence is available\n",
|
|
"[here](https://www.deeplearningbook.org/contents/generative_models.html). To\n",
|
|
"anyone interested in learning more about GANs it is a highly recommended read.\n",
|
|
"Direct quote: \"In this best-performing formulation, the generator aims to\n",
|
|
"increase the log probability that the discriminator makes a mistake, rather than\n",
|
|
"aiming to decrease the log probability that the discriminator makes the correct\n",
|
|
"prediction.\" [Another interesting read](https://arxiv.org/abs/1701.00160)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2fee38bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Writing Our First Generative Adversarial Network\n",
|
|
"Let us now move on to actually implementing a GAN in tensorflow. We will study\n",
|
|
"the performance of our GAN on the MNIST dataset. This code is based on and\n",
|
|
"adapted from the\n",
|
|
"[google tutorial](https://www.tensorflow.org/tutorials/generative/dcgan)\n",
|
|
"\n",
|
|
"First we import our libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "004a0b53",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import time\n",
|
|
"import numpy as np\n",
|
|
"import tensorflow as tf\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from tensorflow.keras import layers\n",
|
|
"from tensorflow.keras.utils import plot_model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af161",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define our hyperparameters and import our data the usual way"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "8cbaf16a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"BUFFER_SIZE = 60000\n",
|
|
"BATCH_SIZE = 256\n",
|
|
"EPOCHS = 30\n",
|
|
"\n",
|
|
"data = tf.keras.datasets.mnist.load_data()\n",
|
|
"(train_images, train_labels), (test_images, test_labels) = data\n",
|
|
"train_images = np.reshape(train_images, (train_images.shape[0],\n",
|
|
" 28,\n",
|
|
" 28,\n",
|
|
" 1)).astype('float32')\n",
|
|
"\n",
|
|
"# we normalize between -1 and 1\n",
|
|
"train_images = (train_images - 127.5) / 127.5\n",
|
|
"training_dataset = tf.data.Dataset.from_tensor_slices(\n",
|
|
" train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "822b8cc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### MNIST and GANs\n",
|
|
"\n",
|
|
"Let's have a quick look"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "52b5965c",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.imshow(train_images[0], cmap='Greys')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "21c5199c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our two models. This is where the 'magic' happens. There are a\n",
|
|
"huge amount of possible formulations for both models. A lot of engineering and\n",
|
|
"trial and error can be done here to try to produce better performing models. For\n",
|
|
"more advanced GANs this is by far the step where you can 'make or break' a\n",
|
|
"model.\n",
|
|
"\n",
|
|
"We start with the generator. As stated in the introductory text the generator\n",
|
|
"$g$ upsamples from a random sample to the shape of what we want to predict. In\n",
|
|
"our case we are trying to predict MNIST images ($28\\times 28$ pixels)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "356759c7",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_model():\n",
|
|
" \"\"\"\n",
|
|
" The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to\n",
|
|
" produce an image from a random seed. We start with a Dense layer taking this\n",
|
|
" random sample as an input and subsequently upsample through multiple\n",
|
|
" convolutional layers.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
"\n",
|
|
"\n",
|
|
" # adding our input layer. Dense means that every neuron is connected and\n",
|
|
" # the input shape is the shape of our random noise. The units need to match\n",
|
|
" # in some sense the upsampling strides to reach our desired output shape.\n",
|
|
" # we are using 100 random numbers as our seed\n",
|
|
" model.add(layers.Dense(units=7*7*BATCH_SIZE,\n",
|
|
" use_bias=False,\n",
|
|
" input_shape=(100, )))\n",
|
|
" # we normalize the output form the Dense layer\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" # and add an activation function to our 'layer'. LeakyReLU avoids vanishing\n",
|
|
" # gradient problem\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" model.add(layers.Reshape((7, 7, BATCH_SIZE)))\n",
|
|
" assert model.output_shape == (None, 7, 7, BATCH_SIZE)\n",
|
|
" # even though we just added four keras layers we think of everything above\n",
|
|
" # as 'one' layer\n",
|
|
"\n",
|
|
" # next we add our upscaling convolutional layers\n",
|
|
" model.add(layers.Conv2DTranspose(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(1, 1),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 7, 7, 128)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 14, 14, 64)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=1,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False,\n",
|
|
" activation='tanh'))\n",
|
|
" assert model.output_shape == (None, 28, 28, 1)\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "854bcd6b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"And there we have our 'simple' generator model. Now we move on to defining our\n",
|
|
"discriminator model $d$, which is a convolutional neural network based image\n",
|
|
"classifier."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "41473304",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_model():\n",
|
|
" \"\"\"\n",
|
|
" The discriminator is a convolutional neural network based image classifier\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
" model.add(layers.Conv2D(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" input_shape=[28, 28, 1]))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
"\n",
|
|
" model.add(layers.Conv2D(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same'))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
" model.add(layers.Flatten())\n",
|
|
" model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af567",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Let us take a look at our models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "f899d4e3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generator = generator_model()\n",
|
|
"plot_model(generator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "87ef384b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"discriminator = discriminator_model()\n",
|
|
"plot_model(discriminator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2bef82d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we need a few helper objects we will use in training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "e397847a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
|
|
"generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n",
|
|
"discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "db3396cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The first object, *cross_entropy* is our loss function and the two others are\n",
|
|
"our optimizers. Notice we use the same learning rate for both $g$ and $d$. This\n",
|
|
"is because they need to improve their accuracy at approximately equal speeds to\n",
|
|
"get convergence (not necessarily exactly equal). Now we define our loss\n",
|
|
"functions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "931eaced",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_loss(fake_output):\n",
|
|
" loss = cross_entropy(tf.ones_like(fake_output), fake_output)\n",
|
|
"\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "0c4a44bb",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_loss(real_output, fake_output):\n",
|
|
" real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n",
|
|
" fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)\n",
|
|
" total_loss = real_loss + fake_loss\n",
|
|
"\n",
|
|
" return total_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fcf8f066",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a kind of seed to help us compare the learning process over\n",
|
|
"multiple training epochs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "eea2bbee",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"noise_dimension = 100\n",
|
|
"n_examples_to_generate = 16\n",
|
|
"seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94a6e341",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have everything we need to define our training step, which we will apply\n",
|
|
"for every step in our training loop. Notice the @tf.function flag signifying\n",
|
|
"that the function is tensorflow 'compiled'. Removing this flag doubles the\n",
|
|
"computation time."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "8d48470b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tf.function\n",
|
|
"def train_step(images):\n",
|
|
" noise = tf.random.normal([BATCH_SIZE, noise_dimension])\n",
|
|
"\n",
|
|
" with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n",
|
|
" generated_images = generator(noise, training=True)\n",
|
|
"\n",
|
|
" real_output = discriminator(images, training=True)\n",
|
|
" fake_output = discriminator(generated_images, training=True)\n",
|
|
"\n",
|
|
" gen_loss = generator_loss(fake_output)\n",
|
|
" disc_loss = discriminator_loss(real_output, fake_output)\n",
|
|
"\n",
|
|
" gradients_of_generator = gen_tape.gradient(gen_loss,\n",
|
|
" generator.trainable_variables)\n",
|
|
" gradients_of_discriminator = disc_tape.gradient(disc_loss,\n",
|
|
" discriminator.trainable_variables)\n",
|
|
" generator_optimizer.apply_gradients(zip(gradients_of_generator,\n",
|
|
" generator.trainable_variables))\n",
|
|
" discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,\n",
|
|
" discriminator.trainable_variables))\n",
|
|
"\n",
|
|
" return gen_loss, disc_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "62015b88",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a helper function to produce an output over our training epochs\n",
|
|
"to see the predictive progression of our generator model. **Note**: I am including\n",
|
|
"this code here, but comment it out in the training loop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "b189ed96",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_and_save_images(model, epoch, test_input):\n",
|
|
" # we're making inferences here\n",
|
|
" predictions = model(test_input, training=False)\n",
|
|
"\n",
|
|
" fig = plt.figure(figsize=(4, 4))\n",
|
|
"\n",
|
|
" for i in range(predictions.shape[0]):\n",
|
|
" plt.subplot(4, 4, i+1)\n",
|
|
" plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n",
|
|
" plt.axis('off')\n",
|
|
"\n",
|
|
" plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')\n",
|
|
" plt.close()\n",
|
|
" #plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ba2c82d5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Setting up checkpoints to periodically save our model during training so that\n",
|
|
"everything is not lost even if the program were to somehow terminate while\n",
|
|
"training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "a0e2fc8a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setting up checkpoints to save model during training\n",
|
|
"checkpoint_dir = './training_checkpoints'\n",
|
|
"checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')\n",
|
|
"checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n",
|
|
" discriminator_optimizer=discriminator_optimizer,\n",
|
|
" generator=generator,\n",
|
|
" discriminator=discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d93a0f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our training loop"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "a1275556",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def train(dataset, epochs):\n",
|
|
" generator_loss_list = []\n",
|
|
" discriminator_loss_list = []\n",
|
|
"\n",
|
|
" for epoch in range(epochs):\n",
|
|
" start = time.time()\n",
|
|
"\n",
|
|
" for image_batch in dataset:\n",
|
|
" gen_loss, disc_loss = train_step(image_batch)\n",
|
|
" generator_loss_list.append(gen_loss.numpy())\n",
|
|
" discriminator_loss_list.append(disc_loss.numpy())\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epoch + 1, seed_images)\n",
|
|
"\n",
|
|
" if (epoch + 1) % 15 == 0:\n",
|
|
" checkpoint.save(file_prefix=checkpoint_prefix)\n",
|
|
"\n",
|
|
" print(f'Time for epoch {epoch} is {time.time() - start}')\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epochs, seed_images)\n",
|
|
"\n",
|
|
" loss_file = './data/lossfile.txt'\n",
|
|
" with open(loss_file, 'w') as outfile:\n",
|
|
" outfile.write(str(generator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write(str(discriminator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ff3a75a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"To train simply call this function. **Warning**: this might take a long time so\n",
|
|
"there is a folder of a pretrained network already included in the repository."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"id": "371ed41a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"train(train_dataset, EPOCHS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "654399f1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now to avoid having to train and everything, which will take a while depending\n",
|
|
"on your computer setup we now load in the model which produced the above gif."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"id": "dec4b560",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))\n",
|
|
"restored_generator = checkpoint.generator\n",
|
|
"restored_discriminator = checkpoint.discriminator\n",
|
|
"\n",
|
|
"print(restored_generator)\n",
|
|
"print(restored_discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "296bfa5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We have successfully loaded in our latest model. Let us now play around a bit\n",
|
|
"and see what kind of things we can learn about this model. Our generator takes\n",
|
|
"an array of 100 numbers. One idea can be to try to systematically change our\n",
|
|
"input. Let us try and see what we get"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"id": "eecfbb1f",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_latent_points(number=100, scale_means=1, scale_stds=1):\n",
|
|
" latent_dim = 100\n",
|
|
" means = scale_means * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" latent_space_value_range = tf.random.normal([number, latent_dim],\n",
|
|
" means,\n",
|
|
" stds,\n",
|
|
" dtype=tf.float64)\n",
|
|
"\n",
|
|
" return latent_space_value_range\n",
|
|
"\n",
|
|
"def generate_images(latent_points):\n",
|
|
" # notice we set training to false because we are making inferences\n",
|
|
" generated_images = restored_generator.predict(latent_points)\n",
|
|
"\n",
|
|
" return generated_images"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"id": "333a593d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def plot_result(generated_images, number=100):\n",
|
|
" # obviously this assumes sqrt number is an int\n",
|
|
" fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),\n",
|
|
" figsize=(10, 10))\n",
|
|
"\n",
|
|
" for i in range(int(np.sqrt(number))):\n",
|
|
" for j in range(int(np.sqrt(number))):\n",
|
|
" axs[i, j].imshow(generated_images[i*j], cmap='Greys')\n",
|
|
" axs[i, j].axis('off')\n",
|
|
"\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"id": "2f5f0154",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generated_images = generate_images(generate_latent_points())\n",
|
|
"plot_result(generated_images)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff581bf2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We see that the generator generates images that look like MNIST\n",
|
|
"numbers: $1, 4, 7, 9$. Let's try to tweak it a bit more to see if we are able\n",
|
|
"to generate a similar plot where we generate every MNIST number. Let us now try\n",
|
|
"to 'move' a bit around in the latent space. **Note**: decrease the plot number if\n",
|
|
"these following cells take too long to run on your computer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"id": "d3617ad8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 225\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=-5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=5))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a074f93",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Again, we have found something interesting. *Moving* around using our means\n",
|
|
"takes us from digit to digit, while *moving* around using our standard\n",
|
|
"deviations seem to increase the number of different digits! In the last image\n",
|
|
"above, we can barely make out every MNIST digit. Let us make on last plot using\n",
|
|
"this information by upping the standard deviation of our Gaussian noises."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"id": "4ef8937d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 400\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=10))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "385a2d0a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"A pretty cool result! We see that our generator indeed has learned a\n",
|
|
"distribution which qualitatively looks a whole lot like the MNIST dataset.\n",
|
|
"\n",
|
|
"Another interesting way to explore the latent space of our generator model is by\n",
|
|
"interpolating between the MNIST digits. This section is largely based on\n",
|
|
"[this excellent blogpost](https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/)\n",
|
|
"by Jason Brownlee.\n",
|
|
"\n",
|
|
"So let us start by defining a function to interpolate between two points in the\n",
|
|
"latent space."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"id": "57de87b8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def interpolation(point_1, point_2, n_steps=10):\n",
|
|
" ratios = np.linspace(0, 1, num=n_steps)\n",
|
|
" vectors = []\n",
|
|
" for i, ratio in enumerate(ratios):\n",
|
|
" vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))\n",
|
|
"\n",
|
|
" return tf.stack(vectors)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfb76bb6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have all we need to do our interpolation analysis."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"id": "e25decef",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 100\n",
|
|
"latent_points = generate_latent_points(number=plot_number)\n",
|
|
"results = None\n",
|
|
"for i in range(0, 2*np.sqrt(plot_number), 2):\n",
|
|
" interpolated = interpolation(latent_points[i], latent_points[i+1])\n",
|
|
" generated_images = generate_images(interpolated)\n",
|
|
"\n",
|
|
" if results is None:\n",
|
|
" results = generated_images\n",
|
|
" else:\n",
|
|
" results = tf.stack((results, generated_images))\n",
|
|
"\n",
|
|
"plot_results(results, plot_number)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.15"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |