Morten Hjorth-Jensen
2939 lines
161 KiB
Plaintext
2939 lines
161 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d2db899",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html chapter13.do.txt -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4908a97",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Recurrent neural networks: Overarching view\n",
|
|
"\n",
|
|
"Till now our focus has been, including convolutional neural networks\n",
|
|
"as well, on feedforward neural networks. The output or the activations\n",
|
|
"flow only in one direction, from the input layer to the output layer.\n",
|
|
"\n",
|
|
"A recurrent neural network (RNN) looks very much like a feedforward\n",
|
|
"neural network, except that it also has connections pointing\n",
|
|
"backward. \n",
|
|
"\n",
|
|
"RNNs are used to analyze time series data such as stock prices, and\n",
|
|
"tell you when to buy or sell. In autonomous driving systems, they can\n",
|
|
"anticipate car trajectories and help avoid accidents. More generally,\n",
|
|
"they can work on sequences of arbitrary lengths, rather than on\n",
|
|
"fixed-sized inputs like all the nets we have discussed so far. For\n",
|
|
"example, they can take sentences, documents, or audio samples as\n",
|
|
"input, making them extremely useful for natural language processing\n",
|
|
"systems such as automatic translation and speech-to-text.\n",
|
|
"\n",
|
|
"More to text to be added"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43cb4913",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A simple example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "cf6b9dab",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACdNklEQVR4nO2dd5wU5f3HP7PXD65Q7yhHE6Q3QfDAgooiEpXEGDUmoFGMBqKI0Yg1agxGY0tijT8liQU1KpYoBkFA5aQfUgREQeod9Spwbef3x97sTnlm5pm2O7P3ffs6uZud8uzMM8/zfb5VEEVRBEEQBEEQREAIJboBBEEQBEEQViDhhSAIgiCIQEHCC0EQBEEQgYKEF4IgCIIgAgUJLwRBEARBBAoSXgiCIAiCCBQkvBAEQRAEEShIeCEIgiAIIlCkJroBbhMOh7Fv3z7k5ORAEIREN4cgCIIgCA5EUUR1dTU6d+6MUMhYt5J0wsu+fftQVFSU6GYQBEEQBGGD3bt3o2vXrob7JJ3wkpOTAyDy5XNzcxPcGoIgCIIgeKiqqkJRUVF0Hjci6YQXyVSUm5tLwgtBEARBBAwelw9y2CUIgiAIIlCQ8EIQBEEQRKAg4YUgCIIgiEBBwgtBEARBEIGChBeCIAiCIAKFp8LLs88+iyFDhkQjf4qLi/Hxxx8bHvPWW2+hX79+yMzMxODBg/HRRx952USCIAiCIAKGp8JL165d8fDDD2PNmjVYvXo1zjnnHFxyySXYtGkTc//ly5fjyiuvxLXXXot169Zh8uTJmDx5MjZu3OhlMwmCIAiCCBCCKIpiPC/Ytm1bPProo7j22ms1n11++eWora3Fhx9+GN122mmnYdiwYXjuuee4zl9VVYW8vDxUVlZSnheCIAiCCAhW5u+4+bw0NTVh3rx5qK2tRXFxMXOfkpISjB8/XrFtwoQJKCkp0T1vXV0dqqqqFD8EQRAEQSQvngsvGzZsQOvWrZGRkYEbbrgB7777LgYMGMDct6ysDAUFBYptBQUFKCsr0z3/nDlzkJeXF/2hukYEQRAEkdx4Lrz07dsXpaWlWLFiBW688UZMnToVmzdvdu38s2fPRmVlZfRn9+7drp2bIAiCIAj/4Xlto/T0dPTu3RsAMGLECKxatQpPPfUUnn/+ec2+hYWFKC8vV2wrLy9HYWGh7vkzMjKQkZHhbqMJgiAIgvAtcc/zEg6HUVdXx/ysuLgYixYtUmxbuHChro8MQRBaDtXU4bml3+FgNfs9IwiCCDqeal5mz56NiRMnolu3bqiursZrr72GJUuW4JNPPgEATJkyBV26dMGcOXMAADfffDPOOussPPbYY5g0aRLmzZuH1atX44UXXvCymQSRVNz4yhqs2nkUH28sw3vTxya6OQRBEK7jqfBy4MABTJkyBfv370deXh6GDBmCTz75BOeddx4AYNeuXQiFYsqfMWPG4LXXXsPdd9+NO++8E3369MH8+fMxaNAgL5vpa1btPIKvvjuM35zdGykh8zLhBLFq51EAwPrdFYltCEEQhEfEPc+L1yRbnpced/wXAPDwTwbjilHdEtwaIghIfQYAdj48KYEtIQiC4MeXeV4IZ+w4XJvoJhBJxOGaOvx10bfYX3lc89krX/2A8x5fin0V2s8IgiD8AAkvPqOxKYyl2w6i6kSDYntIUJqMtpRV4WfPl2DljiPxbB6RJNz46lo8vnAbpv1rteazu+dvxLcHavDQR98koGVEIvhy+yHMnLcOR2vrE90UguCChBef8fyy7zH1pZW46h8rFNvV7i5Xv7QKK3ccwc+e12Yf3ltxHNUq4Ycg5EhC78a9+hmp6xrC8WoOkWCuenEF5pfuw59kAmuSeRQQSQYJLz7j7bV7AAAb9lYqtqeoNC9lVSeYx++rOI6xDy/G8AcWetNAwveQYzdhl73NpsJXV/yAQfd9glU7SbNL+BMSXnyGWkiREHS2q5EGm8YwrZpaKlaEl5xMz/NUEgFCGmbuencjauubcNlz+nXlCCKRkPDiM9S+LWbb1ZCmt2URDovYuLcSjU0xE0+qBeElPztN9zPOLkckEQLooRPBgIQXn6E3YZAlgGDx5Kfb8KO/fYG73t0Y3WZF85Kfle5Fs4iAk5WWkugmEIQhJLz4DF3Ni2xC+ma/vpOlCFK9tCT+ung7AOCN1bGCpFY0L2Q2Ilgcb2hKdBMIwhASXnyG3qr50U+24tvyagDA7Hc26B5PZiMiJcT/Wh9vaMLqnUcQJh8pAmQqJIIDCS8+4q3VuzVRRnImPvW56TlIeCGsaF7W7arAT58rwVtrdms+o3ksuTlSW6/R4m7cW4kzH/ksQS0i/E5Dk3/SJ5Dw4iNu+8/Xhp9LEURGqyOSXQg7odLvrN3rQUsIP3PKgwsx8anPFQLM0WMN2HXkWAJbRfiVhZvL0eeuj/HGql2JbgoAEl4CiVHkESWWIuwIL2kp+kPB8fomRTQTkVx89f3hRDeBCABSNu7fv63vthBPSHgJIHq5YADSvBDWzEbRY1LYx9TWNaL/vQtw9mNLHLaK8CtG6510A6GWIBIJ9cwAQk51hBFuaV4EAVi/pwIAsPsIFWlMVowWPJlpNEUQ/oR6po/5w/ubmNsNE9aR6qXFY094IYm4pWJkaubN7E0Q8YaEFx8zd/lO5najSFjK80LomYAMj7EQXk20HMLkQ0f4FBqxAoixw24cG0L4EiOfKD1YAs8nm8pxgpKVJT2PfrJV/0MaTwifQsJLAFGrct9cFcvRQWMNIc/GXH2igeuYNB3Ny6/mrnalTYR/qWvUjySj8YSQ8JsFkYSXBLK/8jjqDQYOPdQuDbe/HcsPQ5oXQt4Hhj+wkOuYFA5T09SXVuK9UsoH05IgsxHhV0h4SRCluytQPGcxfvLslxBF0VJ+FkOzEcdaaeHmcpz16GdYt+so9zWJ4CDvAY2caf95QmKXbjuIm+eV2msUEUhIdiEk/NYXqCpbgnh7zR4AwMa9Vbjiha/QZKG2jNMK01KyoakvrcTXf5jg7GREXKhrbEJGKmelX5NRhiUo28kNQyQ/FABA+BXSvPiAFTuOYPUP/FoQo/BFK9LxiQbKmhoEHv54C/revQAb9ujXvZKjJwev2nkEN89bhwPVdZrPUikZGcHAb6ttgpAgzUsCqDhWj483ltk+Xm+RvGrnEdw9fyP/iWixHQieW/odAODhBd/g1etOAxDRnqzaac3sd9lzJQCA2jptBBHleSFYkPBC+BVabiWAK174CodqtKtfXgSG1LHom/Lo5MR/HiJINDTFZpIl2w7iZ8+zn7eZqn/PUW3hPcrzQrAgsxEhQdFGBLaUVTs6njXPXPtPCmlNduTFEb/89pDufmarZZbDt98GJsIf6PWlt1bvxvPNGkGCSARkNgoghuUBLEATVrCQRw61a52hu5+Z8MIqH0AhsQQLvX5x238i6RnOPLkD+nfKjWeTCAIAaV4CCa/wYiX8mvA/crNR+9bpuvuxnvr76/dFf9+wV+v4S12FYGHWLX44XItv9ldh3spd0fHmREMT7v9gE77crq8dJAinkOYlgPBGtYqisXaF5TtD+JemcMxslJOp/+qyhNabXl9neG4rgm44LCqy+BLJi1m3OFhTjxteWQsAyM5IxcVDO2Pu8p14+cvIz86HJ8WhlURLhDQvAYRb86L6+0RDE/ZVHI/+TWajYNEo07wYTSp2tCgW0gyhvolC7FsSRoLtYVngwaZ9EY2efIwhgo0oinh+6Xf4/NuDiW6KBtK8BBDeMvWRQSe271mPfobyqthgQ7JLsGgI8wkNdiJErPi81DWGkZnGmTCPCDxGGlxW1GQa5QxKGpZsO4g5H28B4L/FLvWygFF5vIHbbPTRxjL8bdG30ZWTXHAhgodC82Kwn+eaFxv1uIjgYtQ1qo43Rn+XzNDpqbFp5bUVuxTaGSJY7D3qXy0aCS9xxqkT7S1vlHKbjW56fR0eW7gNK3cccXRNwh808JqNbJzbSr+sa2zCweo6LNi4XxG+TfiLd9ftwb9Kdjo+j1HfYH1yoiGWBPHOdzfgmrmrHLeBINR4KrzMmTMHp556KnJyctCxY0dMnjwZW7duNTxm7ty5EARB8ZOZmellM+OK05T8i7ccYOZ5MeJQTT1zO6/5ifAHjTKzkZFpyE7Ys5Uj6hvDmPjUMtzwylrMXb7T8rWI+HDLG+tx73ubsOfoMXy29QD2V9pbRau1ckbCzP7K43j5y52KbV9zlrUg/IefpwhPhZelS5di+vTp+Oqrr7Bw4UI0NDTg/PPPR21treFxubm52L9/f/Tnhx9+8LKZceVYfaP5TiZYFTr0Jjof90uCAa/DLutxm6X/D1uwG9U1hqMC8R//+03U72FfxXF8sH6fpSKjhDfIBYx31u7FNS+vwpiHF9s7l6pD6fU9QQA+/eaArWsQ/sTPEameOuwuWLBA8ffcuXPRsWNHrFmzBmeeeabucYIgoLCw0MumJYxGFwZ216JU/dsvCQYNnCYaVg9LSwmhoUlb00jCSresU/m8XPDkMqy++zyc+chnaAyLqDzegF+c1p3/hITryJ+nFCli12KtPo5E05ZDi9W8qKmsjKgP27Zta7hfTU0NunfvjqKiIlxyySXYtGmT7r51dXWoqqpS/PgZN5KBuSUNV59oxNFatkmJ8B9yjYaxw672U7kTJQsrpia1n4ukhZEE8y8MShcQ8UHeB5wumIzD8kmUSWYEnd/9QNyEl3A4jJkzZ2Ls2LEYNGiQ7n59+/bFSy+9hPfeew+vvPIKwuEwxowZgz179jD3nzNnDvLy8qI/RUVFXn0FV3AjDbvVUFijSxY/vMhha4h4IZ+ErDpRppqo66xMQnLHYcKfyJ+QUzOe1mykL0T7bYIjnCHXvPjtrY+b8DJ9+nRs3LgR8+bNM9yvuLgYU6ZMwbBhw3DWWWfhnXfeQYcOHfD8888z9589ezYqKyujP7t37/ai+a7hRgdw06XAqQMx4T9YArKZn5TUpx74YLPp+Rs5880QiUPeBxwLLy6bjURRtO08TMQXP/u8xEV4mTFjBj788EN89tln6Nq1q6Vj09LSMHz4cGzfvp35eUZGBnJzcxU/fsYNNStpagkj7PQPESIO19ThpS93mO7baKJ5WbCpTBEuS8QfeR9wKryohWFdh13O8933/iYUz1mMN1f7e6FJwNeqNE+FF1EUMWPGDLz77rtYvHgxevbsafkcTU1N2LBhAzp16uRBC+OPG4KHVQGIZJ3kw2p5ALMxKCzyT3I8jsOvr9zFdS7CG9wUXtRH28ngLOdfJZHo0UcWRNJm/GfNHry6InkiSpMJH8su3kYbTZ8+Ha+99hree+895OTkoKysDACQl5eHrKwsAMCUKVPQpUsXzJkzBwDwwAMP4LTTTkPv3r1RUVGBRx99FD/88AOuu+46L5saN9wQXtzwmyH8y4GqE9hSVo0z+rTXNfc4nUA05xP5z8jjAFpb5zwlAGEf+dNsYowXVhZA8l3NjrManXKioQm/e2s9AODCQZ3QppV+tXQisQjw10LYU83Ls88+i8rKSowbNw6dOnWK/rzxxhvRfXbt2oX9+/dH/z569CimTZuG/v3748ILL0RVVRWWL1+OAQMGeNnUuOGKw66fehDhOmP/vBhTXlqJBRvLdPexGgFyUofWhtcMh/n7FW/INpE4zDQvlsaQ5n3/XbITI//4Kbbsr9Z8Zg8RVccb3DkVYQuzDNl+TmTqqeaFR7pfsmSJ4u8nnngCTzzxhEctSjx+c9gl/IcUzbPs24OYOJhtLrVaHqCtyYo2LIrc2hxKQud/5Isklo+SlSconeue9yIpK+58d4PsPHxnmvz0l/jblcNR1DZbsb2mTl4biYgnh2vqcNajS3DegAI8cfkw5j5+fiZU2yjOuOOwS5NHS8BuUA9Lu2c2yYRF/tW4mcMukXjkT4jZH6yYjTj3EwT96JTS3RW4a/5GzfZj9THHbupV8eWtNXtQU9eId9ft1d3Hx4oXEl7ijRuLVqunIGEnmBgJHMZJ6rTbzAQhfr0L0MA42Y+f+ZLzaCIeiLJHxPJRsmQ1UnUos5xBelQeUybEFEWl5oV8+eILz1Mk4YWQ4fwFdfsl/8ey77H7yDFXz0k4x0jQNRJIWceZiSaiyC/ksjQv63ZVcB1LxAf582bVrbIyhqgPT02xN22wrih37CbZJb7wCCYtPs8LEcMNzYvbLgcPffQNLnlauXImbU3iMZpgjDUv2oyoZn0mLIrksBtw1u+uwI2vrMGuw8cUz5KpebHweqsF3xQdzYsAwXBC1OSLgVLz4nYEHWEMj2BCGXaJKInI88JzzBFZjaPb3lqPiU99jrpGSjSWUGz2FU1eDg5/FisCMU+oNMm+8eeSp7/ExxvLcOOraxSCgpWK4UzUmheZ8GJJCGLsW31CFlJPfSauWDUJ+e2dJuElzrixurDTiayMX2+t2YMtZdX4bMtB6xciXMNQtW/wEWuFaya8WtG8mIVXEonlh8PHFN3DqeZFfbhc8/KxQTi/2XkEqMxG/E0iXIAnDNrPodIkvMQZN8rC2PF5sXMMmY4Si6FpyOBT9UqbJwGdaCFUupoS0PkaAVCZjbSDjpVFlHpfPYdds3lOY86Eutgod5MIF7Dpd+0bSHiJM25oXuwIIpSbI3hYLQGg95kI8z7DY1qSeH7p93w7EolBUAoKjn1eNJoX/WnDaD5kXVNZoZrGKK8RRTFad4wr2sjb5jiChJc448bqwqogYmViUhxn/RDCRew67LIK6Zn7vNDUkSyEBEHxLFnp/bcfqOE+n7pf2A2V1vZLUdU2W6clLHDdP1ej3z0LsL/yOKfZKA6NsgkJLy5TU9eIc/6yBA98sJn5uRsv6KffHLC0vwiRcigEgHBYxA3/XhP92+4TU8u2PM+fFHPJgyDoC75NYRFby6o10YVGqM2QqSl60UbGsFokmnxOuMuiLZG54+01eyhUmlDyn9W78f2hWrz05Q7m54la39rzefGgIYQuK3cewYJNMQdII58jo2fD0ryYsXBzOSpldWaI4KL2eZHTJIr4/FvrjvhyAcZI82I1VFq+yXFUFGEJ0rwQCsxev0S9n3auS9qa+FLXqHSsNA420n540+vrcLimjp1hl+NZ/mOZe74sfh70kh1BZTaSEw5HzEpWEEWgXhZhZuTzYnwi1iYaYxIFj/XPz8lLSXhxGbOBIVERPBQ5FDysOuy+v34fHvxwsy2fFwCu5vVZtfMoLn++BFvKqlw7J8HHkdp6XS1GkyjqJpnTIyyKCsHaLZ8XQO2PY+u0hE3MTEJHa+sx5+MtcWqNdUh4cRmz9zoRmhdRtBdtRGNJYrGj+fr2QI0muoTX58nqityIpdsOYsWOI7j6pVWunZNQsnFvJXYeqmV+tngL2y+uKSxaDpEVAdTLhBddxYsgGE6IGl8sUe3zQiNOPDF73b+14NSdCFIT3YCkw3QCSJTPi/k+tXWNuHleqedtIdioe45hbSOd7awoElEEGjgqQes5YjqhrOqE6+ds6ew5egy3vfU1Sr4/DADY+fAkzT5Pf7adeWw4LFpOPCaKosJsZDftwq4jxxTlAJpPzvqViANmvcDvyShJ8+Iy6lXN8XqlKj5RLyiP2ej/vtiBT78pt3QM4R2G91/nM7XfDBDRgqz54ajp9Wz7MhBxZfpr66KCix56Pcee2Ug5kTmZ016Q+VWpkyfSaBNf1DLslrIq3PXuBhxoXnDUk/DSspCr3t8r3Yv+9y7Av7/6IbotEWajZ5Z8x3Xdo6qS9URiMXpkVrrRb15dy7WfB4oXwgO+41Dn6wm+4bCIFMvmQWVhTz0TJM9Zq1QRbWGF5oXEFy+RktMBEadutQbugic/x6srdmHWm+sBsKvH+wkSXlxGvqiRTDD3zN8Y3ZaIF3T7gRrsrzxuup8mM6u/+27SYzdU2i6keUke9BYrTaKIkEXNS8ThO3ZCQ7ORyanTZBKyOlSaddaKY/W44d9r8OnmcsanBC9Ha+sx4N4F0b9FUVQstOXP95v9ESd7v1ePp9HKZczsyYkKlT5Wbx5Joq09QtJLIol3X7EbRcLLgo1l+GyrtQSLhBaep6SnHXl7zR5bDrvyvthkIDmbrdZTU5RTjjITsPbYRz/ZigWbynDdv1bzNJXQ4f31+5iFMSWeWLgttr35AzIbtTBMQ6UTJBCw6puoIVElfuw6fAz3f7AJe47G8iiou46h2cgD1YvVFbkVjtTW44ZX1uCal1f53hEwGag4xk44+Jf/bcPWsmpL54oIQrH+pheGXdcYxp3vbjA8l1pANguVLq+q424nweat1btx3/ubFNsiZqPY339drHXw5nHyTyQUbeQyZuN/okwxTRzlrMlsFD+u/MdX2FtxHF9uP4T/3XIWcx9Ds5EHbfJS81J9IjaZNoZFpKZ4dqnkRy3kitYiiPZXWosAE0Wl5kVvIbR5v3lOn1SZabL6RKNCkGWdlZIdOue2/3zN3K5/byMf+H2RQZoXlzFPUhenhqjweT9sceytiPggbSvXd760W1XaLlajUKwgfy8oc7O7WDUvWjYbiWoNib4zsBnqcPx/yYIZ2FWn+dpIWEPt8yJH2kw+Ly0Ms5VCosxGPLkZ1G2jgSOxxLuveCm8yN8Lv6ujg4ZVYdBqnpewqExyqDeW8LRDrd2TJ78jHzt/ID2hep+/p2Q2ijOJq21kfmFtNWIikRhZ+rx4Nl4Kq/JVnt0kZwQb68KLtfN/sf0QymSmJr05jacdRgIy63AyG3mDIAimmhcyG7UwzAbmROUy4NK8aHxe2MfsPnIM4x79DP8q2elCywgJdWp1o8nAi34UL3NOI4f/FaGPesqx+tis5nl5+OMtmLt8Z/RvPfMQz2NNS9GfckjTG1/MugGZjZKUbeXVWL+7QrPdTEZInM+L9QvrHXHHO19j5+FjuPe9TTp7EFaoPNaA+97biPV7KhTb491VvBRe5Of2e/KroBHvMUVP+DQKoZYw1LyQrjeu6NWhkraT2ShJOf+JZQCAtfech7at0qPbzZzWEvWC8qSH550ut+y3FmpJGPPAh5vx9to9mu3xTlLnlTlnW3m1YsVNZiNnqH1WrAqdTm+/nobFjs+LHKbZiLdRhGX0HgU57LYQylRhh2YvcKI0L/ISBXpoBiWdth6upTICbrKtnC0MxruveCVTnP/EMvz8H19F//7g632+t6f7GbW636rw4tTkqHc9vqAAg89Ipo0remYjabPf31ESXmxg9PKbqU79vOjURBuRGjehVB5vwMFqdpIuL54NT6irXeS5RR5ZsBX/LDEXpgk+rD42p+ZBvTGOxxxo6MdF401c0XsUkmbP71GBJLzYwOjdN/d58W+HoCR1/uLbAzU49aFPmZ958Wy+2mFcqdhNPttCZQLsolkwWxZenF1fT8jl0bwY7ULjTXwxu91+NxuRz4sNWA99z9Fj+MsnW5GeaiwP+lnzQiQGOytOL7rRD4ePme9E+A7rPi/Oeo9ehl2eKDKrWaMpVNo7zLqB3+cqEl5sIH/5pZfrhlfWYONe8/TYfs6eom6Zf1tKuEm/whxssVjvhkgcTh12nWo49JPUmR9rZJr0s1Y6GdFbNEndy+/Pw1Oz0Zw5c3DqqaciJycHHTt2xOTJk7F161bT49566y3069cPmZmZGDx4MD766CMvm2kZ1jPdtI9HcPG3alQ9CPq5rS0dN5+Nl5l19SD/BvewukJ2eu/1hCUezYtRW99aswcnGprsNotgYOd+SsKL38t4eCq8LF26FNOnT8dXX32FhQsXoqGhAeeffz5qa2t1j1m+fDmuvPJKXHvttVi3bh0mT56MyZMnY+PGjV421RKsh8r7nH2tilO17ZFPtuBAlbUibkR8cHPy97IgI+E+2iR11vqC01B1vcObHDrsvrZiF55YuM1uswgVG/ZUot89C3Q/13XYbe5hvp6r4LHwsmDBAlx99dUYOHAghg4dirlz52LXrl1Ys2aN7jFPPfUULrjgAtx2223o378/HnzwQZxyyin4+9//7mVT44Zbk06X/CxXzmNExbEGXP9v/WdFuIOdBU7QNS+Ee1iPNnJ4PV2fF+tZvNUs/KbcTpMIBn/5n7GVQ+9RkOaFQWVlJQCgbdu2uvuUlJRg/Pjxim0TJkxASUkJc/+6ujpUVVUpfryG5fPCf6w7bVBXZ3XKgE65zM5cysgiTCQXJLwEC/WYY3VB5HRO0hNS+KKN/JkHKxkxLRKsc7Olw7xMneAGcRNewuEwZs6cibFjx2LQoEG6+5WVlaGgoECxraCgAGVlZcz958yZg7y8vOhPUVGRq+1m4eQFc8sJSq+oll1E8EvaNNklF273JSK+WPZ5cZrnxYHmxWwXyr7sHk7fap+neYmf8DJ9+nRs3LgR8+bNc/W8s2fPRmVlZfRn9+7drp6fhV112uGaOtdUcW7LD6IocgtlJLy4iz2zkXsjS0Icdn0+MPqJA1Un8NdF36I86n+mijayOOEvcphjR9dhlyMviNn4pxZe5PV3vi2vxi4K5+dGHZVm9Ti/a17iEio9Y8YMfPjhh1i2bBm6du1quG9hYSHKy5V2z/LychQWFjL3z8jIQEZGhmtt5cHOIy357jCulKVId4oXq2Wz7xUOi7hp3jrUN/o7eZHf+d8mthbRCsng81LfGEbp7goM75ZvWG24pTPtX6uxfk8l/re5DB/+9gyt2SjOc4wjzYtZ7TeDL3Necz25HXMutD0xtyTMXmt9h90IfteCeTpiiKKIGTNm4N1338XixYvRs2dP02OKi4uxaNEixbaFCxeiuLjYq2ZaRpTN3XqVOdW8sOw7V9vg9oQTFkXT1fyX3x3Ch1/vd/W6LZHfv/2143O4OawkSniZ/c4G/Oz5Evzxw80JuX5QWL8n4iuol0cq3o6VetfjaYep2YjjHDxCEgGYGY50faWaD/P7ffZUeJk+fTpeeeUVvPbaa8jJyUFZWRnKyspw/Pjx6D5TpkzB7Nmzo3/ffPPNWLBgAR577DFs2bIFf/jDH7B69WrMmDHDy6ZaQv3Q3VhJW8XtlYcomk+IJxq0Ghe/JzLyI24mAxzcJc9JUwAAKQlaxUqVtKnOkTU0odJxvn71iUbmdp5aOOZmI+XfrK5ZR5pfLswddnWOA3Copg6f+jzyy1Ph5dlnn0VlZSXGjRuHTp06RX/eeOON6D67du3C/v2x1fyYMWPw2muv4YUXXsDQoUPxn//8B/Pnzzd08o03aoE0EeHEbmvZxej/9GG9C01hEQer6yi5lAXcsCVLA48bShPyYQo28da8OBEezBY7PN+FzNZ8mL3VRrf6trfWu9oWL/DU54VnVb5kyRLNtssuuwyXXXaZBy1yBz9oG9z2eQmLomnIJeuSn28/hOv+uRqn9miDedf7x7TnZ9zoPtKzckMDR8JLsND6vCR+POLFjWijukZaKPFgd44QBAEb9la63Br3odpGNhAVv/MNHG6beVx3WOP4GqxLPvDBZjSFRXz1/RF325PEaMxGNiYf6RA3ugH5PgYbn7smKDDTrKg/Z/VN0rzwYWo20tl+qKYOFccaXG+P25CLvw3kLxhHOQ8A7js/uR4qDXONAMs5ubaObf8m9HFjpSydIag5WgKkLPA9fs+EKsdsGFR/FdZXI58Xd9Abh4IguAAkvNhDlP/KN3As23bQ1Sa47WQZFkVbg+CB6jpX29EScHOqcUOIDdDcRzAI0vMzE9x5zEakeeHDruYlKJDwYgP5+5WogcP1DLsix3cJ5iLfd7iyUhbd83khgoVaAxoszYuJ8MLxXcjnhQ/TNB7B6TZMSHixgZsVfe0Scj3ayPxb0TTJR1NYxDUvr8QDH7Dzl6gXl7Yy7Db/S8+ECJDswmE2Mvd5IbMRH+aalwB1HAYkvNhA/gImatXjerRR2HwQfOUrysfBw8odR/DZ1oN46csd7B3cU7xY7gd/u3K4ZltWWorzBhGus7/yOC54cplmu/qRJ5XmpXlwDYdFzHqjFB9t0ObQIuGFj2TXypLwYgP56iBR44Y34a36X+ZobT0+/cZZTZSWglnOGzcnG6saOHm/6ZCTgfX3nY8QhUr7kj9/vAVbyqo129VPK0jRRmZdX/ouS789iHfW7WXu86f/foOS7w673LLkw0melyBAwosN5A89Uc/f/Qy7xoUZj1MSOm7MIsvUwosd9a10jFXNi3z//Kw05GWlBX4QS1Z43zm3hOGfj+6GjFRvpwTeth6IFqHU8u2BGlfrxCUr5LBLaJC/fy9+/n1C2pDiRai0yecEH2bVdd24l7E8L1aFl9jv0qFBt323NNTPPOJs7/wZDu6Sh4W3nOX4PEbwCi+Vx4MRrutnSPNCaJC/gIkqVOjU5yUvK03xd5Ds5n7HTPPi5FZ/tiViurPrsCs3G0WjERLw6ElgMoe36Ksoiq6Yjnq2b+V5wkLedlYdp/xRTjFb2AT9HSThxQZ+eOROzUbqw8ur6ijhnEvIc1XwrIitCDPXzF2FLWVVtmsbkX9LcOB9xcOiO4uPnu1bOT6HGbwaIiual6oTDfjNq2uwYGP8C+T6GaPukwy+vCS82MAPWgqncxDr8BU7KMW/GzTIzEZeOFN+f7A2+rtVDZw8uWEyDGDJDL/wYi/BpJp2rdK917xwBAodrK5D1Ql+4eXpz7bjow1luOGV+BfI9TU2q0oHBRJebOCHh+402iioaeWDgFzz4pWga7cwI2t3H3RnwgKsUGneMiVGpIQEz8NreZLQvbFqF6pP8GuBD1XXO2lS0mI0xoti8N97El5skfjH7lT4INnFOxo8Fl4EINoFnTxHaaJKRFViPywA/AQrLT6vzwtcMhsJAvcVbRPmUEXmZ6crtJdmpNAsxsT0WQb8JaTHbgM/5FVw7rtg7fhETHBBpUluNuIYg+3c2VhhRmvHyacn6Td6soml8ngDRv/pU8x6o1T5gc6z1Wpe3BOSvV7U8BSozc9OM91Hjjc5r4IPhUoTGvwwjzv2eaH33TMavda8yJ6dEw1cIvsA9b8Y75XuxaGaek1SNt5bFHYp2ihyzcSbjay+MsmeSdYuRs8yGRajJLzYwA8Ou06rStNixTu8Fl4AITr4uCG8JKI7++AV8g1WnyCzMKNbwotJY87p19HR+ZuazBsaFkVLAonTsTBZMbotXIV4fQ4JLzbww0N3HCpt2Wzk6HItikarZiOLN1cQZM/Dic8LlXX0Bzrvsp5gqt4ckV1cMhuZfJ7mMDsmj9mIpx1yyGzExmiKCLuU2DCRkPBiA19oXhw+OVqseIfnZiPIfV6sPci2rdJdbw/hDL0naCVU2rVuZnJNpwIvz/tg3WxkszFJj4HZKPAp6kh4CSyOo40s7s8z6Dy39Dt8d7DGXoOSiLiESttIUnffRQPQKiNWQTpWHoBIJFZfZfXuboa9mgknVguBquHRvFh9Z8hsxIbMRoQGP2henGfYdd9s9PDHW3DuY0tttig5+O/X+/G3xdujf3M5KFq8hvzZWRFi+3TMYUcb+aA/t2R0zUOcx0c0L/GJNnI67vCESouiNYGOzEZsjO6KWS27IJCa6AYEET+M9fGONvKDwOZnjtTW48oXvsLW8mrFdi9uW8RsJCWps3CcoNpfyvPiXtO4od4UQ/5IwmExmgZBT1BQb4+ESrvfFjufm9HI4QTGO9ZI90qeNkK06OybzBgnqXPWYY7W1qNNgk3QpHmxgR8mcqerDavvd+K/sb955rPtGsEF8C5UOlpV2sJ0ohZeaIj3B/Jn0iCb3Hmfjyi658FgNvE7NVe7mfdIMkHJzUYNHNFMLQVzs5H9ezXyoU9tH+sWJLzYwA+vh3OfF0pS5yZ1jexRmZU51SnKPC/8x4UEZfp3WqD6A/m72Ghj8g27aAMw1bw47DM8mhfe7yK9W3LNi5XMvMmOsdnIWYfxYlyzCgkvNvDDRO5UeLGquPHBVw4kXPfNanSFbFiy0g9CqvTv0d/p2SYW2UNRCC/c1QHcixwx605Oxx2eSS8silxfvUkU0dgUxp4jx6LbSHiJYaRFc8NhN9H3moQXG/hhImcJHyEBuP/igVzHW7UL+0DQDiRemI1ExBw0rUR/hNRmo6jPCz1cv1DPMSGo39ywi5EjZhpZp8o6Nx3Ym5pEzHpzvSIzMc/9I5r7jMP3/nhDk0utsQcJLzbww0TOqm30zYMXYOqYHlzHWx2EaIIzRk8W9EK9Gg7LB3grPi8Cc+XsB2G8JSOPwGlU+LzwPVs3fV5M87wIgiPTEY9ZjLc/NobDeH/9PsW2m15f5wvNuB8w9Hlxob8cryfhJXD44eVgTUKpVpbhVqONaEFjCx7ZxWpvkicls2L+EwSwzUZEQpHnPpFP7rqTj2r7weo6bNxb5UpbzEOlnZ2f12xk91xffX8EX++ptNyuZMRQ+HVBW3cswcILhUpbYOehWizecgC9OrRKdFOYk5a06ddn9cLzS783PN6NJHUhQX9yFkURx+qb0CqjZXQxvYHAC0FXrvK16vMChdnI7ZZZIPHyv2+QT8JyPwLex/PH/37jWlvMrhkSlBmercKTpE4En1n7QHUdc7ue83xLw1jz4vwVPFbf6PAMziDNiwXG/WUJHvhwM5789NuEXL+4V7vo76xQaamzzp7YH/075Rqey6njnV4bJG6aV4qB932Cb/a7syIMKjw2fqvIBSIrjzEy8ciT1DX7vJAgkVAUmheOyd0tmfOUbvmachFmQoPT8gBcTp6cHbLqRIOjtgSZRo77aKSVFV0oKUFmowBSursiIde9YlRR9HfWICPflp5q/GjdSFKnJwCFwyI+aLZF/98XO6xdKKDo3U8vCjPaddAMCYJyQEuk5oVsVlGaZJ1EoXnRuUduJWEbVtQGrVWaUVPNS8jZ9fnMRnznshNWngw8smAL+t2zANsYeaXkGD0nNxx2E202IuElQMiFBccZdl2oKq0nvEx86nM7TUpKvIg2Cov2hp1IkjqGw24ibDgtc95h0qgwG8ldsb2V8ARB++zN5RJnreIyG3G+M1w5Y5KQZ5Z8h8awiEc/2Wq4n2GelyTwefFUeFm2bBkuuugidO7cGYIgYP78+Yb7L1mypNmbXflTVlbmZTMDgzI5WXwz7Or5vLCQZ5pt6Qtsz4SXaIZdftR5XqR5i8xGiUWuQWjk0by4dN2QoNUMmhZmFKyPHXJ4zB1hke87tvRsuqZCnsfRRpXH6x2fwwmeCi+1tbUYOnQonn76aUvHbd26Ffv374/+dOzY0aMWBgu5wGJWHsA8U6ZFzQurPRzqn5aexdWraCM7qMsDxEPj8pNTunh+jaCjp3nxGkEQNBOg19FGvA67XOcyuVcNTeGk9osxl12Mk9Q5Ze7yH3AigblePA0FmThxIiZOnGj5uI4dOyI/P9/9BgWIW8afjKPH6lG6uyLqY6NwV3BsNrIGS8rn0f54rfr2OzyChtWBJLJatj76qMsDRK9v+Uz8jOzeFu+s3Wu+ow5byqpQW9eEEd3buNgqfyH3eVHkedH1eXHnunaihiLaO/vxRjw+L5HiiubnMjMbXfDkMnx3sBar7hqPDjkZvE0MDGZ30ri2kfNK5H0LWie0zp8vfV6GDRuGTp064bzzzsOXX35puG9dXR2qqqoUP8nAyQWt8YeLBypeOnlnTImz2YjVR6kUvTlhL5LUycxGVs7uVOVvB70uwqv1ueDJz3Hps8txUCcsNhnQy/PiudFV0L7XppoXh5fk83nhO5eZluq7g7UAgKXbDvKdMGCYCQ6GPi9wpn353fkn44nLhyE7PXGpMHwlvHTq1AnPPfcc3n77bbz99tsoKirCuHHjsHbtWt1j5syZg7y8vOhPUVGR7r5BR75qNtN6uK3+ZY05PLJLSzEb6X1N+X1rCovYX3nc8bXkznZWBo+M1BRFO6MCkIeLJye+WfKVYXnVCTea40uaZJNwPZfPizsvlQBB67BrVh5AnenQIjw+L5E2mV+kidNh14sFhB8wNRsZal6caVy7tsl2LerNLr4SXvr27Ytf//rXGDFiBMaMGYOXXnoJY8aMwRNPPKF7zOzZs1FZWRn92b17dxxb7B2sfiHfxONvYoTVSYWlYuTpvC1FeNFDrib/7etrUTxnsWYfq74nYVk6+JzMVLx09UhMHFRoelyrjNS4DzhWLneopg6P/28rdjcX2pPfOzfyEvkVfc2Lt7CSTPIsepw8CS4fMBHgmVp5/YMSadrwEnPNi4HPi0NjsR+07r4SXliMGjUK27dv1/08IyMDubm5ip/kQJtAzN1QaWuwBh0+01XiO3kikQt9H21wJ2pO/SzO6VeAcX07mB6XnZ6i02+cDWSZafrDiBWh46bX1+Gvi7fjihe+AqCc1GvrG7Fq5xFflOZwmyZFkrqYNkHvHXfN54VlNjI5JuSwthEPvHmM9LQ46vZ5kSgyCBjNEU6LeaalJH5c973wUlpaik6dOiW6GQmEnU3VseRrOdqIP1Razusrd2HND0csXSuZ8EJjrQiVFqR/zUNcM1JDzNWY07H9H1NG6l/Xwgiz/LvDAIC9FRHTmnxSv+y5Elz2XIkj51+/krA8LwzHW/MMu97DW2aSX/PirD1+IBwWNRltTTVKBs9SdJikLsXKi+0RnragpqYGpaWlKC0tBQDs2LEDpaWl2LVrF4CIyWfKlCnR/Z988km899572L59OzZu3IiZM2di8eLFmD59upfN9CWsfidfxTpV/1uPNmKcg7MNlz5bYvFqyQPPqs9ytBEjSZ3Zk8hOT9VUBJbO0UaVIt4q6hTzynaxW8a3stbupK4iHCTCYZGpLVBEG/Gkz3cJEdoU8eYpFrwXqiI+XRxmIx2fF/WhyeDz8uNnl6P/vQtwuCbmuG56iwx3cFYeINUHZiNPXYVXr16Ns88+O/r3rFmzAABTp07F3LlzsX///qggAwD19fW49dZbsXfvXmRnZ2PIkCH49NNPFedoKTC7hiJJnbPzWz2eoo3s4UmSurA8SV3kGZgJktnpKZpt0gRxx8R+KK86gc+/PWSrPUamIUcJzZIsg+qPn12O8soTWHb72YryHQrNi+x3vZWxW35LTWFt/zQ7dWOYL4zZCSJnBukmTs0LT3i231nfnC7js62xyCmzocVQdHHosOuHsd9T4WXcuHGGEvTcuXMVf99+++24/fbbvWxSYJDump7Pi4PkipHPLY5ArEnYDx3YD4TDIv5Z8gPzMy98NORjdtRsZHKMJLywBI32rTPw72tHo8cd/7XVHvkp1X4UeoINT/djTTpBnYZEUYxOQFvKqjCka370M0VVaVlFZK/n3LAoMhx2jR9MPCo286aub+C8QcnksCsfT5x8L6f3JNUHPi+JC9ImDGHl8VCGuTrrfJbNRqxzJL7/+oKPN+o74rpYRDfKgx9ujub/kR6B2bPIag6pZpmNnKLI/CwIaJR9IT3hhW2GjG3ffqAGWQxtUVAxipxq1HHYZd2jv3yy1bVK7U1h64nKqo43eO73IoKvb/Ka2JJKeJH/bvC1tpVX4+nP9ANdRNHkBCakJrvPC+Eu8jEvP9vYT0G+gmJllzzRaC2ts5Wq0lZIBpXu9gM1up95NXCqk7Yp615p989oNlN4MfHIzykIyuvbVc6Nf3wp3lmzx1G7/ITcuVT93oR1HHZZgsXfDSYkqzSFrbtsVh5v8DzcPsyZ/bWBU3iJoxuR94jyX/Xv0flPLDPU3PEKiHr4QetOwotviXQtUWcVm5OZilV3jcdPR3Q1PVNRmyzNtq1lxuXUdZqjwGmW3z1Hj2HY/f/Dnz76xtF5Es2RWv3sr16H9sbMRvIweu1zkRzsvJh41OeUD2xOrvePz7+3fazfkDuXqhetilBpmfDitcagKWzd8SEumhfONulFG72/fi8u+fsX0b+TSfMix8nXclpV2g8OuyS8+BSW1CzvLplpKeiQk4H2re3V7MjLSrPYHm2DnM6DT3/2HarrGvHCsmBPUodq9aur8qz6pNBgO8QcdmPbWAkMJQdRL8YcdT+wk4+IFRFSdaLRSbN8hdyXRaN5kX11udnIa6Vkow3Ni1vPpG9BDgpzM5mfiSJfJIye1vaVr3Zh/Z7K6N/JEG0k8cmmmInaiVAmcgeksyHNC2EJ+So2Ky2leZv5cToxC+jRLpv72swkdQ47cDzDQr3kSI2+8OL1qo+V54X1WDKj/YXf6ZsXRQg/BMWqjFfz8tKXO9xpjE9pMNCoyP9Wmo28bVM4LFrun5XHG1yxPYZCAiYOZmeFjpg0zNvFG40WtCR1a344in9/9QNTa7toy4Ho73rfiq/4pbP+RcILoQvTYVfWX6KTke3zi/j45jOx6NazuPdX49TnJRn8XYDmAV2HeKms5U+C9VwyUrWvupOWTR7WmXlt9fV5x7iXv9zpoDX+R+6fUd8YxmP/24rl30VC05XCi9xh1/gJXXt6T+Rk2o+5aOLUcMgpasu/4DEiJOibnbmjjZI0Sd2lzy7HPfM3YslW44KSrO+1cHM5Bt63wPQaTs3ZJLwQuohRn5fYtpBNzQuLJlFEVnoKTurQmmt/u4UZjeCpMBsEjISwuAkvsmfBmhRYwotbKIQlQe08zNdJeENwg1oeQC6UPLf0O/xt8Xb8/B8rAKh9XuRmI+PvOuakdo58D6xqXi4Z1hl/vWKYKz4vKSFBdwIMcwpVvA67QTUbfXdQPxAAYL8L0/61GicaeIpfOlu8+KHWGAkvPkUvlFRCqiejl+1SGVat/Vz+Qt95YT+eFmm22C0OKYoibv/P+kBnS5VjpJb2Os+aZJaR9wPWuJKR6l3YscbnJWTcFkDbm+osRr/5mR2HanHe40vx7rpYtJRcS6CucaXQvMjeS7M5VxCA1BT7Q7hVn5enrhiO7u1aueL0LQj6wgtvm6TFT/d22ejVvpXufmoBrfJYA95Zuwe1dcH2qUqkwy5pXloYdh64vH/Jnf4y0401L/LtrNWZ/Lzj+nY0bQdb82KvA28rr8Gbq5MnDNZoZXfrW+sx10N/DlaeF1Y/y2AVT3QweilLVSjbY2bCilxaee26xjDXij7eVbHtcNe7G/DtgRrc8sb66DYjLYFcwJVrXsyejgCBO+Lv5ILWuGxEV7z56+LoNjvRRoA7+Z0E6I+HvHV3pHtl1hz14uKGV9Zg1pvrcee7G3ia6lucONzKK9PbgaKNWhhWhBdWt5Kr1rMs+Lyks/wdZBfgOQdrnrPbf5NplQ2Yq/f/8MFmz64tTSTK3CraByP5SLl3YXkblNdT+rywO8naXRWKv+uTyGx0vEHbv+sNhBf55GopVFrgH1Pys9Lx6GVDMapn29h1w84mMCc0hUVDwdZK7SszgVa9uCj5PlIA9L3SYGt+HWleov+zB2leWhhG0uqgLrmKv1mDtHzST5PUxRzLIJbwIh8YzfpwU1hkOqX6we7pB/wRzWAc4TOwc65mm7Or6TvlqssFtDRY2pA9R/XD4eXver0Fh10B1qp2q2lilAcAIo7AQGyBxLquU+obw4ZmI543SjKxmWpekiOoUYMjVx6HQxYJLy0MI+Hl/emn4783na7ZLj+E5dTI04VYzppWHPV++txyporVDx3YDySyfiDbbKTd7+KhnTXb3BK51P42go5JqaWgfi9KvjuMm15fp7u/bpI6k34lCPxmI5aGJaxTHuCuC/vjf7eciVvPP1n3uk6pb9IXXsJhETsP1ZqeI2o2MmnOS1/uwFfN2pYgYXaf1c/OilbSqc7ND2M/CS9xRO+BZ6enIBQSMKCTdnV8748GoENOBu6e1B91DC9ynnGE5axpRVmwTqXilyDNS4REZvCMOezGUD+X6WefxBwInTw/o3IEZiasZEf9nv+rZKfh/vpJ6ng0L5zCC+NUeg67oZCAkwty9Itqcl3RmKawqDsevvjFDhyo1s9aLcFrNgKAK174yloDA4D6mVrRxITDzkywrdITXxYx8S1oQZgN5KwEYr06tMbKO8+FIAjYfeQYAKBtq1hdI71oIznMHB8uzLf2o42024wGM7+TyHw1rCR11aosqL89pw/zWNec7tRmI5NSBcmOvB83NoVNhRDdJHUm14nUkXIQKm3iW+Ll6xgWRcflRaSSCy2vh0UwSnZohgjrOX4kfnf+yb4omkqalzgytnd7AEC7VsZFFQGlmleamIraZuOr2efiy9+fI/vM/Losnxc3/DTcHNx+/e817p0sziRU86L6FwBqZCGgHXIydJ11nZS119P0CBBcKcwYZOT3Y9xflpgKtwqzUdiKz4sVs5GWRs4kb5rruvBMw2FRs/gpyLVW6iSmebF2bT9EyriB+ulZWUSJ9gLNAAAjurc13ykOkPASRx6cPAi3TeiL+dPHmu6rN24V5mUqpF6911C++mWZjdyYcN1cVX/6Tblr54o3fsiBZcdxM81BjhA58l6g9XlJjonCCnLNy56jx3HEoPYVoOw/yjICxtcRBPvaT8B8AeNlt24SRYUQ8ep1o3HZiCJL52iIhkpbuwdeJmyMJ2rh1sqY7mTMSlSEmprkeIoBIS8rDdPP7s2VYtuxbCF7n81Cpe2aa+wKL/7o+uY0NIWxbtdRHKqpw+XPl+Ct1bvR2BTWrHD4aol49K0ZSep4cWKmk38bdT8w8odpCajvh9l7Ig/lbbCQYVcAv8mE1f/MMs/q92vnD7UprBS8WmWkWtagSEnqrB6X4XbagAShfqRWNC+8WYz9DPm8+ADWu8fbr3hWXizhRY5RdkojXFq4+5bZ72zAf9bEkumt2HEETyzchvTUED773bioVoFnxWOU58MJguYXfpyozxV5gjQOuy1d86L82+wWyPuPfP4x7VaCMxOOWXkOveu78UhFlc+LAOt9JRZtZO24zIBpXjburWRu1/q88J+zoSlsX4PiE6EnWE+R0CBPOqWHmZpUEAQ8+tMhlq+d7M6YcsFFYl/lCew8fEyRiIyndgpv7R6rRB12bRyb6iBJCMsnS6Kl+7yo76tZ95Cbb0SFIGPu88L7CrLOZMWRWHldY/7048EYbTIuNYmiQsgTBL4+PKhLLmadFwnhlkxsAk+jZARF8yJ9pR/97Qvm5+qnY6WGU0NTmEvz8quxPbnPGW9IePEBrD7Ea2Y4tUdbvD7tNCy/4xzdfXhsvHZWyLbNRkHXV0JppuEZM04wsq66iZ3n58RhFzqaF/UK2kofSRYtjdocZyokhNkCi2mCXYHfXMg6V1uTwAG7Tv2je5kvqJqaRIXPFa8gliKriSQ5N1s2GwVM86KHxmxk4XnxVuQ+pXs+fjayq/K63FfxluR4ikmIlQ5SfFI7dM7PMvzcDDvThhNnwWSCZ9Bg5ehxA2nyarCh2XEr6sLY58X9PvLO2j2Yt3KX6+e1S31jGO+V7sXhmkhuEq3wYny8/HO5dZHL58XG7X356lMxtnc7/HHyIMOJXG8lb3bNkCCYCl5NoohWGTGvBe6wb0GI9ls70UZNYTEwmhcznIRKRzQv9sQQv6w9yefFr7jYQQZ2zsN708eiU14mRv1pkWvnbcmyi3zA5DMbeaN5kdphxyyV4iS3vLwNBn+7dIko9Y1hzHpzPQBg/IACtG9tLbzWC6a+tBIl3x/Gtaf3xD0/GqCZhK04xooWNC+hEL+btvxUZ/friLP7RYqxvjbtNNz5zgbcd9EAzTF2I1JCAp9JKidTKbzwfhdJOFREG3G0ddO+Slz2XAmO1cfexdq6RoUQFSQ0SeosDAG8tcRYPcwv0UbBfGoB46QOrTCka77u517IAOpzDi3Sv75dkj3ayAj5wMGjeTle763Drh3hKM2B2cgo2oinMKNd5BP98Xp/FPiUCv29v34fJg/rgrfXKn2lzKJA9PxceDQvTr1nR3Rvg09uOZP5mb7Pi/E1BQim70Q4DORkpCmO4VK8ADKzUeQavIuoe+ZvVAguALDjUC0GdcnjO4GPqW8M45kl27n3r61vxMqdR033YwkqHXMyLbXNK8hs5DHpKSF8OussPHH5MEvHOZVu4+E+YN/nhW+/99fvw6iHPsXaXeYvWbyRPx+e78OqNOwmJ2yYpayGSqfLfBTkE5uyEKOgEl4sN8sQPwu+J3VohYv+rnWuNBNC5BO9UgtjfD0r2go7un4es1HXNlpztSDwOSm3VmteOMYTQYDGbMQ72LHO/93BGq5j/Yi8X81dvgP/KvmB+9jdR47jm/1Vlq/550sHo29hjuXjvICElzhgxxEx3nZFO3KI/VBpvi930+vrcKC6zpfZd60+nwavQqWjZiPrwpFVnxe5g6deqLTWD8Nd6UXUEZoShfy+69V7MTehxH6X72ruwyBwC4d2hhM9AUR+yQ9/ezpeuXa09npmAltYRGuVuYZf8xIZeBTlATiOZd0reTZqv2ElxH7zPuuCiB0uP7VbXK7DAwkvHmNXgxLvFaYdYcmu5sWqLb3RhzXtrd4u7/K8RJ6BHc1Lqon0eUq3fMXf6qgDdRuif3uoeVkrKxLqh+ik2rqY8KJX78XMbMSKNqqta8TqH4w1jrzaCrvwOIDmZ6fj9D7tNdsvHNzJ9Fi5z8ux+iau8USQOeyK1hQvbP8NP6vyTJC3PdnTVrAg4YWwjd2B00o+Aol/LPseN89bl9AiiHKseurbiQbiovkRjOGIKFNjFCq96q7xePPXxYptvz03VuBR6fOiPNZuVekgDr81siKYeu8DT+SNhNS93yvdZ3ptK/fLziStq3kxeaaiCFx3ek+8OGVkdBtLiJVHOtXWNXJ/H3WUI/d9YOzoj9EkhpWFmrztXgqxdrJ3xwMSXnxKEFYEds1GduSPhz76Bu+V7sP/NpXZu6jLSF+BVxAzy2ZqF2lYGVqUbznyxshs1CEnA6kpIbz562KM6N4G/73pdEVejp6yrMyCym7kdai0Hgs2luHy50uwr+J43K4pNzvoCbRW8rxIwjmP1kMQvJ1WrITerrzz3OjvIkSkpoQwfkBBdBvLv0reb9q2Suc2G6n7LW8fY3Z3UcRHG/bj31/x+4t4RcWxeox86FPu/f1mQo03JLz4FL+EoxmRiCR1+ypP2D6WF572SbvwCiVe+bzImXZGT0v784RKj+rZFm/fOAYDO0ciMlbedS6W3jYObbJjkSJazUtsg5eDqvrUN7yyBit2HME98zd6d1EVCuFFZx/TyBuFz0vkD54VuJU8L3bGEysa0pzMNOb2M5pNSleOYvtKvHLtaPxx8iAM6pLHaTbSCkKOzEYAfvPqWtwzf2PCnXffXrsXFccauPdXmo08aJDPIeHFY+zO00Fw2I2Xz4t8hVZxzLhCr1OeW/odTn3oU+w4VGu8Y/N34DVjeVceIHZvfnV6Tzw4eRAW33pWdFs7gyyqaTZGvI45mejeTlkLS62yVuZ58VCdrXPqIx73ETm1PJoXk0fPqm10lGMSs+LzYmc80RO6WJeUb5Mf9uwvRuD/po7EXZP6M891ep/2+MVp3bnbJEDQaF549U9msvpD//2Gux1eYHVRp4j4c0kHd+kpbL82P0LCCwHA3uBmJU/IG6tiGVGtqKPVHHV5Yqo81qAQQB7+eAsO1dTjof9uNjxOWsk2cmaG8izaSPZ7WkoIvzytO3p1aI1//WoURvVoi6evOkX32BQneV6MIlE8dNhVXot98ni6RVUrhBf2PmYCrvxzSWDgEdIFC1OWnVfObmFG+WGtM1Jxbv8CZKSaZ7XlEsQEhkDMeRNYiy35d1y85UBcNKR6WB0XFX5nLs3kj/1sqDsnigOeCi/Lli3DRRddhM6dO0MQBMyfP9/0mCVLluCUU05BRkYGevfujblz53rZRN/idPyNh5OVWbVqOb9/e0M0q6Mz4YVfrWrGzkO1GPrA/3DZc8stHyta1Lw0ctYSsYreeH/myR3w5g3FOKlDa91js1Rp0od0jZiGhnEkNOT9Nlb6oWsmpjiqLeWO2Hr92koBRGn1faS5nxv5MQmCFbORdfT6NuuZyv2h2rc2rpmkB6+gq9W82Ef9HRPpa6iWm8y+l9ys54fIu3jjqfBSW1uLoUOH4umnn+baf8eOHZg0aRLOPvtslJaWYubMmbjuuuvwySefeNlMf+LwLbLal+30/TSLHruStsLJV6t0UXiZX7oXgDL8NgbfDeH1ebn/g02crfKe357TG4O65OLyU4sU21+cOhK/O/9kvDBlhOk5jFTcCrNRAsbUeGpeeAopWsnzIv1+rFmjk5/N9iWR8HKR0oWRgE6PlJCApbeNw6ezztT1fzGD55sI0Pq88DvsavdTP5tE+hpq22KMItrI9db4H0/LA0ycOBETJ07k3v+5555Dz5498dhjjwEA+vfvjy+++AJPPPEEJkyY4FUzPUWvA6aGhOjEx5Ka87PtrV7iiWXhpflmWNW8yO+Om9WZjQY9XtU4r5rZqwnVjtB56/l9cev5fTXbO+ZkYsY5fRhHWGmPfs4Xt9E7dTwnIIXworuP8TmaGHlepH/TDd4xS5oXGyuGq8f0wMMfb2Fel4XaF8oqPP5RkQy7Ic02Hlj7qRcfidS8WE0hYTXPy7zrT8PHG/bjnxYy8foZX/m8lJSUYPz48YptEyZMQElJSYJaFMNJhAyLZbefHf1dPiE//rOhuHJUN64kT0ac01x4LS/L3iqIB6sZWmPCi/1ruvkU5K0XRRHXzl3F/IzZjuYvw1vgzCsSlYPByB9CGSod/zZYKVDnFFakkHYf/V6rPiYWKh352ygXD289ILtkpqVgMKPujzozrh0mDCzQbOPTvAgOoo20+MlspB4XH/1kK3YfOaa7f01dI37x4gqs2nmE6x6c1qsd7r9kkKOaZn7CV4UZy8rKUFCg7NQFBQWoqqrC8ePHkZWlVWPW1dWhrq4u+ndVlTdpku1HDbEP7Jwf+y5y6f8np3TFT1zw+L56TA90ysvCyB5tuPa357BrTfZVryrt4KYQKX/hD9XUY9GWA/ztaP433sLLZSO6Ymzv9pj5RikAf+Z3UJqNlA3s0S4bOw/rD8hW0NOwOOlfVuExGxn5RelNntJ5jepPRWobedsBzu7XERv2VirMV09dMQzTX12Hm8fb09IV92qHx342TPuB3VBp3mgjltlIff99ZDY6Vt+EK//xleExX2w/hC+2H8LVY3pwXyctJYSGJn8UNXWCr4QXO8yZMwf333+/59fxYkDsmJOBA9V15jvaIDUlhElDnGlvzEiz4LALxCZ8qwKIVxO03KShbpOp2ah5d6/S/usRErxdbfPCO8irJ4xLhnXBU4u+dacNCVwlSyj9VdgNMmqnWq5RC/hG2k1rZiO+/dRMP/skFLXJUpQA6N0xR7cSNQ+ThnRiam94tXQszQuX1obDbOTUvHusvhENTaItjTdrXNxz1P2Ei22y03Gsnv+8fhhvWPjKbFRYWIjy8nLFtvLycuTm5jK1LgAwe/ZsVFZWRn92797tSdu8GCcfvnQwAOD6M3t5cHZr2HLYtWgTiA3MVq8kEzKsHsqJ+rxmqzlp8o635iUUsl4N2guMJkO5UCioRhirpkY7OFlohMMifvHiCsxq1mxJLNxczqxuLp9w9K5qpHlRtzVqNmruVmr/DjlGffSasT3wnxuKdT/nJSM1BZeNLEKnPH7nXbvwaFDkVaVj2+z3Ka3my5lW+Ed/+wKnP7wYS7YewP5Ka4KHWTJDI6zk3OqSz36WCxkCqR8WCHr4SvNSXFyMjz76SLFt4cKFKC7WfwkzMjKQkWEtLbod7A6IRked068Aq+4abzu00A5Di/KxfneFK9c0K+ynRmwekJ2ZjWwfqiGk0LwoPzMdCyTNiwvCS5+OrfHtAb7snoIgqDLYJsjnRWe7ujXqv41yy0QmL/4HrLenkz6ypawaX2w/BAB4/PJhAIDdR45h2r9WAwB2PjxJsb988tM1Gxk0SD151jWGMfudr6PHGPq8GCSpu++igYq/g5Cxmy/NC8PnxcE11c/GieblREMY3x+MJLe8+uWI/5y6vxjh5NpW1gRd2mQBO7Xb+xTk2G9AAvBU81JTU4PS0lKUlpYCiIRCl5aWYteuSMKy2bNnY8qUKdH9b7jhBnz//fe4/fbbsWXLFjzzzDN48803ccstt3jZTC68ypTbIScjrhPQ878YgavH9MBbN4xxfC6rjl/SAOqT2orKrKCqwZ032sgNs9G0M/g1byFBlX7f8dXdRy6cqjUHKS72db1VspPuxdKSyBMjqoVVHrORURQJ65jXV+7G2uaK0oZmI/A/fz+toPW6AM8ErFcegO/rmfu8OOk8xx1GQjpZ1G0tr+bet1NeJve+ggBMGFQIIOLm4Cc81bysXr0aZ58di6qZNWsWAGDq1KmYO3cu9u/fHxVkAKBnz57473//i1tuuQVPPfUUunbtihdffNEXYdJ+evmdUJiXiT9cPFCz3c73M1Jps5DGCUeqWYPP/rboW6zceQT/N/VUrgR6ymgji+1o3t+NjJxtDVL4qwkJgmKQT5Q92uh+yT9Taw7cNHnZzatihDphnCAICv+MiuP16JiTqdjHyjk1n+l0H8kXw6j+VErIH/5PVtEzD/E63rqZpE7r82KsJdt+oAYnF7RmLjidpnFwMsd8/u0h7n3VySnNGHdyB7w3fSx6tHcWCu82ngov48aNM3y5Wdlzx40bh3Xr1nnYKnvEM4IhKBiptFnYjTYSOKWMxxZuAwC8v34ffjrCPGJLqXlRfRZHn5e2KhPemJPaYfl3h5n7hlRmo0ShZ4YQBEHxiVrTEg+fFyevqvxQUdQKh5XHGhTCi7IukV70k/71zN4FI+1m64xUfs0L537xYGhRHvsD2w67Lvq8GOx79/wNeH3lbtx63sn47bl90BQW0RQWowslp8KL1TwvdrFab0wQBAzlyLodb3zlsOtn/PTye4G9DLs287x47ON6rL7RfCcozS+agYMz2siNgovq4om3nn8yuupkNw0JSpu/7/K8QKmNUA+URpoXtyK35Ndftu0gtnP6EwHspHPyr6ouTyFvspvFDyWM7lerjNRApYX/4vdn460biqMVytXwVZUWtEnqOK/POr0Vh93XV0aCQaRouQuf+hynPvQp6hojQsuJBm3/fW7pd5ytc+awawUrCwg/r9lJeOHEirbAqlouqFg1G4k2NS+Kc3DsIw1Izy75Dhf97QtUnTAvKaBWH3P666LBhZpFarORYBAOHVKFx/px7jJ6vEZmELeuIz3KTfsqMeWllRj/+FIL59RqUuTb1AUTefK8GGG22jaaaLLTUyz4vCR+FuraJhun9mir+zlXuDO0RQgFwb4I7yRUemt5NSqPN2DL/oi/yYlGreaFlaFYj3g9Ij9EK7oBCS+ciJyLwvX3no91957nbWN8gnWzUeRfR2p9jmMl4eXPC7Zgw95KvPzFTuZ+8lVro2rVb7aidTPDrjrnhdEKNBRSal6y0/0nKBsJp26ajfRMV9J2aVKxdE6GA658W4VK88ISdqxgNlkaRfRFhFy++5l40cUcHrnWUXkAxjY3ktT95tW12H3kGE7UJ85h1wp+MDu7AQkvnPB26rzsNGS2EM2L3cKMTmob8aBeTdXrZJOUn9eqBiWapI6x2rLCS1eP1ExARvO7oIo2yvWw/IMRRit5Y81LHBx2m+VJO2O0Mt1/87+yz9WmLcX+1i9n+i6YCXtJMg8B4DeBehkqzTM0qe/53orjuGneOqbmBQDW767gaku8hBcr76Cf+xcJL5z4JbzXT1hdRUv30Mm9PFTDzkgs14AYJQXTo1HliMP7zZz6afRgFLMzWhl1b9tK8bmXtauMMKptZHT3XRVeTD63M/CKDDOQUfFFdXSSVcwmLDv36/KRRdqNARi/+PK8sJLU2T9/icox3q78sOvwMabPCwBc8vSXXOeI1xxjxWHXB9ZGXUh44cQPNmO/Ybm2UbTonPG9fOCDzThPx09hf+UJ/N8XOxTbDlSfwMD7FkT/blRpUfZXnmCeS/4Oq0OeeQdEN3xe1Ohd+7fn9MbPRiqjqHIzEyO8GGH0rsTD3i71LzueEEpNitZspB7N5WYHO0OEdIzefTF7x1hH/fmnQ6w3JCAIgqBJdOjEaX1vhTILrhPtR1CijdzMtZRISHjhhDQvEYbJQub6Fuage7tsy+cwEwRf+nKHYcbZBz/crPj7zVW7FUJEk0qL8s7avShjCDDyVqiFEFOHXRejjdToDca3nt8XqSkhRTRVblZikmQbPUGjx+uqz4tuaLKIcFiMFq+0dE7ZN2P5aGk1L/qf8WBWgNEts1EQhi+ehYAA7eTrog+47fskgh1tZIX4mY3ichnPSZKv4T2keYnQrzCWQjotJYTFt47DlaO6cR1rv7aRNdQ+LwCwjlGXRm5eUmtrTB12PaxtZDYY19TFhJeM1MT4V+m/DoLhIGw1x4RhG/S2i+Aut8A6Nva7VlOo/mo8eV6MkLqg3mpYr5zCbRP6AuB3vgzC+FVbZ57iwElVaR4qjzVg1hul+HK7ftI3AQLzfjrPsOvocG7IYbeFYbVfJUk0mgZ1v08JCchM4+tGMZ8X997Sv3yyNZqcToLX50XejgaLyWekQ9VRSlZhtTQIg4uRA3vcoo30HHZF+7V8lNFGrM9VZiOGg6+160UO0rsvrO3n9OuI6Wf3BuBvh0qr1HLmZ9LcE+5oI/Md73x3A95ZtxdXvbjCcD/Ws3ZsNvKhw66fIeGFE6sdy2rRwiDwwi9HMLfzrv5FUcQDH2zG/R9sNt+ZgyO19fj7Z9s1AwlL88Ia5OW7aTQvJteW9vZitWQ2tlwwqBDj+nbA7In93L+4Q8wcdt0UzH44XItqZg4f+w8lzLARqR125do25w67kX/1NCysvDjK2lbJEyrNo3mBy4UZ1ZTqRAbJTbX1TWHN/RRFEXUJLA9gBRJeWhhWJ6lkcYqSePnqU3H+wEKwhooMjjpCQOQevvTlDvMdVbBu/YmGJt3BjlfzojQbqTQophl2+ZyP7WBmsspITcHca0bh12ed5Pq1eTH82kafufhaXPvP1TjnMa1jt1vlAVh5XlbtPIKT7/44mjlV7mRpJ0Oq1I/0hLo0s4kmiYaZ2jrzyV+qpK2s78V3E5wMyffM36T4+4P1+zT7nHBoQrYTJWmHIGh2eSDhhROrq6p41HBxE7MOLUnrrN14iiAC9u3urMMe+9/WaFpuNaxBgHUO+cSjDnk2r20kndfZgJOToXW4DULP0fvWAowFOre/28Fqbeh8WBQ1z4/3OZmVB/hoQxmAWOZUI+0dD01R4YX9uZ5GRoI/w66FRiWIglzzasfS95UnquMdap3M2W+v3aP4+8UvvtfsY2Q2+u/X+02vQWYja5DwwonVfmU1+2yiuWBQIXp3bK37uVGH59W82H01WbldPt5YpuvdzzIbsZCvlLUOu8bHxnKAcF2KyZ8vHYyOjAE7ECsjoyR1BofFoxYP6/q8z4mVMdewKrTsMzsrZ+kYvftiHm3Eazbyv/Ry9ZgepvtIX1dZ3yvxiACOG2TYnf7aWhyoUkY81tQ1onR3RbTPxUvANBtf5O4BPX1WSVoOCS+cWO1YbtZwiQeZaSlYeMuZGNG9DfNzN4QXN1cW4bCoG6asDpXWQz7YWE9S58xsdEaf9rj8VHaUlnpw+euVw/Hub8bYuk4iiKfmhXl9hhDB+5xYhRaNDpULO/bMRpF/9e6LWf0wP0zcbpGVnoLTemlrH00a3EmzTS7UWSmT4CVmZqNVO5URj5c+sxyTn/4SH2+MaPP8onk5f2Ah3p8xFs9cdQoGdcmLS5vsEKwZNoFYdtgNoGrOqMCZ9H1Yn/M77NprF4uGsKgbpsyjeampa8Tzy2KqX3WOibfW7FEfosCp5sVosFV/dPHQzhjejS1UJgpDlxeDD+Mxx7Cub8cPKiaY6B8rF1i+P1jLdQ3W9fRWw2YaXO48L/5XvADQTqwvX30qLpQJL9IIJb8vZrfAbp+zGj1ktv+2cmWtra3Nf0smpXj5vPDEkgzpmq+4736EhBdOrAovQbUr6lYzNvB5yeAOlXbv5WwKi5Z8XtTtXrXjiOLvXUeOWbq+1jPCGka9w81cKF5hWB7ASHiJg65AhPZ58/Y9ls+L0ZzidL4x9XlhfhC7qP97ijXUQlxR2yxmFfXsdJmvmGDs02T3HvW7ZwHeWctexKgvJ4rmwos6i7dEq4zI4o/X3G3E1OLupsVaA2GW5oCEF06sdqug+bxI6E0uRpqkdM6wcDdXf01hUdfnhWcFo067/sIyrQOeEVHNi80AA6PxIwCyi6EPhdFEEo/vFhZFRjI5vmPlfYcVbaTGqcN22MTnJc3MbMSdpM5auxKFVlhTRxZF/pVP0GYCsXSPJGdrK8x6cz3XfpFQaePBQG9ckgQxN4SXNq3STZPlsQTie340wPG14w0JL5y0yU7H784/2XCfUT1j9tqgal5+qqqdIyFJ66yBglfz4rbwoqd54Yn6SHMoXNqtkC1hdHWh+T8/06u9vnO34R1x+WvlZaVh5Y4jGPvw4tj1ReBXc1cp9jtYXYe/fLIVPxw2Nu2wsukaOuw6TLAcNRvpvEJmi6Ask1W2BG8iyUSjTjEhhUarkQsvfokmNxMa9MoftG6OOOT11TMiJAim46z6Hv/958Nx7ek9HV873gSjR/uAtq3SMeOcPszPCnIz8MnMM/Hva0dFt916XiR9t7qQnt+5bERXxfeQMBpE01P4BlA3zUaN4bCuwy7PCsatJIJe+LwEgcnDu+C2CX3x1g3Fiu2CSXkAt4UyQQCuevErRYE9EaKm4N7MN0rx98+248fPLDc8nxXNy7byaltOuorrmeR5MVsEzRzfR1GyQ83TPz8Fvdq3wt+uPMV+I+OI2mQqgJ2UTy60mb1KcfGzgrnZSB4UIM8r1apZeLETaq+GZ82s7lNmTuF+JZit9hkCBPQtzFE4rk4a0gkr7jwXf740WBVeBUFgOodK0rpffF7CYf26QjzXcax5iUaiuKN5+ex342KfBUCuSQkJmH52b5zaQxkdIgjGAp3b3y0cFjUrWtb11zdnTj1SW294PqXDbvO/Orqk859Y5rw8hMNoo445mVgw80zdzycN6YTFvxuHAZ1zbbYwvmg1L0qzkXSjWsl8XgQYRxvFS4t5QkcTLCH3eZEn5JN8Xtxw2OVZFKmFl/TUAAw4DBJTkraFwJN0KSikGEYbeZvnhYVjzYvD1QaPScEI9RjTJjst9pndRvkEY4dd76/FCpXmhSm8GJxOndzQ7vXsal6SDdb3VTjsNv9rRfMSrxfKrKp0fWOsI8k1g9Kzb3BBeOFxxlVrt9T+f0EhmK0mPIXV/Y0GUa8z7LIIizCINmINIu6OYDGfF7tnULZHcWsCP18ZmI1cVr2whEcnGj65GYjHYdepql+6nt5tYWkIg+J8aweW2UjeZ6Tfsy0IL07lv908kYiieZFWueblH5/HAgSkcdEdnxfzfdTaLTIbEUlNrDwAw2GXM8+L22kM9Lz7v9x+GBOeWGZ4rNOMo25rXhSfBVx6iavZiKV5cfBo5VqbWJYX/RPqhb/y0tREmhc5allNENj3Rh4qbSYQO32fHl6whWs/s34n9ZXj9U14d91ezXFu+LzwvF/JYjYi4YXQwDIDuaN5sd0kJkY25q2qhFASB6pPoCmsDaW1i93zBHO4MEeAuxo2M9xOe89y2DWalPQiSLivZ+KwG9RVsV20mhdVqHTzv8pQaWOcCsysatGsc5r1e0l4eeijzYrt/9tcBlEUXfF5CQkC3rj+NNN95AS1jwWz1T4jCHVDrJCaEsKCmWfg3H4do9v8Vh4A0Ne86LF211GMemgRfvl/KxwLL25rXpKpBxlqXuJ4LTvIZREep2ynPi+xPC/sz4OaL8ouzFBpWa+RPs5KsxBt5LBN3dtp6/toktTB/B2ub+5cr6/crdj+5fbDWLLtoGsOu6N7tcOfLx2su496LCefFyKp6FeYi8Fd86J/qwcVObyaF7eFF6vpu1/56gcAwPLvDjs3G3Hkebni1CL07tgavTpoBz+jqseCAJzSLR9AMBLWqYmr5sXla8nNRiXfHYpcw2B/q9FGQ4vy0b1ddux6UrSRzcKMyQZrkcTSvKTJxhzeJHV2KcjN4NvRzGzUHGDQKU8byLHuh6NocNHnxeieqGUVp5GXiYKEF0IXuXpRUi0yQ6U5hRe31Qt60UZ6hBmRJHbhqW007cxe+HTWWeiYox381PcxJzNN9nsq7r94EG46pzf+d8tZzhoaZwRBiKsWyX3NS+yE97y3CYCxgGQ1K+r5AwqwUPZM7ZUHSF40ZiNVkjrpd7m2wGvNCy9mPUHK89ImO137oSBE/Z+cEB2zDb602mxEmhci6ZB38ZQUKVRa+1bwlgcwG+c376vibRoA65oXhUnA0pFaos6cHDVVWP4M6k3pqSGUzD4HJbPPQUZqCvKy0zDr/L7o3VE/k61fMRMMn7nKvYRpbmvzWKp7w1BpiwJ0ZDKO/R02CZUO6sRiF9M8L83ItQWm4cEOpReeLiaKoqkWUDIb9SnQvtMC3CkPEDKXXbRJ6kjzQiQb8jHByGzEUstef2YvzTazieaiv3/B3zhYE14EQRmK6HTSkwarTQYCl3RfmMILY3jplJeFTnlZjtrlB8zu7TkyXyqnuG2hYuWIMXbYtSi8qJ58LM8Le3/WxJJM/lFq1BOrAKU2Jmo2siDUOZ2aWfeb9d6bRhs1C7osU2BIEFxNUmck0Kk/4118+o1gttonDOiUi7QUAU9dMTzRTfEEuVASC5XmO5ZlKjGb1Ky+vGZJoTTXl+3u2GwE4P++2IH9lSd095FuFfOeBXOxw4XZvfVzBmFWun+jFbXVaCO1GSSW54V8XgDtxBpx2JVviPwjvy+meV4c3kPehY6ZH50k6LJCogXBLc0Le5z+j6yMh1bzEkwxIJit9gm/LO6Obx64AKf1apfopniOFdv7pCGdmNtdd9g1ScetRjkxOc/z8rxJJeqo+ZkxuiZLWXo1kfIAxvfWz3lsWJoXNx12BSgn47CJ5iWFM4x1xtm9AQA/H93NUnv8hnoeVaf+l/qOwufFzGGX47qFBtnQucxGHPtFhRdGHxNgvS+xiJqNZF/6ycuHYaSsjIc22si/76MRJLw4JKhSKw/yF4DHlgoAEwYW4PGfDWVO2DzO9FaiR6yGSrvpsAuI5vklIKlwWZ8lF13yI+au8f0LTMVCPysTrGpe6m04WSp8XmTRRix5llfzMuu8k/HRTWfgwUsGWW6Pn2A57CqijZp/VzvsGt0lO/V+7GDWEyQtHUvD7JbmRfqqimKWqq+m/qZB9auKS6uffvpp9OjRA5mZmRg9ejRWrlypu+/cuXObX+TYT2amP2sE+XgMdgVlfgU+s9HQonxkpKYw7w1PBV4rQoUTzYsbUSq81WyTvZ8AwLu/GYM/XzoYd0zsZziKq80mfoO1+DXqk5Z9XnTMRiGB7VfG9HlhNCgUEjCgc27go5M0DrtgaylTFQ67xufkuSNG942nVpbIkeilPqp5CWuuKbjk88J1CtVXJeFFhzfeeAOzZs3Cfffdh7Vr12Lo0KGYMGECDhw4oHtMbm4u9u/fH/354YcfvG4mwcDKHHPjuJPQo102rhrVXfdYrkGA/5KWNS/KonvOzUa85g+eaKOg0zE3E5ef2g2ZaSlo35oRCirDz1+dZfIy6il2HHblVJ9oiGwXBKZvRtCFEatovq+gfFdiDrvKid/oGdlJmS+HZ6QQm/8z4mB1Hc57fCkOVNcB0Jpr3NC8SP3RSpXtoPYxz4WXxx9/HNOmTcM111yDAQMG4LnnnkN2djZeeukl3WMEQUBhYWH0p6CgwOtmEgyYfqY6L8XvL+iHJbedjbzm6sj2NS/8L7DVUGn5qY0cbbnOBQualxZgNpLzf1efilN7tME8ZppytnnEL7CjjQzyvNhw2JXzyIKtACLaA5aJKKip2+2icdiFwDSBKH1ezDDfw2j+5h2SeGSPbw/UYN2uCgDeaDyk0H351/GzptMJnr4Z9fX1WLNmDcaPHx+7YCiE8ePHo6SkRPe4mpoadO/eHUVFRbjkkkuwadMm3X3r6upQVVWl+CHcwcmky3pheNSiVhYfVpPUya8/841SS8eqiWhejImZ2lqOwy4A9O+Ui7duGBN1ZH/0p0MUn/t5MGV1P0OzkQtZUYFIf+A1GyUzLC2AUvMS+UMh1HEuIqxeV4I72siiNlceoswrBN9+QV/DzxsYhT7V3ywnMxXJgKfCy6FDh9DU1KTRnBQUFKCsrIx5TN++ffHSSy/hvffewyuvvIJwOIwxY8Zgz549zP3nzJmDvLy86E9RUZHr34OwDtNsxKN5sWA4arQ4cfBofngRRVExCU8a0gln9GmPX57WPbpN+pQ5LragOemioZ0T3QRu1H1UFEXDfmu1S+kJbiEds1FLC5XW5HkR2IK+vBKyG9FGRosJLrORaD1+Ua554RWQ5Jm4WUQ1LwZfOjMtBQtmnoH7LhqAT2aeyXVdP+I7nWRxcTGmTJmCYcOG4ayzzsI777yDDh064Pnnn2fuP3v2bFRWVkZ/du/ezdyPsI6TkFam2Ygr2oj/GlYXvTw+N7yoz3Tx0M7497Wj0U7m7xFz2G1ZE5Aavylalm07GP39QNUJHKqpi/6t7n9uOFHK0bsVgqBX10e77e4fDXC1TX5CazZiR87INS+85lsjDP0+uDUvXLtFkdeEcyuNRNTnRbaN9f37FebimrE90bcwx5XrJgJPhZf27dsjJSUF5eXliu3l5eUoLCzkOkdaWhqGDx+O7du3Mz/PyMhAbm6u4sdLTusVi5f326DsNo6SqzFDpd2dCBKreVGHkjebiKBdEbLcFlqSQOO37zrlpUi044mGJoz60yKM/OOnukJKkyi6msVXb8yIpME3d9i97vSeOKlD8EpG8KLJ86IKIY8KL6poI6MexmOiNTYbmR5uK2uU3GGXV0g2+yYsh12/vX9u4anwkp6ejhEjRmDRokXRbeFwGIsWLUJxcbHBkTGampqwYcMGdOrETnwWb169LuaEGMfiub6B90Ww67BrZQViVRZyV/MiMqMgUhVRENK/rBW1a03xPazJxw8crq2P/n682flb7bcQDrubXFHv64cE7cQNaEOHgxoZwkvl8QbF3xHNi3qL0l/EDbORkR8WlynbRheRm414F1Zm7099k9Zs5Kd3zk0899yZNWsWpk6dipEjR2LUqFF48sknUVtbi2uuuQYAMGXKFHTp0gVz5swBADzwwAM47bTT0Lt3b1RUVODRRx/FDz/8gOuuu87rpnKR7IOHHCf2dts+Lx4JhAJcrkCsCpWWtCvyCtuC6l9Fe1pON/Ltuk/evaW+qe4i7mte9H1eWA67mk1+vZkusa9CGQWozosT07xYMRuZCDeC8WKCZ9ywI+AqzEYuDU4ss1Gy4rnwcvnll+PgwYO49957UVZWhmHDhmHBggVRJ95du3YhJNOrHz16FNOmTUNZWRnatGmDESNGYPny5RgwwH923mSfgC4d0RVzl+/EuL7WC+mxVkNu53mxipv+C+pQaWmAzEhLkW2M/CPldWip+DW6SC4sNDVHaajnoLBoxYXcHL1bERJilduV+wuq/fx5L91ib8Vxxd/qUGkJ+cLK6YIyJBjrbvgCDayjdNjlO8ZMy1SQE0noqjQbJSdxiZmaMWMGZsyYwfxsyZIlir+feOIJPPHEE3FoFWFGTmYaPvvdOObKxwzWfjxJmJwmjzPCTfW/OlRaGmAzU7XqbGY68KQdUrT49ZvKn4rUN9WiSjhsHG1kFX2HXbbmBQAe+vEg3PXuRsPjk4WZ5/bBz19cEdug8meRfpdrLdq3znCUpE7tFKyGq7aRjT6iyLDLeYyRKJ2TmYprz+gJgF1SIdnwXbQR4S/UKz9uf13GttLdFabHqef5uyf157yiOW5ksJQQISoGPGmwyJRpXqSPWZNSS8o95tfBUy6USM7fzGijOPi2CdCvfnzegFiqCb/eS7cY07s95k8fG/1bHSodizaKbevAqGAvx+yehQTjxIlu56eSkH8HN7rYoz8dguz0iD5C+X2Ss9O0oCGUiCesweC90n3mB6reYjdNDm467D7z2Xf4/lBt9G9pgGX5vLCjopJzQGHhR7PR3xd/q5iUpCRhTJ+XOEgvej4vgMq3yof30m3atZKlGwA7SV2a7D1rk51uXJjRNIudcR91U/Mmx23/SWViuuTvJyS8EJbgNhvZfHnUE4WbCUbdDJUu+f6w4m/pvmQoNC/NZiPGZVvAHORr/vK/bQoti+ToqPF5Cbvs6K1DKKSdzHp1aAVAub2lRTiqaz6xNC9tstMwsLN+iozjJmVEzEKt3bjn/Qpz0LaVsuaXXNjgvYbRuMqKdFT/nkyQ8EJ4g80XRv0S66nS7eDVCgow0bxYLN6XzPA+TXXROi+Qa15ivyv7iNvRRnoIgoB7L4oEJfxm3EnYdP+EaPbTzLRYn6qzWEk9GWANAVlpKehbkINOeZno0b4V/nDxQN3jD5o4zId0cuxIuOHoLwgCTu/dXrHN7ezJSvMaOewSBrQE1ZwaJ3leeFALGG6pyQVBsJyR1wpSO+WOhFLT2Q67hBGtM1Jx9FiD+Y4OkPe1Br1oI5cddvUICQLGnNQemx+YEPVbkMhMjWnzrNbzCiLqV16Z+FHaR8CHN50OIBK1k5+djouHdsb76zlM0+rrMa4pxw2NrQCtsOLmwgxQZR2WXztJVS+keSEswR9tZO+FWfTNAcXfbtr43U71LkcahzIY0UYsR+GW4LvghFYZ3q+reB12Xe01unleIv+qBRdAOclZraSeDCgjZ2J/pKWEFOHGc34y2Ob5jTUvbkRAhkJaYUXPx8nJNSRYCTSTDRJeCE+w+8Lc/vbXir/tLE5+cVo35nY3fV7URPO8pGrzvLDMRiS7GBOPgoTyx6IXKt0kinFxNOH9ticaWoLmRe4LInIvhFplpNp7rwQTzYsLix6WQ7bbGj35+VvC4oiEF8IS3KHSLr07dlSrvz2nj2bbtH+tNrV9O4GpeZGEFzIbRenSJgsA8NMRXQ33i4dfqkLzomM2ilSV9r4tvJNNS/B5kd8JEdYWMHbkAbN778bzZ4XCy4UiNwQZucNuVro2ZUOyQcILYQ3ON8E14cXGiRKx6ohpXrQpv700VwWFhbeciXd/MwYdmzOA/vK07ob7iyLwyKVDkJeV5lmblKHSzWYjzT7eJk6U4NUutASfl445GWiTnYaOORlonZ7quc+GOpeMGjcEC0EQNLWrGl0WXuTfITcz9t4kq/BCDruEJUZ0b8O1n1vOzHasB4moPyVdMlc22Ur2eKbmJVlHFB36FOQo/jZ7RiJE/OzUIrTOTMVvXl3rSZsUodJhtubFdZ8XHXi7bEvweUlNCWHFneMjQkVIUNwbLwRJsyR1buSHEgStj0ufjq2xdNtBAO4scOQOu7mZyT+1J/83JFzlrJM74MUpI9GnoLXhfm7NzXYEEbcd4XiIlgdIS8Fr00YjHI45nTKjjVqW7KLBTDsmzVFeyqFyH6gmyWFXXR7Ad2aj5Ne8AOqoPZkPjAfXCplqXty4hjJfzW/GnaT4Lq5cQ6bZkS+iauuSU+Al4YWwzHhZunKvsaOhYBW48xp5M8ecpMzn0MBy2G2xXi8RTDUv0cHcu/skFyqlUGn17NgUFuNiNuItF9ESHHbVKDUvXlzBeWFGM0IqzUt6akjxLN3Q7sg1L3LzdfWJRsfn9iPk80J4glzocGLGsWU2SqDmhQX5vGhR2//1kN/Wc/t1dFUTI7IcdlX7zC/dG6estpyalxZgNlKjyETrge4lJHhfHkCAoBgH1eMFbySk0dCWogiVju1YU+dtvqREQcKLA/oW5pjv1EKRv2NOJhw7gkgiTDJGwgs7z4uXrfE/5majyD2T7/X7if08y/sTy/OifFYvf7kTn21V5h4ClCtbN+DtD/FImOc3BI81L0IcQqUl/x2JkKAUxNx4rik66jvSvBBRPpl5Jl6cMhJDi/IT3RTfIh8MnEw4tsxGCXTYZUE+L1rMnpF0y9Rpzt0UXuSPRU/zAgCb9lVptsV74fL4z4aifet0PHH5sLhe1w+wqkq7fX6j7uiWz4s8d1EoIr3ErsF5ESMZR73QO7VHJLjikmGd+RsaIMjnxQZ9C3NI62KCW5Vw7cghiZALjIQsufCSk5mK6hONOKdf/PyG/Ahvn5DvZbZCtgpL88KaQ1iCVqasAKcbmPmy/OSUrvjx8C4tLkoNUD5zeQiwW4QEwdDrxa1oI3mf15qNHF9C4zc17/piVByrR7vWGc5P7kNIeCE8Qf5uOtGE2Dk2EQM8bzO/uP0c7DpyDIO75nnbIJ/DEyoNqIUV45BWq/x18bfR36MZdhlLW5agleWy8HK8wVy13xIFF0B5/73K+2MYKu1SDpYUjdlIdg1OAcmonakq6SUlJCSt4AKQ8EJ4hFs+L3a1NoIQl6zuUXjbmZedhsHZLVtwAfijjdS31U2z0codR6K/G5mNWE3NTndXeElWvwQ3kN/+/GwPNC8h70OlBQHGDrtulCBoYU4gLezrEvHCrWgjO3OVAH+l33+4uWDc3ZP6J7gl/oE3JbuiorAA5Hu08o6Gs3Oajdx22K2pI+FFD3lfyc9Od/38AgTDAcOtDLtK3x1BoeVzo+6aWvOS7JDmhfAEuXOanhc8D/bMRs3CUxxVL0ZXumJUN0wc3MnTVPdBg7vwoqD89R9TR2LSX79wvT16hRkBtqDldr6VGtK86CK//V4Ir+ZJ6lzK8xJS/q24hgual0SkiEgkLUtUI+JGikJ4sX8eO8ncBJPoAS8wS2RGgosS84KbkfupXq0O7OyNya1JpzwAwBagO+dnuXp90rzoI9fiqstMuEHEYVcft6pKGzvsuiC8JCA5ZyIh4YXwBPnk5GRFYFcIibdzY8vLvuEM7lBp2TYvn6gkfPIKLyerymPcMbGfo+uT5sWYz28/GwtvORNtW7lvNoJgPM6EXVCyCVD7vCj7mht+NaR5IQgXkL9I5qtsA+wKL/avaIt4pJBPJswG2miSujg9SGnyYJmN9NowcVBh9HcbVjAFNfUkvBhR1DbbE60LIBVm9NZsJKiijdTXc8Vs1MIyX5LwQniC3M3FWXkAe8e6GZXCA8ku1mA9ngsGxoQB6XaqHXa9ImykeWFcWIRx3g6rXDmqm6PjCfsIiEeotLa/yM/Ka5oy6mUtTXghh13CE+QDvqNoI7vHxdvnJb6XCzx2QqW9LGYZ07xoYbVVFN3JZfSTU7pg4qBOOKNPe/OdCU8wS1LnRgI5QVA6qau7S8n3h7nOY9SUFia7kOaF8IYUl3xe7PquePEeZ6bpvy7d2mZ7cMXkJS0lhGln9NTVOLBqG3kpkBr5vDD3h6jom3Y1L60zUnHegALXM/YS/AgmPi9umIRDgqCqbSS4rq1taUkMSXghPEEZbRR/h10vzEZ3XNCPuUJ+5NIhNPnY4K5JAzCnOQcOoCrAF90Yn7bETAPaGWVLWbVmmygq+6Zdv654mzcJLYJgnLnZtQy7qhpNFw4uNDiCDfWWGCS8EJ5glE3SCnYP9WJO0FvZ5GaR9dV1GEnqvCRqNuLWvKh9GPiPI/xFSDDuZ01upPQRtGPiyB5tcVqvti6cvGVCwgvhCSEOn5fx/Qvw6E+HGJ7HttnIA+klmvwuDtdqibA0Lwqfl3g47Fo4Rt4cu6ZR0rwkHkEwTq3vRiSQxmzUfL0+HZ1HUJ15cgc8e9Upjs8TNEh4ITyBx2x054X9MGlIJ8Pz2B3avXBeE8BuD01A7iBf/bJ9Xry7z2JU88KrelH5vNg2G9k6jHCRyPvrbah0SADSU5SaF2m7U64e0x0TBxuPo8kICS+EJ/AIL4KJl3/nvEwHhRk9sRsxV/80AbmPNF3IhQIvb3M4bE3zEjEbxf62HdJPnSfhCDB+h93IfitAWXtIGp/cGKdaapoGEl4IT+D1BzB6d+dPH2vf58XeYabnJM1LfAjHOdrIqs/Lj4d3UYVK8x2n/grUdRKPmcOuG8JBSBCQJivmSTKrc+IivDz99NPo0aMHMjMzMXr0aKxcudJw/7feegv9+vVDZmYmBg8ejI8++igezSRchEfzYqSiL2qbhY65/tK86Pu8uH6pFkkuo/5T/DLs8mtefn1WL+RkprmSpK4wN9PWcYR7xKMwIwQgTWY2ammp/L3Ac+HljTfewKxZs3Dfffdh7dq1GDp0KCZMmIADBw4w91++fDmuvPJKXHvttVi3bh0mT56MyZMnY+PGjV43lXCRVIeh0k6jTDyxGum0ijQvznji8qEo7tUOvzv/5Og2kRErHY/II0mgTjMocpfCUPlb7QPP/3IEfj66G64a3d1GKwk3EeJUmDEtRWs2sgrrMDIbecTjjz+OadOm4ZprrsGAAQPw3HPPITs7Gy+99BJz/6eeegoXXHABbrvtNvTv3x8PPvggTjnlFPz973/3uqmEiygKM+q48ovQFzKk7XblAk8cdnXOScKLM348vCtev/40tGudEd3GzLAbh2gjiTQDOxCrb1oV0CcMLMSffjwY6alkuU80IR2NqoQ7ZiNln5K6i1WtDmv31BZWTVrC0zenvr4ea9aswfjx42MXDIUwfvx4lJSUMI8pKSlR7A8AEyZM0N2/rq4OVVVVih8i8SjKA+i8W6Kov5qOeeP7JwRVAHsCJfu1e0iJu649oyeA+FWVVtc2+vWZJ+nuy+qzJMAGi6vH9Ij+Ho8kdQIEVXmAyO+8p552Rk+8feMYxbZfn9kLZ/RpjzP6dHDcviDiaXatQ4cOoampCQUFBYrtBQUF2LJlC/OYsrIy5v5lZWXM/efMmYP777/fnQYTriFXthgN7GZjvq8cdgX2mSnPi3s8/rNhuGZsJYYX5QNQ3dt4OOw2e730aJ+Nkd3bYPUPRzX7soRVEmCDw92T+iv8qzJSQyZJ6lwwG4Wg0LJJ4yOrijmLuyYNAAB8Wx7L9jz7wv6O2xVkAq+znD17NiorK6M/u3fvTnSTCCg1L/rhoKLpfOQrh11QqLTXZKal4NQebZHarGKP162VfF3CzdlUBUHQL/kg+bzINrW0ir7JRHpKyPAdljQv7Vun276GoOPzYlUuonVSDE+Fl/bt2yMlJQXl5eWK7eXl5SgsZNd1KCwstLR/RkYGcnNzFT9E4knhyM/RMTfTVMiw+6568pILOqHSNHF5RtyqSjcLLdJKWM9EKH2mhvpAsJA/rfTUkInZKHbU/RcPtH09J2YjQounwkt6ejpGjBiBRYsWRbeFw2EsWrQIxcXFzGOKi4sV+wPAwoULdfcn/InZYL7o1rOQm5lmOh256ZXvBqR5iS9ygSUu5QEYjsJqWNpAXg0hrZwTjygqx5X01JDh85PMRpFUCfauKQgqs5EgtYWkF7t4XlFu1qxZmDp1KkaOHIlRo0bhySefRG1tLa655hoAwJQpU9ClSxfMmTMHAHDzzTfjrLPOwmOPPYZJkyZh3rx5WL16NV544QWvm0q4iHyVwXo9T+rQGoB3Pi+eOeySz0tciZPLi8znRbqWoPtcmdFGnH2A5ip/IH9aGakhw84lCbYhHc0rDykqs5EE9Qf7eC68XH755Th48CDuvfdelJWVYdiwYViwYEHUKXfXrl0Iybw7x4wZg9deew1333037rzzTvTp0wfz58/HoEGDvG4q4SL8K1Hj/XwVbaRbHoCEF6+I162NroBlmhe9S7PNRl60ivAKeb8y07xEtXEwV720a5WOw7X1SA0JaJQ5tIRCgiKkuSlajoKkF7t4LrwAwIwZMzBjxgzmZ0uWLNFsu+yyy3DZZZd53CrCS5w6MLIK81nBI5cXMhslEC81XLEMuxw+LyTAJhXpKSlcSeqMBFqJrPQUfH3b+WhsEnHKgwuj20OCgHSZ5kU6pwuBTC0WWi8QniBXozsZ1m3PCR7MJZHBi3J8JIq4mI1kmhe95xotqidrEfWB4CBCtKR5kdfZ4jFz52amITNNObWGBKUpvTEqvJD0YhcSXghPcBp9IU0Q/jIbgTmD0rwVH+LisBu7mr7ZiPEBb2FGwh/In26GabSRpHnRKxCiPa96v1BIUGijw2onK8Iy9MoRvsSpF743ZiP20JVKDg9xwctQ6ajLi2geWULat2ATiTaK/R0JlTaPNgLsBxiEBKUDeGNYLSwTVqFRl/A1djU4nhRm1DlnHqMaMuEO8SjGCGg1L4Ls/2qkLinvDyS8+IO515yKgZ1zUZCbYb5zM5EMu/qEOZy4JaR+oO4O6mEs6rBLZiPbkPBCeI6T19PulODVZMJaoeVnk/ASF2w+0qFd80z30eZ50a93wzYbkfDiB8b17Yj/3nQGBnTST1aqHo/MktRJhExqIAH6XVQdSt9o02E3XsJ8ECDhhfAcJ4sLX5UH0Dmnbhp5wlXsPNJubbO5+oI2z4t+FJkTsxEpaPyBvE9kmDjsxo7hEB4YWjn19eTwOOw+efkw031aIiS8EJ7jxKM+noUZz+jTHmNOamd4TlLzJg47z/SVa0dz9aFYnhd5WKxetJG2PeT25C+Gd2tj+Ln82aWbmI0Ux3BqXjQOu82d5hendcNJHVpFq6fzjCY5mbGMJpQXJkZc8rwQLRtHwovN4+xMJv++djQe+u9mLP/uMLstAjnYxRtFhl0bkmy3dtlcq2qN5sXIYddBeQAiPvz6rF7ITk/BWSd3wC1vlmLj3irF5wqH3ZQULt+6SLSR+T4spGi0P04eDFEUo/vxLIaob7Gh9QLhOU4SMdmubeSBbVgASS/xxo1xm3WK3h1bK/7W+LzoVBDXOx+5vPiLjNQUXHdGL/QpyMGr156m+Vw+PqSl8D08bu0MjM1G8t951nUhRZFb6mgSJLwQnhN2IL0oV972jrOC0WAS0bwod3jgEntVZgnr2B22efpCTPMSlV7MzUaKE9Ok4lfyVA716nc8LYXP54XnEUejjXS2q+ERXnjrZrU0SHghPMeJ2Uj+0lt5hb1KJa/+KlOKe3hyHUKL24/0goGF0d9FjebFOGeH3baRy1TiUWfYDYXMo4iAZp83s32Ygq1+/+AZG0mrx4aEF8JzCnIzLR+jzLcRwYrt14sXXn3Ka8b2cP8ihC5uq8yf++WIaCSHyl+3OVTaWPMih/wSgoX8aaWGBK7xQrDg9KY+nd7peZTSTrOVJyvksEt4xstXn4rS3RXoW5iDd9fttXUO+aQQ+Z1v9LD7uhudXRCUK+c7JvazeRXCDl7IB9LEwEpSp3c51naaXoKFOsEgj2AcYpiN1egpUvSFW3LYtQtpXgjPOLtfR9xy3sm2BvboC2vT58WLFz4kCIrBiwYV7/HaQVFa1MYcdjnKAzgwG1GX8QuxB5Gawms24olai/UfxbE6h3L5vNAszYRuC+FLpHc9pFohcR/vgcNuempI8TkJL8FHeoZq9b3RWpz12CkKJDio33F13SE91JpXFvIijlxt4diHxhk2JLwQvoTl+MYy/eqlZfdiMklPDSE1xbg9hHd4YjZqPqfGYVfQnzSilYNtaAWHFeXbaSbhMvLnlRLiHy3MhA29z/X6Ep/Drryjme7eYiDhhYg7/VV1R2aO76PZhxVyyBoAUvWEFw9e8ozUFGSkxkoBeBXRRLBxUyCN5eNQal4Uan9ds5H5tmln9FT8vfyOc/D2jWMwsLN5nSXCe+w67JrKGro+L+ztao1f+9bagpKKBRpFq0Uh4YWIK/Onj8V/bihWbJs8rItmP2kysLvqsG02MhgdMlJDyEilVyae2M3zY4b0lGNmI7XDrr7hKFpVWva5XJD9xWndcNekAYpjOudnYUT3Ns4bTriCUqPLaTbiOK+eJkUvYkidYffsvh2Qna6slUZrJDY0EhNxZVhRPlplKIPcjEJP1VEBakb1bMu8jhd24nQSXuKO1+N2zGE38i+Xwy6zMGPsd0oq5n8UmpcUXs2LebSRXugzb48ICYJGM63QvFDXikIjMZFwWJOBwBRelPu0a5Wu6/PiifCSEqIK0gnEzScacwiP/CZaCJVmfaCnhSH8h1rbETKSVFX78TrsauD0eQmFtO0jYZgNCS9EwmFHb0j/KtW7yuOsRYTwYDQ4ZaSR5iWR8AoF+ap08MbnjPwbnUQUSerYxzC1gtQtAoX82fH7vJjvo5/nhW9/1vHyfp9OcdNRKEkd4Tm5WcaTCdNs1PyOKjSmqh1Dgvur3MI8/WzA6SkhZJDmJWHwPmkrPSLq8xKO/C2vKq0fbaQllaSXQKGNNuLzebHrL6t3frXmpTGsNUzJtcsTBxfiXyX5GNWznc2WJA8kvBCeM+akdphS3B0nF+QwP2cJILHVrX5osiDoT1R2zUZXj+mBHQdrMbJHG9z2n68Vn6WnhmjlE2fsPEYrAq3GYVfyeTG4Nmu7XLWvVvsT/kMuTKRwal54Er3omY14k9SxitjK+1ZGagre+c1Y43a2EGgkJjxHEAQ8cMkg/OK07uzPdY4B1JoX9XH6qn27CpnMtBT8+adDcL6saJ9EemoIGWn0ysSTjrK6WNxZbC2cP5bnpflfxbV0NC/RaKMYKbL8P00kvPgfteaFx2wEc82LbrSRzvnPPLmD4u/6prBGoCGXFzakeSESjqHPi6a2EWR/y/dkH28XVv6Y9BTyeYk3uZlpWHjLmUhPDXFrVKwM9oJG8xL9xFJVafnqmKfYHpE41MIBt9mIo1/pRxuxD552Ri90ysvErDfXAwDqG8OaffSCElo6NBITCccs9DS2jeGwa2GCYZGn44/TKiMVvxrbU3M9eZI6Ij70KchB93atPDm31M+awlK0kSxU2sJ55BMMS/VP+Acx+r8IKQbjiBwB5uUBdE2GOudPTw3hJ6d0jf5d1xjWaHeoPAAbEl6IhMMjqADaiA6jCYZ3lV6Qq81oKXHvRQNQmKt04C1qm811XiKR8K+ipX7y/aFa/OH9TbHyAFBXNJcfq402UggvZDbyPU0yATMU4q1tJJj6M1mvKq2krrFJcxLyBWdDZiMi8bDMRia5NAApM6bOKTkXKx1zMrGtvEb380bVKnpE9za490cD0L0dCTF+xcpCVS6UzF2+E13ys5rPIWiEk3CTqDmGdR5SvPibEd3boLauMfo3b6h0SHBS24ivbSyzERX9ZEMyHZFwjJLUKbcZ/y2Hd7AwSzrXGNYOJr86vSfO7V/AdwHC1+ituAUoZWr5ytmsv5Lmxb9kpaVgbO/2imfE77BrvpPVaCM1bLMR37EtDRJeiITDLg/A2E+zj76jHe9qxSzy+Vh9E9d5iGCi7mfK8gDKcFoJtclJDcku/mVc30h0T5NsTRIJleZzejHNsKujduM1G9U3sqKNSHphQcILkXCYodIc2hjD+jPcYbXGO7LUuIS/sRYqrdxbXphRjjyayGwl3ER2I9+j0LzwRrGBw2zk8NHXMcYb0rywIeGFSDg8zrmAdlISoC+k8K50aFGTfFjzeVEJL9HyAMrPqmU+EmbiEZmN/Iv0aOTPKMSpeeEqD6B7LL/mxe6xLQ1PhZcjR47gqquuQm5uLvLz83HttdeipkbfORIAxo0b1+wsF/u54YYbvGwmkWDYZiNzu5Gh2cjl9z2LygIEBisOjup+Ik/ObqbV07sKCS/+R/2M1M/67L4d0ElVKkQAT7SRtSR1auoamzSVq0nzwsZT4eWqq67Cpk2bsHDhQnz44YdYtmwZrr/+etPjpk2bhv3790d/HnnkES+bSSQY3smG5fOid6jbq5V2rdNdPR/hHW5pXvROk5NhHKTJ8PEmEszVY3oAAG4e3weA0ucF0PaDl68ZhXd+M0axzZHmhXOMq1P5vPRs3wqt0ikomIVnwss333yDBQsW4MUXX8To0aNx+umn429/+xvmzZuHffv2GR6bnZ2NwsLC6E9ubq5XzST8AK/mRX2YwQTj1mpl9sR+SAkJeOyyoe6ckPAc6Vnd86MBpvuqzZMHqusAGJeeGNGjjeE5qTyA//jDxQOx9Y8XoH+nyFyidqxlPWq1L0xIEEx9WvoV6tVv42un2mz06ayzECLVCxPPhJeSkhLk5+dj5MiR0W3jx49HKBTCihUrDI999dVX0b59ewwaNAizZ8/GsWPHdPetq6tDVVWV4ocIFtzRRswMu/qhrjyYDUa/PuskbH5gAkb3oiquQWFM7/b49qGJuPb0nqb76laOFrT97cpR3fDE5UNjWZZ1OpnadEAuC/5Anh1bLWAyx6CQerzRP/fY3pHx4fcX9GN+zit/tG2VrhiTqDSAPp7po8rKytCxY0flxVJT0bZtW5SVleke9/Of/xzdu3dH586d8fXXX+P3v/89tm7dinfeeYe5/5w5c3D//fe72nYivrAmEGaeF81xRpoX9156KgkQDB67bCg65Uf8FNI4q3/rzQ2s7jPnJ4O5zknRRv5H6/PCCBpgbFP7o0g8c9UIHKqpw0kdWutc0Xg8+s8NxXj44y34w8UDNdXsCTaWhZc77rgDf/7znw33+eabb2w3SO4TM3jwYHTq1AnnnnsuvvvuO5x00kma/WfPno1Zs2ZF/66qqkJRUZHt6xP+gK+2kcEJaMHS4rh0RFfznVS00vFf4XHO1INkF/9jZDb6zw3FALRjkGBgNmqVnoI8XcHFXPs2skdb/OfGMcY7EQosCy+33norrr76asN9evXqhcLCQhw4cECxvbGxEUeOHEFhYSH39UaPHg0A2L59O1N4ycjIQEaGfn0awv+wVqo8GXYNywNwSC+kkW25SP0jP4vtiB3iSEim18c0q3rrzSM8Rj3myBdGBc31zDRmI9gPhabiiu5jWXjp0KEDOnToYLpfcXExKioqsGbNGowYMQIAsHjxYoTD4ahAwkNpaSkAoFOnTlabSgSEjFStip/nVRcEfRGFRzCh/AktF0n9n5nGNi8Jgn0NCmVI9T/qZ8sqssnS9FosGh3FykLJrsavpeGZw27//v1xwQUXYNq0aVi5ciW+/PJLzJgxA1dccQU6d+4MANi7dy/69euHlStXAgC+++47PPjgg1izZg127tyJ999/H1OmTMGZZ56JIUOGeNVUIsFkpqVg7jWn4uZz+0S38fjBCIxtRsdr97HWTsIfXDO2BwCgTXaa7XPICzCyEWznayGfF/+jfrbKCuKR39XRRhHNi04eF5PBhDQv7uNpnpdXX30V/fr1w7nnnosLL7wQp59+Ol544YXo5w0NDdi6dWs0mig9PR2ffvopzj//fPTr1w+33norLr30UnzwwQdeNpPwAeP6dsQt550c/ZunDLxRqLRbhdYI/3HPpAH4dNaZuHGc1oxsxmvTRuOCgYV4+FLjxVBE82IshOj5BJPZyP9ofJ1kD0mSQ1hmattKEeoEruNp9pu2bdvitdde0/28R48eChVZUVERli5d6mWTiIDArG2k+ttxkjoaUAJJKCSgd8ccfLKpPLrtkZ/yaWbHnNQeY05qb7qfAHPh5drTe2H+un24eFhnxXbpuL//fDjunr8Rz1x1ClfbiPjxk1O6YMnWAzi9T8QFQlE1XNA3G9nFyqEsMzqhhVL3Eb6EnXdBtQ/0tSc8Aw2ZjYKNPKHXz0bajzAsapuF3UeOK7YZRZZItG2Vji9+f7ZGUJasRj8a0hmTBncinxcfkpGagud/OZL5meTzos2xIuCykV3x5Kfb0NBkTQVjxWz06GVDcd0/V2Pm+D7mO7dgSMQjfAnT50UlqBhHG/FdQ8+GTfifenWOd5u89WttiGpE8xL7Wwqf1ezH6IBynxcSXIKB3AIgySxq2aUpHEbHnExsuv8Cy/5WVrrByQU5WHb72fjJKdZD/1sSJLwQvoRZl1GTd8FZkjqaVoJNA6MCrx1ys7QK6EhkSWxCG9mjLff5KFok2EgCp1rwlBIfpqeGLAuleVn2ncsJNiS8EL6ErXlR/S3or2h4TEKOHPCIhNPgkuZFT8tnO9qIOlXgkD8xvbEjXeaLYkV0SU8NYUjXfDvNIgwg4YXwJXw+K/o7kcNu8lNv0e9AD1b9GCd5XqiqdADhqCckLxNiRfFyqyyKknAPEl4IX8KMNmJ4/ztx2CXZJdiM6tnGlfOo83lI2NW82D2OSBxy3ze9RZE8oSH5MiUeijYifAm7qrR6H4flASjcKNBcMrQL0lNSMKRrnqPzsPqBUTZVM0h4CTZ6Y4pC8xKnthD6kPBC+BI+nxd94aVzc3VhI2gACjahkIBJQ7wpGyII9n1eKMFu8JA/aj3Ni8LnhQaPhENmI8JX9CvMARBJIqXGSnmAXxZ3x1Wjuxlei1S/hB7qUGkrkOYlePAILxkKh10aOxINaV4IX/HejLEoqzyB7u1aaT7TZtgF0gyc6x768WBsP1CDFTuOMPehSYbQg6c8gB6d87Jcbg3hNTzRRsO65Ud/p3VP4iHNC+ErMlJTmIILi5AgIFWvwEwzJxqadD9rClOKOoKNAMFyvpZ515+G8wYU4M+cpQoIf6KnkR13cofo71RoMfGQ5oUIDKwkdWkmwsuxen3hJUzOCYQOdhx2T+vVDqf1audNgwhPMRNUzzy5A5mZfQZpXojAoLYzC4KA9BTjAeW4keaFzEaEDgKUaf6J5MbsSbMSZBKJhYQXIjB0bav0JQgJ4DAb6WcMo2RihC4OktQRwcNsHaMWVkZ0dyfHEGEfEl4I3/OvX43C9Wf2wpWjlNFDAgSkmmheJgws0P2MNC+EHnZ8XoggY/ys1aPMAxcP8q4pBBckvBC+58yTO+DOC/sjNaQ2G+lnR5W4a1J/PKLjQElmAUKP9JQQRaMRUdT+LnnZacjNJJfRRELCCxEY+jbngJFolZGqW4dEIjs9FT8bWeRls4gkJCs9hSLRWhB25FRa/CQWEl6IwJCdnoqN90/A/RcPxOAuebj9gr4a4SWHVkOEC6SnhsjnpQVh1WEXMDc7XzaiK/Kz02jx5BE00hOBonVGKqaO6YGpY3oAUFaAffrnp+BUl4r1EQT5vLQcrDrsAuYO/49eNhRNYdFUO0zYg4QXItDIk0Wd278jMtNSDPbWQvMToQf5vLQchhaZFffUCiA8Dv8kuHgHCS9EoJEPDnayXt55YT+s2nkE087o6WaziCSAQulbDl3bZGPxrWchPzud+TlraCGfl8RCwgsRaOTRRnZWOb06tMa6e85DiFZISctVo7tj1c6jGNWzraXjSPPSsujVoXWim0BYgIQXItCEFJoX5+cgko9LhnVGv0456Nmer2aWBMkuhAQNEf6DhBci0Mhzv1DtEYKFIAjoV5hr+TjSvBAS6tIkROKhUGki0PBqTchxjjAijZGpmYQXgvAvJLwQgcYsw67V/YiWSWZqLErtpyO6AgBuOrcPAODSU7ompE2Ef2ANH8O75QMATi4gX5lEQGYjItDwalRIdiGMyEhLQXVdIwDgz5dGykmM69sRa+4ej7at2BEoRMuBNX48/4sReHXFLk3NNSI+kPBCBBpe4cVOGDXRcshMiymh5X2qXeuMRDSH8Bksn5eOuZm45byTE9AaAiCzERFwUjh7MPm8EEZYTW5ItCwkExHhH0h4IQLNqJ7tkBIS0E9VtFGNXHZ58vJhAID7Lx7oYcuIICHXvBCExKezzsKDlwyMliMh/AOZjYhA0zojFZvun4A0ExWMPG5k8vAuOH9gAbLTqfsTEeQOuwQh0btja/TuSA65foRGbyLw8Kj8q080Kv4mwYWQ05qqkRNEoPBMV/rQQw9hzJgxyM7ORn5+Ptcxoiji3nvvRadOnZCVlYXx48fj22+/9aqJBEEQACImxE55mbh7Uv9EN4UgCA48E17q6+tx2WWX4cYbb+Q+5pFHHsFf//pXPPfcc1ixYgVatWqFCRMm4MSJE141k2gh9C2I+MRcNZrCGgkt3du1Qsnsc3HdGb0S3RSCIDgQRNHbNJJz587FzJkzUVFRYbifKIro3Lkzbr31Vvzud78DAFRWVqKgoABz587FFVdcwXW9qqoq5OXlobKyErm51lOCE8nJ3orjWPRNOX42sogiSwiCIHyIlfnbNy72O3bsQFlZGcaPHx/dlpeXh9GjR6OkpET3uLq6OlRVVSl+CEJNl/wsTCnuQYILQRBEEuAb4aWsrAwAUFBQoNheUFAQ/YzFnDlzkJeXF/0pKirytJ0EQRAEQSQWS8LLHXfcAUEQDH+2bNniVVuZzJ49G5WVldGf3bt3x/X6BEEQBEHEF0vxgbfeeiuuvvpqw3169bLn8FZYWAgAKC8vR6dOnaLby8vLMWzYMN3jMjIykJFBKbwJgiAIoqVgSXjp0KEDOnTo4ElDevbsicLCQixatCgqrFRVVWHFihWWIpYIgiAIgkhuPPN52bVrF0pLS7Fr1y40NTWhtLQUpaWlqKmpie7Tr18/vPvuuwAAQRAwc+ZM/PGPf8T777+PDRs2YMqUKejcuTMmT57sVTMJgiAIgggYnqWVvPfee/HPf/4z+vfw4cMBAJ999hnGjRsHANi6dSsqKyuj+9x+++2ora3F9ddfj4qKCpx++ulYsGABMjMzvWomQRAEQRABw/M8L/GG8rwQBEEQRPAIZJ4XgiAIgiAIHkh4IQiCIAgiUJDwQhAEQRBEoCDhhSAIgiCIQEHCC0EQBEEQgYKEF4IgCIIgAoVneV4ShRT5TdWlCYIgCCI4SPM2TwaXpBNeqqurAYCqSxMEQRBEAKmurkZeXp7hPkmXpC4cDmPfvn3IycmBIAiunruqqgpFRUXYvXs3JcDzELrP8YHuc/ygex0f6D7HB6/usyiKqK6uRufOnREKGXu1JJ3mJRQKoWvXrp5eIzc3l16MOED3OT7QfY4fdK/jA93n+ODFfTbTuEiQwy5BEARBEIGChBeCIAiCIAIFCS8WyMjIwH333YeMjIxENyWpofscH+g+xw+61/GB7nN88MN9TjqHXYIgCIIgkhvSvBAEQRAEEShIeCEIgiAIIlCQ8EIQBEEQRKAg4YUgCIIgiEBBwgsnTz/9NHr06IHMzEyMHj0aK1euTHSTAsWcOXNw6qmnIicnBx07dsTkyZOxdetWxT4nTpzA9OnT0a5dO7Ru3RqXXnopysvLFfvs2rULkyZNQnZ2Njp27IjbbrsNjY2N8fwqgeLhhx+GIAiYOXNmdBvdZ/fYu3cvfvGLX6Bdu3bIysrC4MGDsXr16ujnoiji3nvvRadOnZCVlYXx48fj22+/VZzjyJEjuOqqq5Cbm4v8/Hxce+21qKmpifdX8S1NTU2455570LNnT2RlZeGkk07Cgw8+qKh/Q/fZOsuWLcNFF12Ezp07QxAEzJ8/X/G5W/f066+/xhlnnIHMzEwUFRXhkUcececLiIQp8+bNE9PT08WXXnpJ3LRpkzht2jQxPz9fLC8vT3TTAsOECRPEl19+Wdy4caNYWloqXnjhhWK3bt3Empqa6D433HCDWFRUJC5atEhcvXq1eNppp4ljxoyJft7Y2CgOGjRIHD9+vLhu3Trxo48+Etu3by/Onj07EV/J96xcuVLs0aOHOGTIEPHmm2+Obqf77A5HjhwRu3fvLl599dXiihUrxO+//1785JNPxO3bt0f3efjhh8W8vDxx/vz54vr168WLL75Y7Nmzp3j8+PHoPhdccIE4dOhQ8auvvhI///xzsXfv3uKVV16ZiK/kSx566CGxXbt24ocffiju2LFDfOutt8TWrVuLTz31VHQfus/W+eijj8S77rpLfOedd0QA4rvvvqv43I17WllZKRYUFIhXXXWVuHHjRvH1118Xs7KyxOeff95x+0l44WDUqFHi9OnTo383NTWJnTt3FufMmZPAVgWbAwcOiADEpUuXiqIoihUVFWJaWpr41ltvRff55ptvRABiSUmJKIqRly0UCollZWXRfZ599lkxNzdXrKuri+8X8DnV1dVinz59xIULF4pnnXVWVHih++wev//978XTTz9d9/NwOCwWFhaKjz76aHRbRUWFmJGRIb7++uuiKIri5s2bRQDiqlWrovt8/PHHoiAI4t69e71rfICYNGmS+Ktf/Uqx7Sc/+Yl41VVXiaJI99kN1MKLW/f0mWeeEdu0aaMYN37/+9+Lffv2ddxmMhuZUF9fjzVr1mD8+PHRbaFQCOPHj0dJSUkCWxZsKisrAQBt27YFAKxZswYNDQ2K+9yvXz9069Ytep9LSkowePBgFBQURPeZMGECqqqqsGnTpji23v9Mnz4dkyZNUtxPgO6zm7z//vsYOXIkLrvsMnTs2BHDhw/HP/7xj+jnO3bsQFlZmeJe5+XlYfTo0Yp7nZ+fj5EjR0b3GT9+PEKhEFasWBG/L+NjxowZg0WLFmHbtm0AgPXr1+OLL77AxIkTAdB99gK37mlJSQnOPPNMpKenR/eZMGECtm7diqNHjzpqY9IVZnSbQ4cOoampSTGQA0BBQQG2bNmSoFYFm3A4jJkzZ2Ls2LEYNGgQAKCsrAzp6enIz89X7FtQUICysrLoPqznIH1GRJg3bx7Wrl2LVatWaT6j++we33//PZ599lnMmjULd955J1atWoWbbroJ6enpmDp1avRese6l/F537NhR8Xlqairatm1L97qZO+64A1VVVejXrx9SUlLQ1NSEhx56CFdddRUA0H32ALfuaVlZGXr27Kk5h/RZmzZtbLeRhBci7kyfPh0bN27EF198keimJB27d+/GzTffjIULFyIzMzPRzUlqwuEwRo4ciT/96U8AgOHDh2Pjxo147rnnMHXq1AS3Lnl488038eqrr+K1117DwIEDUVpaipkzZ6Jz5850n1swZDYyoX379khJSdFEY5SXl6OwsDBBrQouM2bMwIcffojPPvsMXbt2jW4vLCxEfX09KioqFPvL73NhYSHzOUifERGz0IEDB3DKKacgNTUVqampWLp0Kf76178iNTUVBQUFdJ9dolOnThgwYIBiW//+/bFr1y4AsXtlNHYUFhbiwIEDis8bGxtx5MgRutfN3HbbbbjjjjtwxRVXYPDgwfjlL3+JW265BXPmzAFA99kL3LqnXo4lJLyYkJ6ejhEjRmDRokXRbeFwGIsWLUJxcXECWxYsRFHEjBkz8O6772Lx4sUaVeKIESOQlpamuM9bt27Frl27ove5uLgYGzZsULwwCxcuRG5urmYSaamce+652LBhA0pLS6M/I0eOxFVXXRX9ne6zO4wdO1YT7r9t2zZ0794dANCzZ08UFhYq7nVVVRVWrFihuNcVFRVYs2ZNdJ/FixcjHA5j9OjRcfgW/ufYsWMIhZRTVUpKCsLhMAC6z17g1j0tLi7GsmXL0NDQEN1n4cKF6Nu3ryOTEQAKleZh3rx5YkZGhjh37lxx8+bN4vXXXy/m5+crojEIY2688UYxLy9PXLJkibh///7oz7Fjx6L73HDDDWK3bt3ExYsXi6tXrxaLi4vF4uLi6OdSCO/5558vlpaWigsWLBA7dOhAIbwmyKONRJHus1usXLlSTE1NFR966CHx22+/FV999VUxOztbfOWVV6L7PPzww2J+fr743nvviV9//bV4ySWXMMNNhw8fLq5YsUL84osvxD59+rToEF41U6dOFbt06RINlX7nnXfE9u3bi7fffnt0H7rP1qmurhbXrVsnrlu3TgQgPv744+K6devEH374QRRFd+5pRUWFWFBQIP7yl78UN27cKM6bN0/Mzs6mUOl48re//U3s1q2bmJ6eLo4aNUr86quvEt2kQAGA+fPyyy9H9zl+/Lj4m9/8RmzTpo2YnZ0t/vjHPxb379+vOM/OnTvFiRMnillZWWL79u3FW2+9VWxoaIjztwkWauGF7rN7fPDBB+KgQYPEjIwMsV+/fuILL7yg+DwcDov33HOPWFBQIGZkZIjnnnuuuHXrVsU+hw8fFq+88kqxdevWYm5urnjNNdeI1dXV8fwavqaqqkq8+eabxW7duomZmZlir169xLvuuksRfkv32TqfffYZc0yeOnWqKIru3dP169eLp59+upiRkSF26dJFfPjhh11pvyCKsjSFBEEQBEEQPod8XgiCIAiCCBQkvBAEQRAEEShIeCEIgiAIIlCQ8EIQBEEQRKAg4YUgCIIgiEBBwgtBEARBEIGChBeCIAiCIAIFCS8EQRAEQQQKEl4IgiAIgggUJLwQBEEQBBEoSHghCIIgCCJQkPBCEARBEESg+H/X4Jrl8f3E0AAAAABJRU5ErkJggg==",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/src/layers/rnn/rnn.py:204: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n",
|
|
" super().__init__(**kwargs)\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential\"</span>\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[1mModel: \"sequential\"\u001b[0m\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
|
|
"┃<span style=\"font-weight: bold\"> Layer (type) </span>┃<span style=\"font-weight: bold\"> Output Shape </span>┃<span style=\"font-weight: bold\"> Param # </span>┃\n",
|
|
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
|
|
"│ simple_rnn (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">SimpleRNN</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>) │ <span style=\"color: #00af00; text-decoration-color: #00af00\">1,184</span> │\n",
|
|
"├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
|
|
"│ dense (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>) │ <span style=\"color: #00af00; text-decoration-color: #00af00\">264</span> │\n",
|
|
"├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
|
|
"│ dense_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>) │ <span style=\"color: #00af00; text-decoration-color: #00af00\">9</span> │\n",
|
|
"└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
|
|
"┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
|
|
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
|
|
"│ simple_rnn (\u001b[38;5;33mSimpleRNN\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m) │ \u001b[38;5;34m1,184\u001b[0m │\n",
|
|
"├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
|
|
"│ dense (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m8\u001b[0m) │ \u001b[38;5;34m264\u001b[0m │\n",
|
|
"├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
|
|
"│ dense_1 (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m1\u001b[0m) │ \u001b[38;5;34m9\u001b[0m │\n",
|
|
"└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">1,457</span> (5.69 KB)\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[1m Total params: \u001b[0m\u001b[38;5;34m1,457\u001b[0m (5.69 KB)\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">1,457</span> (5.69 KB)\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m1,457\u001b[0m (5.69 KB)\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 1/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 2s - 48ms/step - loss: 0.5174\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 2/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.4188\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 3/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.4065\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 4/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.4041\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 5/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.4032\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 6/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.4008\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 7/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.4000\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 8/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3989\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 9/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3963\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 10/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3964\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 11/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3957\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 12/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3965\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 13/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3965\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 14/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3953\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 15/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3943\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 16/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3938\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 17/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3920\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 18/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3935\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 19/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3949\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 20/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3922\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 21/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 9ms/step - loss: 0.3927\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 22/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 9ms/step - loss: 0.3914\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 23/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3924\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 24/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3885\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 25/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3896\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 26/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3901\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 27/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3885\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 28/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3905\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 29/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3884\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 30/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3904\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 31/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3889\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 32/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3874\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 33/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3899\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 34/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3886\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 35/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3886\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 36/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3862\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 37/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3873\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 38/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3875\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 39/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3857\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 40/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 9ms/step - loss: 0.3860\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 41/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3860\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 42/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3850\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 43/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3863\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 44/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3836\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 45/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3844\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 46/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3841\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 47/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3828\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 48/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3843\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 49/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3837\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 50/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3850\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 51/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3847\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 52/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3809\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 53/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3835\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 54/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3830\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 55/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3813\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 56/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3826\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 57/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3835\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 58/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3823\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 59/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3820\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 60/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3814\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 61/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3825\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 62/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3817\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 63/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3815\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 64/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3796\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 65/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3802\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 66/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3805\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 67/100\n"
|
|
]
|
|
},
|
|
{
|
|
"ename": "KeyboardInterrupt",
|
|
"evalue": "",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
|
|
"Cell \u001b[0;32mIn[1], line 58\u001b[0m\n\u001b[1;32m 55\u001b[0m model\u001b[38;5;241m.\u001b[39mcompile(loss\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmean_squared_error\u001b[39m\u001b[38;5;124m'\u001b[39m, optimizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrmsprop\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 56\u001b[0m model\u001b[38;5;241m.\u001b[39msummary()\n\u001b[0;32m---> 58\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrainX\u001b[49m\u001b[43m,\u001b[49m\u001b[43mtrainY\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m16\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 59\u001b[0m trainPredict \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(trainX)\n\u001b[1;32m 60\u001b[0m testPredict\u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(testX)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py:117\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 115\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 116\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 117\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 118\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 119\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/src/backend/tensorflow/trainer.py:320\u001b[0m, in \u001b[0;36mTensorFlowTrainer.fit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq)\u001b[0m\n\u001b[1;32m 318\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m step, iterator \u001b[38;5;129;01min\u001b[39;00m epoch_iterator\u001b[38;5;241m.\u001b[39menumerate_epoch():\n\u001b[1;32m 319\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_begin(step)\n\u001b[0;32m--> 320\u001b[0m logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 321\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_end(step, logs)\n\u001b[1;32m 322\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstop_training:\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 150\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 152\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:833\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 830\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 832\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[0;32m--> 833\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 835\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[1;32m 836\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:878\u001b[0m, in \u001b[0;36mFunction._call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 875\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m 876\u001b[0m \u001b[38;5;66;03m# In this case we have not created variables on the first call. So we can\u001b[39;00m\n\u001b[1;32m 877\u001b[0m \u001b[38;5;66;03m# run the first trace but we should fail if variables are created.\u001b[39;00m\n\u001b[0;32m--> 878\u001b[0m results \u001b[38;5;241m=\u001b[39m \u001b[43mtracing_compilation\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_function\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 879\u001b[0m \u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mkwds\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_variable_creation_config\u001b[49m\n\u001b[1;32m 880\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 881\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_created_variables:\n\u001b[1;32m 882\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCreating variables on a non-first call to a function\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 883\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m decorated with tf.function.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py:139\u001b[0m, in \u001b[0;36mcall_function\u001b[0;34m(args, kwargs, tracing_options)\u001b[0m\n\u001b[1;32m 137\u001b[0m bound_args \u001b[38;5;241m=\u001b[39m function\u001b[38;5;241m.\u001b[39mfunction_type\u001b[38;5;241m.\u001b[39mbind(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m 138\u001b[0m flat_inputs \u001b[38;5;241m=\u001b[39m function\u001b[38;5;241m.\u001b[39mfunction_type\u001b[38;5;241m.\u001b[39munpack_inputs(bound_args)\n\u001b[0;32m--> 139\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunction\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;66;43;03m# pylint: disable=protected-access\u001b[39;49;00m\n\u001b[1;32m 140\u001b[0m \u001b[43m \u001b[49m\u001b[43mflat_inputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfunction\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\n\u001b[1;32m 141\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/concrete_function.py:1322\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[0;34m(self, tensor_inputs, captured_inputs)\u001b[0m\n\u001b[1;32m 1318\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[1;32m 1319\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[1;32m 1320\u001b[0m \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[1;32m 1321\u001b[0m \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[0;32m-> 1322\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_preflattened\u001b[49m\u001b[43m(\u001b[49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1323\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[1;32m 1324\u001b[0m args,\n\u001b[1;32m 1325\u001b[0m possible_gradient_type,\n\u001b[1;32m 1326\u001b[0m executing_eagerly)\n\u001b[1;32m 1327\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py:216\u001b[0m, in \u001b[0;36mAtomicFunction.call_preflattened\u001b[0;34m(self, args)\u001b[0m\n\u001b[1;32m 214\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcall_preflattened\u001b[39m(\u001b[38;5;28mself\u001b[39m, args: Sequence[core\u001b[38;5;241m.\u001b[39mTensor]) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[1;32m 215\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"Calls with flattened tensor inputs and returns the structured output.\"\"\"\u001b[39;00m\n\u001b[0;32m--> 216\u001b[0m flat_outputs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_flat\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 217\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfunction_type\u001b[38;5;241m.\u001b[39mpack_output(flat_outputs)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py:251\u001b[0m, in \u001b[0;36mAtomicFunction.call_flat\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 249\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m record\u001b[38;5;241m.\u001b[39mstop_recording():\n\u001b[1;32m 250\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_bound_context\u001b[38;5;241m.\u001b[39mexecuting_eagerly():\n\u001b[0;32m--> 251\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_bound_context\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_function\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 252\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 253\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mlist\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 254\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfunction_type\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mflat_outputs\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 255\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 256\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 257\u001b[0m outputs \u001b[38;5;241m=\u001b[39m make_call_op_in_graph(\n\u001b[1;32m 258\u001b[0m \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m 259\u001b[0m \u001b[38;5;28mlist\u001b[39m(args),\n\u001b[1;32m 260\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_bound_context\u001b[38;5;241m.\u001b[39mfunction_call_options\u001b[38;5;241m.\u001b[39mas_attrs(),\n\u001b[1;32m 261\u001b[0m )\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/context.py:1500\u001b[0m, in \u001b[0;36mContext.call_function\u001b[0;34m(self, name, tensor_inputs, num_outputs)\u001b[0m\n\u001b[1;32m 1498\u001b[0m cancellation_context \u001b[38;5;241m=\u001b[39m cancellation\u001b[38;5;241m.\u001b[39mcontext()\n\u001b[1;32m 1499\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cancellation_context \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m-> 1500\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1501\u001b[0m \u001b[43m \u001b[49m\u001b[43mname\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdecode\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mutf-8\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1502\u001b[0m \u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnum_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1503\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtensor_inputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1504\u001b[0m \u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1505\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1506\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1507\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 1508\u001b[0m outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[1;32m 1509\u001b[0m name\u001b[38;5;241m.\u001b[39mdecode(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mutf-8\u001b[39m\u001b[38;5;124m\"\u001b[39m),\n\u001b[1;32m 1510\u001b[0m num_outputs\u001b[38;5;241m=\u001b[39mnum_outputs,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1514\u001b[0m cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_context,\n\u001b[1;32m 1515\u001b[0m )\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:53\u001b[0m, in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 51\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m 52\u001b[0m ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[0;32m---> 53\u001b[0m tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 54\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 55\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 56\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Start importing packages\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras.models import Model, Sequential \n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"from tensorflow.keras import optimizers \n",
|
|
"from tensorflow.keras import regularizers \n",
|
|
"from tensorflow.keras.utils import to_categorical \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"# convert into dataset matrix\n",
|
|
"def convertToMatrix(data, step):\n",
|
|
" X, Y =[], []\n",
|
|
" for i in range(len(data)-step):\n",
|
|
" d=i+step \n",
|
|
" X.append(data[i:d,])\n",
|
|
" Y.append(data[d,])\n",
|
|
" return np.array(X), np.array(Y)\n",
|
|
"\n",
|
|
"step = 4\n",
|
|
"N = 1000 \n",
|
|
"Tp = 800 \n",
|
|
"\n",
|
|
"t=np.arange(0,N)\n",
|
|
"x=np.sin(0.02*t)+2*np.random.rand(N)\n",
|
|
"df = pd.DataFrame(x)\n",
|
|
"df.head()\n",
|
|
"\n",
|
|
"plt.plot(df)\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"values=df.values\n",
|
|
"train,test = values[0:Tp,:], values[Tp:N,:]\n",
|
|
"\n",
|
|
"# add step elements into train and test\n",
|
|
"test = np.append(test,np.repeat(test[-1,],step))\n",
|
|
"train = np.append(train,np.repeat(train[-1,],step))\n",
|
|
" \n",
|
|
"trainX,trainY =convertToMatrix(train,step)\n",
|
|
"testX,testY =convertToMatrix(test,step)\n",
|
|
"trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
|
|
"testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(SimpleRNN(units=32, input_shape=(1,step), activation=\"relu\"))\n",
|
|
"model.add(Dense(8, activation=\"relu\")) \n",
|
|
"model.add(Dense(1))\n",
|
|
"model.compile(loss='mean_squared_error', optimizer='rmsprop')\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)\n",
|
|
"trainPredict = model.predict(trainX)\n",
|
|
"testPredict= model.predict(testX)\n",
|
|
"predicted=np.concatenate((trainPredict,testPredict),axis=0)\n",
|
|
"\n",
|
|
"trainScore = model.evaluate(trainX, trainY, verbose=0)\n",
|
|
"print(trainScore)\n",
|
|
"\n",
|
|
"index = df.index.values\n",
|
|
"plt.plot(index,df)\n",
|
|
"plt.plot(index,predicted)\n",
|
|
"plt.axvline(df.index[Tp], c=\"r\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "897a47c7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## An extrapolation example\n",
|
|
"\n",
|
|
"The following code provides an example of how recurrent neural\n",
|
|
"networks can be used to extrapolate to unknown values of physics data\n",
|
|
"sets. Specifically, the data sets used in this program come from\n",
|
|
"a quantum mechanical many-body calculation of energies as functions of the number of particles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6776ae2a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"# For matrices and calculations\n",
|
|
"import numpy as np\n",
|
|
"# For machine learning (backend for keras)\n",
|
|
"import tensorflow as tf\n",
|
|
"# User-friendly machine learning library\n",
|
|
"# Front end for TensorFlow\n",
|
|
"import tensorflow.keras\n",
|
|
"# Different methods from Keras needed to create an RNN\n",
|
|
"# This is not necessary but it shortened function calls \n",
|
|
"# that need to be used in the code.\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras import regularizers\n",
|
|
"from tensorflow.keras.models import Model, Sequential\n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"# For timing the code\n",
|
|
"from timeit import default_timer as timer\n",
|
|
"# For plotting\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"\n",
|
|
"# The data set\n",
|
|
"datatype='VaryDimension'\n",
|
|
"X_tot = np.arange(2, 42, 2)\n",
|
|
"y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,\n",
|
|
"\t-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, \n",
|
|
"\t-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35227d36",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The way the recurrent neural networks are trained in this program\n",
|
|
"differs from how machine learning algorithms are usually trained.\n",
|
|
"Typically a machine learning algorithm is trained by learning the\n",
|
|
"relationship between the x data and the y data. In this program, the\n",
|
|
"recurrent neural network will be trained to recognize the relationship\n",
|
|
"in a sequence of y values. This is type of data formatting is\n",
|
|
"typically used time series forcasting, but it can also be used in any\n",
|
|
"extrapolation (time series forecasting is just a specific type of\n",
|
|
"extrapolation along the time axis). This method of data formatting\n",
|
|
"does not use the x data and assumes that the y data are evenly spaced.\n",
|
|
"\n",
|
|
"For a standard machine learning algorithm, the training data has the\n",
|
|
"form of (x,y) so the machine learning algorithm learns to assiciate a\n",
|
|
"y value with a given x value. This is useful when the test data has x\n",
|
|
"values within the same range as the training data. However, for this\n",
|
|
"application, the x values of the test data are outside of the x values\n",
|
|
"of the training data and the traditional method of training a machine\n",
|
|
"learning algorithm does not work as well. For this reason, the\n",
|
|
"recurrent neural network is trained on sequences of y values of the\n",
|
|
"form ((y1, y2), y3), so that the network is concerned with learning\n",
|
|
"the pattern of the y data and not the relation between the x and y\n",
|
|
"data. As long as the pattern of y data outside of the training region\n",
|
|
"stays relatively stable compared to what was inside the training\n",
|
|
"region, this method of training can produce accurate extrapolations to\n",
|
|
"y values far removed from the training data set."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "7dc577d1",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# FORMAT_DATA\n",
|
|
"def format_data(data, length_of_sequence = 2): \n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" data(a numpy array): the data that will be the inputs to the recurrent neural\n",
|
|
" network\n",
|
|
" length_of_sequence (an int): the number of elements in one iteration of the\n",
|
|
" sequence patter. For a function approximator use length_of_sequence = 2.\n",
|
|
" Returns:\n",
|
|
" rnn_input (a 3D numpy array): the input data for the recurrent neural network. Its\n",
|
|
" dimensions are length of data - length of sequence, length of sequence, \n",
|
|
" dimnsion of data\n",
|
|
" rnn_output (a numpy array): the training data for the neural network\n",
|
|
" Formats data to be used in a recurrent neural network.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" X, Y = [], []\n",
|
|
" for i in range(len(data)-length_of_sequence):\n",
|
|
" # Get the next length_of_sequence elements\n",
|
|
" a = data[i:i+length_of_sequence]\n",
|
|
" # Get the element that immediately follows that\n",
|
|
" b = data[i+length_of_sequence]\n",
|
|
" # Reshape so that each data point is contained in its own array\n",
|
|
" a = np.reshape (a, (len(a), 1))\n",
|
|
" X.append(a)\n",
|
|
" Y.append(b)\n",
|
|
" rnn_input = np.array(X)\n",
|
|
" rnn_output = np.array(Y)\n",
|
|
"\n",
|
|
" return rnn_input, rnn_output\n",
|
|
"\n",
|
|
"\n",
|
|
"# ## Defining the Recurrent Neural Network Using Keras\n",
|
|
"# \n",
|
|
"# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.\n",
|
|
"\n",
|
|
"def rnn(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with one hidden layer and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer\n",
|
|
" hidden_neurons = 200\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to \n",
|
|
" # the network immediately after the input layer\n",
|
|
" rnn = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\")(inp)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1978b6a1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Predicting New Points With A Trained Recurrent Neural Network"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "d4ad417a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_rnn (x1, y_test, plot_min, plot_max):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" x1 (a list or numpy array): The complete x component of the data set\n",
|
|
" y_test (a list or numpy array): The complete y component of the data set\n",
|
|
" plot_min (an int or float): the smallest x value used in the training data\n",
|
|
" plot_max (an int or float): the largest x valye used in the training data\n",
|
|
" Returns:\n",
|
|
" None.\n",
|
|
" Uses a trained recurrent neural network model to predict future points in the \n",
|
|
" series. Computes the MSE of the predicted data set from the true data set, saves\n",
|
|
" the predicted data set to a csv file, and plots the predicted and true data sets w\n",
|
|
" while also displaying the data range used for training.\n",
|
|
" \"\"\"\n",
|
|
" # Add the training data as the first dim points in the predicted data array as these\n",
|
|
" # are known values.\n",
|
|
" y_pred = y_test[:dim].tolist()\n",
|
|
" # Generate the first input to the trained recurrent neural network using the last two \n",
|
|
" # points of the training data. Based on how the network was trained this means that it\n",
|
|
" # will predict the first point in the data set after the training data. All of the \n",
|
|
" # brackets are necessary for Tensorflow.\n",
|
|
" next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])\n",
|
|
" # Save the very last point in the training data set. This will be used later.\n",
|
|
" last = [y_test[dim-1]]\n",
|
|
"\n",
|
|
" # Iterate until the complete data set is created.\n",
|
|
" for i in range (dim, len(y_test)):\n",
|
|
" # Predict the next point in the data set using the previous two points.\n",
|
|
" next = model.predict(next_input)\n",
|
|
" # Append just the number of the predicted data set\n",
|
|
" y_pred.append(next[0][0])\n",
|
|
" # Create the input that will be used to predict the next data point in the data set.\n",
|
|
" next_input = np.array([[last, next[0]]], dtype=np.float64)\n",
|
|
" last = next\n",
|
|
"\n",
|
|
" # Print the mean squared error between the known data set and the predicted data set.\n",
|
|
" print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())\n",
|
|
" # Save the predicted data set as a csv file for later use\n",
|
|
" name = datatype + 'Predicted'+str(dim)+'.csv'\n",
|
|
" np.savetxt(name, y_pred, delimiter=',')\n",
|
|
" # Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
" # for the training data.\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.plot(x1, y_test, label=\"true\", linewidth=3)\n",
|
|
" ax.plot(x1, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
" ax.legend()\n",
|
|
" # Created a red region to represent the points used in the training data.\n",
|
|
" ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn(length_of_sequences = rnn_input.shape[1])\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2cad4fc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Changing the size of the recurrent neural network and its parameters\n",
|
|
"can drastically change the results you get from the model. The below\n",
|
|
"code takes the simple recurrent neural network from above and adds a\n",
|
|
"second hidden layer, changes the number of neurons in the hidden\n",
|
|
"layer, and explicitly declares the activation function of the hidden\n",
|
|
"layers to be a sigmoid function. The loss function and optimizer can\n",
|
|
"also be changed but are kept the same as the above network. These\n",
|
|
"parameters can be tuned to provide the optimal result from the\n",
|
|
"network. For some ideas on how to improve the performance of a\n",
|
|
"[recurrent neural network](https://danijar.com/tips-for-training-recurrent-neural-networks)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c39f1516",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer, increased from the first network\n",
|
|
" hidden_neurons = 500\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Create two hidden layers instead of one hidden layer. Explicitly set the activation\n",
|
|
" # function to be the sigmoid function (the default value is hyperbolic tangent)\n",
|
|
" rnn1 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=True, # This needs to be True if another hidden layer is to follow\n",
|
|
" stateful = stateful, activation = 'sigmoid',\n",
|
|
" name=\"RNN1\")(inp)\n",
|
|
" rnn2 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False, activation = 'sigmoid',\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN2\")(rnn1)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn2)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn_2layers(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "842c7602",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Other Types of Recurrent Neural Networks\n",
|
|
"\n",
|
|
"Besides a simple recurrent neural network layer, there are two other\n",
|
|
"commonly used types of recurrent neural network layers: Long Short\n",
|
|
"Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short\n",
|
|
"introduction to these layers see <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>\n",
|
|
"and <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>.\n",
|
|
"\n",
|
|
"The first network created below is similar to the previous network,\n",
|
|
"but it replaces the SimpleRNN layers with LSTM layers. The second\n",
|
|
"network below has two hidden layers made up of GRUs, which are\n",
|
|
"preceeded by two dense (feeddorward) neural network layers. These\n",
|
|
"dense layers \"preprocess\" the data before it reaches the recurrent\n",
|
|
"layers. This architecture has been shown to improve the performance\n",
|
|
"of recurrent neural networks (see the link above and also\n",
|
|
"<https://arxiv.org/pdf/1807.02857.pdf>."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "6f0e9b62",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons on the input/output layer and the number of neurons in the hidden layer\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input Layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)\n",
|
|
" rnn= LSTM(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True, activation='tanh')(inp)\n",
|
|
" rnn1 = LSTM(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True, activation='tanh')(rnn)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn1)\n",
|
|
" # Define the midel\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the model\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with four hidden layers (two dense followed by\n",
|
|
" two GRU layers) and returns the model.\n",
|
|
" \"\"\" \n",
|
|
" # Number of neurons on the input/output layers and hidden layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden Dense (feedforward) layers\n",
|
|
" dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)\n",
|
|
" dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)\n",
|
|
" # Hidden GRU layers\n",
|
|
" rnn1 = GRU(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True)(dnn1)\n",
|
|
" rnn = GRU(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True)(rnn1)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Define the model\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the mdoel\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Change the method name to reflect which network you want to use\n",
|
|
"model = dnn2_gru2(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)\n",
|
|
"\n",
|
|
"\n",
|
|
"# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)\n",
|
|
"# \n",
|
|
"# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"# Reshape the data for Keras specifications\n",
|
|
"X_train = X_train.reshape((dim, 1))\n",
|
|
"y_train = y_train.reshape((dim, 1))\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Set the sequence length to 1 for regular data formatting \n",
|
|
"model = rnn(length_of_sequences = 1)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(X_train, y_train, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict the remaining data points\n",
|
|
"X_pred = X_tot[dim:]\n",
|
|
"X_pred = X_pred.reshape((len(X_pred), 1))\n",
|
|
"y_model = model.predict(X_pred)\n",
|
|
"y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))\n",
|
|
"\n",
|
|
"# Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
"# for the training data.\n",
|
|
"fig, ax = plt.subplots()\n",
|
|
"ax.plot(X_tot, y_tot, label=\"true\", linewidth=3)\n",
|
|
"ax.plot(X_tot, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
"ax.legend()\n",
|
|
"# Created a red region to represent the points used in the training data.\n",
|
|
"ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0752ba7f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Generative Models\n",
|
|
"\n",
|
|
"**Generative models** describe a class of statistical models that are a contrast\n",
|
|
"to **discriminative models**. Informally we say that generative models can\n",
|
|
"generate new data instances while discriminative models discriminate between\n",
|
|
"different kinds of data instances. A generative model could generate new photos\n",
|
|
"of animals that look like 'real' animals while a discriminative model could tell\n",
|
|
"a dog from a cat. More formally, given a data set $x$ and a set of labels /\n",
|
|
"targets $y$. Generative models capture the joint probability $p(x, y)$, or\n",
|
|
"just $p(x)$ if there are no labels, while discriminative models capture the\n",
|
|
"conditional probability $p(y | x)$. Discriminative models generally try to draw\n",
|
|
"boundaries in the data space (often high dimensional), while generative models\n",
|
|
"try to model how data is placed throughout the space.\n",
|
|
"\n",
|
|
"**Note**: this material is thanks to Linus Ekstrøm."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "784138f8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Generative Adversarial Networks\n",
|
|
"\n",
|
|
"**Generative Adversarial Networks** are a type of unsupervised machine learning\n",
|
|
"algorithm proposed by [Goodfellow et. al](https://arxiv.org/pdf/1406.2661.pdf)\n",
|
|
"in 2014 (short and good article).\n",
|
|
"\n",
|
|
"The simplest formulation of\n",
|
|
"the model is based on a game theoretic approach, *zero sum game*, where we pit\n",
|
|
"two neural networks against one another. We define two rival networks, one\n",
|
|
"generator $g$, and one discriminator $d$. The generator directly produces\n",
|
|
"samples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a42f89ee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" x = g(z; \\theta^{(g)})\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abe7212f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The discriminator attempts to distinguish between samples drawn from the\n",
|
|
"training data and samples drawn from the generator. In other words, it tries to\n",
|
|
"tell the difference between the fake data produced by $g$ and the actual data\n",
|
|
"samples we want to do prediction on. The discriminator outputs a probability\n",
|
|
"value given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0821eaee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" d(x; \\theta^{(d)})\n",
|
|
"\\label{_auto2} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55e0ccf6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"indicating the probability that $x$ is a real training example rather than a\n",
|
|
"fake sample the generator has generated. The simplest way to formulate the\n",
|
|
"learning process in a generative adversarial network is a zero-sum game, in\n",
|
|
"which a function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f37ece14",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto3} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d6d6d5fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"determines the reward for the discriminator, while the generator gets the\n",
|
|
"conjugate reward"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c74b45c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" -v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto4} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "605ac8c9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"During learning both of the networks maximize their own reward function, so that\n",
|
|
"the generator gets better and better at tricking the discriminator, while the\n",
|
|
"discriminator gets better and better at telling the difference between the fake\n",
|
|
"and real data. The generator and discriminator alternate on which one trains at\n",
|
|
"one time (i.e. for one epoch). In other words, we keep the generator constant\n",
|
|
"and train the discriminator, then we keep the discriminator constant to train\n",
|
|
"the generator and repeat. It is this back and forth dynamic which lets GANs\n",
|
|
"tackle otherwise intractable generative problems. As the generator improves with\n",
|
|
" training, the discriminator's performance gets worse because it cannot easily\n",
|
|
" tell the difference between real and fake. If the generator ends up succeeding\n",
|
|
" perfectly, the the discriminator will do no better than random guessing i.e.\n",
|
|
" 50\\%. This progression in the training poses a problem for the convergence\n",
|
|
" criteria for GANs. The discriminator feedback gets less meaningful over time,\n",
|
|
" if we continue training after this point then the generator is effectively\n",
|
|
" training on junk data which can undo the learning up to that point. Therefore,\n",
|
|
" we stop training when the discriminator starts outputting $1/2$ everywhere.\n",
|
|
"\n",
|
|
"At convergence we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfec7462",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto5\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" g^* = \\underset{g}{\\mathrm{argmin}}\\hspace{2pt}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto5} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37c65d4c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The default choice for $v$ is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c868a092",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto6\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)}) = \\mathbb{E}_{x\\sim p_\\mathrm{data}}\\log d(x)\n",
|
|
" + \\mathbb{E}_{x\\sim p_\\mathrm{model}}\n",
|
|
" \\log (1 - d(x))\n",
|
|
"\\label{_auto6} \\tag{6}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ad465af3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The main motivation for the design of GANs is that the learning process requires\n",
|
|
"neither approximate inference (variational autoencoders for example) nor\n",
|
|
"approximation of a partition function. In the case where"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "27858a4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto7\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto7} \\tag{7}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86006023",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"is convex in $\\theta^{(g)} then the procedure is guaranteed to converge and is\n",
|
|
"asymptotically consistent\n",
|
|
"( [Seth Lloyd on QuGANs](https://arxiv.org/pdf/1804.09139.pdf) ).\n",
|
|
"\n",
|
|
"This is in\n",
|
|
"general not the case and it is possible to get situations where the training\n",
|
|
"process never converges because the generator and discriminator chase one\n",
|
|
"another around in the parameter space indefinitely. A much deeper discussion on\n",
|
|
"the currently open research problem of GAN convergence is available\n",
|
|
"[here](https://www.deeplearningbook.org/contents/generative_models.html). To\n",
|
|
"anyone interested in learning more about GANs it is a highly recommended read.\n",
|
|
"Direct quote: \"In this best-performing formulation, the generator aims to\n",
|
|
"increase the log probability that the discriminator makes a mistake, rather than\n",
|
|
"aiming to decrease the log probability that the discriminator makes the correct\n",
|
|
"prediction.\" [Another interesting read](https://arxiv.org/abs/1701.00160)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2fee38bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Writing Our First Generative Adversarial Network\n",
|
|
"Let us now move on to actually implementing a GAN in tensorflow. We will study\n",
|
|
"the performance of our GAN on the MNIST dataset. This code is based on and\n",
|
|
"adapted from the\n",
|
|
"[google tutorial](https://www.tensorflow.org/tutorials/generative/dcgan)\n",
|
|
"\n",
|
|
"First we import our libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "004a0b53",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import time\n",
|
|
"import numpy as np\n",
|
|
"import tensorflow as tf\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from tensorflow.keras import layers\n",
|
|
"from tensorflow.keras.utils import plot_model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af161",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define our hyperparameters and import our data the usual way"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "8cbaf16a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"BUFFER_SIZE = 60000\n",
|
|
"BATCH_SIZE = 256\n",
|
|
"EPOCHS = 30\n",
|
|
"\n",
|
|
"data = tf.keras.datasets.mnist.load_data()\n",
|
|
"(train_images, train_labels), (test_images, test_labels) = data\n",
|
|
"train_images = np.reshape(train_images, (train_images.shape[0],\n",
|
|
" 28,\n",
|
|
" 28,\n",
|
|
" 1)).astype('float32')\n",
|
|
"\n",
|
|
"# we normalize between -1 and 1\n",
|
|
"train_images = (train_images - 127.5) / 127.5\n",
|
|
"training_dataset = tf.data.Dataset.from_tensor_slices(\n",
|
|
" train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "822b8cc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### MNIST and GANs\n",
|
|
"\n",
|
|
"Let's have a quick look"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "52b5965c",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.imshow(train_images[0], cmap='Greys')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "21c5199c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our two models. This is where the 'magic' happens. There are a\n",
|
|
"huge amount of possible formulations for both models. A lot of engineering and\n",
|
|
"trial and error can be done here to try to produce better performing models. For\n",
|
|
"more advanced GANs this is by far the step where you can 'make or break' a\n",
|
|
"model.\n",
|
|
"\n",
|
|
"We start with the generator. As stated in the introductory text the generator\n",
|
|
"$g$ upsamples from a random sample to the shape of what we want to predict. In\n",
|
|
"our case we are trying to predict MNIST images ($28\\times 28$ pixels)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "356759c7",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_model():\n",
|
|
" \"\"\"\n",
|
|
" The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to\n",
|
|
" produce an image from a random seed. We start with a Dense layer taking this\n",
|
|
" random sample as an input and subsequently upsample through multiple\n",
|
|
" convolutional layers.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
"\n",
|
|
"\n",
|
|
" # adding our input layer. Dense means that every neuron is connected and\n",
|
|
" # the input shape is the shape of our random noise. The units need to match\n",
|
|
" # in some sense the upsampling strides to reach our desired output shape.\n",
|
|
" # we are using 100 random numbers as our seed\n",
|
|
" model.add(layers.Dense(units=7*7*BATCH_SIZE,\n",
|
|
" use_bias=False,\n",
|
|
" input_shape=(100, )))\n",
|
|
" # we normalize the output form the Dense layer\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" # and add an activation function to our 'layer'. LeakyReLU avoids vanishing\n",
|
|
" # gradient problem\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" model.add(layers.Reshape((7, 7, BATCH_SIZE)))\n",
|
|
" assert model.output_shape == (None, 7, 7, BATCH_SIZE)\n",
|
|
" # even though we just added four keras layers we think of everything above\n",
|
|
" # as 'one' layer\n",
|
|
"\n",
|
|
" # next we add our upscaling convolutional layers\n",
|
|
" model.add(layers.Conv2DTranspose(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(1, 1),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 7, 7, 128)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 14, 14, 64)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=1,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False,\n",
|
|
" activation='tanh'))\n",
|
|
" assert model.output_shape == (None, 28, 28, 1)\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "854bcd6b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"And there we have our 'simple' generator model. Now we move on to defining our\n",
|
|
"discriminator model $d$, which is a convolutional neural network based image\n",
|
|
"classifier."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "41473304",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_model():\n",
|
|
" \"\"\"\n",
|
|
" The discriminator is a convolutional neural network based image classifier\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
" model.add(layers.Conv2D(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" input_shape=[28, 28, 1]))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
"\n",
|
|
" model.add(layers.Conv2D(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same'))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
" model.add(layers.Flatten())\n",
|
|
" model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af567",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Let us take a look at our models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "f899d4e3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generator = generator_model()\n",
|
|
"plot_model(generator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "87ef384b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"discriminator = discriminator_model()\n",
|
|
"plot_model(discriminator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2bef82d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we need a few helper objects we will use in training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "e397847a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
|
|
"generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n",
|
|
"discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "db3396cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The first object, *cross_entropy* is our loss function and the two others are\n",
|
|
"our optimizers. Notice we use the same learning rate for both $g$ and $d$. This\n",
|
|
"is because they need to improve their accuracy at approximately equal speeds to\n",
|
|
"get convergence (not necessarily exactly equal). Now we define our loss\n",
|
|
"functions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "931eaced",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_loss(fake_output):\n",
|
|
" loss = cross_entropy(tf.ones_like(fake_output), fake_output)\n",
|
|
"\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "0c4a44bb",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_loss(real_output, fake_output):\n",
|
|
" real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n",
|
|
" fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)\n",
|
|
" total_loss = real_loss + fake_loss\n",
|
|
"\n",
|
|
" return total_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fcf8f066",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a kind of seed to help us compare the learning process over\n",
|
|
"multiple training epochs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "eea2bbee",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"noise_dimension = 100\n",
|
|
"n_examples_to_generate = 16\n",
|
|
"seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94a6e341",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have everything we need to define our training step, which we will apply\n",
|
|
"for every step in our training loop. Notice the @tf.function flag signifying\n",
|
|
"that the function is tensorflow 'compiled'. Removing this flag doubles the\n",
|
|
"computation time."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "8d48470b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tf.function\n",
|
|
"def train_step(images):\n",
|
|
" noise = tf.random.normal([BATCH_SIZE, noise_dimension])\n",
|
|
"\n",
|
|
" with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n",
|
|
" generated_images = generator(noise, training=True)\n",
|
|
"\n",
|
|
" real_output = discriminator(images, training=True)\n",
|
|
" fake_output = discriminator(generated_images, training=True)\n",
|
|
"\n",
|
|
" gen_loss = generator_loss(fake_output)\n",
|
|
" disc_loss = discriminator_loss(real_output, fake_output)\n",
|
|
"\n",
|
|
" gradients_of_generator = gen_tape.gradient(gen_loss,\n",
|
|
" generator.trainable_variables)\n",
|
|
" gradients_of_discriminator = disc_tape.gradient(disc_loss,\n",
|
|
" discriminator.trainable_variables)\n",
|
|
" generator_optimizer.apply_gradients(zip(gradients_of_generator,\n",
|
|
" generator.trainable_variables))\n",
|
|
" discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,\n",
|
|
" discriminator.trainable_variables))\n",
|
|
"\n",
|
|
" return gen_loss, disc_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "62015b88",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a helper function to produce an output over our training epochs\n",
|
|
"to see the predictive progression of our generator model. **Note**: I am including\n",
|
|
"this code here, but comment it out in the training loop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "b189ed96",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_and_save_images(model, epoch, test_input):\n",
|
|
" # we're making inferences here\n",
|
|
" predictions = model(test_input, training=False)\n",
|
|
"\n",
|
|
" fig = plt.figure(figsize=(4, 4))\n",
|
|
"\n",
|
|
" for i in range(predictions.shape[0]):\n",
|
|
" plt.subplot(4, 4, i+1)\n",
|
|
" plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n",
|
|
" plt.axis('off')\n",
|
|
"\n",
|
|
" plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')\n",
|
|
" plt.close()\n",
|
|
" #plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ba2c82d5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Setting up checkpoints to periodically save our model during training so that\n",
|
|
"everything is not lost even if the program were to somehow terminate while\n",
|
|
"training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "a0e2fc8a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setting up checkpoints to save model during training\n",
|
|
"checkpoint_dir = './training_checkpoints'\n",
|
|
"checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')\n",
|
|
"checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n",
|
|
" discriminator_optimizer=discriminator_optimizer,\n",
|
|
" generator=generator,\n",
|
|
" discriminator=discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d93a0f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our training loop"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "a1275556",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def train(dataset, epochs):\n",
|
|
" generator_loss_list = []\n",
|
|
" discriminator_loss_list = []\n",
|
|
"\n",
|
|
" for epoch in range(epochs):\n",
|
|
" start = time.time()\n",
|
|
"\n",
|
|
" for image_batch in dataset:\n",
|
|
" gen_loss, disc_loss = train_step(image_batch)\n",
|
|
" generator_loss_list.append(gen_loss.numpy())\n",
|
|
" discriminator_loss_list.append(disc_loss.numpy())\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epoch + 1, seed_images)\n",
|
|
"\n",
|
|
" if (epoch + 1) % 15 == 0:\n",
|
|
" checkpoint.save(file_prefix=checkpoint_prefix)\n",
|
|
"\n",
|
|
" print(f'Time for epoch {epoch} is {time.time() - start}')\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epochs, seed_images)\n",
|
|
"\n",
|
|
" loss_file = './data/lossfile.txt'\n",
|
|
" with open(loss_file, 'w') as outfile:\n",
|
|
" outfile.write(str(generator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write(str(discriminator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ff3a75a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"To train simply call this function. **Warning**: this might take a long time so\n",
|
|
"there is a folder of a pretrained network already included in the repository."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"id": "371ed41a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"train(train_dataset, EPOCHS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "654399f1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now to avoid having to train and everything, which will take a while depending\n",
|
|
"on your computer setup we now load in the model which produced the above gif."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"id": "dec4b560",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))\n",
|
|
"restored_generator = checkpoint.generator\n",
|
|
"restored_discriminator = checkpoint.discriminator\n",
|
|
"\n",
|
|
"print(restored_generator)\n",
|
|
"print(restored_discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "296bfa5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We have successfully loaded in our latest model. Let us now play around a bit\n",
|
|
"and see what kind of things we can learn about this model. Our generator takes\n",
|
|
"an array of 100 numbers. One idea can be to try to systematically change our\n",
|
|
"input. Let us try and see what we get"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"id": "eecfbb1f",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_latent_points(number=100, scale_means=1, scale_stds=1):\n",
|
|
" latent_dim = 100\n",
|
|
" means = scale_means * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" latent_space_value_range = tf.random.normal([number, latent_dim],\n",
|
|
" means,\n",
|
|
" stds,\n",
|
|
" dtype=tf.float64)\n",
|
|
"\n",
|
|
" return latent_space_value_range\n",
|
|
"\n",
|
|
"def generate_images(latent_points):\n",
|
|
" # notice we set training to false because we are making inferences\n",
|
|
" generated_images = restored_generator.predict(latent_points)\n",
|
|
"\n",
|
|
" return generated_images"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"id": "333a593d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def plot_result(generated_images, number=100):\n",
|
|
" # obviously this assumes sqrt number is an int\n",
|
|
" fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),\n",
|
|
" figsize=(10, 10))\n",
|
|
"\n",
|
|
" for i in range(int(np.sqrt(number))):\n",
|
|
" for j in range(int(np.sqrt(number))):\n",
|
|
" axs[i, j].imshow(generated_images[i*j], cmap='Greys')\n",
|
|
" axs[i, j].axis('off')\n",
|
|
"\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"id": "2f5f0154",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generated_images = generate_images(generate_latent_points())\n",
|
|
"plot_result(generated_images)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff581bf2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We see that the generator generates images that look like MNIST\n",
|
|
"numbers: $1, 4, 7, 9$. Let's try to tweak it a bit more to see if we are able\n",
|
|
"to generate a similar plot where we generate every MNIST number. Let us now try\n",
|
|
"to 'move' a bit around in the latent space. **Note**: decrease the plot number if\n",
|
|
"these following cells take too long to run on your computer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"id": "d3617ad8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 225\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=-5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=5))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a074f93",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Again, we have found something interesting. *Moving* around using our means\n",
|
|
"takes us from digit to digit, while *moving* around using our standard\n",
|
|
"deviations seem to increase the number of different digits! In the last image\n",
|
|
"above, we can barely make out every MNIST digit. Let us make on last plot using\n",
|
|
"this information by upping the standard deviation of our Gaussian noises."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"id": "4ef8937d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 400\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=10))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "385a2d0a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"A pretty cool result! We see that our generator indeed has learned a\n",
|
|
"distribution which qualitatively looks a whole lot like the MNIST dataset.\n",
|
|
"\n",
|
|
"Another interesting way to explore the latent space of our generator model is by\n",
|
|
"interpolating between the MNIST digits. This section is largely based on\n",
|
|
"[this excellent blogpost](https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/)\n",
|
|
"by Jason Brownlee.\n",
|
|
"\n",
|
|
"So let us start by defining a function to interpolate between two points in the\n",
|
|
"latent space."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"id": "57de87b8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def interpolation(point_1, point_2, n_steps=10):\n",
|
|
" ratios = np.linspace(0, 1, num=n_steps)\n",
|
|
" vectors = []\n",
|
|
" for i, ratio in enumerate(ratios):\n",
|
|
" vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))\n",
|
|
"\n",
|
|
" return tf.stack(vectors)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfb76bb6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have all we need to do our interpolation analysis."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"id": "e25decef",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 100\n",
|
|
"latent_points = generate_latent_points(number=plot_number)\n",
|
|
"results = None\n",
|
|
"for i in range(0, 2*np.sqrt(plot_number), 2):\n",
|
|
" interpolated = interpolation(latent_points[i], latent_points[i+1])\n",
|
|
" generated_images = generate_images(interpolated)\n",
|
|
"\n",
|
|
" if results is None:\n",
|
|
" results = generated_images\n",
|
|
" else:\n",
|
|
" results = tf.stack((results, generated_images))\n",
|
|
"\n",
|
|
"plot_results(results, plot_number)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.15"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |