Morten Hjorth-Jensen
2841 lines
157 KiB
Plaintext
2841 lines
157 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d2db899",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html chapter13.do.txt -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4908a97",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Recurrent neural networks: Overarching view\n",
|
|
"\n",
|
|
"Till now our focus has been, including convolutional neural networks\n",
|
|
"as well, on feedforward neural networks. The output or the activations\n",
|
|
"flow only in one direction, from the input layer to the output layer.\n",
|
|
"\n",
|
|
"A recurrent neural network (RNN) looks very much like a feedforward\n",
|
|
"neural network, except that it also has connections pointing\n",
|
|
"backward. \n",
|
|
"\n",
|
|
"RNNs are used to analyze time series data such as stock prices, and\n",
|
|
"tell you when to buy or sell. In autonomous driving systems, they can\n",
|
|
"anticipate car trajectories and help avoid accidents. More generally,\n",
|
|
"they can work on sequences of arbitrary lengths, rather than on\n",
|
|
"fixed-sized inputs like all the nets we have discussed so far. For\n",
|
|
"example, they can take sentences, documents, or audio samples as\n",
|
|
"input, making them extremely useful for natural language processing\n",
|
|
"systems such as automatic translation and speech-to-text.\n",
|
|
"\n",
|
|
"More to text to be added"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43cb4913",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A simple example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "cf6b9dab",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAACWt0lEQVR4nO2dd5wU5f3HP7t7Hbg76h3l6EqRKs0DCyqKSIwYY9QYUaIYDSSi/iwkxhI1GI2xxILGKDGKqIlgB5EqeoAgR5WjS72j33EH13bn98exe1OemXmm7czsft++TnZnnpl5duaZ5/k+3/YEBEEQQBAEQRAE4ROCbleAIAiCIAjCCCS8EARBEAThK0h4IQiCIAjCV5DwQhAEQRCEryDhhSAIgiAIX0HCC0EQBEEQvoKEF4IgCIIgfAUJLwRBEARB+IoUtytgN5FIBPv370ezZs0QCATcrg5BEARBEBwIgoATJ06gXbt2CAa1dSsJJ7zs378fBQUFbleDIAiCIAgT7NmzBx06dNAsk3DCS7NmzQA0/Pjs7GyXa0MQBEEQBA8VFRUoKCiIjeNaJJzwEjUVZWdnk/BCEARBED6Dx+WDHHYJgiAIgvAVjgovr7zyCvr16xfTghQWFuKLL77QPOaDDz5Az549kZGRgb59++Lzzz93sooEQRAEQfgMR4WXDh064Mknn8Tq1auxatUqXHTRRbjyyiuxceNGZvlvv/0W119/PW655RasWbMG48aNw7hx47BhwwYnq0kQBEEQhI8ICIIgxPOCLVq0wNNPP41bbrlFse/aa69FVVUVPv3009i2c845BwMGDMD06dO5zl9RUYGcnByUl5eTzwtBEARB+AQj43fcfF7C4TBmzZqFqqoqFBYWMssUFRVh1KhRkm2jR49GUVGR6nlrampQUVEh+SMIgiAIInFxXHhZv349mjZtivT0dNx+++2YPXs2evfuzSxbWlqKvLw8yba8vDyUlpaqnn/atGnIycmJ/VGOF4IgCIJIbBwXXnr06IHi4mKsWLECd9xxB2666SZs2rTJtvNPnToV5eXlsb89e/bYdm6CIAiCILyH43le0tLS0L17dwDAoEGD8N133+H555/Hq6++qiibn5+PsrIyybaysjLk5+ernj89PR3p6en2VpogCIIgCM8S9zwvkUgENTU1zH2FhYVYsGCBZNv8+fNVfWQIgiAIgkg+HNW8TJ06FWPGjEHHjh1x4sQJzJw5E4sXL8a8efMAAOPHj0f79u0xbdo0AMCdd96JCy64AM888wzGjh2LWbNmYdWqVXjttdecrCZBEARBED7CUeHl4MGDGD9+PA4cOICcnBz069cP8+bNwyWXXAIA2L17t2TlyOHDh2PmzJl48MEH8Yc//AFnnHEG5syZgz59+jhZTYIgCIIgfETc87w4DeV5IQiCIAj/4ck8L4Q1Dp6oxvQl23Gkku0vRCQ3x6pqsXBzGerDEberQhAE4TgkvPiEX8/4Dk9+sRl3vP19bFtVTb2LNSK8xJUvfYNfz1iFGd/ucrsqRBKx/VAljlbVul0NIgkh4cUnbNjXkDl45a6jAIBP1+3HWQ/Pwz+X7nCzWoRH2H30JADgs/UHXK4JkSzsOXoSFz+zBGc/Nt/tqhBJCAkvPuXu99YCAJ74/AeXa0IQRDJSvOe421UgkhgSXnxKeio9OkJJwO0KEAnNws1luO+/a3GqNqxZrrKmHgkWC0J4DBoBfUpGasjtKhA+YM/Rkxj7wtf48Pu9bleFSAB+PWMV3l+1F68u3Q410aSk9AT6PDwPk2euiWvdiOSChBcf8Nk6pR9Dego9OkKbj4r3YdxL32Dj/grc/f5at6tDJBCl5dWq+95YthMA+V8RzuL42kaEdSbN/F6xjTQvhBbFe47jzlnFbleDSFDIIkS4DU3ffQppXggWgUCD18uOQ5Uu14RIVgLkeEXEARoBfQppXpKb6rowHv1kI77Zdtj0OcpP1uGTtftRXaftfEkQRhBrZb610D4JQgsSXnxKBkUbJTX/WrYTb36zCze8vsL0OSbMWInfvbsGj36yycaaEclMdV0Y763aE/v+SwvtkyC0oBHQA7yz4kf8p2iXoWMyUkjzksz8eKRKcz+P6v773ccBAHPW7LOhRkSywQqF/rh4vws1IZIRcth1maqaevxx9gYAwE/7t0dOVirXcZTnhWARiP1LjgdEfDlVG0YNra1FxAkaAV2mtr7xZa8J8/seGNG8hCMCdh7WnqkT/oKiPQhewhEBry7ZjrU2ZsQVZFle5m8qQ6+H5mL64u22XYMgtCDhxWXEXYCR2bJY81KnM9v5/btrcOHfFuP907bo5TuO4BfTi1BSesJQXQnvoCa7UKQHIee97/Zg2hebceVL3zh2jf/7oCGP0L7jpxy7BkGIIeHFQxgZeFJD/MJLNFlUdFZ03WvLsXLXUfx6xnfGK0l4mu92HXNkld9whFQ9fqWktMLxa9BSAES8IeHFZcRp241MmsVlzfYbh07UmDuQ8DQvLdpm6/mOn6zFkCe+wn3/pSy9BEF4AxJeXObxzxpXhQ5wqF5YRWjOk3xoCay19RFbzUfvr9qDo1W1eH8VrY9ENCA3cVMfRMQbijZyETPJwVhjUoRX9UL+EEmB3JmSSG54JkWWoSaXNOw5ehKzvtuN5llpuPW8rq7VgzQvLnL8ZJ3kuyAI2HGoEj99cRnmbSzF5tIKPP/VVkmZaEck7pDI3ExY4VRdWNOnhdqXv3HCH0UuIHNPoAjfs+/4Kby0aDtmrtztaj1I8+IicsfKhz/eiE9PryD9m/+sZh7DnEOZ7TdIE0Oc5vrXluP92wsBNDjnhoIi4ditShG+gfy5k4eonOr28EGaFxc5flIqvEQFFy3YPi98PYfbjY2wD61nvvCHg4ZXlF656ygA4K2iXej10Fys3HnUSvUIDxEPsxFpXpKPuJgjNSDhxUWOnjQe0srKBUP9BiFmf3m16WMf+mgjausjmDJrTWwbta/k4IUFW3HRM4tNhdrzCi8HT1RTWLXP8YpPHQkvLlJVU2/8IIo2IiwgCAKKth/RHaCCQdLTJRt/n78FOw5V4Z9f7zB8LI/Z6OO1+zH0iQV4cM4GE7UjPAOZjQgzdmJWg4nOZARBoGRiCcyKHUfwQXTFXpOPee6GUlz/z+W4+JnFin3Pzt8S+yz1eVFerKK6TrK0BZE4RLUon607gI37yw0do8XT8zYDAN5Z4a6jJ2GN6JN2O5s3Oey6iBntqVael2tfW479x09h4T0jkZailEvlNkq3JWfCGNe+thwA0CO/melzzN9UBgA4Jot0A4DnFzRGtkmEF1k7PVpVi7Mfm4+CFpn4+r6LTNeF8CahQACrfzyGSTO/BwDsenKsooy8TfD0ZUG3RzvCFhoddsnnJWkxYztkNZiyimpc/vzXWLnzKPYeO8U9WyL8yd5jp0ybCsOcEnNIY6BZvuMIAGDPUVrHJhEJBgLYUqa/7pnRyRcJL4lBdNxy+3GS5sVFrGhexE5vT80twaYDjeuXqJmOqOtIDKw8R16rYkjD50VuIth77CTa52a6Hn1AmGPDvnK8LvJzCQadmVNT8yDshDQvLmJm9hx9/8XHnqqVZuqtVxNeqPNICKw8R96oEK1Zsrh5vbtyN8796yL8YTY5YfqJ4j3H8cKCraitj+An/1iGOcX7Y/tCgYAjWhLSvCQGMbORy8+TNC8+I9pgJGOQrA2R026iY77T4A1Tlfq8SI8Rf3/myxIADULMtJ/1NV0vIj7c9V4xbj2vC8a99A0AICNVOX8NBuCImpYC2BIDr4wuJLy4iQm7UaPmpfHYymppyLWa5kVxLupMfEkgYD7le4QzQEgrVFp8aUrZ4S9mr9mH2Wv2xb5vLlX6toRCzpiNSPOSGET7HrefJpmNXMSU2Sjm89K4TezvAgARVZ8X/eb245EqXPzMYrz/3R4TtSPigTWfF16H3cbP8kPE5yDZxecwHqCWs7b4MKMBB26bGQh78EqotKPCy7Rp0zBkyBA0a9YMbdq0wbhx41BSUqJ5zIwZMxAIBCR/GRkZTlYz7giCgLKKapMOu/otpi5sPv/Gnz7aiO2HqnDf/9aZPgfhLFYGAd4mp7W2UUSieSHxxetoNRfW0wue7nfthsxGCULM58XdajhqNlqyZAkmTZqEIUOGoL6+Hn/4wx9w6aWXYtOmTWjSpInqcdnZ2RIhJ9Ek9j/O2YCZK3ajf0Gu6XNoDRn/Xb0XLZumYVCnFrJjBNTUh1WOaqC6Tns/4T6WNC+cJkWxil9L80J4H63HxXqWTkUbkdnI3wiCgO2HKmNtxu08L44KL3PnzpV8nzFjBtq0aYPVq1fj/PPPVz0uEAggPz/fyaq5yszTGSbX7jlu+Nggw2wk58tNZfhyU5kiudSWskqc9dA8w9ckvEUgYN5cw202Ot3QIhFBcYxAZqOEgdUcggEgqKOT/+/qvRhgcPJFsou/eXnxdjw9rwTtchosIW4/z7g67JaXNyRPa9GihWa5yspKdOrUCZFIBGeffTb+8pe/4KyzzmKWrampQU1NTex7RUUFs1yi0KiF4hs29h+XJhITO/O6LTkT/Ig1Jg0OuybPYyDPiyAIGPfyN1i3V5r0kBQv/sKo2SgUDHD1DWLHXy1q6yO4979rFe2I8BdPz2uwhlhZ+NVO4uawG4lEMGXKFIwYMQJ9+vRRLdejRw+88cYb+Oijj/D2228jEolg+PDh2Lt3L7P8tGnTkJOTE/srKChw6id4gpjowjmA/OpfK0ydn/AWYu2HFaHTSJ6XEzX1zAFHfAYSZPwNy2eJ17zDG9X4weo9+EiUR4bwJ3KfJbfHirgJL5MmTcKGDRswa9YszXKFhYUYP348BgwYgAsuuAAffvghWrdujVdffZVZfurUqSgvL4/97dmTHFEyPIOGIAjYcajK+coQjiMZJyz0GrzCBm+GXXLY9TesxxcKBvhMApzPvvyUch0twn8ohNpkSFI3efJkfPrpp1i6dCk6dOhg6NjU1FQMHDgQ27ZtY+5PT09Henq6HdX0BUacl2sMrvr74fd7sWLnUaNVIuKAWGCQdyKvLtlu6jxa8GbYJdHF37DCnXlCpQF+EyQ56iYGDc9RrAF2F0c1L4IgYPLkyZg9ezYWLlyILl26GD5HOBzG+vXr0bZtWwdq6D9ieV44hg2jwsvd7681UyUiDkjNRlKhYdoXm7nPw6950ToHiSx+wqiZccmWQ5i3sVS3HG+eF15hiPA43lK8OKt5mTRpEmbOnImPPvoIzZo1Q2lpwwuRk5ODzMxMAMD48ePRvn17TJs2DQDw5z//Geeccw66d++O48eP4+mnn8aPP/6IW2+91cmq+gYjPi96YdFuNz6CH7tWfOBeVToYwFaVlYUjpHrxFVpCBqs5fLb+AN95OZ899TOJgdd8XhwVXl555RUAwMiRIyXb33zzTdx8880AgN27dyMoiss7duwYJk6ciNLSUjRv3hyDBg3Ct99+i969eztZVd/QqHnRp6bOfLI6wluI16uyoobn1ZoEAwFc/UoR+xwqnwn/YSVnjx0rlBP+wWvRqY4KLzwd5eLFiyXfn332WTz77LMO1cj/RBsQn+aFhJdEQfwuWVtVmq/cCdl6WWbOQXgDrUGH1/Gf5WtnZpFPwr8oNC8uq9RobSOfYaS96JmNCG9RXRdWzXAsFhgCsLAwI+dxS7YcUt0nULRRwsDz9KwKq+SwmxjIn6PbT5WEF5/BWlVaDdK8+IdwREC/R75En4fnoZ6xNpXYbGRlLLFDayIWgEgL4294ZE81gTdsYqkJwr94LFKahBe/ETDg9KLn80JdincoP1WH2nAE9REBxxl5MaTaDvMCjB2aEqm/Lkkv/kb/+akJL7xPnqxGiYHcTOS2DwwJLz6Fy2GXzEa+QdzBswYLuwQGOxZVFJ+CNC/eR2uGzPP81JoMt/M3SS8JgeIxkuaFMEJM8cLRcdSS2cg3iGcxrEcblmlezBKxoUlIhCcSXjyPVnvh6UfUhRe+65Pokhi47aArh4QXBzhQfgrn/nUhphvIfMqLoVBpEl58Q0D0JrIGBXFulYgFu9H+8lP6hXSQal7YFZm9Zi8+/J69HhnhHfgcdq2ZjYjEwGOKFxJenODvX27B3mOn8KSBzKe8GLEzkvDiT1iDhWCDsmPX4SocP2l9nZmIjvPwydp63PXeWtz9/lpUVNO6Nm6juaq0BYddO0yQhH9Q+LyQ2Sjx4F1t1QyNZiP9sno+L1W1YVz0zGK8VbTLesUI21hUclASyVEXjkgz45psXku3qoc/GyGio3kRO4pTokRvwyOAqHVn/Bl23Z6jE3agiDYih93Ew8l3tTFUWh+egWPHoSo89NFGS3UizLNixxHsOXpSMhD8cfYGvLFsJwCgsqYeQ5/4ChPeXBnbb9ZhN1VrwSIDRDT8b8pP1eGxzzbZch3CI6g0N9K8JBZ6/k/KJHUOVoYDEl4cwEmJNJozgcfRjsxG3mbDvnJc+9pynPfUIsUA8fHa/QCAhZsP4tjJOuw6cjK2LxIxJ8Ck2SS8aLW9J7/YjA+/32fLdQjnsWI2Itklcfh66yEMeeIrLPihTLWMIkkdCS+Jh6MP1ZDDLoVKe5n1+8pjn+XCSPQ7a0Ves2PGur3HTR7Jf/0tKos5Eu6h1R3xCMFWNSys68/4Zicqa9SXoCDiy43/WonDlbW45d+rVMsoM+yS2SjhiIPswpekjjQvnoYlmESJhjSzlCVmE839u+hHU8fJ0RrMtH4T4Q5areWbbUd0j68LC1j941HFZIhXqGGVeuSTTXjkYzJXE+ZxdGFGwl3IWdLbiJN3yccBAcCOQ5W45/21iuMERvl4ouWPTovwJR7PfFmCD1bvxU/7t5Ns522DEZUGo7V+FuE9grKJlNvzFNK8OICTDzXm88K1thGZjbyMWKsif5qCIOBXr69AVa3yGbq9GKLWjDslRMKL17D6RD5Y3ZCvJ+qHFYXX78rJ6EsifrhtJpJDwovNzFyxGws3H3Ts/MZCpUnz4mXENmSWQLK/vJp5nCC4p3l5ceFWzWuT5iV54JVJwuTZmxB47dUms5GN7DxchT/MXu/oNaLSL09/QMsDeBvxQK/UvKgf5+ZQ8Lcvt6BzyyzV/Sle6+GSgIMnqtGySXrcBUdeDaBaOWop/sJrq4OT5sVGKhirAdtN4/IAZDbyO1LNi3Sflmlm5+EqzN1Y6lS1dNl7jL3EwDXTv8Weo9aXHyD4+X73MQx9YgFu/NeKuF+bV6ESJrOR61TXhXHda0X4+/wt5k8ik13cVqiR8GIjmWmhuF2LzEb+Ryy8yDt4rcf79LwSh2rEh5pg9d2uYyiRhUpPeuf7eFQpaXln+W4AwLfb1aOGnJow80YbmRVeIhEB/1iwFd9uP2zqeKKR977bg+U7juKFBVtNn4M0LwlMeor67ayuC+PVJdst58Ewkmqboo28jVjNXxeWPiu3nXK1MDIWrdx11LmKEFyCiVNNife0ZoWXj9fuxzPzt+CX/4y/VinR2HGo0tRxc9bsw+KSBh9Ob4ku5PMSN/65dAeemb8F0ywu1mhoeQAyG3kacbSRPCLDu6IL4SXcHFDUQqDlmI02+lGUVZqwxr7jxs25e46exJT3igEAu54cS5qXREZrhiPOpmoFijZKHMSdgVLzEu/aEH5j9Y/HYmHMWjg15vA2UXnb5sVjY6Wv2X+cHbmoxaHKGsl3rz0PEl5sROtlTtMwKRmhsQHpdx1OOMpt3F+Op+dtptTeNiAWXurDMs0LSS+EDle/8q2r1+dtomrCi95gSIFr9mFEC79m9zEArLWMvPVASHiJE3YtimckVNoJ4WXsC8vw0qLteHquNfMXIaU+ItO8uFQPguCFV8BWS9mgd7jXBstk4aqXG4Ri8d0XBEFhojS7ur1dkPBiI1ovs92aF55m42RyqE0HKhw7d7IgjtYwq1o3S6umaXG9HuEeTnUDvHOjujCJ4m5j5gnIoyHlywO4jceq42/iYzbin43wOtQR7iB+OvIO3mmrEVmlkgenHjXv5EjPbHS4soY58SPFi42Ibu9v31mNb7bph5+L739YEMhhN5FhvcvRl9Ius5H8vFpQWm5vI36Gcp8X3hwaBKGHY5oXzsmRVqbvD1btweDHv8Jf5ypzF3ltLR0/I+5PPl9fihte1w8/F8sqguA9Mx4JL7aifJmj77ddmhf1KymJULCRJwlHBJysrZcMKnVynxenNS/Onp7wEE4JwlY1LwDw6CebAADTl2zHws1luPu94lgwgMfGyqRDbjby2uOgPC8OE44ICAUDSLVd86Jfhmbv3uSnLy7Dxv0VeOrn/WLb5JoXp6FoJv+TnhJ0NR0Cf7SRekFxH/XrGasAAO1yM/F/o3tQtJGNWPV5iQiC54RJ0rzYCOtljr6ctvm8RK/FUdbJNUVIpWuejfsbnJ2XbjkU21Yf5wy7JLr4H57lSMpP1WH+prI41EYdNQErgABzgnWkqia2n7AHM92JWHiMRGh5gISG1T6i2SVtC5U20H7s0LzsPnKSmdPF7TC5RED8fGrlwovD146n4oW0PM7A0xX86vUVprKr2omW2Yg1v0pPaRDKPDZW+hqj/XVFdR32lzcmtouwQqVpYcbEgfUwT9U2JAey3eeFx2HXgOaFdb5tBytx/tOLUPiXBYbqRvAhfj7yZ+W0yS+eAsWzVlayJSxhV2ZvK2imAWAJL6kNfaXXHET9Rm19BP9Z/iN2Hq4yfGy/R77ETW+sjH2naKMEhyXdDnniK7yz4kfb7LdGTmNEeGEV/Xprg1njBGXTdQTx81Fm2HX22vGcNL2wcFscr5Y8+GVw19a8KFtiRlTzItp2sKKaNHgG+efXO/CnORtw4d8WW+5Pks7nZdq0aRgyZAiaNWuGNm3aYNy4cSgpUYbEyfnggw/Qs2dPZGRkoG/fvvj888+drKbj/HH2BtsHCz6HXf7zsTqRkIbERfZo64iFl3ibjdzgP0W78MGqPW5Xg4gztZwOu1EaNS+N24b+ZQEe+mij7XVLZFaJVnS3LLwkm8/LkiVLMGnSJCxfvhzz589HXV0dLr30UlRVqauxvv32W1x//fW45ZZbsGbNGowbNw7jxo3Dhg0bnKyqLWg1ELsnDTw2TGOaF2VZrzXWREPcpysddh2+eJylo4MnqvGnjzbi3v+uU/xWwhx+eTu18rywuqio5kXe//xn+Y+21ivRsVMz50XNi6Oh0nPnzpV8nzFjBtq0aYPVq1fj/PPPZx7z/PPP47LLLsO9994LAHjssccwf/58vPjii5g+fbqT1eVi2dbDWFxyEPdd1lPhx6ItvNgzWkQbpN2h0qyiWpoXcti1jiRJXURuNkqsaKOqmsaF4ajlmKM+HEGKzSkX4oHRhRkzUslh1w7E3bfV/iQcUfq8uP184vomlJc3OI+1aNFCtUxRURFGjRol2TZ69GgUFRU5WjdefvWvFXh92U68VbRLsU9rQLerwzbSXozkgGBrXgxcjDCMWDOmWB7A4WvH23+A/BWs8fbyH9Hrobn4dntjWne1wWP93nIcraqNU830MbpuV8rpjoe6H2uIhQ2rb19Dhl3lNjeJm/ASiUQwZcoUjBgxAn369FEtV1pairy8PMm2vLw8lJaWMsvX1NSgoqJC8hcP9hw9qdim9TDtTrlid8Nh1U/LbEQ+L9aROuxSnhdCnQfnbEBdWMDv3y0WbZW+g4IgYPWPx3DFi8twzjTvRAhqmY1YxCaBbk/tfY5EeLH4wntxqZm4CS+TJk3Chg0bMGvWLFvPO23aNOTk5MT+CgoKbD2/Xdg9GNlttjHqsEtYR3zL5cJjIuV58dK1/U69xpof4YgQixA0KjA4iVHNS/RdoN7HGuJVoK2OFxFB8FzG9rgIL5MnT8ann36KRYsWoUOHDppl8/PzUVYmzQhZVlaG/Px8ZvmpU6eivLw89rdnT2JHM0SFadsdgBn9CwkvziKezcg7BudDpd3riMhfyjzikHq5YiIsCDGTi5cwKkhF3wVSvFjDTu14JCIgJegtfytHayMIAiZPnozZs2dj4cKF6NKli+4xhYWFWLBAqvKcP38+CgsLmeXT09ORnZ0t+XOLeEQbRRuk3d0/S6r2Sx4JvyI2G8k1c16b5dhJAv80x6msqccGlcRzkQgQ8tgAA2ivbcQi+lpQtKM15KtCy/nbvBK8991urnNFBKBphreWQnS0pU+aNAlvv/02Zs6ciWbNmqG0tBSlpaU4daoxXfX48eMxderU2Pc777wTc+fOxTPPPIPNmzfjkUcewapVqzB58mQnq2oLWjNKuwaj2DVs93lRnlDcSUYcXCcpWYlINC+ynU5rXuL8OKn12MdP/rEMq388qphXhwUBXgxGkucwiqImmkQFeRJdrKHnsPviom24/3/r+bO1e+wldrSpv/LKKygvL8fIkSPRtm3b2N97770XK7N7924cOHAg9n348OGYOXMmXnvtNfTv3x///e9/MWfOHE0nXz/gseeugCWbvLZ0R+yzPJSXsI5Y86IwGzl8bXqa3mbBD2VYv1c9tf+izYeUZiNZOKtXIryMLhD79NwSlJSeILORRYI6mpcoRzgi0yKCoJicu928HNUD8bw8ixcvVmy75pprcM011zhQI2eJh9lIiClebHYA1qmgkytUJyviW6pw2E2k9QEgnUW73el5nS1lJ3DLv1cBAHY9OZZZhvX+N/glNN5pv64pdaKmHqOfW4qnft7P7ar4mqDE/0n9pdvPsXBnRBA89956UMnoX7SerV3CRvQs8QiVFqMV5UCYI+Kq5iW+PVH5qTrXru03Nu7XX0xREJQOmQ1mo8Ztfl9TihQv1uANld5/vFp952nCERJeEhqt2bL9ywPYi55PDsku9iOJNlJk2HX22vHuiK56+dv4XtDHlJ+s0y3DmmxEIoJstu1vKGDAGnx6F3AlNGxwefGW9ELCi0lYL5am5sUuh93T57HbrKAnvPBqXk5U16GiWr/zJaQCi9JfN3GT1HltBuc1yk/pr+IuQLnWjFdDpc2SOL/EHXj9n3j6Gi9GP3or9slHyBtDTX0YX20qUylto8+L7F+70Ksfj89LOCKg7yNfAgC2PD5GsfYTIUWseZHfXw/2FbaRwD/NFriEf8ZNDEcET4ZKE+7A2xR43BkjZDZKXKZ9vhkvL96uut+u5+5UA9LXvOhfuKq2ccZ4pKrGcp0SHa08L85n2HUxSZ3XekGPUXFKX3gRoNRMNOR5caRKjqBnFqJWYo0A79pGPKHSglI/47YZyUdN3dt8sEo7s6/tPi9xdthVaF4Y/Q6peY2htTyA43lenD09YYFyHuFFEBSDf1hQrvzrZfbpRLmQkGsN3lBpnrssCN7TBpPZyCb0ZhH2JamT/msXevUzGirttYbuRcSdt/z+O21jdnVtI/cu7Qt4VoNnPT95nhe/Q+3EOIIg4NZ/r0JmWggtmqRJtqvBk4C0of/31hMhzYtJ5MKKXp9h22OPJXqJb54XhdmIUZxbTUkoiPfCjG5Cgq1x5P0L6xZGGKp9P0OaF+PsPXYKCzYfxKfrDkjWlNK6k28V/ah7XsrzksDozngMPvn+BbnapzN0Nn30Fn7l0byIOxvqeIyh8HlJ5Pun8tMWbi7DzBV8a60kG/IoIjXNSyK1mwT6KXFDrLGVTLA17uWOw1Vc5/Xa4yCzkU3oRSgaffAtslJVz/Ppuv1Yp5E63AyGQ6UZv1cs3yR7x3OsqhYZqSFkpoUk29UGF0W0kWM18y6/ntGQVfbsTrnome/eAqtuw0r7L282zFDpiODJkFazJM4viQ+CIEgWwQzyyS5cRCLem1CR5sUk8gepp3kx+tzVfGgEAZg8c42xk3FQWaOdW4IrzYu32rZrlJ+sw8DH5mPAn79U7FNrB8rlARyomEfQm8MdOpF8kWqvf70DI59ehAPlpxTzgsueW6ow27Lah1XV/uBOzc0f7ACJ/A44weSZazDq70ti36WrSlu7mWEPPgzSvNiE3Q67qiuuOiQhHKzQHjB4ktQl0qzPCutPr8bNcrxUu0PeU8o6h14zSSSnU14e/+wHAMDT80oU+7YerFRsiwhqmhfzdfDa+5tM74QdfLb+gOS7fPkIKwisUGmXHw9pXkwiF1bsNhup9d9ONZiyCu31Lbh8XsSfk7jfEb/mx0/W4i+f/4DNpRUN+1RuDI/Hf6Kg90uTT3RpRKz214K1tlFEsCaAeG3l+GTuQ+yA0+WFi3DEe8+DhBebiJfZ6FRt2NiJODmoo6rn6djEHSfNmhr400cb8drSHbjsua8BqHciHhs3CI8jnP5PsdWK8MIpOMULb9XG31jVqrGS1LkNCS82oa95Mfbo1c7Hk8DKDIcrtYUXHs2A+P2oqY/g5cXbsGGfvY7FfkB8H+S/X93nxWtdg3PoDrDJrHqBLOWAyr1iJQ2LWEwk5rk26LX6+Azxa2R1YV1B8F4kGwkvNqGb6tqo5kWlB3dq0cN6nVhpueZl5c6j2Fp2QrJNLKC9sWwnnppbgp/8Y5l9lfQhvC+8x/oFR9E3GyWv9CL/5WrmWgHKNiMI1jR4njMbuV0BnyMek6w63BpNUhoPSHixCb1FsIxKrWqyEK9N3CiGlwcAcMmzSyXfxT9x4/4KO6rlSwSVzw3fVXxekkh60VW8JK/sAkA2Y1a5Vw2aF2VuICvtyGsDVBK9Eo5j9dla1eo5AQkvNmG3z0u8Iy70qsczKxP/Ru9ZSOOH5vLzZDZiIr5nySy7BAJS4U29XbA8XqxpK7QGODcESq+ZKZKZklLlZNTtp0Oh0jahK7wYPaHFziItJShJD62HHWsb0QDcsFiemagrj016HYUl2CZ501FF7b4IjMgiljbGCFrveEow4JjWVw1qEt7hn1/vdLsKCkjzYhM2rw5gefa54O4L8JN+bfkPMGE20jpFsvktlJZXY8gTC/DkF5slN0I+UKsPRknUVTN+qmpa8yRHbULActgVBMFSyL3WOx7Si0hwgGR6JZwg0V8jEl5sQk/zYlQrYdVs1CQ9Bd3bNOUub3h5ANY5RJ1fspmNZny7C4crazB9yXbV3y4IAuZtLGXuSybNy9vLf1RoBSWCb4J3umLW7D6G619bHvsu/+mqwguUCekEWGtHV/RXn+yk6Dn1OUCya3KtkugTSBJebMLuiYnVDjwlFEBaCv/jtcNslMw0ka1hFEV8WxduPogp7xUzyyVTR/3Cwm2YvmS7ZFskSX1ernr5WxTtOCLb2ngH1OYMDa8jw2xkoS7jBrZX3ZcSSqankhgk+gSShBeb0HfYNRhtZKUyaLBRZ6WyB1QW9qwqzX25hCM7s3EhTYnjsujzqh+PqR6fbMLh11sPSb4nc9vRQstspNS8WMvFodWHyVe1jgfUJqzh9P1zW5wl4cUmdPO8GDyfVbNRKBhAVhq/P7bdDrvJ1vHkiIQX8SKXvBoVp+5Xs3Rv+uRX18nMRqLfn8w+L4FAgCute0lZBY5W1Uq2sZx4jV1bfZ8b602xNAcNK2wnWediEqcnRG4/BRJebEJvYmK4U7HYV4QCAWSqmDJY6Pu8GHPYTTbSRSa6Y6JBRU0LI8cps1FeToYj57VKdZ10mQupw268a+MtpHle2O1iwz526KqVZqTlI+GK8ML4LcP+sgCTZ66Je138SKKbokl4sQnb1zayIL1MurAbUkJBNEk3YjayV/OSbAOQ+PYcPVkn2s734J3qaLw6S62ulwov3qyl+xhpFxHB2qrSWu+sK3leGNsOV9YoVk9ORurCEXxUvE9zQd1Et0R7U6fsQ+xeVdqKifne0T0BAJmp9pmNjCapSzbEKu4qkdlIb8HLKIne0ciRm42S1WFXzo9HqvD97uOx74beKYtmIy28onkhGvjXsp148ovNaJ6VijUPXcos49WJi12Q5sUm7F7byI7OIsuA2UhPs8K3MGNivyxaiG/Ph9/vNXy8U/fOq09EbjYin5cGxIILYMxvwWp0idZdd0fzov17TtWGsXTLIdTItHjJwNItDQ7vx06qr3VHZiOCC31NSfwbkhGzkV4fadTnJcHfGwVi4UOrQ1Ej2aKNahQOu6R5YWFkABIEvkmGGp5z2NX5KXe/X4zxb6zEY59uik+FPEQGRySpXgSp3yHhxSZ0k9QZbEh29BWZRqKNdH1e9H8Ar3NqIsIzyGjNJJ2UXdIN5PuJF7Vhudmo8XMSK14UGHmPrK4qrSU2uhAprcsXGxoSPr69fLfLNYk/mSLhJaqFkZPomnDv9Wo+YbkssZT+2kYG87zY0INnGsjzYofPS6KrKbUwKpzKcezOCcDLN5zt1NltQ6p58eBI6RKGNC8Gy8vxnubF2G9ZufOoagbrREOseRn/xkqcqFZqex3vj13u7kl4Mcnm0hPYtL8xXNHK2kb3ju6BT393LrIzGjUldvQVhnxedBoin89L4+dkmz3zvMfHqzTs0w6pXgQAF/fKw7sTz3Hk/HaRZFYzbozcl4hgzevFa6+s0bH3F68W4Tf/WY3dR046UyEPkZEqHbqrapR+P4n+TpHwYoEN+8pjn62sKh0KBtCnfY5k8TOts10zqANX/TJSQ5h0YTecmae/xpHe4EmaF214fvt7q/ZYOt4K6aneftXFw26ipzU3glGfFyumgkAgoGoecuPdNnvFfcdP2VoPLyL3eWG9M4neHzvaoy1duhRXXHEF2rVrh0AggDlz5miWX7x48ekMk9K/0lLvqwL11i3jaUhiU5Fmqu4Q/2O7d3RPPHLFWbrlzOR50ZLXEvy9UWDVvuz0LMlrs2o5yewvpYWxeyFYMxtB3VztxiMx+1sS3dcDULoEsH5yot8GR4WXqqoq9O/fHy+99JKh40pKSnDgwIHYX5s2bRyqoTXE0q6uTZijIYnPoHW6NKOLpHEUN+PzIj9tor8sWlgVPhwLlT59Xjd8FoyQ6LNEsxjXvJi/ViCg7pjriubF5CUT0VxSWl6Ne95fi3V7jwNQmo1YJHoEo6NJ6saMGYMxY8YYPq5NmzbIzc21v0I2YyQ3BU8zEp9C62xGVotuOJf+wKXXUbBeBPmAmMwDkNXf7vS987jsItW8uFcNz2HcYdf8tQKn/2M9ATdebbOXDCdgP3TPB8X4ZtsR/O/7vdj15FiuUOlE7489aQgfMGAA2rZti0suuQTffPONZtmamhpUVFRI/txAN8Ou0VWlbTIbNZxLv4zeC89jNlI7QzKoca3+RKcmSdHT+knzkgzthRcjUWzWfV6gOmty5ZEY0jo1lk3EQXvbwUrJ91SOMcDp++C2b5qnhJe2bdti+vTp+N///of//e9/KCgowMiRI/H999+rHjNt2jTk5OTE/goKCuJY40asOOw2InLY1TgdT8Nln1UdMw67co2O2suSgH2JAss+Lwmu4mXx6xnf4Zrp3yISEUjzooLxtY2s3T21vsLrDrtSn6nEa0F6P4m1O9G7FE+tbdSjRw/06NEj9n348OHYvn07nn32WfznP/9hHjN16lTcfffdse8VFRVxE2DEbUN/VWn980nNRuonNOrzwpMzRl/zwpgCyjUvKqcICwKCnncZtYbVjsK5hRkb/vWi5mXh5oMAgNKK6qQQcM1gKEmdwfIs1JqJK2Yjg2Hisc8JnlkW4BPQElEDJcZTmhcWQ4cOxbZt21T3p6enIzs7W/LnBvprG6k3pOgu8Rm0hCEnzEZmHHZr6yNYufNo7Lv4N4pLJ7rjGGC9o3D6DnlQdomRlhKUmY1crIzHMOawa31VabVJkxsDYVVtvX6h04h/dyL6vMh/keI74zcnerfreeGluLgYbdu2dbsaTMTtRU/z8uk6/WXcJZqXuJuN9PYLzBfkF68WxT6rvSuJPgMArHcUTgt4XhZeBEHedhK/vfBidCC2lmE3oK55MX1W87z5zS7uspI8QUnQ3/D8xES/D46ajSorKyVak507d6K4uBgtWrRAx44dMXXqVOzbtw9vvfUWAOC5555Dly5dcNZZZ6G6uhqvv/46Fi5ciC+//NLJappG/MJYSWnOcnwKaSSOMW420i+jN3jWy/wSWKjtr48IuP+/69CzbTNMGNEltj0SERD04qIpJrDaUTjVz0TblhfNRlEEWX6SBO9zDWGkXVk2GUG9nXh9IBRXLxE1DvLbr9S8KI/5euthx+rjBRwVXlatWoULL7ww9j3qm3LTTTdhxowZOHDgAHbvblxUq7a2Fvfccw/27duHrKws9OvXD1999ZXkHF5FL0kdD2IBKEVjUDeqeeHRvfBEG+n1CWqzvqVbDsWyy0aFlwPlp3DN9CIUdm2Jp6/pr1s/r2M92shZnxfvii7s/CSCIKBoxxH0ys9G8yZp7lTMAxgZiOVCoFEazEYq5/a4QCD+3clgpvaCMOl2FRw1G40cORKCICj+ZsyYAQCYMWMGFi9eHCt/3333Ydu2bTh16hSOHDmCRYsWeVpwMZLnhQfxKS7q1QZpKkKKEz4vei9DfUS/Y5TcD9H2ilNK2/V9/12HvcdO4YPVe/Ur5wOsCh88yy+IeXBsL0Pl7WifThERpCZJAcDHa/fjl/9cgcueX6oov+twFR75eGNSpIE3EoUWiVg0G0FdelE77+Pj+ii2XTs4/hGfEYnmxf2BnXAez/u8+AU7hgbxOZpnpWH9o5dieLeWinKpRs1GHGX0ZivycFYWarkWWJ1JWUU1R638g9XJ3qETNYbK92przDHdw7ILdh6uwnvfNa77JAiIrQ5cViG9L3uPncTIvy3GjG934dZ/r4prPd2Ap1lF10QTYDFJnYbmRe28uVmpim3XD+tovhImSfQ8L+QHpsRTodJ+xh7Ni9RslJ4SYtqg1TQyVuqm1+nVRwTdpERqEUYsrY6WT48fiXeHydvavBwqHeWX/1wh+S4IgmqbPfevi2KffzjgTkLKeMLTrgZ1ao6VO4+e1mybv5bW2kZq9UhhvMdutDSJ5iUpQqXdroH7JNYIEmei7ef97/bgk7X7bT13ioZ2xajPi5b/TBQ99XSYQ/MisTtLZkLadaqtT4Lexm7sX97KU4Q8LGzFE56BOHqvGvK8WJNe1LoK8WlDokIsLbAbj05Q6XsSFflEMgl+sgISXmzgvv+ts/2c0Q6C1RGkGlzbSCzsFLTIRE6mUtWr98LXRyKGoo3EwhDLJCXuAK+Z/q32iT1KOCJg5c6jOFUbjnuGXKOaFD/JAgL0Uw8kCzyal9i7JFj3eeHRvIgFlhDjQelFXnZp1cRkDdVJ9Ay7cpLgJ+pCwosVbGpBrNOkaphVjPq8iLU4oUAAb9w8WFFG3+dFfy0L8V49Bzpxp7d2b7nmeb3K9CXb8YtXi/Cbt1fHfbbHbzbyfqi0HEHwV32dhEcYiaYbiOiYjX5zQVf0bZ+jul9rVemTteHYZ3HfxHpOeo9uYMdc7QImiOhoev2Obqh0EvrEkPBigT99tNFWu7tYgAidFjhuObeLopxRs5HYRyagsvqaXh/Jo3kRdyC7j55kbo/CmrH5jbeKdgFoCAV/7qutps5h1PE2SqLkx2EhQN3nxU/sPXbSckQUj0wsUrygpOyEarnrh3TE1Mt7qu4PAHj1xkG61xNrfk0JmQ6Msx8VN5rtkyNUWvt7XOoQ/0tKIOHFItdML9IvxIk4XDY6uxnZo43CZ0UsvNw8vLPuecWalwanPGUZO/K8qBUIM+z2ieDTYMdvMCuDGD3MeG4gd/G7bFZTH8a5f12EEU8uRE19WP8AFbg0L6fb4f7jp7BOQ4vJ01wHdWqBOZNGaJYR90dmnpMTg96fP90U+6wlvBw/WYs9oolVouC2IOEG/urRPEhlDf/6G3qIFz8UaybyczIk5cRmoxoOZ1dJRIBKOCRPhl29jlRtP2u7lkOyXwjZ8BvMmkeMHpaZFjJ1HVdIALNR+am62OeTNVaEF/0y0Xu1cb+2Fljvnka1XWKBpOtp/5TrhzaGP0uEF5bPi86jc9onRStn0tgXluG8pxb5LkeQnpkoGfx85JDw4iHEL51WhJBYGOGJ1BGbjSIRcyp5vmgj9nbWi+X3wQmwSfNiUsXA+wyjdz7LgPDSqWWWiRrZhwD/m8VYJmCr51Ej+nrrCydAp5bqzrKB2L+N57m8b1t8fd+FkmR0YrORGYddp606YY0QrajQ8sV6/bXmvEp9OIIXFkjN1ElgKVNAwouHEHdU4s5bPvaL+4s6lk1GhljLURcWTIXNhiOCpm6ypPQEJr7FThomrmJUkOEJ3/Y6dgywTpuNom3HiNlo8f+NNFwfu/F786gP2+XMr3+eqLCgd88CgQDa52bikt55Kvul/0Y/F7TIkggpemYjt+clPNmqfzig7hvkFWrrIxjz/Ne494O1ku0zV+5GdZ28308+6YWEFwM4rZrjdTQTz7p5NC/igcusM1tYJ0ndH2evV93HigTw68z6z59swpNfbAZgjwBmVgPlpObKbWfZRIg2kiw0aSGNEc/byhI6WESb6/lntmafh1MkFj+bpunKtAu6ZiOuq+icQ6MvDqsIjuK+73ClsYzWblC04wh+OFCBD1bvlfzezaVKwcsNq5HbbygJLwZwuoHwChbiMbOeI4uV2EemPhIxNTNq8HlR369Vc2kIdcM3P5poD1ZU441vdmL6ku04VRu2ZYA1a3riPcyPIZQCBMlM/3fvrsGDc9bHPZeOFcSzf7VnUMXhL8fjsBttC3U62p5oe71mUAcUdm2J+y+TRR6xNC8q13v4it64Y2Q39MhvptyvM6zZMQn8+/wtqvvqVNqJWEt9stY+X0WnkJj7RT+JJZy58Wa4/TbS8gAGcOphRV9mNXWn/GUPBgK4dnABSspOYFCnFvjqh4Oa5xfPpOsjAvcMS0yDz4u5OyAedBo7Y7ebvnHkztF2OB2bXSXB75oJLQRBOoBGs1d/+P0+l2pknLCkzbPLPPTRRt3z8Mhr0fdZL6opekszUkN497ZzIAgC/jp3c+P+qPAi7h8Y7SwYCMRWh2deR9dhV3s/D/9YuE11n5rPS61EeDHvRB0vxE721XWN9WWNE36cDFqFNC8G8IpHdzAQwF9/3g9zJo1AdqYx+bM+LJjSvOiFSmudUqJCF6T/+hl7QqXtF0J4cnV4HdZ98cOAE0U8y1frN77cVKp7Hi6fl9O3qkbhByEvF9D+LjufeJsYXcdgzb3ApWfloZ0sgtJO1CaBdfX+El4yUhuHZ/HEiSWcJeZilNqQ8GIArzQP8Wz954M6qDrgseAxM7GPM7/om3Sdo9NmI3OnchVJCnIItvjtmE3WpzV+jD4r32RtvIEA4/elui5sKZ+K3YgddrcerGQX4ngJeEzJUSFEL20Cj0Nvw7/ibYzzWBw1stJS8PX9F+Gcri2snUgFNZ8XsVmtQhTK7gV+PFKFHYek7URNQ84Szkh4ITyJvFmKZz7pKSH8c/xg5GfzzWTMRkGEIxHNWaDWYBphqNC9osUyglj3JAj2aF7MOsfyamys3ubM1Pjnh2lYVZq/fH04goF/no8hj3/lGb8Y8SThuteWY/aavYoyPDXlMxs1IDYtsODVmIgHTdYA2q9DrvZ5OJ6dkxm21TQv4uCG46fqPNMH1YUjuODpxbjomSUSXxw1XTdLoPXIT4krJLwYwK0GogyVNp4YKkp9xJzZqJ4nw64KrHWO/PiuCbLf4WqoNLfDrjXkCRLjgQB+4exwZQ2++uEgTtWFUVFdj1M6A3i8kDvPvvnNLlPn0ZtR/+umwY1mIx3Ni94t1YtamjflfEy+sDumjlFfYoAHp7211IQSsc9LOCJ4pq2IkxiKkxuqPXrSvDRAwosBvBK5wcytYOB4Mw67EStmI3Hkxen+w4/vmiD7bEeotFntjd7gfmZeUwDA2L5tY9vm33U+fndRd67z/+WqvhjVKw/jCzuZqp9VeO/L4Me/wu1vr248ziMh+PL8S6z2zjPz1yvTull67G3WddjV1bycNhtJjmn83CO/Gf5vdA80y5CGR987ugdaN0uXnEm7HtLr2Y3aHZM/E6+sgVQn0tKJE5CqPXqWdtGNn+K25oqEFwN4ZcC1mofDvObF3A0QGD4vfkT8O4SIPQOl2Wepd9Ss2wrx3LUD8IBolnxGXjPcc2kPrvP/clhHvH7TYEOZeW1DMK+R8krz4jHPGjEbtWySxlxNPhgIxARZ6z4v0n8BvknRpAu7Y96U8zlKSs/v1GRQrQ3Ic2J5pKlI2op4QqSWgJQdbeSVXxM/SHjxAEbbndUx02y0kZZ0rzWLkmss5Nv8gvR3CLZECpk3G2kf2KJJGsYNbI8Miz4rboRkJ8Kq0jyZr3mICvtqt0O8SHw02ujinm1UyvLeU5HPC+chZp5W86w0E0fpozZB4tGGuYHYPypapXkbS3HlS98wy7M0RmYmhX7Pck7CiwGcaux6p5XPUNwYUOojAibP/N70sVEak9R5pOcwgNTnxSazkQPRRnbihhlGEMxf1yumXbnPC6tePK9A46sTYE4QgoHG7VGzUXoqu1tn3VK99P68Ao+R9hit78NXnIU2EnOTPajd1lq5QOmNpiLRvET7x9/8Z7VacWbEqBmzUZN0aZqNnExltmQvQ8KLAdzqGHkcdo1g1ta8ZvdxrZOqEma8nH5ELHDtOXoSC0u0kwPyYHpVaca21FAAE89TTx5mBrd8SMxe1SvNiyclAU9/Iog1L0zhIxATHKIDWHoKW9vGamtiH4tGXxRr6Dbp0/vzczLw3m8KLV5NiVob4BEo3UCieTEZPm8myk7+ap+Z1xQjurc0fB63oAy7PiTAEDmNqNnjrbipY7ycXhlkjCDuH9RUukYx+yxYA9Gy+y9CHmfIPC9umG8Ewfyk2CvNSi9VP8CpeTnd6NSeQiio3Jeewp6Tsh5lKBgATvv5xhx2TTxz8YTIyLsdT5ddhc+LRxpLbb3Il46jBTN9XkxcV96HBFS0e16FNC8GiFdj15vtmpmtNxWpCOPdPMMss5Fnhhl+nNAa2Wk2cuK52pHLxigCvDOwmMUun5dHPtkEQMvnJaDoD9T8nFgDk7j9BRmaFxNuMoaKOtG81JRe8mfiFS2wpF6nq6TVL9jl88ISUo08D7fvHgkvBnDqYXVr3VTyXZ4YTH5dM+Nd/4Kc2Od4h9VJfV4a/vVIv2EIJ0IrbV1V2oGBIORCDyEIgumBxSu+VPU8odIGzqc2JxabjaKoaV5Y/Yb4fkUHTGm0kQM+L6LCTsz01SZGCodd269sDslSEqf/1erjWZFsZpq94hoB91eUNwIJLwYw0jFeP7SAu+zlfaXp3OUzJzt8Xp78WT/kZqVi0oXdFOpTO9Cqkbgjj6rBxb8pO8Mf1ksnxkU7+wonBgIjndl9l/GFYetRbiH7qXcGJI6aGDGvqDyGYEC5T114UZ5EXIXY8gAmoo3E6CbDM1DWDGpNRx5K7hE5V+JIHK2TVh9vn+ZF9h3x18pbgYQXA/A2j/a5mRIzjRY3FXZSDBCZadqPxYzwUtAiC98/eAnuHd0TtWF7M0t+snY/Vuw8qrpfkqSOcRNbNbU/4sAJwk6YjcxqXmxwpFULqRVjpH7ndLXH2e/e/65DrcllLLwCj9nIDtNpkOHJm65mNmI8StagZzTPi1Y5J6KJ9FC7q/JB3yum6zpGQIOW2ciuaCP5ZCfAEIS9DAkvBjAydqmtr6E4J2Ob3noyZset6IAnXn3Wjtny795do7mfGSotXifIcg3igxM2cjujjYyeKidLPzTSyCJ8dobwV1SbWzjPK7Np3vefF7U7yxpw1B12GZoXnWpy53lRKXjziM6a51Q7f/vcTL4LM1D7TYrtHmkr4tWuG81GbmhelP5TWvRtn6NfyEFIeDECZ/sQBIHbP4LV5pTCi36eFyPjRqqoc7uwh/7s2ypMh13RT/KKn4Ia//52F8Y8/zUOVlTbfm6rK/SKMSo68IRXGunM7IyqNt0mPNKU+JYH4D9fgOHbAkTzvEhR07ywYFXBlKlIdTujr5KYpaT7bx7eGd/9cRQ6t8oyXonTSDJha9xkjzQVqc/L6fpq+rzYlGGXOQHiOO6F6wfi9xefgfsus7bGlVX84WzgMwQY0bwoy2XqpGRn+moa6HAGd2qOnw/qgO5tmjJTjtvN11sPxz7HHHZNnqthxeH46jYf/ngjAOCvc0tsP7cVbUVmakiyuJzR+8JjmTEmvNj3XMw6R3vFFMAjGNpRU/HyAFHUNC+8lbDqUKt3hJZZKiIIaN0s3VJbEgDsPnISv3i1CM0yUvDJ785FRmpI0Ta8Mmdi+bxomo1sctiV9xcBTofdnvnN8NP+7Yxf0GZI82IA3o5REPiTBvFoXuRlrK9tFMDfrumP2y/oJklSFQ9iMwSx5oXz2K82lWHIE19h6ZZDtteLh8OVNbafk6eTfufWYYptAoDlUy/G27c07uNtFeMGtEOzjBSuDsitJHUeWTPPNDyTFyOz5UCALUgEg/xmI2YdGG+fGYdas12S/Ljq08K4JeFFELB691GUVlRj68FKbD9UeXq7rJxHBF2Wc7dWH89eVdp6PQIBPu2pV9xiSHgxgBM+L6xics2Lk69YShw0L2IaNS/aTrwsbn1rFQ5X1mL8Gyttq8+WshOYtXI3l7BZWVNv23Wj8AgHedlKp0dBEJCTlRpbPRrgH0Ceu24g1vzpEjTjiPI6q10230lhr+bFtNXIG+ORwrnbarXUnCmZZiOVDLssWM3enNmo8SC93yoRjmS1j0YEWRGaBUhzvew/Xs2sl1faijj6UxCAB+esx9GqWtXyYabDrj0/hufZe8Wp11HhZenSpbjiiivQrl07BAIBzJkzR/eYxYsX4+yzz0Z6ejq6d++OGTNmOFlFQ/A2DwFGfF6U5fKz1Z3V0mxOvBFvzQvL58VNLn12KR74cD0+XLNPt6wTdeYZ8FlNiVUXIyr+lFCQa+Xj3Kw0rHpwFNc57ezU/Boqvf1QJcpP1tluNlJ7tg15XqT7MlTWNmKhN+g5sbaR+Kc4oXmJCNLfNfGtVThWVat4adxuK1HEPi8RQcDby3drlmeNLaZ8XmS3WLxOls6Rhq/lBI6OXFVVVejfvz9eeuklrvI7d+7E2LFjceGFF6K4uBhTpkzBrbfeinnz5jlZTW54G4ggGPB5ERV77toBGNmjNaZccoakzFUD2wMAurZqgrUPX8pXWU7irXlhWI08ob5dv/e4K9flmWCyBhg7BKk6jvV3AHYoe7fWTRTb7NS8bNhfbtu54sX2Q5W4+JklGPDYl1CsAWjxGWrleZGTZsRspCMEm3miJuUYAMCp05GQVqyVgiAofte3248wNC/x73cqquvw0qJt+PFIVWwbK0mdFiyB0lSotOw0KcEAVwCBVzQvjjrsjhkzBmPGjOEuP336dHTp0gXPPPMMAKBXr15YtmwZnn32WYwePdqpanLDr3lhq/bYZRvPOm5ge4w7LaiIue+yHhjSuTkKu7bSdeY1Sqprmhd+s5EgCIoEU3bjlvjEk6+F1ZTsyxFijIIWmfj52QX4xZAOKJy2UHY+y1WKsaWs0tRxbkauLd9x5HQd7A+r14rmUTrsWusjWjRJQ7ucDBw8UYM+DoTDakUb1ZzWvFg2G8nu/4b95ciXrfvlRlN55KON+HDNPryyeDs2PNowptUyoo2MYkd7CwWDvsqw66loo6KiIowaJVVRjx49GlOmTFE9pqamBjU1jY6UFRUVTlXPmM8LZ5ItnnOmp4RwWZ+2mmWev24gxv9rJe4fYyx8zS3NixHunFWMj9fut78yHoDPbCTg3Ynn4Jtth/Hiom0AgNzMNEU5owLNiG4tMbxbS5yorsf6fXyajvzsDNw56gzF9lZN09C5lVIbE2/c1OGJB2W7l5IIMHxbACA1hX95AF7SUoJYfO+FOFUb5soF1FA//vNr5XmpPj1JsZSEUVBqIo5U1iDPhYR5cqLJPMX+c5K0ESbPa6a5yfse1iKfzGt5xJveU8JLaWkp8vLyJNvy8vJQUVGBU6dOITNT6Qsybdo0PProo/GqIheCwN952dUMzu7YHOsevtTwSx9/h93TmhfRNvHLu+foSazdexyX92kb+y12Ci619RHM21iKc7q2RGsPdGY8LkyCABR2a4nCbi1xUa82qKuPcA8qWqSEgpg58RwAwIZ95Zg083vcN1pb+FWbERdNvRipbiyEJMNNXyrxWGB3Nma1tzQtFFTsU1uY0QhpKUFD5icjBFQ+A+I8J1Z8XgSlw7TgXYddI1pooGH5DK1z8CK/wymcmhenteC8uN/bWGTq1KkoLy+P/e3Zs8exa/HPbJUvj2pJG18gM7MV98xG7P3nPbUIk2eucUzT8tKibfjdu2tw1cvfSLZH61N+qg6/eLUIbxXtcuT6ckIc979Vs0Yty9kdm2OYTWn4xfRpn4Ml916Isf20NXxqDt5eEFzEfL7+AGat1HZ8tBvx6xeOw/IGoWAAKaGg7ZoXqxgxPcjLTvtZXwCAlTmVICgHcwHeDZWW1it+ZiP5vQ8GA1ym35p6e5eXMYunNC/5+fkoKyuTbCsrK0N2djZT6wIA6enpSE+P0wyas30Y0ry4LP7bsUaOEcoqavDgnPXYXNpo3mPdg5W7jjL9f6wyd0MpAGDvsVPM/f9dvRcrdx7Fyp1HMb6ws+3Xl5Om00ufd0YrtM3hS5XuxMKMctzK+8JLdED67TvfAwDOPaMVOjQ3n63VCBKzkd3vNSOBWDTyUL493UC0kRtIk+A18vx1A3BWuwYfG2tJ6gSFaYOleREXqakPW/YVMou4XmYtMpwulhKUmpcA13iUFnLnPsnxVCsvLCzEggULJNvmz5+PwsJCl2okhfVY1d4xnoXZ1M4Zb1LiOCDd834x3l6+G9Wi9ZX2l1fju13ShR156vToJxtx479WGPIv0JuhZIkcoqPPsGd+M+7zGyVFR2PBs3hiPBE/F0/KMbLHe/ykuTWSTCG6H3b7BQSgHGyiZh359nhrU/V4bFwfyXc1nxexYGxlUiUwfF4aIpDkAk3D97kbSnHWQ/Pw/irntPa8mJV5TTnsMkKleYTGPu35cz85iaOtvLKyEsXFxSguLgbQEApdXFyM3bsb1LlTp07F+PHjY+Vvv/127NixA/fddx82b96Ml19+Ge+//z7uuusuJ6vJDat9qC0z7xfNC6C/HIGdVNWyVY7XTC+SfOfJP/PmN7vw9dbDWL7jCI5V1eLmN1fi8/UHNI9Re8mjM/bWorBgNe2MEfT6YD0hzWve/+IBxmumIqO8vHgb/jh7vW3voPhJ2e7zwmgHMeFFtq8pR/JBu0kLBdG/IBfd2zRFxxZSTdeN53TC6+MHx76rJakTvwtmV1sH2NFGauUAYMp7a1AfEXDff9eZvqYaz3xZgskzv9deY0nisGuu3djR2kJB6HrsfjRphGf6JEd7n1WrVmHgwIEYOHAgAODuu+/GwIED8dBDDwEADhw4EBNkAKBLly747LPPMH/+fPTv3x/PPPMMXn/9dU+ESQP6qbRj5YwszGixTnbQJE3Z2Y3tq+374DRGHIlr6sN4+ssSLC45FDMXqKEbli36fKD8FNcxWujNZKwKAGLB0ykHSzFi4cXuhIl20ODbwOcA+dTcEryzYjc27rcnQlHcqcvff6uyDKsVNZqNGre9c+swW/Pt8BIIBDDnt8Px5ZTzEQoGYtrKMX3yAWg4EYuqKp6wWFEesfK8MH1eTn9v2cQ5t4N/LNyGT9cdwKofj6mWMZNtXHEOEwcqo42CmqbnjyaNQP+CXMPXcQpHRfSRI0dq3lRW9tyRI0dizZo1DtbKPMyEToxnbUzzYq1OdpAVR80LL0Z8K+rCAo5WqqfTFqM3IxO31+gztOLYpzeO6Alpesc3y0jFP64fiGAgEBcNmkTzkhIE7F/uyRLyx8szA7crekL8qGzP88KwG6UzzEZpKcroIy3+9JPeeOzTTbjzYmX4u1HEK19/8rtzcaK6Hi2aNDibi7P+qpqNRO+CtbWNGMszCALjPW743i43A/uOS7WsByuq8cHqvfjF4AJmVGIkIhgybdVqtDGJ5sW02cj4MfLah4LammIvCS6Axxx2/UjDbEs5y+JfVdp9stIZg57LmsFUAx2DMZ8X9X0fr92P37/bKDjHhBcLD4nVPsTY4Z9wRRxXeBVrirypeZHOulXNhKLtdvnuiB+l7XleGC9ko9mocVtKUJn3BQB+djbb+f2Wc7tgbN+2zPWzrJAaCsYEF0CeOI/tsCsxG9mcpC4iqGte8kTJ68IRAaFgADe/+R02HajAos0H8d87hkuO23W4Cle8uAy/HtEFd11ypul6iuvb+Nn+aKNWTdNwmGNylxIMoi7ghRGJD+/1Ph6G6bCrUtZPPi9ZqUoZ1m2rJk8IcZT6iMCdJEt9MINEcAHEmhfz6FUrNcXtO22M3m0bnfXinSOIF/HzUjMJid9PuyKopEnq5HWy9p6z2m2jw67Yb4Sdq+MvV/VVPXd+Tobjfgxq6y2JrysxG1lcVZppNpKXi1238VrR+7zpQEO7YZl7/jp3M05U1+P5BVtN1xEATtbW4zf/WYU5onXVnNC83H1JD3x934UKs/KVA6STnpCK4OtVSHgxAEvQYJqNBAEXcUaJuC+6sDUvbjtl6Q2M4mgOraUY6mWjiJHOoVHz4qDZyGORIWp8Mvlc3Du6B24e0Tm2zZOaF0H6vB6cs4FZTqwZtctHRJKkzkzsqgZbD1Yqo41O33+x7KX23rg9KIl9XiRmI1GZFJvMRhFBGe3FEmiq68J48ovN+G5Xo4CybOth09flQfyz/vX1TszbWIbdR0+K6mnuvFp9VCAAFLTIUrSfdrmZsSUKABJeEhq9Rcxi5QD8/uIzMJUjVb8nNC8MXwm327DebFg8E61TSQi2YV85evxpLp77aktsm5H7bYvZSOdOpnpUeyGnb4ccTLqwu8Rs5MVoI+YMm/EAxcKLExqkOOSoa5xJS7QXKsKLy2+0OPeMJNpILVTaQnUbzEaybYJS+zV9yXZMX7Jd4u8yYcZ35i/MUzdRFY4xwvhNm400VC9qtzIQAJqmN2rdG4QXf/RHAAkvlmE+a6FhpvHzQR10j/eA7IL8bL4kaE4jHmT0QojFDnlqJrrHP9uEcETAc181qnfV3nHW5ug1LJmNLGpevNyVeNXkpebbIEacAddKWK4Ycccfj/VfWHle1PIGuT0miTUv4smGWKgS+39Z8nkRBIWZTe4LBVhZ/NO5Y/5T9KPxk0O7j1J79nKBNqSyfhbgbK4rs5DDrgHYmhd1eKRYLwgvv7mgK8KRCAZ0zMVd760F4E5nJ+7U9IQXsVZebhrSPM6U5sX8Q9INlXY5lbsVvGk2Uj6riCBAHoNTL2pAdmWZFp+lPmLeXKl6fllbigq+coddvbq5QYbIYVecXt6JJHWA8j1n3X+7I8Ls4IPVe00dpyUrq4kk8q6pYXkAdtlPf3euqXo5ifd6Hw/DUumpJakD+DoML6yvkZedgUev7IOrBupripxEnJU4pDMwijUvvJFdgLGQwrg47HoyTS0fnjQbMcwDTK2aqCHYNYaJ+wLeVeWNIO9qoiZHifZCVfPibjsTm0dr6tiTDanPi/lrRViaF4Eh2DrQ9daFI1i65RCq66TJOKO336nHoDnBUtO8KIRhdZ8XvUzgbuC9GvkMrcbIM6jyrlsTb9zo6uqNaF7EwouBgUIr2kiOHT4vejdSt1NwW9+vQbMM6ytbOwFPrpc6ifBizygmflS8y4NYISqoiH+fWqJCt1uReKAU59VR0xpZyrArMHxeGGYjM5qX+nAEczeWqu6//rXlGP/GSrz3nXSpAaeVPKYWZpR9D2loXrwICS8GYCepY2heThfUC5e+ckA7W/IEOIEbM7VaseZF5/pinwI1IZGlLjVlNrKSpE5nv18cdlk8fEVvdG3dBE9c1Ue/sIvo+byYfboHK6rx+foDMbOl+EmqOZHbSVRTIe5nVIUXDzUzsVZC/I6KnUetr23E0LzIyplZwmG2KKxZTjgixEKrF5ccNHxuK2ibjRqQF5G3Ca8vuiqHfF4MwGofahl2ASBTLSX2aZ6/bqDlOjmFti+PMzMJsZ+AnsAg7rAN+bwYsBvFHHatRBs5vDyAmxS0yMLCe0a6XQ0JDeYB6TaWwDp3Y+MaWGaf76XPLcXxk3V4cGwv3HpeV0lfoPB5MXcJCfKWFHVwrRNdS00YdttsJKZP+5zY57SUIB6+ojeq6yJoI0oWZ3VVaWUbYLQLE8qxgyfUU0qLn3l+ToZkn9O3X2tSpvbsWQ67pHlJUJh5XpjlGv7NyWpI3S721H7j5sFo0SQNb04Y4lAtbcINh936xvurJ2OIZ01GVPTm8rzwHyPH6vIAhDEEKNPAs9rSXz7fLDnGDNEVqx//7AdsKTsB8UsTT82L+L3xohN1lG8fuAj/u2M4esgiVyaM6II7RnaTbLOiBVDN86LIhG7vM5JkdlbpkpzzeVHfpxUqLabBbGRblRyHNC8GYGteWA67jSWv6N8OWw9WYnPpCQDART3zsPrBUZ6aCRklGAjYvmouIDUbCQJw6EQNFvxQxiwrvnyNTHh5e/mPaMJa8gBaMxTldjtSvOvNZPySpM5PyAUHvUHKjqZ86bNL8eqNg0R1iJ/Pi3jG7+V+pV1uJtrl8vn4WfoZgtIkFFHJumsn4r5FKShJ/7UbvSR1zO2y7ykhfyWpI+HFALyh0noN1MsdTBStpFZO1V7cCUcEAdf/czm2HWTnYhALFvJFz6JZVc/p2kJxnKloIysZdnX26810vN9SvMWfP9mERTJ/A71nPndDKXqJlj0wS1lFdeyz3InciWSUUQdXLUHp5uGd8atzOtl+baex5LCrZjaCfJu9z0TczuKQ5keClqysKryc3n5Wu2xs3F+BS3vn441vdtpfOYegaZ8hGGYjjVDpxg1xbsk2cKquPvZZvMAa4JzqU6z+FgBVwQWQCi9qnbfV3A62hErrCSc+EGT9xILNBxnZVbWf4PMLtkpyj4jZsK8cizbzOV8+9NHG2Gc7NC+/GCxNXSA3pUQj1bRMVH3a56B7m6aW6xJvrCWpYy3MqHR6MSNgaLUl8TXl13fT50Wdhkp9NGkE1j1yKfJzMnCGj9oKCS8W8ZON0AgFzbOQkRpEyyZpyMmUhsQ6lWpc7HioJ/BJVLSC2hpTjG0q52NGpAgC3lnxIw6UVyt3cqN9rxK0+XgKnkFKzUT4k38sw4QZ32kK0izsEF6uH9pR8l0+oEedc404rPsFK0K9ALbwIn/Ctvu8iB+DQoBu+DceeV5G9cqT7NPrr1NCQWSfTnvws7M74IExPfG/O4bj4St6AwCe/Jn6op5uQsKLAbjbusO21XiQnhrCyj+OwjcPXBS3AVY8gOgNOOL9akVZsxFDaxuFBfxxNnthP170NS/W9nuNc7u3crsKCuwYpLYfMia8GEmcqIZ8AJebUqL+UvFwDo43VvzYG5LUybZFgE2yFcbtVojLNS/xXLcueqWstBBev2kw1zGsviUUDOD2C7phUKfmmDCiC9Y/cimukwnRXoF8XgzAaop+Ci0zSrZaErI4eMzrvfg8GVJZwoshnxcbOh+rmrneNvhixJP/3DIUXaZ+7nY1JNjhf2BUk6LwebFeBUXuk1i0UQJqXiybjWQPvWjHEUU5M++31iHi/mZO8X4cqao1fH6zRK/NumtqJiWeO+zVRJQAaV4MwU5Sxygn66quPrvBdl3YtaUT1XIG8Y+V/UanxDXpzIW/rNrQwDqHGZ8XK5g1sX119wV48+YhGNixueU6xJNAwHvhljwzYL0iRgWEWhsECkU0iFx44XDY9St2m41Y2L14pvx0X289HPvsvM+L/j75e+n3iTdpXgzAygfBE23UuVUTrHvkUjRN8+ftlv/GeOQq0OtWeDQvrEHLTJ4XK+gN5Gr16d6mqS8dLb0Iz2N8au5mPHqleqZgN0wzrIXzxKTEQqUT0GxkRQJmmI1UitmKlpAc83lxaOoX07wwOufovtRgENVgL83gR0jzYgDu5QEYx2ZnpNq2em28kf9Gp15AsXCoN1uWO+yyy8jOb7C3skXz4vceIgHgSUL376IfNfe7od1gZUAVk5rAZiNrCzNyal5U1zljb5+1cjdmydYsinKwohq/+tcK3Ws6tRBvo3CkJKphkq9g7/euyZ+qAJfgNRslAq7M5SQ+L9pFJYFJqmajxu2zVu7G377cYqg6dggvyUjAqfUjTGLHY7S8SrQNdVCESp922HViBWu3sXt5AHY5le2MHRXVdXjgw/Wq53p6Xgm2lBlz6rYTLRNYdJfc7OjUJDRekObFIqyXLJ5e5nYzuFODj8VVA9vHtsWriUu9WHQcdg1qXh74cD0OV6qvS6IWKm0VvT5YAPDzQR20CyUw8jB8J+D1bdB6b13RvOgsnMejefGbw3cUu/O8sFArw9p+soadByjKiep6zf1OE2vijNsWMxvJlo7o2baZsrCPIM2LAXhVfv4VXYD3flOIyup65GQ1DiryTjQePi+6axuJfV5Uz2ftSVjRvDRNT0FlTT3OO6M13l25W7Ps367pj/+u3mv6Wn4mHivZ8jaD+oigurChHQ64VlEKL+qh0ov/byT2l59C73b+FF4saV4EvndXbf0h1pHyhTblNM3gG0qd0nas+vHo6fMrEWLCS+PeL+86H21z+JZq8CqkeTEAqxNk9b0+VrwgFAxIBBcAuFiW9Mix9Tmgr01p3G8uVNoIVoSXeXedj8fH9cE9l56pWc7PWjo7iEfEA++ko6ZefYCyapqx4ykrM+w2fL9+WEMejmFdGpfD6NyqCYZ3817OHV6s+Aey1jFSK8eCqYXV6QuaprurB1i3txwA28cuWvcUkeblzDx/a10A0rxYJhkcMu+8+Ax0bdUE9/53naPXkWpetDuL5xdsbTzOQKi06rVtXpixfW4mfnVOJ1TWuKtOdgMjb0Q8FkHmfYy19REgnb3PLrPRgh/K8Lcvt2DShd10y8q7FrnPQnQ9pl8N64i+7XMkq9f7HStrGwH2m430IrrcFl60UPN58TukeTEA78KMiUZGagjXDC5w/DpGRAVxDgW1A93UvETRax9R4bdVU5VR04f87Oz2+oVOE49VtXnbgXyBTzFWQ6WjGrZb/r0KPxyowOSZa3SPkZsYxFqq939TiG6tG0LpA4EABhTkIiOVvZK6H7EYKc31zI08Ub2+oIlHhBeWzBe9F2kpiTXce+OO+wRmnpdkkF7ihCRJnQHBwUjUgBjxKsAs7AmV1t4fHdSW3DsSJ2vDePjjDbFBya88+tM+eH8Vnw9PHGQXbtOctvDiLYfdoV2UK6YnElY02iVlJ1BSdkK3nBFzs97z5xW2nB4vtDLsJprmhYQXA7B9XhKrQbiKwPyof5ggMB3h9GZflz23VHQO5X5boo04dXNN0lPQJD0FL98wyPI13SYzLYR//3ooHv90Ew5X1uDYyTrVslbNAzywHmNuViqOy+pVG1aPKPFCLpV4ODd7BTe7VTM+L7x9hdMubuwkdQ3/psTDRhtHEuvXuADJLvYh1mwZMfmoldTrcLQGVcCuxfUsn8KXXHBma8y/+wIMKMjVLBePxI2sx8hqXtV1zpmNzKAXKp3IuPlLWU/6neXaEYNhD+faiZmNSHhJXrzbPBMD6cKMxo5lhTJaneXYvfYJi/QE8lMwQzw0L7yri2uFQ1vVvJhpSXoZdglnYLWX91axM+tGsUNLawday9WkWFmq24OQ8GIAVoeXTGajR67oDQB4/roBjpzfyNpGYj4q3o95G8sU273gsKvFb0d2Q/tcf+da0EPPdyEe2gSzDrvi973eA2Yjvy4v4jfMdBu8fYXjPi8sh91oqHQ8HMziSGL9GofxhmztHjeP6IKSxy/DpWflO3J+8f21I/+JVdOT3Q67D58W/gDgvDNa4b7Lelo+v9+JS54XltmIUU4pvDR+dstsJL49ieZwqYWbKSjM9D1eXhwzWrW0lMRqP3ERXl566SV07twZGRkZGDZsGFauXKladsaMGQgEApK/jIyMeFRTF96FGaeOSdxBKT3FOTPHxLdWxT7boYW12p9Y1dwAUtV/3/Y5sc/JorHT+5XxUGVHH2P5qTp8VLwPJ2vrmdKLPEmduIg7CzNK20ky+byIGduvbVyvZ+a1j4eJmQ9GkjohMTUvjkcbvffee7j77rsxffp0DBs2DM899xxGjx6NkpIStGnThnlMdnY2SkpKYt+9kwiOZTZq/Hz7Bd3wq3M6okPzrDjWKTGxQ3Cwego7uiO1ppssA5F+qLjzdYi2pd++sxrfbDuCcQPacWpeGktZ9nkx8TsDAelQlCxtRs5z1w7AZ+sOxO16ZpqkVzQvrPct2o7J58Ugf//73zFx4kRMmDABvXv3xvTp05GVlYU33nhD9ZhAIID8/PzYX15enmrZeKIXKv3AmJ4kuNiEHV2BIbMRo+jikkOW66DWXSSL5kVP93Ky1vkMxNF28M22IwCAOcX7VRx2paHSUs2L84NT11ZNFNtI8xJ/R2V5v8FjRuL2eTFVI3608rxQtJEBamtrsXr1aowaNarxgsEgRo0ahaKiItXjKisr0alTJxQUFODKK6/Exo0bVcvW1NSgoqJC8hdPkmYMijN2aF7sOIdVxFpDcVtJsH5EFdb7IR6DT9Zqr9ZrB4aWBxAh9Xlx3mykFE6kqpdkjTaK98+Wdxs8ggmv8OJGjxSt2hX92wEAOjRPjCABR7vQw4cPIxwOKzQneXl5KC0tZR7To0cPvPHGG/joo4/w9ttvIxKJYPjw4di7l52xc9q0acjJyYn9FRQ4l8be/aEwibDD58UDqly1fjdZZ9FAg0AX1TIM6RyPTLGMUGlGKYXwAhvNRhwNWq6NI7NRA/F2G5A/K54waM+ESmssDzCieyt8/vvzMHfK+XGulTN4bv5XWFiI8ePHY8CAAbjgggvw4YcfonXr1nj11VeZ5adOnYry8vLY35492vH4VuB12CWsY0dXYKQ/4V152ChqzSPZ2828u87HukcuRYsmaY5fizdJncJhV1QmHj4NbC0VmY3ijbxtVNfqC65eSVLHyugt1tj1bpft6UUkjeDor2jVqhVCoRDKyqQ5OMrKypCfzxdum5qaioEDB2Lbtm3M/enp6UhPj8+idizbJ3UnzmCH1sRrZiMxyWICYP3KYABIDQWRGgrGxSTAakssYVUuvIjRWvfILuTCSQByU2NytBm3kXcbf/uyhF1QhJ7mpboujHBEcN7nhXGBied1dfiq7uCo5iUtLQ2DBg3CggULYtsikQgWLFiAwsJCrnOEw2GsX78ebdvGN1yOBat5JskYFHfscdi14SQOkSzjEOv9EM8O4+G4zKt5cdvnRWk2ks6jczJTHa+DV3Dz9ZBPej5dt1/3GD2fl1vfWoWf/GOZpXqZoX+HHDSPg3bTDRzXH91999246aabMHjwYAwdOhTPPfccqqqqMGHCBADA+PHj0b59e0ybNg0A8Oc//xnnnHMOunfvjuPHj+Ppp5/Gjz/+iFtvvdXpqupCCzPGD1tCpT3spZQs2VL1FqaMx21gtQOmz0tY3efFqtmIpznL74U8z8vE87tixc6jMcdLwhnkj4rn2fM47P5woAKdWjobjapw+U7g8clx4eXaa6/FoUOH8NBDD6G0tBQDBgzA3LlzY068u3fvRlCUPOfYsWOYOHEiSktL0bx5cwwaNAjffvstevfurXaJuMGafSVu03AXOyw+WgvtKS9o/XpGSBqzEetnira5lWGXhabmJQ5mI5ZAK7492RmpeP83fBprwjxy94B6Dn8W3mij6jpno+vkwkoidzNx8dyZPHkyJk+ezNy3ePFiyfdnn30Wzz77bBxqZRyWTTyRG4eb2LE8QLxp2SQNR6pqNUokn/Ml22yk9sUZmFo8psOuRp6XONggWdFGyaKhk9NHlI063jgZKl1xyvm8RslCYrgdxwmW056eWpwwh/9EF2OCbLIMSqz3IxBnzQvT54XRwpzMsMuDXBsXSOLeJT8nA4v/bySyXfDzkQsvdYwV6+XwmhVPOax5kZPIbg2eC5X2MvKZGUCaF6eId6SQtsaEF/7GkCxmIxbiITked0EQBIUmj8thV/SZx3SgXQf9MvKlZxoWZkzedtK5VZNYKP3/7oifuUwu2PI8Oy9ENgLK8SiRWw9pXgxAZqP4Ee++YMkW60sB6NEut3GB0WQxG13Usw0+W38AzTJScKK6QWUeb82LIEgX/QQ4HXYF9X1OwLoXSdJMdBnUKR7JDBswYiF8et5mNElP8ezaRok8PpHwYoAahsoveRW7bK4fWoB3V1pPFOiNrsAYah3Fp787Fyeq69E2pzEtdyKrc8VcNbA9WjRNQ592ORjyxFcApLPBeAzOEUHAVz8c1C1XI3fwdn15ACCx587exIi/3UuLtgMACru2dKo6lkjk8YmEFwOwNC8Jtsq4ZTjMw1z40WFXrZtgOR+2bJqYuRfkBIMBXNhDunq8dL0n451rSjBgaKbLzvPCWphRPVRaEBqOMWvG4WnPrEiRJJFxPYUZJUq9XR2fRRTCSgK3Hxp6DRAVXlqKkv4ksmSrxfy7zsebE4Yotttl+/Wh7MI10Dx7bX+M6ZOPX4/o4nyFPIpU82L8/dnw6Ghc1LONfsHTsAQHVvPSWh4AsJbrhedYluKFzEZuYPw513pleYDkkV1IeDFC1GF3YMfc2LZEWaHTKGfkNVPMqAH7stp6xQHOCDyC7FUDO+CVXw1CZlooDjXyPlqyy5DOzZnbM1JDaJbBrzQ2nWFXZ78ReEJpFdFGgWSON3IPM11PPPIA8aBMUudKNeICmY0MELWJZ6Q2Djw3De+MtJQgcyBPRuwy9/hQdknojsJOpA677DKhYACF3Vrhu13HVPfzwtsmtUKlAWsRRyyfmdRQAHWicyrMRgAyUml+GW/MTMDi4dBthkQWfunNMEC0gWaKhJe0lCD+fGUfXGhAjZ3I2LU0fETwn8rcZ9V1DV6fF637aSTUnHcw0kpSB1gboFiaF7kAplgeIAC8+Muz0T43E89fN8D0tQljmFlWJB4O3TxE36ffjuwGAJh6eU83q+MopHkxQFTzIlb504AlxS6zkYDTzpE+UsEkc04Os2jdMi1/mJQQ/73mdabUCpU2ch4WrAy9KcEggMZzsrRJfdrn4JsHLjJ93WTg1nO74PVlO207n5nH7DWz0X2X9cSUUWciLSVx9ROJ+8scIDozkwgvNGBJsC1KyD8yC2EQnjwvAWgLNkbMRrzmHqXPi31mo3rGzFz+ExTLA9DUiIt7Lu1h6/nMaF72l1fbWgfTiJpMIgsuAAkvhohGI4jNRtS9SNGSXX52dnvu80QEwXf3luRYPnjzvGjtM2I24ta8yGfPsrZs1GyUIvoBEQGIyLQvcgEsmRKM2Ynd98lryl4jzunJBAkvBogKL2KHXUKKVpRQqoGkOAL813knS+I5szx8RcPK8M9eOyC2TU27oJcaP2SgLdWZ1rxIMap5SQ1J6yhfI0cuvCRL1mW7SXThpUkav/CSTC2IRDoDRJ2yMkTqOD+G9DpJpoZgZ8RPwSPZtg1Bsos2E0Z0wQ3DOknU2Vr3TGufIZ8XTo2JXp4Xo06ZKaEAUCc+XnpCubCrXJiR4MHuSYMZs5GTGPl5yeTGQJoXA0Q7r7SUxgHaW83cfc5qn4Obh3dm7ksxGN7qN5u/v2rrDnI7vNrA8/pNQzSfvyGfF05JuD4iSEw78kHMqPCi0LzUa2teFCuNU4PiwnbhxWOduiAAN57Tye1qeA4SXgwQVRunizpgrzV0txEEAY/89CzmvpSQAbORAOq8kwA1GeSCM1vb5vPCazYCpH4tVjPsyoV1udlIPugqZRd6AXiw29rmRW36LefyZeROphZDwosBojOv9FSx8OK9hu4mWrfDkOYFfnTY9VuN3UehbRBhX7QRv8ZEbDqSN2XLmpewtsMu+UyZw+73zms9ekPaCL6yydSESHgxQF1M80JmIzW0Zi2GfF68kTbBEEnUb9iGfIAXo5nnxQGzESBNVCefmBjR4ADK9q5nNpKTTAORl/DzhDSZtHUkvBggpnkhs5EqWrfDSISIkdmGZ/BbfT3AuAHt0btttuHjtDQ2cowklxNHHCnMRhZCpVn1kLdv+fWoObmD1/p0QUguoYQXEl4MEJ3BSYQX0r1I0A6VNuKw678X1l+19QaZaSF8fud5zH1a2hVDmhcjPi8amVIXbj6IB+esR3VdWLWMmBSZsF5bLzMbyaQX+btDZkh38GKPztsUSspOOFsRD0Gh0gaIdmxppHlRRdPnxYDDrj9DpWmwsZNUjQyhRnxe7HLYfWfFbgBA66YZuHPUGbrnUpiNwhGIV7yQ/wY/tvlERJ5M0G28VRvvQJoXA0TVvmI7fW5WqlvV8SRa9mIjs2X40Gw0pHNzt6uQUKRpCLtGhJewWbORyrCx7/hJrnMpHXYjsuzC0t8gf3d81vzjSvvcTMfOHS9hgbT21iDNiwGiM7jUUBBv3DwYJ6rr0TbHuZfID/Rpn40N+ypi37UmLUaT1Pmt8777kh7o3LIJRvXOc7sqCYHW2ixGBGHWoohq1Gj4vOhtlzOkc3MU7zneWA+daCOl2YjvOslG19ZNUF3LZ7ozQ7y06bzXEQRqCyxI82KAqMNuaiiAi3rm4coB/Gv1JCrv3HIOXr1xUOy7ZrSRwSR1fiMrLYTfXNAN3Vo3dbsqCYFWJJKh5QEMrPhbqxEqHUWeiVeNnMxUbHx0NM5o09Ae5KHWcqdj/7X4+PLKDWejW+smeOmXZztqYrPS98yZNMLGmjRCJmklpHkxQKPwQjJflJysVIw+Kx8TRnTGZ+sOYHxhZ9WyRn1e6IVNbrTMRk6FSkt9XtjHfbx2P3rkN9M9VyAQQJP0FGSdXoVeLrzIf4G8mn5zWHeaMX3bYkzftgCcNblYObOR5In8yfBIrGVBo7AB6kVmI0LKw1echeVTL0aLJmkAgEeu6K0IgTWWpM5/ZiOStczzBSPiSMth11iotIHOX2B+VPD0vBLdU0V9WqL9xSOfbJScUxkqTbHSvKiN+2fmWdd6Wsmwa6QPMHIZagpKaBTmRBCEWCdoxHcjmRAPKDeP6KIIgdXyYZAjCP5zeqGZsnl6tc1WCLepGu+ZEUHYiMOueOCyarmMVjHaX+w5ekpyTqXmhWbYvKjdKTvewbj5vHBqVMjnhQ0JL5yIne1I82KOlk3Sucv6sR+nDsYa8kee7kKotLTdWWuEcs2LHpEIcP3QAkvXTBbUTHp2vINWnroTmhcfdoVxgUZhTsTZMbVmhIQ6rZqlcZf1w9pGF/ZojSv6t3O7GgmLtsOuEc0Lf/f/7fYjuPiZxVix44hlATo6kKn57sh9ugQImDLqTGsXTRLUno0dfnKWzEYGei0j1kzS6ioh4YWTunrSvFildVN+zUtdWPC8w+5ff94P55/RKvbd49X1HXYJL0YWVHzjm53YfqgK17623PKMN9p+1czMeg67NOVWR91s5ODJOQgGtR3NzVxIEPyX8yoe0CjMiXg5e2PJ1ogozbP4NS8rdx5F+ak6B2tjD+KBiWZH1pCbAuzK82JE8yKtj6nDRMdrO/izHHapBfHhrNnImuZl3l3n4+bhnfWvQw67liDhhRNxjhevawS8ipEIET8QQECyfg01C2vI+3K7MuwaijaS1Mce1YdaXeXCriCQsoUXJ++TlRXtAwGgS6sm+P3F+stH8K6RRW2CTVyEl5deegmdO3dGRkYGhg0bhpUrV2qW/+CDD9CzZ09kZGSgb9+++Pzzz+NRTU2iYdLyxdYIY3x934VuV8E2AgGp/xPJLvZil9nI6GrQURx3Gpf9BIWvBTUoVdSejXzJBVPntnBs9Oo8EalzivcbPzERw/GR+L333sPdd9+Nhx9+GN9//z369++P0aNH4+DBg8zy3377La6//nrccsstWLNmDcaNG4dx48Zhw4YNTldVk1qR5oUwT0GLLPTI00/w5QcCkGZ6JY2cvWiZjeKiebFJeFF1LpV9J58XftTMRmLtbptm/D52YuzI82Kna0FD1gjl+YyknkhEHP/1f//73zFx4kRMmDABvXv3xvTp05GVlYU33niDWf7555/HZZddhnvvvRe9evXCY489hrPPPhsvvvii01XVpElaCn52dnuM7UfRJVZJlDE+EAjIfF4IKzRNkyb81s7zwt91rdtbbrgugUD8F86LkM8LN3L54tcjuqBr6yb4xeAOtp/bGKedtElD7ziO3uHa2lqsXr0ao0aNarxgMIhRo0ahqKiIeUxRUZGkPACMHj1atXxNTQ0qKiokf06Qn5OBv/9iAKb9rK8j508mEkVDEQCQSj4vtvHWLUPRrXUTzJgwBICe2cjZuoQCAcfNRkqHXSA9NRT7nppCDUoN+aN56IreWHjPSDRNT1EtY/7s/AQd0bxQtBELR9c2Onz4MMLhMPLypKvs5uXlYfPmzcxjSktLmeVLS0uZ5adNm4ZHH33UngoTcSFR/HYDAVm0EfUwlhjYsTkW3DMy9l3LYTctFFLdZwd2OJdHhR/ebLARQUBOZiqm/awvggEgK42WnlNDPdrIjjwv5o+NXt/u4ATqWZT4Xrc1depUlJeXx/727NnjdpUIHRJljG+INkqQH+NBtAaAZhnODux2al54w3qjxa4f2hHXDuloz8UTFDUBw4630cpzd6I3EEATIxaOCi+tWrVCKBRCWVmZZHtZWRny8/OZx+Tn5xsqn56ejuzsbMkf4W38mg9l3IB2GDdA5PMUMLZSNmGcBfdcwMyZkZ2Z6uh1U4IByz4vesfLxyNa24ifm0d0BgCM6tVGst2OaKN4LcxoFX/2ovbhaM+blpaGQYMGYcGCBbFtkUgECxYsQGFhIfOYwsJCSXkAmD9/vmp5wn/4dRLx3HUDMbhzi9j3QABom5PhYo0Sn26tm6JP+xzFdsc1LyEbNS+85Uh24eaeS87EzInD8OIvz5Zsd31tIydECpU1apO9uThuVL377rtx0003YfDgwRg6dCiee+45VFVVYcKECQCA8ePHo3379pg2bRoA4M4778QFF1yAZ555BmPHjsWsWbOwatUqvPbaa05XlYgTPpVdFAQA5GVn4M0JQySOgoS9sNqL00t0hAIBy4ODUWGENC/8pISCGN6tlWK7PWYjb2leBI7zvnHzYPx6xioM79bS/gp4FMd73GuvvRaHDh3CQw89hNLSUgwYMABz586NOeXu3r0bQVHExvDhwzFz5kw8+OCD+MMf/oAzzjgDc+bMQZ8+fZyuKhEn/Gy/FVc9+jsu7NFGpTRhB240l1AwYGkQMwMJL9ZhaenkZKWFcLJWPbutJZ8Xl7q2i3rmYfnUi9GqKf8SLH4nLtPFyZMnY/Lkycx9ixcvVmy75pprcM011zhcK8It/OzjKlYL+/hnEDqEgtY1LzFUTiTPBUKii3UKWmThizvPQ/OsNFzx4jJmGV3hxcraRg5ILxmpQaY5Sr4lP8lM2ORtSMQdr2tepv9qkNtVIDhIdzDDaIPmxdo5hNi/KtlgA0DR1Iti362E6BKN9GqbrTmQZ6Zph9lbWtvI/KGq/HP8YJopMSDhhYg7XlePa/mvSM1GcagMoXqfnfR7aVh+wJ52qpWvpm1OZuxzvM1UyUpWqrbBwUvRRu/ddg4GdmxOfQ0DEl6IuFNncqG8eKFl1gpIPlOP4iZOdui2aF5OH3/vZT0NlSecRVfzYkV4sblP0NJSJ3tzIeGFiDu19R4XXjSkF9K8eAc7cnqoYUe0UZT2uZm4pHeeYrv8/GGyG8WFM/Oaau63MrdyqklSV6OEhBci7nheeCGpxFOozWaddPy2Q/MihqeuXjenJgqdWjasn/XJ5HOZ+8MWnF7s7jqi52NpYJK9l6LkFETc8b7wor5PEm2U7L1HnOjehj1TdlTzYnOodMjPIXYJyEiN9AZWNGC2m41sPVtiQZoXIu7UetznhTcainxe4kOf9jm4on87xXYno9ZSbAiVFkcZseoql41I8+INwrLH0D43E6khzj7BKc0LY1+yZ/cm4YVwlL9e3RcA8ODYXrFtfta8gHxeXOH5awfgqav74cu7zo9tc1KZEbTZbBTiaCwku8QHPY1aRKZ5mXXbOVj14CVc53aqSbKaz+s3DcaI7i3x/m+Sc+kcMhsRjnLtkI64vG9bNMtoXEivTj618Rgsc8TvLz5DsY1kl/gRDAbwiyEF0m0OSo/BgA0LM4oO1xK0OjTPxN5jpzCK4dRL2MPahy9F/0e/5CpbtOOI5HsoGEAO50Kg9msDA6f/rzxv9zbN8M6t59h8Pf9AwgvhOGLBBXDebNSiSRqOVtWaPp7ln3DnaeFFEipNqhdXcdyNxKKM3T63MYeLVgTb7N+OwDfbDmNM33xrFyQknNe9FT5csw+tmqZzCx8AsHDzQcl3I6+57aILdTGqkPBCxB2nQ0KdcI5knZL6FXdxUni0mqJu8oXd8VORn46Wlqh1s3SMG9jewtUIFo9eeRZ6tc3G5f3aSrYbNc8Z0fDxlk0NBbg00NGzkRCjhIQXIuHg8S/QgtUBRQdK8YBJHYq7BB302AsEzPugdG/TFP83uodkm9U2SRinWUYqJp7f1fJ5DD05zsIZqSHUhev1T0ftRhVy2CU8wdDOLWw7l1XNC4W1+gMnfV4C4Pd5GdVL6qvCcgh1UtAiHMaI2YizbGaqdpZfQh96pQhPkKGTstsIVgcK7uUBaFbkKk4nE+TVvPCEOLPq6m239cRFfN/nTBqBp37eT7N89Nmd272V7rl5W2QWZ39HZiN1SHgh4s7tF3QDAPx2ZLfYNjuVHSkWpRctoYQ6Ee9gx7O49dwuABryukhPzi9cyH24WMdR1mbv0DO/WezzgIJc/GJwgUbpRgHilV+drbviPO+EJkND8/LajY3XaMzzQu1HDvm8EHHnvtE9cPXZ7dGtdVO8vHg7AHs7d6uCkJbZiMYg72BHm/nj2F6YeH5X/GPhVry9fHdsewD8qzzzaF7IFOk+n/3+XGzcV8FcZ0qLaDtrlpGKy/poR4TxPmUt4aWJaFX7qNBC/Y4SEl6IuBMMBnBGXjPJNjvfTasDBfUT/sAOeSAQCCAvm52pNCqSiCNDMlKDqK6ThvorhBeGLEOaF/c5q10OzmqXY/g4I4+O9zmnpahrh6mt8EFmI8IT2Ok/ovfyn6GyVg7P8aS+9Q52dvJM+eP0NrEwzJox84T+k+LFvxjpm3iLKsyU4n2ipQi0lgdIdkjzQngCOycbepoXvWtpuczQpMg72CnwysUPQRBi0UYNPlQN2pZ0xoxZvggxS5Qhs5F/ceKd12oPWqkagIbIzGuHaPvpJAMkvBCewM6+vXlWms619IQbGmj8gJ1tRrlIYuNn8UyYFeIa5vB5oTblX4xo+HiLap1TrJVhFXvkp2ehd7ts7jolKmQ2IjyBnSYAq2nW5TV59UbtCAPCHez1DZAKIBFBiAk0TdIa53gsXwVFtBFDmAlRT+tbjLQysVlZq3lqmY1YWhlpigYDFUpg6JUiPAFrIHp8XB+mml6L8YWdkKoTKq03Cxbv7t8hB6PPojVnvIiTmhdBAJ6eV9JwHVFzYqV0lwsvF/VURrMw87zQMtK+wNDaRqKyWodprXUlFl4o2kgdMhsR3oDxcgow+dLq+bRY6AhI/e8dbPV5UQgvAjaXngAA7Dl6Kra9pi6sOFYcbfTXq/viygHKdYoogsS/mF3bKKCxxoTWchEhhtmI+h0lJLwQnkCtg8hIDSlCU7UQBB6fFu1ziFW/NDf2LrZqXhRmI3Y51oroYs3LtUM6Mo9jtclsAysdE/4goPJZjpbDLkt4EdM2hx3an2yQ2YjwBGqvctN04/K13phmRbih+Y93cFKboZZ4rqZeKbzwJalr/Py7i7pjUKfm+NPY3qbrR8QPeTv73x2FqmUlZiON5qklvKj5wyy45wJ89vtzkasTkJAskOaF8ASsFz0A48KLAEER6pydkYKK6sYVXPWGPK3ZE2lvvYOW34BRtKKNxLCEF548L2K1/1UD2+OeS3tolCa8hPyd10p0x2ve4Q2VFmuBu7XWzk+VbJDmhfAEagvXmdO8SM/Vs60srFDfbiSpg9a5Cfew80nI5Y+aeqVvCwC0bCKd9b4+frCqoCNGPFhZXXuLsJ8esozfYuR9E+8ERq2veOKqPtqh0owkdYQSeosIT6CmKm1iUHjJyUzVT0JHHUJC4KTZ6GSNVHj5102DMbxbSzzy07Mk21s0TTOcYZdkF+8xc+IwnKWSO8V0K1M58IZhnTRD50MSzQuhBr1GhCfIUXFebJqhFF6+uPM8xbZxA9rh4p5tcPsF3fRDoXXqIsnVIN9HvYlnsFMIkDvsVtXUS75f3CsPMyeeg/a5mZLtAfCZjY5W1cU+a61rQ7hDy6bpuGqgMkoMUL7zvNpXsw67FFnEB/m8EJ6AJaRAENA0TbmdNeO+elAHnHdGawDKTkPunW9k+QCKNvIutmpeZA+6sraeWUx+yWAgwJWvpX1uYxts3TTdcPUI51F7jHJhgttsZNJhV7yP+h91SHghPIF8IGqbk4FxA9tLZqxaiFWt8nNdO6QAHxXvV90vR2svzYm8g7NrG7HLsfwfeJYHuHpQB5yqC+Mn/drRzNqj8DxHI2hpaLTyvIjlGspjqA7pLwlPIH5Jx/Zri2/uvwjNMlIx8fwuuLBHa0lZ1nuvlhvhzovPQKrMwExrGyUG9mbY5RslWJoXRuoXBVlpKbjt/G5oJzM7Ed6BJ+Qd4J/AaHUjWpFy4n28dUpGSHghPEf5ybrYC5yVloI3JwyV7Ge99mLhRdwvBALK8rpmI619JNd4BjvNRrxDBGs2PfK0cC33hyH8Ba+cYMfkRlvzQsILD44KL0ePHsUNN9yA7Oxs5Obm4pZbbkFlZaXmMSNHjkQgEJD83X777U5Wk/AA4ne5bwf1PApqSGcy0jwJXWX5EYz0PadqpVEn2RmUFdUr2Cq8cI4Rcrk3GAjgkZ+ehUeu6I3/3THctvoQ8SfCcLx+6uf9FNu4NS8a+0Ih9b1iwYZkF3Uc9Xm54YYbcODAAcyfPx91dXWYMGECbrvtNsycOVPzuIkTJ+LPf/5z7HtWVpaT1SQ8wqL/G4n5m0rxq3M6aZZjmo0CbM1LMAC0aJKGh37SG3/+dFNDWQMOuxXVUp+bwm4t8atzOuJMjbwQRHxw0F+X+5qBQEMuoptHdLGvMoQrsHxeRnRvZfp8WhoaTc2LSKVAwos6jgkvP/zwA+bOnYvvvvsOgwcPBgD84x//wOWXX46//e1vaNeuneqxWVlZyM+nlXyTjS6tmuC287txlFS++FKfF6X/S6eWjQKwvsNu4/6KU9Kok0AggMfH9eWoI+E0bvi8yNseLbiYOLAi3llPlz9JnTq8GXbJbKSOY2ajoqIi5ObmxgQXABg1ahSCwSBWrFiheew777yDVq1aoU+fPpg6dSpOnjypWrampgYVFRWSPyKxYXUe0pTa4rIBxTG6A45o9ynGKsKEN3DF54WheSESA5bZiLlsiWzjT/urT8TV0Gq7IRJeuHBM81JaWoo2bdpIL5aSghYtWqC0tFT1uF/+8pfo1KkT2rVrh3Xr1uH+++9HSUkJPvzwQ2b5adOm4dFHH7W17kT8sToGSBx2GSK5WJuild0SoAHJL9i5thGv9CIfdChbc+JgRlBo0SRNks5fjFY/opZRfFSvNpLjeJaeSFYMa14eeOABhUOt/G/z5s2mK3Tbbbdh9OjR6Nu3L2644Qa89dZbmD17NrZv384sP3XqVJSXl8f+9uzZY/rahD9gRxuJ9zPCpo1oXghfYK/mxWyYLLWlRIHla6eXTTcYUC+j5fPCErxfHz8Y/xw/WHIcvzkz+TCsebnnnntw8803a5bp2rUr8vPzcfDgQcn2+vp6HD161JA/y7BhwwAA27ZtQ7duSn+I9PR0pKdTxspkgtUpsPxcAHbHojdjDwAY2qUFVu48itbNqG15FTvFBv4wWe3vhH9pl5uJoZ1bYOWuo7FtuhZmkw2A5fMSDCrPR6KLOoaFl9atW6N169a65QoLC3H8+HGsXr0agwYNAgAsXLgQkUgkJpDwUFxcDABo27at0aoSSYRkMTOGICPuEniS1L34y4F4dckO/HJYRzurSdiIGyYbpdmIpJdEIj1VaozQe7rBgLqAo7k8AGMnS4DmWTcrWXHMYbdXr1647LLLMHHiRKxcuRLffPMNJk+ejOuuuy4WabRv3z707NkTK1euBABs374djz32GFavXo1du3bh448/xvjx43H++eejXz9lvD2RnBhKUhf9V9RZaKRYiB3TplkG/vST3ugmyxFDeAc7fV7MaudJdEksLu4p9dPUe8DBgLphyWy0EQBc2jsPZ+Y1xaBOzbUrkMQ4muflnXfeweTJk3HxxRcjGAzi6quvxgsvvBDbX1dXh5KSklg0UVpaGr766is899xzqKqqQkFBAa6++mo8+OCDTlaT8BnsCADRZ4bPi0TzYiDPC+FdetiYa4c7NTwjSR2RONxY2BnPLdiK4ycb8jvp+7wENDQvGhFFOn3Qa+MHQxAEWqpEA0eFlxYtWmgmpOvcubPEIamgoABLlixxskqERzHyjjL9WFSS1EXLGgqVJnzBhBFdcKoujAvObI2rXv42tj0jNYjqOo4Fh0xgdoVhwh+EggFc3rctZq7YDYDH58XcdXi0hiS4aEOrShOeICfTWtr9oFT10vgxpnkRm434k9QR3iUtJYgpo86UbEtPCSInMxXVdTWGzqWld7lB5PckH3NofEk8WGZn9bIB1f5CSz7h9Xkh1CHhhXCVp37eD0u2HMIvhhRwH8NOUifazwpoFWteKM9LwhLQCF3VQi0k9bohBXh8XJ/G80OueaHGkmgY0cxqOexqiT6sPC8kuxiDhBfCVX4xuAC/GMwvuKgRUDMbRTPsisqS2ShxCUDdB8EMLZumqYbhA5SkLhER+z+l6Mx0zPYltiZYTFIcXVWaIOIFS2ABREKLAZ8Xkm38R3ZGwzzs7E65ugt7slBT2csHL0WeFzIxJhw1In+pzLSQZtmA2VBpGnktQ5oXIiFokt7YlKWCzOl/JcsDkM9LovHR5HMxa+Vu3HpeV7RokoYhnVtg28FK/GH2eq7j1VT2qbK4ennboAl04lFT3yi8yJ+/nIaJELuM1pGsCRRl0zUGCS+ErxnTJx93jOyGjNTGGRLLd5eijRKbLq2aYOrlvWLfh3ZpgX3H1Rd0ldOqaRpze0pIW/NCcm7iUVPfuBirnk+Tdqi0+nF65ihCH7qDhO8QT1D6dchFvw65kv1SHwWWz4v2+Um2SQzUJrKZqUpTwP2X9WSWlTtWypsGCcKJh1jzokeDg7hxyGxkHbqFhK/p0qqJYhtLUJFk2GVIL91aN56HhqPEplPLLMW2lk3TmetYpcpGGbmwQm0l8ag1ILxoaV60BFsSeq1DwgvhS/53RyEe+klvjD4rT7FPmvNFmaSOpQqe9rN+mvuJxMHIwJEi93mhDLsJjxHNi5b1R6tlsCZQ5PFiDPJ5IXzJoE4tMKhTC+Y+ps+LaD9LZSveRsNRYmPE3SBVHm2kyPNiR40IL2FU82KECSM6A9APGiD0Ic0L4Qu+eeAi7rLMtY1EfQUruyXNoJMHtWfdIkvptJuaIhNWZD0mRaYlHtV1Yf1CpwloZNiVa3AzUoN4+IqzAJDwYgckvBC+wMjyAexFGpVOvGLEnQnJMYmBmsOu2uN98ZcD0b8gV7JNkedFfi7qQRMOI2ajlCB/UkQjS5QQ+tCrR/gCI686w+VFAmvWI56Nk89LgqPyfM/Ia4aPJo2QOIEr8ryQw27CY8RsFAqq697kzUwQebWwMuxSmhdjkPBC+AJxRyDouLZJBBHG8SyNbQYjfJZITPQ09uJQaj3NC5kbE4/zz2wFAGifm6lbNjUUUJ3saGfYbdzZsUUW0lKCGNG9pbGKJjnksEv4AiO+BVlpyoR14nwdrFlPt9ZNcP3QAuQy/B6IxEKvJYlTwsujjRSh0iS7JBx/+klv9Gqbjcv65OuW1RJeFc7dou9i7c7nd56H1FAA6Sk0gTICCS+ELzAySGRnNPrHhE/3EeKOgdXhBAIBSbg0kbjoaUvEwq88zwuFSic+zTJSMWFEF66yrNWh1RBrjE/VNjoFN0kLkanaBGQ2InyHnm24aUajTF5VUw8ASE9pbOrkLJccqDUTvRV9MyRmI2orhDopoaCp5QGqDSxBQLAh4YXwBUbeb/Fs+UR1HQAgTSS80HL0yY2u2UgsvOhk2CXNS3KTEtQIldY4rlfbbGcqlESQ2YjwBWbzaZxgaF6I5EZP4BALumk6ZiOSXZKbkEaotFyjItYYd2vdFLN/OxxtsjMcrF1iQz064TuMRBSeqD4tvIhm0+FIBNcOLrC5VoRf0MuwKxZeFMsDyMqSWSm5SdEKlZZ97ynTtgzs2JwroolgQ8ILkdBUVis1L3VhAX/5WV+M7dfWrWoRLqKreRFpW+R5XsTH3nZ+V/JXSHJCBhY3evmGs52tTJJBwgvhC8Qz3OZZ+tl2h3ZuWPfoF0M6KI6vDwsIBQPo3rqpzbUkvETP/GbM7b85vxsAYNyAdsz9YkFXkedFNCDlk8o/6dHSvMn3kJbFXsjnhfAFwWAAcyaNQE1dmCsXy39uHYq9x06h22kBRTxDro9ETm9zpq6EN+jTPgddWjXBzsNVku292jbDxkdHS0KixWiajUSNhhIbEqFQAO2bk1DiBiS8EL5hgGzdGS3SU0IxwUVO7enkL5SOO/H55/hBGPX3pWjTLB0HT9QAaBBAmqSrd31Ss5G6cjozjRTXyU5KMIAbhnXCrsNVuKBHa8k+Mik6CwkvRNJRHyapJVno3qYZ1j58KcoqqnHps0u5jkmVmI3UB6AMyoia9ISCAaSlBPHolX0U+8Qt5xXyd7EdmjoQSUfdac3Lzwc1+MPQmiKJTU5mqkSbojcfFssr8jwvYshsRGj6vIh2jelLwQF2Q5oXIumoO615KWiRhQ2PjkYWDUIJj1iboqfNF0cUyaONxJDwQmgJt2ZzUxF8kOaFSDrqw42LojVNT6GMu0lAqugZ6/k6BSTCi7KLjK4I3LsdZUlNdrSWGiGXF2chzQtBEAmPWAipi0Q0SsrMRgzBtvihS1BbH0FOpn7IPpHYhGji4xqkeSGShgfG9ET73ExMueRMt6tCxJlMUVh0uo6jrdhsxIoYaZaRipZN0+2rHOFbKMOye5DmhUgabr+gG26/oJvb1SBcICM1hNduHIT6iKCrMaHxiOBF0+eF7EaOQpoXgiCSgkvPysflHFEfNOgQvGhpXv56dV9kpYXw4NhecaxR8uCY8PLEE09g+PDhyMrKQm5uLtcxgiDgoYceQtu2bZGZmYlRo0Zh69atTlWRIAhCgd7aRwQRRcvnpV+HXKx/ZDRuPa9rHGuUPDgmvNTW1uKaa67BHXfcwX3MU089hRdeeAHTp0/HihUr0KRJE4wePRrV1dVOVZMgCEICmY0IXuTLR8ghh17ncEx4efTRR3HXXXehb9++XOUFQcBzzz2HBx98EFdeeSX69euHt956C/v378ecOXOcqiZBEISEn/Rvh3Y5GfjZ2e3drgrhcc7MYy/+STiPZxx2d+7cidLSUowaNSq2LScnB8OGDUNRURGuu+465nE1NTWoqamJfa+oqHC8rgRBJC5N01Ow7P6LKP8PocrHk0dg+6FKnNOVsnO7hWccdktLSwEAeXl5ku15eXmxfSymTZuGnJyc2F9BQYGj9SQIIvEhwYXQol+HXFw1sIPb1UhqDAkvDzzwAAKBgObf5s2bnaork6lTp6K8vDz2t2fPnrhenyAIgiCI+GLIbHTPPffg5ptv1izTtas5z+r8/HwAQFlZGdq2bQxnLCsrw4ABA1SPS09PR3o6JYwiCIIgiGTBkPDSunVrtG7d2pGKdOnSBfn5+ViwYEFMWKmoqMCKFSsMRSwRBEEQBJHYOObzsnv3bhQXF2P37t0Ih8MoLi5GcXExKisrY2V69uyJ2bNnA2hIDDVlyhQ8/vjj+Pjjj7F+/XqMHz8e7dq1w7hx45yqJkEQBEEQPsOxaKOHHnoI//73v2PfBw4cCABYtGgRRo4cCQAoKSlBeXl5rMx9992Hqqoq3HbbbTh+/DjOPfdczJ07FxkZGU5VkyAIgiAInxEQBL0F4v1FRUUFcnJyUF5ejuxsWrKeIAiCIPyAkfHbM6HSBEEQBEEQPJDwQhAEQRCEryDhhSAIgiAIX0HCC0EQBEEQvoKEF4IgCIIgfAUJLwRBEARB+AoSXgiCIAiC8BWOJalzi2jamoqKCpdrQhAEQRAEL9Fxmyf9XMIJLydOnAAAFBQUuFwTgiAIgiCMcuLECeTk5GiWSbgMu5FIBPv370ezZs0QCARsPXdFRQUKCgqwZ88eyt7rIHSf4wPd5/hB9zo+0H2OD07dZ0EQcOLECbRr1w7BoLZXS8JpXoLBIDp06ODoNbKzs+nFiAN0n+MD3ef4Qfc6PtB9jg9O3Gc9jUsUctglCIIgCMJXkPBCEARBEISvIOHFAOnp6Xj44YeRnp7udlUSGrrP8YHuc/ygex0f6D7HBy/c54Rz2CUIgiAIIrEhzQtBEARBEL6ChBeCIAiCIHwFCS8EQRAEQfgKEl4IgiAIgvAVJLxw8tJLL6Fz587IyMjAsGHDsHLlSrer5CumTZuGIUOGoFmzZmjTpg3GjRuHkpISSZnq6mpMmjQJLVu2RNOmTXH11VejrKxMUmb37t0YO3YssrKy0KZNG9x7772or6+P50/xFU8++SQCgQCmTJkS20b32T727duHX/3qV2jZsiUyMzPRt29frFq1KrZfEAQ89NBDaNu2LTIzMzFq1Chs3bpVco6jR4/ihhtuQHZ2NnJzc3HLLbegsrIy3j/Fs4TDYfzpT39Cly5dkJmZiW7duuGxxx6TrH9D99k4S5cuxRVXXIF27dohEAhgzpw5kv123dN169bhvPPOQ0ZGBgoKCvDUU0/Z8wMEQpdZs2YJaWlpwhtvvCFs3LhRmDhxopCbmyuUlZW5XTXfMHr0aOHNN98UNmzYIBQXFwuXX3650LFjR6GysjJW5vbbbxcKCgqEBQsWCKtWrRLOOeccYfjw4bH99fX1Qp8+fYRRo0YJa9asET7//HOhVatWwtSpU934SZ5n5cqVQufOnYV+/foJd955Z2w73Wd7OHr0qNCpUyfh5ptvFlasWCHs2LFDmDdvnrBt27ZYmSeffFLIyckR5syZI6xdu1b46U9/KnTp0kU4depUrMxll10m9O/fX1i+fLnw9ddfC927dxeuv/56N36SJ3niiSeEli1bCp9++qmwc+dO4YMPPhCaNm0qPP/887EydJ+N8/nnnwt//OMfhQ8//FAAIMyePVuy3457Wl5eLuTl5Qk33HCDsGHDBuHdd98VMjMzhVdffdVy/Ul44WDo0KHCpEmTYt/D4bDQrl07Ydq0aS7Wyt8cPHhQACAsWbJEEARBOH78uJCamip88MEHsTI//PCDAEAoKioSBKHhZQsGg0JpaWmszCuvvCJkZ2cLNTU18f0BHufEiRPCGWecIcyfP1+44IILYsIL3Wf7uP/++4Vzzz1XdX8kEhHy8/OFp59+Orbt+PHjQnp6uvDuu+8KgiAImzZtEgAI3333XazMF198IQQCAWHfvn3OVd5HjB07Vvj1r38t2fazn/1MuOGGGwRBoPtsB3Lhxa57+vLLLwvNmzeX9Bv333+/0KNHD8t1JrORDrW1tVi9ejVGjRoV2xYMBjFq1CgUFRW5WDN/U15eDgBo0aIFAGD16tWoq6uT3OeePXuiY8eOsftcVFSEvn37Ii8vL1Zm9OjRqKiowMaNG+NYe+8zadIkjB07VnI/AbrPdvLxxx9j8ODBuOaaa9CmTRsMHDgQ//znP2P7d+7cidLSUsm9zsnJwbBhwyT3Ojc3F4MHD46VGTVqFILBIFasWBG/H+Nhhg8fjgULFmDLli0AgLVr12LZsmUYM2YMALrPTmDXPS0qKsL555+PtLS0WJnRo0ejpKQEx44ds1THhFuY0W4OHz6McDgs6cgBIC8vD5s3b3apVv4mEolgypQpGDFiBPr06QMAKC0tRVpaGnJzcyVl8/LyUFpaGivDeg7RfUQDs2bNwvfff4/vvvtOsY/us33s2LEDr7zyCu6++2784Q9/wHfffYff//73SEtLw0033RS7V6x7Kb7Xbdq0kexPSUlBixYt6F6f5oEHHkBFRQV69uyJUCiEcDiMJ554AjfccAMA0H12ALvuaWlpKbp06aI4R3Rf8+bNTdeRhBci7kyaNAkbNmzAsmXL3K5KwrFnzx7ceeedmD9/PjIyMtyuTkITiUQwePBg/OUvfwEADBw4EBs2bMD06dNx0003uVy7xOH999/HO++8g5kzZ+Kss85CcXExpkyZgnbt2tF9TmLIbKRDq1atEAqFFNEYZWVlyM/Pd6lW/mXy5Mn49NNPsWjRInTo0CG2PT8/H7W1tTh+/LikvPg+5+fnM59DdB/RYBY6ePAgzj77bKSkpCAlJQVLlizBCy+8gJSUFOTl5dF9tom2bduid+/ekm29evXC7t27ATTeK62+Iz8/HwcPHpTsr6+vx9GjR+len+bee+/FAw88gOuuuw59+/bFjTfeiLvuugvTpk0DQPfZCey6p072JSS86JCWloZBgwZhwYIFsW2RSAQLFixAYWGhizXzF4IgYPLkyZg9ezYWLlyoUCUOGjQIqampkvtcUlKC3bt3x+5zYWEh1q9fL3lh5s+fj+zsbMUgkqxcfPHFWL9+PYqLi2N/gwcPxg033BD7TPfZHkaMGKEI99+yZQs6deoEAOjSpQvy8/Ml97qiogIrVqyQ3Ovjx49j9erVsTILFy5EJBLBsGHD4vArvM/JkycRDEqHqlAohEgkAoDusxPYdU8LCwuxdOlS1NXVxcrMnz8fPXr0sGQyAkCh0jzMmjVLSE9PF2bMmCFs2rRJuO2224Tc3FxJNAahzR133CHk5OQIixcvFg4cOBD7O3nyZKzM7bffLnTs2FFYuHChsGrVKqGwsFAoLCyM7Y+G8F566aVCcXGxMHfuXKF169YUwquDONpIEOg+28XKlSuFlJQU4YknnhC2bt0qvPPOO0JWVpbw9ttvx8o8+eSTQm5urvDRRx8J69atE6688kpmuOnAgQOFFStWCMuWLRPOOOOMpA7hlXPTTTcJ7du3j4VKf/jhh0KrVq2E++67L1aG7rNxTpw4IaxZs0ZYs2aNAED4+9//LqxZs0b48ccfBUGw554eP35cyMvLE2688UZhw4YNwqxZs4SsrCwKlY4n//jHP4SOHTsKaWlpwtChQ4Xly5e7XSVfAYD59+abb8bKnDp1Svjtb38rNG/eXMjKyhKuuuoq4cCBA5Lz7Nq1SxgzZoyQmZkptGrVSrjnnnuEurq6OP8afyEXXug+28cnn3wi9OnTR0hPTxd69uwpvPbaa5L9kUhE+NOf/iTk5eUJ6enpwsUXXyyUlJRIyhw5ckS4/vrrhaZNmwrZ2dnChAkThBMnTsTzZ3iaiooK4c477xQ6duwoZGRkCF27dhX++Mc/SsJv6T4bZ9GiRcw++aabbhIEwb57unbtWuHcc88V0tPThfbt2wtPPvmkLfUPCIIoTSFBEARBEITHIZ8XgiAIgiB8BQkvBEEQBEH4ChJeCIIgCILwFSS8EARBEAThK0h4IQiCIAjCV5DwQhAEQRCEryDhhSAIgiAIX0HCC0EQBEEQvoKEF4IgCIIgfAUJLwRBEARB+AoSXgiCIAiC8BUkvBAEQRAE4Sv+H042GiLBSdKvAAAAAElFTkSuQmCC",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/src/layers/rnn/rnn.py:204: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n",
|
|
" super().__init__(**kwargs)\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\">Model: \"sequential\"</span>\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[1mModel: \"sequential\"\u001b[0m\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
|
|
"┃<span style=\"font-weight: bold\"> Layer (type) </span>┃<span style=\"font-weight: bold\"> Output Shape </span>┃<span style=\"font-weight: bold\"> Param # </span>┃\n",
|
|
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
|
|
"│ simple_rnn (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">SimpleRNN</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">32</span>) │ <span style=\"color: #00af00; text-decoration-color: #00af00\">1,184</span> │\n",
|
|
"├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
|
|
"│ dense (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">8</span>) │ <span style=\"color: #00af00; text-decoration-color: #00af00\">264</span> │\n",
|
|
"├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
|
|
"│ dense_1 (<span style=\"color: #0087ff; text-decoration-color: #0087ff\">Dense</span>) │ (<span style=\"color: #00d7ff; text-decoration-color: #00d7ff\">None</span>, <span style=\"color: #00af00; text-decoration-color: #00af00\">1</span>) │ <span style=\"color: #00af00; text-decoration-color: #00af00\">9</span> │\n",
|
|
"└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
|
|
"┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n",
|
|
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
|
|
"│ simple_rnn (\u001b[38;5;33mSimpleRNN\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m32\u001b[0m) │ \u001b[38;5;34m1,184\u001b[0m │\n",
|
|
"├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
|
|
"│ dense (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m8\u001b[0m) │ \u001b[38;5;34m264\u001b[0m │\n",
|
|
"├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
|
|
"│ dense_1 (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m1\u001b[0m) │ \u001b[38;5;34m9\u001b[0m │\n",
|
|
"└─────────────────────────────────┴────────────────────────┴───────────────┘\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Total params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">1,457</span> (5.69 KB)\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[1m Total params: \u001b[0m\u001b[38;5;34m1,457\u001b[0m (5.69 KB)\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">1,457</span> (5.69 KB)\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m1,457\u001b[0m (5.69 KB)\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"font-weight: bold\"> Non-trainable params: </span><span style=\"color: #00af00; text-decoration-color: #00af00\">0</span> (0.00 B)\n",
|
|
"</pre>\n"
|
|
],
|
|
"text/plain": [
|
|
"\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 1/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 2s - 41ms/step - loss: 0.5362\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 2/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.4106\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 3/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.4015\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 4/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3978\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 5/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3953\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 6/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3941\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 7/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3899\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 8/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3917\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 9/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3893\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 10/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3903\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 11/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3860\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 12/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3843\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 13/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - 11ms/step - loss: 0.3874\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 14/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3868\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 15/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3842\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 16/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3855\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 17/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - 13ms/step - loss: 0.3861\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 18/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 10ms/step - loss: 0.3809\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 19/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3829\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 20/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3809\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 21/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3824\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 22/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3804\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 23/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3797\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 24/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3807\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 25/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3796\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 26/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 2s - 31ms/step - loss: 0.3783\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 27/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 9ms/step - loss: 0.3775\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 28/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3789\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 29/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3774\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 30/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 9ms/step - loss: 0.3759\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 31/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3768\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 32/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3741\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 33/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3755\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 34/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3733\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 35/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - 22ms/step - loss: 0.3736\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 36/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - 10ms/step - loss: 0.3721\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 37/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 10ms/step - loss: 0.3736\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 38/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3728\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 39/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3719\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 40/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - 11ms/step - loss: 0.3718\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 41/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3704\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 42/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3730\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 43/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3704\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 44/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3698\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 45/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3712\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 46/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3691\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 47/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3672\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 48/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 8ms/step - loss: 0.3657\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 49/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3678\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 50/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3634\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 51/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3667\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 52/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3685\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 53/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3669\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 54/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3654\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 55/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3635\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 56/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3643\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 57/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3649\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 58/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3617\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 59/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - 7ms/step - loss: 0.3648\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 60/100\n"
|
|
]
|
|
},
|
|
{
|
|
"ename": "KeyboardInterrupt",
|
|
"evalue": "",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
|
|
"Cell \u001b[0;32mIn[1], line 58\u001b[0m\n\u001b[1;32m 55\u001b[0m model\u001b[38;5;241m.\u001b[39mcompile(loss\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmean_squared_error\u001b[39m\u001b[38;5;124m'\u001b[39m, optimizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrmsprop\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 56\u001b[0m model\u001b[38;5;241m.\u001b[39msummary()\n\u001b[0;32m---> 58\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrainX\u001b[49m\u001b[43m,\u001b[49m\u001b[43mtrainY\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m16\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 59\u001b[0m trainPredict \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(trainX)\n\u001b[1;32m 60\u001b[0m testPredict\u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(testX)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/src/utils/traceback_utils.py:117\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 115\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 116\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 117\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 118\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 119\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/src/backend/tensorflow/trainer.py:320\u001b[0m, in \u001b[0;36mTensorFlowTrainer.fit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq)\u001b[0m\n\u001b[1;32m 318\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m step, iterator \u001b[38;5;129;01min\u001b[39;00m epoch_iterator\u001b[38;5;241m.\u001b[39menumerate_epoch():\n\u001b[1;32m 319\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_begin(step)\n\u001b[0;32m--> 320\u001b[0m logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 321\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_end(step, logs)\n\u001b[1;32m 322\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstop_training:\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 150\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 152\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:833\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 830\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 832\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[0;32m--> 833\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 835\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[1;32m 836\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:878\u001b[0m, in \u001b[0;36mFunction._call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 875\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m 876\u001b[0m \u001b[38;5;66;03m# In this case we have not created variables on the first call. So we can\u001b[39;00m\n\u001b[1;32m 877\u001b[0m \u001b[38;5;66;03m# run the first trace but we should fail if variables are created.\u001b[39;00m\n\u001b[0;32m--> 878\u001b[0m results \u001b[38;5;241m=\u001b[39m \u001b[43mtracing_compilation\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_function\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 879\u001b[0m \u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mkwds\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_variable_creation_config\u001b[49m\n\u001b[1;32m 880\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 881\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_created_variables:\n\u001b[1;32m 882\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mCreating variables on a non-first call to a function\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 883\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m decorated with tf.function.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py:139\u001b[0m, in \u001b[0;36mcall_function\u001b[0;34m(args, kwargs, tracing_options)\u001b[0m\n\u001b[1;32m 137\u001b[0m bound_args \u001b[38;5;241m=\u001b[39m function\u001b[38;5;241m.\u001b[39mfunction_type\u001b[38;5;241m.\u001b[39mbind(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n\u001b[1;32m 138\u001b[0m flat_inputs \u001b[38;5;241m=\u001b[39m function\u001b[38;5;241m.\u001b[39mfunction_type\u001b[38;5;241m.\u001b[39munpack_inputs(bound_args)\n\u001b[0;32m--> 139\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfunction\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;66;43;03m# pylint: disable=protected-access\u001b[39;49;00m\n\u001b[1;32m 140\u001b[0m \u001b[43m \u001b[49m\u001b[43mflat_inputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mfunction\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\n\u001b[1;32m 141\u001b[0m \u001b[43m\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/concrete_function.py:1322\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[0;34m(self, tensor_inputs, captured_inputs)\u001b[0m\n\u001b[1;32m 1318\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[1;32m 1319\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[1;32m 1320\u001b[0m \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[1;32m 1321\u001b[0m \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[0;32m-> 1322\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_preflattened\u001b[49m\u001b[43m(\u001b[49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1323\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[1;32m 1324\u001b[0m args,\n\u001b[1;32m 1325\u001b[0m possible_gradient_type,\n\u001b[1;32m 1326\u001b[0m executing_eagerly)\n\u001b[1;32m 1327\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py:216\u001b[0m, in \u001b[0;36mAtomicFunction.call_preflattened\u001b[0;34m(self, args)\u001b[0m\n\u001b[1;32m 214\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcall_preflattened\u001b[39m(\u001b[38;5;28mself\u001b[39m, args: Sequence[core\u001b[38;5;241m.\u001b[39mTensor]) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m Any:\n\u001b[1;32m 215\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"Calls with flattened tensor inputs and returns the structured output.\"\"\"\u001b[39;00m\n\u001b[0;32m--> 216\u001b[0m flat_outputs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_flat\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 217\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mfunction_type\u001b[38;5;241m.\u001b[39mpack_output(flat_outputs)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py:251\u001b[0m, in \u001b[0;36mAtomicFunction.call_flat\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m 249\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m record\u001b[38;5;241m.\u001b[39mstop_recording():\n\u001b[1;32m 250\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_bound_context\u001b[38;5;241m.\u001b[39mexecuting_eagerly():\n\u001b[0;32m--> 251\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_bound_context\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall_function\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 252\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 253\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mlist\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43margs\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 254\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mlen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfunction_type\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mflat_outputs\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 255\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 256\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 257\u001b[0m outputs \u001b[38;5;241m=\u001b[39m make_call_op_in_graph(\n\u001b[1;32m 258\u001b[0m \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m 259\u001b[0m \u001b[38;5;28mlist\u001b[39m(args),\n\u001b[1;32m 260\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_bound_context\u001b[38;5;241m.\u001b[39mfunction_call_options\u001b[38;5;241m.\u001b[39mas_attrs(),\n\u001b[1;32m 261\u001b[0m )\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/context.py:1500\u001b[0m, in \u001b[0;36mContext.call_function\u001b[0;34m(self, name, tensor_inputs, num_outputs)\u001b[0m\n\u001b[1;32m 1498\u001b[0m cancellation_context \u001b[38;5;241m=\u001b[39m cancellation\u001b[38;5;241m.\u001b[39mcontext()\n\u001b[1;32m 1499\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cancellation_context \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m-> 1500\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1501\u001b[0m \u001b[43m \u001b[49m\u001b[43mname\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdecode\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mutf-8\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1502\u001b[0m \u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mnum_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1503\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mtensor_inputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1504\u001b[0m \u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1505\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m,\u001b[49m\n\u001b[1;32m 1506\u001b[0m \u001b[43m \u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1507\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 1508\u001b[0m outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[1;32m 1509\u001b[0m name\u001b[38;5;241m.\u001b[39mdecode(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mutf-8\u001b[39m\u001b[38;5;124m\"\u001b[39m),\n\u001b[1;32m 1510\u001b[0m num_outputs\u001b[38;5;241m=\u001b[39mnum_outputs,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1514\u001b[0m cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_context,\n\u001b[1;32m 1515\u001b[0m )\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:53\u001b[0m, in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 51\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m 52\u001b[0m ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[0;32m---> 53\u001b[0m tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 54\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 55\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 56\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Start importing packages\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras.models import Model, Sequential \n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"from tensorflow.keras import optimizers \n",
|
|
"from tensorflow.keras import regularizers \n",
|
|
"from tensorflow.keras.utils import to_categorical \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"# convert into dataset matrix\n",
|
|
"def convertToMatrix(data, step):\n",
|
|
" X, Y =[], []\n",
|
|
" for i in range(len(data)-step):\n",
|
|
" d=i+step \n",
|
|
" X.append(data[i:d,])\n",
|
|
" Y.append(data[d,])\n",
|
|
" return np.array(X), np.array(Y)\n",
|
|
"\n",
|
|
"step = 4\n",
|
|
"N = 1000 \n",
|
|
"Tp = 800 \n",
|
|
"\n",
|
|
"t=np.arange(0,N)\n",
|
|
"x=np.sin(0.02*t)+2*np.random.rand(N)\n",
|
|
"df = pd.DataFrame(x)\n",
|
|
"df.head()\n",
|
|
"\n",
|
|
"plt.plot(df)\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"values=df.values\n",
|
|
"train,test = values[0:Tp,:], values[Tp:N,:]\n",
|
|
"\n",
|
|
"# add step elements into train and test\n",
|
|
"test = np.append(test,np.repeat(test[-1,],step))\n",
|
|
"train = np.append(train,np.repeat(train[-1,],step))\n",
|
|
" \n",
|
|
"trainX,trainY =convertToMatrix(train,step)\n",
|
|
"testX,testY =convertToMatrix(test,step)\n",
|
|
"trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
|
|
"testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(SimpleRNN(units=32, input_shape=(1,step), activation=\"relu\"))\n",
|
|
"model.add(Dense(8, activation=\"relu\")) \n",
|
|
"model.add(Dense(1))\n",
|
|
"model.compile(loss='mean_squared_error', optimizer='rmsprop')\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)\n",
|
|
"trainPredict = model.predict(trainX)\n",
|
|
"testPredict= model.predict(testX)\n",
|
|
"predicted=np.concatenate((trainPredict,testPredict),axis=0)\n",
|
|
"\n",
|
|
"trainScore = model.evaluate(trainX, trainY, verbose=0)\n",
|
|
"print(trainScore)\n",
|
|
"\n",
|
|
"index = df.index.values\n",
|
|
"plt.plot(index,df)\n",
|
|
"plt.plot(index,predicted)\n",
|
|
"plt.axvline(df.index[Tp], c=\"r\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "897a47c7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## An extrapolation example\n",
|
|
"\n",
|
|
"The following code provides an example of how recurrent neural\n",
|
|
"networks can be used to extrapolate to unknown values of physics data\n",
|
|
"sets. Specifically, the data sets used in this program come from\n",
|
|
"a quantum mechanical many-body calculation of energies as functions of the number of particles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6776ae2a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"# For matrices and calculations\n",
|
|
"import numpy as np\n",
|
|
"# For machine learning (backend for keras)\n",
|
|
"import tensorflow as tf\n",
|
|
"# User-friendly machine learning library\n",
|
|
"# Front end for TensorFlow\n",
|
|
"import tensorflow.keras\n",
|
|
"# Different methods from Keras needed to create an RNN\n",
|
|
"# This is not necessary but it shortened function calls \n",
|
|
"# that need to be used in the code.\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras import regularizers\n",
|
|
"from tensorflow.keras.models import Model, Sequential\n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"# For timing the code\n",
|
|
"from timeit import default_timer as timer\n",
|
|
"# For plotting\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"\n",
|
|
"# The data set\n",
|
|
"datatype='VaryDimension'\n",
|
|
"X_tot = np.arange(2, 42, 2)\n",
|
|
"y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,\n",
|
|
"\t-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, \n",
|
|
"\t-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35227d36",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The way the recurrent neural networks are trained in this program\n",
|
|
"differs from how machine learning algorithms are usually trained.\n",
|
|
"Typically a machine learning algorithm is trained by learning the\n",
|
|
"relationship between the x data and the y data. In this program, the\n",
|
|
"recurrent neural network will be trained to recognize the relationship\n",
|
|
"in a sequence of y values. This is type of data formatting is\n",
|
|
"typically used time series forcasting, but it can also be used in any\n",
|
|
"extrapolation (time series forecasting is just a specific type of\n",
|
|
"extrapolation along the time axis). This method of data formatting\n",
|
|
"does not use the x data and assumes that the y data are evenly spaced.\n",
|
|
"\n",
|
|
"For a standard machine learning algorithm, the training data has the\n",
|
|
"form of (x,y) so the machine learning algorithm learns to assiciate a\n",
|
|
"y value with a given x value. This is useful when the test data has x\n",
|
|
"values within the same range as the training data. However, for this\n",
|
|
"application, the x values of the test data are outside of the x values\n",
|
|
"of the training data and the traditional method of training a machine\n",
|
|
"learning algorithm does not work as well. For this reason, the\n",
|
|
"recurrent neural network is trained on sequences of y values of the\n",
|
|
"form ((y1, y2), y3), so that the network is concerned with learning\n",
|
|
"the pattern of the y data and not the relation between the x and y\n",
|
|
"data. As long as the pattern of y data outside of the training region\n",
|
|
"stays relatively stable compared to what was inside the training\n",
|
|
"region, this method of training can produce accurate extrapolations to\n",
|
|
"y values far removed from the training data set."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "7dc577d1",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# FORMAT_DATA\n",
|
|
"def format_data(data, length_of_sequence = 2): \n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" data(a numpy array): the data that will be the inputs to the recurrent neural\n",
|
|
" network\n",
|
|
" length_of_sequence (an int): the number of elements in one iteration of the\n",
|
|
" sequence patter. For a function approximator use length_of_sequence = 2.\n",
|
|
" Returns:\n",
|
|
" rnn_input (a 3D numpy array): the input data for the recurrent neural network. Its\n",
|
|
" dimensions are length of data - length of sequence, length of sequence, \n",
|
|
" dimnsion of data\n",
|
|
" rnn_output (a numpy array): the training data for the neural network\n",
|
|
" Formats data to be used in a recurrent neural network.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" X, Y = [], []\n",
|
|
" for i in range(len(data)-length_of_sequence):\n",
|
|
" # Get the next length_of_sequence elements\n",
|
|
" a = data[i:i+length_of_sequence]\n",
|
|
" # Get the element that immediately follows that\n",
|
|
" b = data[i+length_of_sequence]\n",
|
|
" # Reshape so that each data point is contained in its own array\n",
|
|
" a = np.reshape (a, (len(a), 1))\n",
|
|
" X.append(a)\n",
|
|
" Y.append(b)\n",
|
|
" rnn_input = np.array(X)\n",
|
|
" rnn_output = np.array(Y)\n",
|
|
"\n",
|
|
" return rnn_input, rnn_output\n",
|
|
"\n",
|
|
"\n",
|
|
"# ## Defining the Recurrent Neural Network Using Keras\n",
|
|
"# \n",
|
|
"# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.\n",
|
|
"\n",
|
|
"def rnn(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with one hidden layer and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer\n",
|
|
" hidden_neurons = 200\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to \n",
|
|
" # the network immediately after the input layer\n",
|
|
" rnn = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\")(inp)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1978b6a1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Predicting New Points With A Trained Recurrent Neural Network"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "d4ad417a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_rnn (x1, y_test, plot_min, plot_max):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" x1 (a list or numpy array): The complete x component of the data set\n",
|
|
" y_test (a list or numpy array): The complete y component of the data set\n",
|
|
" plot_min (an int or float): the smallest x value used in the training data\n",
|
|
" plot_max (an int or float): the largest x valye used in the training data\n",
|
|
" Returns:\n",
|
|
" None.\n",
|
|
" Uses a trained recurrent neural network model to predict future points in the \n",
|
|
" series. Computes the MSE of the predicted data set from the true data set, saves\n",
|
|
" the predicted data set to a csv file, and plots the predicted and true data sets w\n",
|
|
" while also displaying the data range used for training.\n",
|
|
" \"\"\"\n",
|
|
" # Add the training data as the first dim points in the predicted data array as these\n",
|
|
" # are known values.\n",
|
|
" y_pred = y_test[:dim].tolist()\n",
|
|
" # Generate the first input to the trained recurrent neural network using the last two \n",
|
|
" # points of the training data. Based on how the network was trained this means that it\n",
|
|
" # will predict the first point in the data set after the training data. All of the \n",
|
|
" # brackets are necessary for Tensorflow.\n",
|
|
" next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])\n",
|
|
" # Save the very last point in the training data set. This will be used later.\n",
|
|
" last = [y_test[dim-1]]\n",
|
|
"\n",
|
|
" # Iterate until the complete data set is created.\n",
|
|
" for i in range (dim, len(y_test)):\n",
|
|
" # Predict the next point in the data set using the previous two points.\n",
|
|
" next = model.predict(next_input)\n",
|
|
" # Append just the number of the predicted data set\n",
|
|
" y_pred.append(next[0][0])\n",
|
|
" # Create the input that will be used to predict the next data point in the data set.\n",
|
|
" next_input = np.array([[last, next[0]]], dtype=np.float64)\n",
|
|
" last = next\n",
|
|
"\n",
|
|
" # Print the mean squared error between the known data set and the predicted data set.\n",
|
|
" print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())\n",
|
|
" # Save the predicted data set as a csv file for later use\n",
|
|
" name = datatype + 'Predicted'+str(dim)+'.csv'\n",
|
|
" np.savetxt(name, y_pred, delimiter=',')\n",
|
|
" # Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
" # for the training data.\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.plot(x1, y_test, label=\"true\", linewidth=3)\n",
|
|
" ax.plot(x1, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
" ax.legend()\n",
|
|
" # Created a red region to represent the points used in the training data.\n",
|
|
" ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn(length_of_sequences = rnn_input.shape[1])\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2cad4fc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Changing the size of the recurrent neural network and its parameters\n",
|
|
"can drastically change the results you get from the model. The below\n",
|
|
"code takes the simple recurrent neural network from above and adds a\n",
|
|
"second hidden layer, changes the number of neurons in the hidden\n",
|
|
"layer, and explicitly declares the activation function of the hidden\n",
|
|
"layers to be a sigmoid function. The loss function and optimizer can\n",
|
|
"also be changed but are kept the same as the above network. These\n",
|
|
"parameters can be tuned to provide the optimal result from the\n",
|
|
"network. For some ideas on how to improve the performance of a\n",
|
|
"[recurrent neural network](https://danijar.com/tips-for-training-recurrent-neural-networks)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c39f1516",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer, increased from the first network\n",
|
|
" hidden_neurons = 500\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Create two hidden layers instead of one hidden layer. Explicitly set the activation\n",
|
|
" # function to be the sigmoid function (the default value is hyperbolic tangent)\n",
|
|
" rnn1 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=True, # This needs to be True if another hidden layer is to follow\n",
|
|
" stateful = stateful, activation = 'sigmoid',\n",
|
|
" name=\"RNN1\")(inp)\n",
|
|
" rnn2 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False, activation = 'sigmoid',\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN2\")(rnn1)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn2)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn_2layers(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "842c7602",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Other Types of Recurrent Neural Networks\n",
|
|
"\n",
|
|
"Besides a simple recurrent neural network layer, there are two other\n",
|
|
"commonly used types of recurrent neural network layers: Long Short\n",
|
|
"Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short\n",
|
|
"introduction to these layers see <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>\n",
|
|
"and <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>.\n",
|
|
"\n",
|
|
"The first network created below is similar to the previous network,\n",
|
|
"but it replaces the SimpleRNN layers with LSTM layers. The second\n",
|
|
"network below has two hidden layers made up of GRUs, which are\n",
|
|
"preceeded by two dense (feeddorward) neural network layers. These\n",
|
|
"dense layers \"preprocess\" the data before it reaches the recurrent\n",
|
|
"layers. This architecture has been shown to improve the performance\n",
|
|
"of recurrent neural networks (see the link above and also\n",
|
|
"<https://arxiv.org/pdf/1807.02857.pdf>."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "6f0e9b62",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons on the input/output layer and the number of neurons in the hidden layer\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input Layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)\n",
|
|
" rnn= LSTM(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True, activation='tanh')(inp)\n",
|
|
" rnn1 = LSTM(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True, activation='tanh')(rnn)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn1)\n",
|
|
" # Define the midel\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the model\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with four hidden layers (two dense followed by\n",
|
|
" two GRU layers) and returns the model.\n",
|
|
" \"\"\" \n",
|
|
" # Number of neurons on the input/output layers and hidden layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden Dense (feedforward) layers\n",
|
|
" dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)\n",
|
|
" dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)\n",
|
|
" # Hidden GRU layers\n",
|
|
" rnn1 = GRU(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True)(dnn1)\n",
|
|
" rnn = GRU(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True)(rnn1)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Define the model\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the mdoel\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Change the method name to reflect which network you want to use\n",
|
|
"model = dnn2_gru2(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)\n",
|
|
"\n",
|
|
"\n",
|
|
"# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)\n",
|
|
"# \n",
|
|
"# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"# Reshape the data for Keras specifications\n",
|
|
"X_train = X_train.reshape((dim, 1))\n",
|
|
"y_train = y_train.reshape((dim, 1))\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Set the sequence length to 1 for regular data formatting \n",
|
|
"model = rnn(length_of_sequences = 1)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(X_train, y_train, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict the remaining data points\n",
|
|
"X_pred = X_tot[dim:]\n",
|
|
"X_pred = X_pred.reshape((len(X_pred), 1))\n",
|
|
"y_model = model.predict(X_pred)\n",
|
|
"y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))\n",
|
|
"\n",
|
|
"# Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
"# for the training data.\n",
|
|
"fig, ax = plt.subplots()\n",
|
|
"ax.plot(X_tot, y_tot, label=\"true\", linewidth=3)\n",
|
|
"ax.plot(X_tot, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
"ax.legend()\n",
|
|
"# Created a red region to represent the points used in the training data.\n",
|
|
"ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0752ba7f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Generative Models\n",
|
|
"\n",
|
|
"**Generative models** describe a class of statistical models that are a contrast\n",
|
|
"to **discriminative models**. Informally we say that generative models can\n",
|
|
"generate new data instances while discriminative models discriminate between\n",
|
|
"different kinds of data instances. A generative model could generate new photos\n",
|
|
"of animals that look like 'real' animals while a discriminative model could tell\n",
|
|
"a dog from a cat. More formally, given a data set $x$ and a set of labels /\n",
|
|
"targets $y$. Generative models capture the joint probability $p(x, y)$, or\n",
|
|
"just $p(x)$ if there are no labels, while discriminative models capture the\n",
|
|
"conditional probability $p(y | x)$. Discriminative models generally try to draw\n",
|
|
"boundaries in the data space (often high dimensional), while generative models\n",
|
|
"try to model how data is placed throughout the space.\n",
|
|
"\n",
|
|
"**Note**: this material is thanks to Linus Ekstrøm."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "784138f8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Generative Adversarial Networks\n",
|
|
"\n",
|
|
"**Generative Adversarial Networks** are a type of unsupervised machine learning\n",
|
|
"algorithm proposed by [Goodfellow et. al](https://arxiv.org/pdf/1406.2661.pdf)\n",
|
|
"in 2014 (short and good article).\n",
|
|
"\n",
|
|
"The simplest formulation of\n",
|
|
"the model is based on a game theoretic approach, *zero sum game*, where we pit\n",
|
|
"two neural networks against one another. We define two rival networks, one\n",
|
|
"generator $g$, and one discriminator $d$. The generator directly produces\n",
|
|
"samples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a42f89ee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" x = g(z; \\theta^{(g)})\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abe7212f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The discriminator attempts to distinguish between samples drawn from the\n",
|
|
"training data and samples drawn from the generator. In other words, it tries to\n",
|
|
"tell the difference between the fake data produced by $g$ and the actual data\n",
|
|
"samples we want to do prediction on. The discriminator outputs a probability\n",
|
|
"value given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0821eaee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" d(x; \\theta^{(d)})\n",
|
|
"\\label{_auto2} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55e0ccf6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"indicating the probability that $x$ is a real training example rather than a\n",
|
|
"fake sample the generator has generated. The simplest way to formulate the\n",
|
|
"learning process in a generative adversarial network is a zero-sum game, in\n",
|
|
"which a function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f37ece14",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto3} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d6d6d5fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"determines the reward for the discriminator, while the generator gets the\n",
|
|
"conjugate reward"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c74b45c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" -v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto4} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "605ac8c9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"During learning both of the networks maximize their own reward function, so that\n",
|
|
"the generator gets better and better at tricking the discriminator, while the\n",
|
|
"discriminator gets better and better at telling the difference between the fake\n",
|
|
"and real data. The generator and discriminator alternate on which one trains at\n",
|
|
"one time (i.e. for one epoch). In other words, we keep the generator constant\n",
|
|
"and train the discriminator, then we keep the discriminator constant to train\n",
|
|
"the generator and repeat. It is this back and forth dynamic which lets GANs\n",
|
|
"tackle otherwise intractable generative problems. As the generator improves with\n",
|
|
" training, the discriminator's performance gets worse because it cannot easily\n",
|
|
" tell the difference between real and fake. If the generator ends up succeeding\n",
|
|
" perfectly, the the discriminator will do no better than random guessing i.e.\n",
|
|
" 50\\%. This progression in the training poses a problem for the convergence\n",
|
|
" criteria for GANs. The discriminator feedback gets less meaningful over time,\n",
|
|
" if we continue training after this point then the generator is effectively\n",
|
|
" training on junk data which can undo the learning up to that point. Therefore,\n",
|
|
" we stop training when the discriminator starts outputting $1/2$ everywhere.\n",
|
|
"\n",
|
|
"At convergence we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfec7462",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto5\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" g^* = \\underset{g}{\\mathrm{argmin}}\\hspace{2pt}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto5} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37c65d4c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The default choice for $v$ is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c868a092",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto6\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)}) = \\mathbb{E}_{x\\sim p_\\mathrm{data}}\\log d(x)\n",
|
|
" + \\mathbb{E}_{x\\sim p_\\mathrm{model}}\n",
|
|
" \\log (1 - d(x))\n",
|
|
"\\label{_auto6} \\tag{6}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ad465af3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The main motivation for the design of GANs is that the learning process requires\n",
|
|
"neither approximate inference (variational autoencoders for example) nor\n",
|
|
"approximation of a partition function. In the case where"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "27858a4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto7\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto7} \\tag{7}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86006023",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"is convex in $\\theta^{(g)} then the procedure is guaranteed to converge and is\n",
|
|
"asymptotically consistent\n",
|
|
"( [Seth Lloyd on QuGANs](https://arxiv.org/pdf/1804.09139.pdf) ).\n",
|
|
"\n",
|
|
"This is in\n",
|
|
"general not the case and it is possible to get situations where the training\n",
|
|
"process never converges because the generator and discriminator chase one\n",
|
|
"another around in the parameter space indefinitely. A much deeper discussion on\n",
|
|
"the currently open research problem of GAN convergence is available\n",
|
|
"[here](https://www.deeplearningbook.org/contents/generative_models.html). To\n",
|
|
"anyone interested in learning more about GANs it is a highly recommended read.\n",
|
|
"Direct quote: \"In this best-performing formulation, the generator aims to\n",
|
|
"increase the log probability that the discriminator makes a mistake, rather than\n",
|
|
"aiming to decrease the log probability that the discriminator makes the correct\n",
|
|
"prediction.\" [Another interesting read](https://arxiv.org/abs/1701.00160)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2fee38bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Writing Our First Generative Adversarial Network\n",
|
|
"Let us now move on to actually implementing a GAN in tensorflow. We will study\n",
|
|
"the performance of our GAN on the MNIST dataset. This code is based on and\n",
|
|
"adapted from the\n",
|
|
"[google tutorial](https://www.tensorflow.org/tutorials/generative/dcgan)\n",
|
|
"\n",
|
|
"First we import our libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "004a0b53",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import time\n",
|
|
"import numpy as np\n",
|
|
"import tensorflow as tf\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from tensorflow.keras import layers\n",
|
|
"from tensorflow.keras.utils import plot_model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af161",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define our hyperparameters and import our data the usual way"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "8cbaf16a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"BUFFER_SIZE = 60000\n",
|
|
"BATCH_SIZE = 256\n",
|
|
"EPOCHS = 30\n",
|
|
"\n",
|
|
"data = tf.keras.datasets.mnist.load_data()\n",
|
|
"(train_images, train_labels), (test_images, test_labels) = data\n",
|
|
"train_images = np.reshape(train_images, (train_images.shape[0],\n",
|
|
" 28,\n",
|
|
" 28,\n",
|
|
" 1)).astype('float32')\n",
|
|
"\n",
|
|
"# we normalize between -1 and 1\n",
|
|
"train_images = (train_images - 127.5) / 127.5\n",
|
|
"training_dataset = tf.data.Dataset.from_tensor_slices(\n",
|
|
" train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "822b8cc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### MNIST and GANs\n",
|
|
"\n",
|
|
"Let's have a quick look"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "52b5965c",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.imshow(train_images[0], cmap='Greys')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "21c5199c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our two models. This is where the 'magic' happens. There are a\n",
|
|
"huge amount of possible formulations for both models. A lot of engineering and\n",
|
|
"trial and error can be done here to try to produce better performing models. For\n",
|
|
"more advanced GANs this is by far the step where you can 'make or break' a\n",
|
|
"model.\n",
|
|
"\n",
|
|
"We start with the generator. As stated in the introductory text the generator\n",
|
|
"$g$ upsamples from a random sample to the shape of what we want to predict. In\n",
|
|
"our case we are trying to predict MNIST images ($28\\times 28$ pixels)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "356759c7",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_model():\n",
|
|
" \"\"\"\n",
|
|
" The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to\n",
|
|
" produce an image from a random seed. We start with a Dense layer taking this\n",
|
|
" random sample as an input and subsequently upsample through multiple\n",
|
|
" convolutional layers.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
"\n",
|
|
"\n",
|
|
" # adding our input layer. Dense means that every neuron is connected and\n",
|
|
" # the input shape is the shape of our random noise. The units need to match\n",
|
|
" # in some sense the upsampling strides to reach our desired output shape.\n",
|
|
" # we are using 100 random numbers as our seed\n",
|
|
" model.add(layers.Dense(units=7*7*BATCH_SIZE,\n",
|
|
" use_bias=False,\n",
|
|
" input_shape=(100, )))\n",
|
|
" # we normalize the output form the Dense layer\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" # and add an activation function to our 'layer'. LeakyReLU avoids vanishing\n",
|
|
" # gradient problem\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" model.add(layers.Reshape((7, 7, BATCH_SIZE)))\n",
|
|
" assert model.output_shape == (None, 7, 7, BATCH_SIZE)\n",
|
|
" # even though we just added four keras layers we think of everything above\n",
|
|
" # as 'one' layer\n",
|
|
"\n",
|
|
" # next we add our upscaling convolutional layers\n",
|
|
" model.add(layers.Conv2DTranspose(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(1, 1),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 7, 7, 128)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 14, 14, 64)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=1,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False,\n",
|
|
" activation='tanh'))\n",
|
|
" assert model.output_shape == (None, 28, 28, 1)\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "854bcd6b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"And there we have our 'simple' generator model. Now we move on to defining our\n",
|
|
"discriminator model $d$, which is a convolutional neural network based image\n",
|
|
"classifier."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "41473304",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_model():\n",
|
|
" \"\"\"\n",
|
|
" The discriminator is a convolutional neural network based image classifier\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
" model.add(layers.Conv2D(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" input_shape=[28, 28, 1]))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
"\n",
|
|
" model.add(layers.Conv2D(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same'))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
" model.add(layers.Flatten())\n",
|
|
" model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af567",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Let us take a look at our models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "f899d4e3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generator = generator_model()\n",
|
|
"plot_model(generator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "87ef384b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"discriminator = discriminator_model()\n",
|
|
"plot_model(discriminator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2bef82d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we need a few helper objects we will use in training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "e397847a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
|
|
"generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n",
|
|
"discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "db3396cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The first object, *cross_entropy* is our loss function and the two others are\n",
|
|
"our optimizers. Notice we use the same learning rate for both $g$ and $d$. This\n",
|
|
"is because they need to improve their accuracy at approximately equal speeds to\n",
|
|
"get convergence (not necessarily exactly equal). Now we define our loss\n",
|
|
"functions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "931eaced",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_loss(fake_output):\n",
|
|
" loss = cross_entropy(tf.ones_like(fake_output), fake_output)\n",
|
|
"\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "0c4a44bb",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_loss(real_output, fake_output):\n",
|
|
" real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n",
|
|
" fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)\n",
|
|
" total_loss = real_loss + fake_loss\n",
|
|
"\n",
|
|
" return total_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fcf8f066",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a kind of seed to help us compare the learning process over\n",
|
|
"multiple training epochs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "eea2bbee",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"noise_dimension = 100\n",
|
|
"n_examples_to_generate = 16\n",
|
|
"seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94a6e341",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have everything we need to define our training step, which we will apply\n",
|
|
"for every step in our training loop. Notice the @tf.function flag signifying\n",
|
|
"that the function is tensorflow 'compiled'. Removing this flag doubles the\n",
|
|
"computation time."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "8d48470b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tf.function\n",
|
|
"def train_step(images):\n",
|
|
" noise = tf.random.normal([BATCH_SIZE, noise_dimension])\n",
|
|
"\n",
|
|
" with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n",
|
|
" generated_images = generator(noise, training=True)\n",
|
|
"\n",
|
|
" real_output = discriminator(images, training=True)\n",
|
|
" fake_output = discriminator(generated_images, training=True)\n",
|
|
"\n",
|
|
" gen_loss = generator_loss(fake_output)\n",
|
|
" disc_loss = discriminator_loss(real_output, fake_output)\n",
|
|
"\n",
|
|
" gradients_of_generator = gen_tape.gradient(gen_loss,\n",
|
|
" generator.trainable_variables)\n",
|
|
" gradients_of_discriminator = disc_tape.gradient(disc_loss,\n",
|
|
" discriminator.trainable_variables)\n",
|
|
" generator_optimizer.apply_gradients(zip(gradients_of_generator,\n",
|
|
" generator.trainable_variables))\n",
|
|
" discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,\n",
|
|
" discriminator.trainable_variables))\n",
|
|
"\n",
|
|
" return gen_loss, disc_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "62015b88",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a helper function to produce an output over our training epochs\n",
|
|
"to see the predictive progression of our generator model. **Note**: I am including\n",
|
|
"this code here, but comment it out in the training loop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "b189ed96",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_and_save_images(model, epoch, test_input):\n",
|
|
" # we're making inferences here\n",
|
|
" predictions = model(test_input, training=False)\n",
|
|
"\n",
|
|
" fig = plt.figure(figsize=(4, 4))\n",
|
|
"\n",
|
|
" for i in range(predictions.shape[0]):\n",
|
|
" plt.subplot(4, 4, i+1)\n",
|
|
" plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n",
|
|
" plt.axis('off')\n",
|
|
"\n",
|
|
" plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')\n",
|
|
" plt.close()\n",
|
|
" #plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ba2c82d5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Setting up checkpoints to periodically save our model during training so that\n",
|
|
"everything is not lost even if the program were to somehow terminate while\n",
|
|
"training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "a0e2fc8a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setting up checkpoints to save model during training\n",
|
|
"checkpoint_dir = './training_checkpoints'\n",
|
|
"checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')\n",
|
|
"checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n",
|
|
" discriminator_optimizer=discriminator_optimizer,\n",
|
|
" generator=generator,\n",
|
|
" discriminator=discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d93a0f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our training loop"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "a1275556",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def train(dataset, epochs):\n",
|
|
" generator_loss_list = []\n",
|
|
" discriminator_loss_list = []\n",
|
|
"\n",
|
|
" for epoch in range(epochs):\n",
|
|
" start = time.time()\n",
|
|
"\n",
|
|
" for image_batch in dataset:\n",
|
|
" gen_loss, disc_loss = train_step(image_batch)\n",
|
|
" generator_loss_list.append(gen_loss.numpy())\n",
|
|
" discriminator_loss_list.append(disc_loss.numpy())\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epoch + 1, seed_images)\n",
|
|
"\n",
|
|
" if (epoch + 1) % 15 == 0:\n",
|
|
" checkpoint.save(file_prefix=checkpoint_prefix)\n",
|
|
"\n",
|
|
" print(f'Time for epoch {epoch} is {time.time() - start}')\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epochs, seed_images)\n",
|
|
"\n",
|
|
" loss_file = './data/lossfile.txt'\n",
|
|
" with open(loss_file, 'w') as outfile:\n",
|
|
" outfile.write(str(generator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write(str(discriminator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ff3a75a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"To train simply call this function. **Warning**: this might take a long time so\n",
|
|
"there is a folder of a pretrained network already included in the repository."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"id": "371ed41a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"train(train_dataset, EPOCHS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "654399f1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now to avoid having to train and everything, which will take a while depending\n",
|
|
"on your computer setup we now load in the model which produced the above gif."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"id": "dec4b560",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))\n",
|
|
"restored_generator = checkpoint.generator\n",
|
|
"restored_discriminator = checkpoint.discriminator\n",
|
|
"\n",
|
|
"print(restored_generator)\n",
|
|
"print(restored_discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "296bfa5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We have successfully loaded in our latest model. Let us now play around a bit\n",
|
|
"and see what kind of things we can learn about this model. Our generator takes\n",
|
|
"an array of 100 numbers. One idea can be to try to systematically change our\n",
|
|
"input. Let us try and see what we get"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"id": "eecfbb1f",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_latent_points(number=100, scale_means=1, scale_stds=1):\n",
|
|
" latent_dim = 100\n",
|
|
" means = scale_means * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" latent_space_value_range = tf.random.normal([number, latent_dim],\n",
|
|
" means,\n",
|
|
" stds,\n",
|
|
" dtype=tf.float64)\n",
|
|
"\n",
|
|
" return latent_space_value_range\n",
|
|
"\n",
|
|
"def generate_images(latent_points):\n",
|
|
" # notice we set training to false because we are making inferences\n",
|
|
" generated_images = restored_generator.predict(latent_points)\n",
|
|
"\n",
|
|
" return generated_images"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"id": "333a593d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def plot_result(generated_images, number=100):\n",
|
|
" # obviously this assumes sqrt number is an int\n",
|
|
" fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),\n",
|
|
" figsize=(10, 10))\n",
|
|
"\n",
|
|
" for i in range(int(np.sqrt(number))):\n",
|
|
" for j in range(int(np.sqrt(number))):\n",
|
|
" axs[i, j].imshow(generated_images[i*j], cmap='Greys')\n",
|
|
" axs[i, j].axis('off')\n",
|
|
"\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"id": "2f5f0154",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generated_images = generate_images(generate_latent_points())\n",
|
|
"plot_result(generated_images)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff581bf2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We see that the generator generates images that look like MNIST\n",
|
|
"numbers: $1, 4, 7, 9$. Let's try to tweak it a bit more to see if we are able\n",
|
|
"to generate a similar plot where we generate every MNIST number. Let us now try\n",
|
|
"to 'move' a bit around in the latent space. **Note**: decrease the plot number if\n",
|
|
"these following cells take too long to run on your computer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"id": "d3617ad8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 225\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=-5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=5))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a074f93",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Again, we have found something interesting. *Moving* around using our means\n",
|
|
"takes us from digit to digit, while *moving* around using our standard\n",
|
|
"deviations seem to increase the number of different digits! In the last image\n",
|
|
"above, we can barely make out every MNIST digit. Let us make on last plot using\n",
|
|
"this information by upping the standard deviation of our Gaussian noises."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"id": "4ef8937d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 400\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=10))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "385a2d0a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"A pretty cool result! We see that our generator indeed has learned a\n",
|
|
"distribution which qualitatively looks a whole lot like the MNIST dataset.\n",
|
|
"\n",
|
|
"Another interesting way to explore the latent space of our generator model is by\n",
|
|
"interpolating between the MNIST digits. This section is largely based on\n",
|
|
"[this excellent blogpost](https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/)\n",
|
|
"by Jason Brownlee.\n",
|
|
"\n",
|
|
"So let us start by defining a function to interpolate between two points in the\n",
|
|
"latent space."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"id": "57de87b8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def interpolation(point_1, point_2, n_steps=10):\n",
|
|
" ratios = np.linspace(0, 1, num=n_steps)\n",
|
|
" vectors = []\n",
|
|
" for i, ratio in enumerate(ratios):\n",
|
|
" vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))\n",
|
|
"\n",
|
|
" return tf.stack(vectors)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfb76bb6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have all we need to do our interpolation analysis."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"id": "e25decef",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 100\n",
|
|
"latent_points = generate_latent_points(number=plot_number)\n",
|
|
"results = None\n",
|
|
"for i in range(0, 2*np.sqrt(plot_number), 2):\n",
|
|
" interpolated = interpolation(latent_points[i], latent_points[i+1])\n",
|
|
" generated_images = generate_images(interpolated)\n",
|
|
"\n",
|
|
" if results is None:\n",
|
|
" results = generated_images\n",
|
|
" else:\n",
|
|
" results = tf.stack((results, generated_images))\n",
|
|
"\n",
|
|
"plot_results(results, plot_number)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.15"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |