Morten Hjorth-Jensen
2775 lines
149 KiB
Plaintext
2775 lines
149 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d2db899",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html chapter13.do.txt -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4908a97",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Recurrent neural networks: Overarching view\n",
|
|
"\n",
|
|
"Till now our focus has been, including convolutional neural networks\n",
|
|
"as well, on feedforward neural networks. The output or the activations\n",
|
|
"flow only in one direction, from the input layer to the output layer.\n",
|
|
"\n",
|
|
"A recurrent neural network (RNN) looks very much like a feedforward\n",
|
|
"neural network, except that it also has connections pointing\n",
|
|
"backward. \n",
|
|
"\n",
|
|
"RNNs are used to analyze time series data such as stock prices, and\n",
|
|
"tell you when to buy or sell. In autonomous driving systems, they can\n",
|
|
"anticipate car trajectories and help avoid accidents. More generally,\n",
|
|
"they can work on sequences of arbitrary lengths, rather than on\n",
|
|
"fixed-sized inputs like all the nets we have discussed so far. For\n",
|
|
"example, they can take sentences, documents, or audio samples as\n",
|
|
"input, making them extremely useful for natural language processing\n",
|
|
"systems such as automatic translation and speech-to-text.\n",
|
|
"\n",
|
|
"More to text to be added"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43cb4913",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A simple example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "cf6b9dab",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAACVtklEQVR4nO2dd5wU5f3HP7N7He6OegU4mtIEUQQVlKKiKFhiLLEFNFEjUWxoUDBRYzQYW4g/o9iJwZYELImIoAKiNOm9KU2446h3tCu7O78/9nZvZnbK80zZmdn9vl+vg92ZZ2aenXnmeb7Ptz2CKIoiCIIgCIIgfELA7QoQBEEQBEHwQMILQRAEQRC+goQXgiAIgiB8BQkvBEEQBEH4ChJeCIIgCILwFSS8EARBEAThK0h4IQiCIAjCV5DwQhAEQRCEr8hwuwJ2E4lEsGfPHuTn50MQBLerQxAEQRAEA6Io4siRI2jTpg0CAX3dSsoJL3v27EFZWZnb1SAIgiAIwgS7du1Cu3btdMuknPCSn58PIPrjCwoKXK4NQRAEQRAsVFdXo6ysLD6O65FywkvMVFRQUEDCC0EQBEH4DBaXD3LYJQiCIAjCV5DwQhAEQRCEryDhhSAIgiAIX0HCC0EQBEEQvoKEF4IgCIIgfAUJLwRBEARB+AoSXgiCIAiC8BWOCi+vvPIKevfuHc+5MmDAAHz++ee6x8ybNw99+/ZFTk4OOnfujMmTJztZRYIgCIIgfIajwku7du3w9NNPY+nSpVi6dCkuuOAC/OxnP8O6detUy2/btg0jRozAoEGDsGLFCkyYMAH33HMPpk2b5mQ1CYIgCILwEYIoimIyL9iiRQs8++yzuPXWWxP2PfTQQ/j000+xYcOG+LbRo0dj1apVWLhwIdP5q6urUVhYiKqqKsqwSxAEQRA+gWf8TprPSzgcxgcffIBjx45hwIABqmUWLlyIYcOGybZdfPHFWLp0Kerr61WPqa2tRXV1teyPIAiCIIjUxXHhZc2aNWjatCmys7MxevRofPTRRzjllFNUy1ZUVKC4uFi2rbi4GKFQCPv371c9ZuLEiSgsLIz/0YrSBEEQBJHaOC68dOvWDStXrsSiRYvw29/+FjfffDPWr1+vWV65IFPMqqW1UNP48eNRVVUV/9u1a5d9lScIgkhzVv90GG9/tw2RSFI9DAhCF8dXlc7KysLJJ58MAOjXrx++//57/O1vf8Orr76aULakpAQVFRWybZWVlcjIyEDLli1Vz5+dnY3s7Gz7K04QBEHgipe+AwAU5mbiqjPauVwbgoiS9DwvoiiitrZWdd+AAQMwe/Zs2bZZs2ahX79+yMzMTEb1CIIgCBU27T3idhUIIo6jwsuECRMwf/58bN++HWvWrMEjjzyCuXPn4qabbgIQNfmMGjUqXn706NHYsWMHxo4diw0bNuCtt97Cm2++iQcffNDJahIEAWBr5RFc+MI8/HfVHrerQhAEoYujZqO9e/di5MiRKC8vR2FhIXr37o2ZM2fioosuAgCUl5dj586d8fKdOnXCjBkzcP/99+Pvf/872rRpgxdffBFXX321k9X0LXWhCLIyKEkyYQ/3vL8SWyuP4u73V+Dy09q4XR2CIAhNHBVe3nzzTd39U6ZMSdg2ZMgQLF++3KEapQ5rfqrC5S99i7vOPwm/u7i729UhUoD9R9XNuQRhRNWJejTNzkAwoB5YQRB2Q9N2n/LUjGjE1t/n/OByTYhU4URd2O0qEB6ivOoE3pj/o2G5XQeP47Q/zsL1r7ElEiUIO3A82oiwTnVNPXbsP45T2xXGt4XCFLZI2MuJehJeiEaufnkB9lTVxL8LUNeqfLxiNwDg++2HklIvggBI8+ILRvxtPi5/6VvM3VQZ3xainAsEByfqwvjX97tQeaRGswy1KUKKVHDRQ9pqnpm5EUlecYZIU0h48QE/HToBAPhUEgUSpoGG4ODPMzZg3LTV+MVkUu0T9hKRCCsvz/0BX22o1ClNEPZAwovHOFobwu3vLMUnK3cn7KuRqPXrw5FkVovwObPWR5M/bj9w3OWaEKlO5RFy/Cach4QXj/HK3K2YvX4v7v1gZcI+qUMlaV4IgvACZCUi3ICEF49x8Jj66tmA3KGShBdCypyNlTj36a+x6McDqvu1nC3VoGhXggfycSHcgIQXz6HdEZyobzQV1UfIbEQ08qsp32P34RO4/rVFls9FuTrSA1EUMXNtOXYdZDMlaqyNm9BjHasNWasYQTBAwovH0JvE1EjNRhQqTdjE1sqjePGrLfHvpNRLD/63uhyjpy7HoGfmWDpPRNFpPTVjA47XkQBDOAvlefERNaFG4YXCWgm7uHjSNzIzJJkB0gMtEyMvas3lh8pjsrxUBGE3pHnxEUYOu1Un6vHk/9Zj7e6qZFaL8AFaKn8gsS2R6JIe2PWc1c6j194Iwg5IePEYepNeqcOumublif+uxxvfbsNl//etE1Uj0gStNhiJiPhk5W5s338suRUiPMU/F+3AJZO+QWV1NIkdKeoINyDhxUdI87yEVPK8rC+vZj7Xq/N+kPk5EIQRH6/cjXs/WInznpvrdlUIGzArdPzh47XYWHEET83YED0P6eoIFyCfFx9RL3HSteLzUlMfxsTPNwIArj+rDEX5OZbrRngbO7T4i388aMNZiFThwNE6AKR5IdyBNC8eg3UWYyXPizQ6oC5EIdfpgGCDEwKF56cafH2IsgUda4goUnPwJp8XwmlIePEpFG1EJJvjtbTqdKrwx/+uw/tLdlk6Ryyfi5rmhScpIkGYgYSXNIdUvgQLk+f9gJnrKtyuBmEDW/Yewdvfbec+7vO1Fag63pgB/FiDMEtdCOEGJLykEGbyc9AyA6mB00lxn27wkSL8z/E6cxq0bfuP4dZ/fB//3mg2SixLZqPUQBSjEYb9//wVVu067HZ1ZJDw4jGSoQmRqnSH/fUb5y9IOE5GgF5lwnmW7jgU/3w8rnmhCVAqMn76Gpz33Fzc+8FKVFTX4LdTl7ldJRnU46Uh0s6mTiXkmvAftB4RkWxifYfahGvUW0soU7PPeX/JTuw40LjuVb3HtPQkvKQouw4e1+w8qE9JPTJIeCEYsfv1V+tn9h2ppX6GcBQSXjwGy/v+0yHjVWAHPTMHb2k45VGfknoEgyS8EGzYrRHROhv1M6mF14RREl48BksD+WFfY3r27iX5muUmNmTAJFKfoIGHpN7urCB1A+mE1TEoP7sxt6koitrLSXhttCNSCuq1fAhrpxDWNBtRp5JqWPF5ocgQgocWTbPin4/WhjQddkl4SS281k+Q8OJDpMKHXv+gtk8UKTYgFeHxebn9naWYPO+H+PeATq9UWV2D8dPXWKob4S2syhQ5GcH45wNH66Dlx0myC+EktLaRx2ARLaRZ2lnKi6KIez9YiU9X7UHLJll4/eZ+VqpIeBAen5fZ6/di9vq9GD3kJAD6OWIe+PcqzN+y32r1CE+h32dMW/YTlu08pLlf2uccOFanXY6EF8JBSHjxIRFGzUuMbfuP4dNVewBEO5s7/umteH3COlbyvOhpXjZWHDF9XsKbGPUZD/x7leY+QYBM0xKOkM8L4Q5kNvIh0s6DpXtQZtHdd6TW3goRSaOmPoxPV+3BgaPyZ2jk86Jnr9bbZ+QITPgPKyJFVjAgM1tHRFHTh46EF8JJSHjxGgzvu9znRf+A+nAE8zbvs1orwiM8P2sT7nl/Ba5/bZFsuxUhQ23F6Vi7ovQxqUfEQrKxrIyArIsSRW1NDokuhJOQ8OJDtDQvap3IjDXlePIz/ZBpij7yD5+tLgcAbKk8KttuJdpI7dBYGwuQ9JJyaEUhshAMCLJ+RoSoHdVIybsJByHhJUnUhsLYWFFti6AQEdntRtv2H9MvALnQM3NtBa5/bSHKq06YrB3hBhkGDrvS9ayUqPm8xEyNWv4w4YiI615diEc+okgkv2GlC4pExIRoRy1NDpmN/Mf6PdUY++FK7DponAjVbUh4SRK3/WMpLpk0H53Gz0B1Tb3xATrIHHYNyjbJMvbJlp5j9NRlWPTjQTz6yTpzlSMcRet5W/N5Sdy5tUGzo3XaxT8ewOJtB/Hu4p261yXcIRIRMWtdBSqqahL2WVlJXhQVml9RJ5+U6asQbnH5S99i+ordukEd1TX1ePzTda4nQXVUeJk4cSLOPPNM5Ofno6ioCFdeeSU2bdqke8zcuXMhCELC38aNG52squNIw03//vVWzXIsL7xM8SITZBKPNpvVpeq4NQGLsMYX6yqweW9ipI/WZFYrz8uJujB+9fYS2QJrStQOHfHifIQjoqbZKOSxRdoIOZ+u2oPf/HMZ+k/8Cl+sq5Dts2I2ioiirE+JiKJmWyDNi/+ICbZbKrWjDI/UhDBlwXa8vWB7kmqljqPCy7x583DXXXdh0aJFmD17NkKhEIYNG4Zjx4xNGZs2bUJ5eXn8r0uXLk5WNakcOq6dG4EFaaew/cBxrNtTpVmWpf9QM2VRKjv3WLr9IO745zIM++s3zMdINS/SmfW7i3dgziZ9h20t01AoEtENoya8y8IfDsQ/3/HPZTIzgBWH3YjCQVfUOR8JL6lJfSjqzOT2siKO5nmZOXOm7Pvbb7+NoqIiLFu2DIMHD9Y9tqioCM2aNXOwdv5F2Vdc+uK3WPP4MNWyTJoc61UibGR9eTX3MdI8L/XhCIKBaBbUIzUhw2O1TEOiSKHSfiU3Kyj7XlFdg7IWeQCsmY0iirWM9DQvJLukJqGGLKlGfnZOk1TRqaoqqiFo0aKFYdk+ffqgtLQUQ4cOxZw5czTL1dbWorq6WvbnJZZuP4j/rd4j26af0p8hw65KmYMamS7ZNC/Axopq1IUawwP0HDwJ99DSiEk1L9LBhEX2UPN5MTqexiVvk6cQXpQChxabDJISilD0UaJ2n0XCi3/Re3b14ehOK4kx7SBpGXZFUcTYsWMxcOBA9OrVS7NcaWkpXnvtNfTt2xe1tbX45z//iaFDh2Lu3Lmq2pqJEyfij3/8o5NVt8Q1kxdylWfzeWHvLFjMPx98vxOPfrIOQ7q2Zrg64RYvfrUFe6vVEwxKfV5CYT4hVEtAiYiipiOwMteQngBEJB+l8CIlrBPCfPEkfVNlJCLvUUSImmsbkdkoNQk1CC+ZLmtekia8jBkzBqtXr8a3336rW65bt27o1q1b/PuAAQOwa9cuPPfcc6rCy/jx4zF27Nj49+rqapSVldlXcQcQBGsdvlpnoTVIsfQfb327DQAomZ3HeWH2Zs19Usfa2MwIYNO8aPm1iKL+0gExznzqS9w8oCPuHpo6fml+J1cRZSgVNq067EqFkkhEezLFKryEwhF8v/0QTisrRB5DdCThPHoO+fXpZDa6++678emnn2LOnDlo164d9/H9+/fHli1bVPdlZ2ejoKBA9ud1vtm8H6f9cRZmri1P2Mfyvts9o6HAEW9hpkuQNgmpTwPLubS0KxFRZMqwu/9oHZ7XEayI5KOnebHbYVfrbKzd1Ctzf8ANry/Cbf9YarpeRPKIOexmuuyw6+jVRVHEmDFjMH36dHz99dfo1KmTqfOsWLECpaWlNtfOPSqqa1BdE8LoqctNHa+qedFS/TN0VKrCEFkBfEbjM6yXmo1YfF50zkgZdv1JTqa8a5e+4VYcdgF5/xPVxKiX0xJeakNhHJHkupq6eAcAYIEkQopIPg/8S3tBTimxjO2Zqezzctddd+G9997DJ598gvz8fFRURPMNFBYWIjc3F0DU7LN792688847AIBJkyahY8eO6NmzJ+rq6jB16lRMmzYN06ZNc7KqvoInSy9v3hjCW3yxrgJ7q2swakBH3XLSZyh32DUvfIgRHZOS6bMSbiDTzFl+4eUZdnnNRudM/BoHjtVhzePDkJ+TqeuDQySPact/MixTF4pgze5o4E3lkcQEiMnEUeHllVdeAQCcd955su1vv/02brnlFgBAeXk5du5szNJZV1eHBx98ELt370Zubi569uyJzz77DCNGjHCyqrZTXVOPLSpJxpTU1Ifx9nfbMbRHEboW5zOdm0ft64YZirCPWKbLfh30I/SkTzDEORpoqv0hUqi0z6iuqVddLFGWWM6i5kUrSaYSrX7lQENk5Nrd1RhwUkvqf3yEtB1Jo1PdwFHhhUVDMGXKFNn3cePGYdy4cQ7VKHlc/n/f6mY1jfHynK148eut+MvMjdj+9KVMM1pNNa3qNnNmIxqyvMXuw/prTUnfNV6HXa3BIyKyHS/l4LE6rNp1GIO7tra0WCTBTzgiovfjswAAE686Vb5TZuqxdh3l8iRa7cfoMrHABa00D4S3UArFbvtK0tpGDsEiuADAil2H45/nbKrE8h2HDI/hmamw5nkhvM2J+rDufi2fBpZoIa3nL4qi9vEax4z423z8asr3eK/Bj4FIHsfqGhMSHjiqHlIPWDcbSY9WJq2TlTO4TkAQ8OWGSkt1IdzDbY0ZCS8e4ldvf284wwb4hA0rmhwg6lzHa4Yg7KemzkB4kTzDWCgjwKZB09LORUTtSCStYyqqo3bwmYr1dIjkkmg2akTNbMTlRyfKP2sNYs/P2owaHaFbEIB/LiIh1y9ENWWN392e85Lw4nHeWbhd9v1EXRgLftivXliFegbBQ6vjqg2FcdofZ2HIs3OZr0c4A4/mJcRrNtJoItFcRAyVU8FqRAthDb27r/ZseCZEsjwvOtFGn6+twHNfyBfilfY1AoC6kH67JrxFxEPSCwkvHufRT9bJvt8xdZnhQntSXpn7g2EZrZnTD5XHUFMfYdIGEc5iKLxIniFvhl3Nc0Jb82IEyS7uonynjZYH4DIBSIre+8FKLNl2ULPoXEXiy7AiEq7WZadPgh1RlMsrbi/eS8KLz/jGgSy4NM54DBV1xwkDs5GUes61jfRCXU2vKk2NKulIn5RSeJQONGqaFx5hk0fQ2VslD6eV+tsEBKC2noQXPyHKtG4uVgQkvLiOnT5PPHZrKWo2cOWYZfbchD387Sv1DNMxZHleOH2U9JKM6a04rX9Oai+uoqN5UXPY5ZlF8zzZI7VRJ+JDx+owc225TNMS1byQ2cgvCEKis7ab0EISKYKVdqQZLaBIRkUpP7yL9FnxJqnTdtjV1ryQ8OI9RI3PStQddjmuY+LR/uLVhdhSeRS3DWzMsi4AZDbyEfuO1Mr86dx+xUnz4mE+WbmbuawV+6NmkjLv+GYRBsg1L3KnSCP0NS/k8+IX9AaT8dPXxD+rKebMOuyysqXyKADg45V74tsCguB6ojMiCqtm/aqXv3O4JuyQ8GITczZV4rFP1tqqBr33g5XMZa1IwWo2cCuOnkTykS8PwLe2kXaeDvMOu2RmTD7Se668/bsPn8Dxhjwwamaj9eXVzBFiVh6tNLpIEEjz4hVYJxvbGfOXJQMyG9nEr97+HgBQ1iIPtw3qnPTrWxkqWGZS0Y6RBBqvItW8xTLshiPaCcSUR2ud0+zCjCS6JB+jiKL6kAhkqZuNrn5lAa7rV8Z0nXqt2HqWYyVawednbULViXqd0kSy8ONkg4QXm9lzmG+xKrvCzaw0PqYsvKbPTiQDpcNuJCLi4knfYGuDul6LSETUnHFHRCDIkWBXOiiSz0vykTtTJu6vDYcBZGo+mw+X7mK7joVHK9UK8qR8IJzFj28rmY1sxi2nVhHmG6CaAJUYbWTy5ERSkCWpi4ioPFJrKLiIoohrX12IQ8fls9+sYLRb+NlL32JPFbswLjVHWJicEyaRmY1U3ulYWLL1VaXNI9W8EN7BTJMoKcixvyIckPBiA09/vjH+2S3DipX+SG2WprcqLeFBFJqXDC2ViYSjtSEsU1lLKzcrCACorglpJiBT0/SFSfPiKrJoIzXNS4N/idVVpQn/IYoidhw4ppvTiZd//Posq9WyBAkvNjB5XmMWW/fCia2YjVh8XkyfnuCEtwk9M3MjlmxvFDJCEbbkclph1GYXg2ZZioJwDpnPi4qAEgsm8LrsMnNtBX7/8RpqTzbyf19vxZBn5+IvMzcZF2akW0m+becyAwkvLmOXUBCxYDdS68wop4t/eFmxBER9WGQSSLV8Xdhyw+ifLzaTq6mnhT2ThSyLrsrzj2levD4RGT11GaYu2okPv2fzwSGMeWH2ZgDyibYUr7cJNUh4sRmWjl9KyKZpkNONz4+NO10JhSNsq4lrtD2zmpeQTHiJCi49H/uCFvZMFtK8TGpmowafF7+Y9PZW8wU/EObhbRNemNxStJHN8D5Tu0LUnPZJIZ8X51jww37sPnQC+4/W4Y7B1sPs6yMiU2ek7bhprmdSal42VhxBOCLSwp5JQh5tpG028gt+EbJSAd477QHZhYQXt7Hr/XTiPRcNZnKEdTZVHMGNry+Of2/VNIv52KhjbqLy9MWvtqBds1zD453UvIiiNzq4dEI62KuZBBvNRsl7ma3M0KnPSR68bYLXwuAEZDZyGbveT+Vy5VZJCJW28dxEI2t2V8m+b957hPnYbn+YiQVb96vuGzdtteHxViyWan1dWLbuiegJ1XI6IU9Sl7g/LrwkqT6A+eUlAO87FqcSvPfaC682CS92w/lU7VKNOm42ommQIyjXdvlsdTlz1tFwRMS9H640fW0tsxFbyvbEY6UJyCKifIkJaj/WEEURd7+/Ag/8a5V2Gcln1Wij+nDDueyunTZBQcDXG/eaOpbaTBLhvNVWhFK7IOHFZnjXBLJrduG4w66zp09b6hR+CHuqavDsF+zhjFY6eC2z0Yl6c74RIZ08LzSLtsb+o3X476o9mLb8J1TXqAu30rag7vMS07wk72EEAsCvpyw1dSz5vCQP7jbhvuxCwovd8AqkXp1dKIUwj1bT99RZDCO2Eq2mFSptdqXfkMxsJH8XaCCyhnSBzFjUkBLpLfZKqLSVGTo1meTBe689ILuQ8OI2tpmNnHDYlcVe2n9+wrygECNsId263Wni5WYjuc8LDUTWCEpuJovAq55ht8FsZFutjAmSz4sv8GOoNAkvNsMfKm3PdSlU2p9YFV6saF6spIlXa7fS04kKnxfSvFhDtmq4RpsxWlU6timpmhezoWugNpNM+EOl3ZdeSHixGV6J1E6fFydNUNSPOEOtRbORFe2J3TNbpc8FaV7sQ3r/pJqX2lBjBmOpgKO+XllsYxJ9XiyFSlOjSRbcZiP3ZRcSXtzGviR1zkLdiDNY1bxY0Z5o+bywoHZkRJR/Jp8X+5DevVibqQ2FccYTs3HB8/OiZaT3X+XZsmheePIMsRC0IL1Qi7HG/1bvweIfDzCV5c7zYqZCNkNJ6myGP9rILp8XCpX2I66ajWx/pvI8L1JnTRJe7COmefmh8hiO1YVx7OBxAPLBXk0wjTAIL3ZjLc8LtRmzbK08ijHvrQAAbH/6UsPy3GYjD6heSPNiM/zRRvZc12gM450BCYIiw66JOhHGuLlyriXNi6pZovFzRBRlYjw5X1pDOnnQijaKKMx2CedoeIv1hAK75QUe4aVlE7nWh9qMefZwLslB0UYEN/bNLvTPY8brX1o3mgQ5g1XNixXsjjaKKIRduc8LNSAryMxGKgKvKIqGGXbjZiPG69gBj6N/u+byJS2ozZiH985RnheCP9rIpusavedmbM+i7DN1JE5gNc+LFSxFG6m0B5nDbkQEZNFGpi9FQOGw2yDwyn2KAKXZLuEcKudKvI69D4rndMqiJLuYh/c50vIABDf2hUrrC0IZZoQXufRCOICrmheHQ6Xl++Qb9h+tddVk5jekwmJtKIwdB47JnDGVmhdVrVrDNr2JiJuvuVGbIdjh1rzQwoxyJk6ciDPPPBP5+fkoKirClVdeiU2bjFOfz5s3D3379kVOTg46d+6MyZMnO1lNe+F8qMlKUmcu34Ko8omwkzoLSeasYkUboudTobZf+n3b/mPo9+SXuPTF+eYrkG5IbmdtfQRDnp2Lx/+7XrbbyGFXTPigchmbmyPP6ZRCFWnrLMB57yhUWsG8efNw1113YdGiRZg9ezZCoRCGDRuGY8eOaR6zbds2jBgxAoMGDcKKFSswYcIE3HPPPZg2bZqTVbUN3mearGgjM2YjvZk0YQ/KtY2SiZW2p5cELbofkJsxGvfNWFMOANi896jp66cbRj4vEYXmRc+hWtfnxUWzUUTxs0jzYh5eM78fHXYdDZWeOXOm7Pvbb7+NoqIiLFu2DIMHD1Y9ZvLkyWjfvj0mTZoEAOjRoweWLl2K5557DldffbWT1XUFO81GevDKLoIgKKKNqCNxghN17gkvVsxGyoEGUEanKR1IG794YUVavyG9l7UqC2eKor7mS7pNT0Cx+y0PqTUU1mtTl2Ma3nGFt39PebORkqqqKgBAixYtNMssXLgQw4YNk227+OKLsXTpUtTXJ66mWltbi+rqatmfm7gVKm10HjONTdrJ0STIGcyu4GwH1rLzGpmN5GOPVE6yknWVaFxgUUo0w3bjd/XnI/9fFZvfcx6ncKVQRZoX83ALLz7UvCRNeBFFEWPHjsXAgQPRq1cvzXIVFRUoLi6WbSsuLkYoFML+/fsTyk+cOBGFhYXxv7KyMtvrzgNvkjr7Muzqn8fMgEH+us7jpvDi7NpGCs2LZCdpXviRrW2kFiqt1HTpaMZ0o43MVlADK34r1OeYh9thl7O8F17hpAkvY8aMwerVq/H+++8bllVqCWIDvJr2YPz48aiqqor/7dq1y54KJwnbnNJE/U7JzIAhKgYjNR7/dB1unfK9pYEwHdh3pFb1HvrVbKSmtZGvbaTc1/jZCx2f32DJ4SLVVKg9n/pwBFMX7cD2A9o+h3b7vPBoT5RFP1m5R1VQO1obwtLtB6nP0UH6HJ/+fKNheX4tl/svcVKEl7vvvhuffvop5syZg3bt2umWLSkpQUVFhWxbZWUlMjIy0LJly4Ty2dnZKCgokP25Cf/CjHZpXvTDbs0JL8ZmoykLtuOrjZVYs7uK+/zpwpfr9+LMp77EA/9albDPbuHlL1efylzW9mgjpeZFwweDNC/8SO/2h98nTtD0ortivPntNvz+47VY/ZP2u2q/5oVDeFG5+oIfEtfmueaVBbhm8kL8a6m/JqpuMXneD4ZleIchL5h+HRVeRFHEmDFjMH36dHz99dfo1KmT4TEDBgzA7NmzZdtmzZqFfv36ITMz06mquoadq0rbbYLgC3MktHjx6y0AgOkrdifsO27zM+MRDKxFGyVuS/B50XTYNX3ZtEU6kditkvpdhLF2hu065o7TgqceatdWaysbK44AUH+fiCj8j5HXYZf7ArbjqPBy1113YerUqXjvvfeQn5+PiooKVFRU4MSJxpdv/PjxGDVqVPz76NGjsWPHDowdOxYbNmzAW2+9hTfffBMPPvigk1W1De4Muzb6vOjN4gMmnrRR6KVUbWtm+QHC/sGCJyReaTa678Iu+HTMuUzHqrVbpZ+F1mBqLudQenK0NoRIRDRsJwnRRialF7ujCnnqURMKY8XOQ7Jt2RlB7QNoxqQJS7+ybMdBXPXyd1j902ETGXbdf4cdFV5eeeUVVFVV4bzzzkNpaWn878MPP4yXKS8vx86dO+PfO3XqhBkzZmDu3Lk4/fTT8ac//Qkvvviib8KkuaONbLpufTiim2qeV1UvQN6RqXVq9ZLRyoxwlO44sXYLj/Ci1LzcO7QLerdrxnasXhI0FaS/1Qthln5gb3UNej32Ba59daFh2cS1jUwKL7ZrXthPuOvgCfz85QWybdkZ1LGYw/i+X/3KQizfeRjXv7bIl0nqHM3zwtI5T5kyJWHbkCFDsHz5cgdq5D3s8nk5buA7Yd1hN/p/dU095m3ah6E9imRlM0h64caJdY2saF54hAp1p1H5RqUZKQYpXtiIJfNbtuOQQcmY5qURs87YXoo2AoAsPeGF2pEmPMPK8bowf54Xzvo4gaPCSzrCO6u0y2P+WG1Id7+ZAUO2qnTD/6P/uQwLfjiAIV1b45q+jc7XQZJduHEi0ojHfGcpSZ1qEjT5dy1NAJkY2ZAKokaDUTTDrrGDvRFurrVF2Ad3qDS35sX9d5iEF5exS017rFZ/IORtbIKgyPPSUNGY9/+8zfswb/O++H6KIOHHivCgBY8/iZW2p36s9gkp2ogf6TtrNDMWoUwKmBoOISnyM5IO733zY3uh+bLL2NVkjtfZr3mRVs6onl6QxP2GE90Fz+rhVjostTwiepoXyvPCj/RRMjnsSspYyZ7sJXSFttT4iY7g+NpGHniHSXgxSWV1TYJnPGBmeQCbzEZO+LxoLKynWjZFOstk4sRsh0fzYvvyADqn09K8ULvRRnafDMqKNpmNvEaq/I5k4/R984LwQmYjk5z1568AAP+7e6BsO28ImV2Wg+MGPi9mkL8AxmprghMHbhqPP4n9ywMwOuxKpkwREQh6oCP0InLNC5/ZyAmTJOEfeJ8+70Qq5UOl04El2w5aOt7L0UbytWr0y9IMiR8nbpmVaCMeWEKltZaXkLZFGmS1Ebg0L9oO0n4mRX5G0uHVaJLZiHBtVekagygBc0nqEqONCPtwxGzEE21k4fJaodLSy8sdSBs/S+uYKoOsEwRl5jX9snZFG3kNu5PmpQPb9x/Dip2HuY7hvcsekF3IbGSVjyymqLar865tSDMvCFppts0kqWvEuJrUyWihdeedGGAyOGwwVsxG6gszArmZwbgWUBZqT5oXbnhe2VQ1G6WKEJZMzntuLvcx3GYjD6heSPNiEeWChNzLA9hUj9oGzYtWRkozjU2m9jfyeaFORpWtlUewSmMhPCdumdm1jV4f1Y/rOmpqaREicjMb07nXSEyZWknqPl65G+8s3M517VQnEhFRUx9WPEv91qJcQiBVNFqp8Su8D7fZyJlqcEGaF5txa1Xp2lB0oMjJDKKmPtGEZK6xcUQbmTp/6nPhC99o7nN7eYCY9qRvh+a46JRiruuor3UlF56Py4QXieZFUsdHPloLADi/WxHKWuRx1SFVue61hVi56zAevbxnfBtLU9FykPYzFI2WLPwnvZDmxWVs83lpEFhyNBYy483zIggCOew6jBP3zEy0kdXsyzFEyDV8xyS5h6Tl1S53pMb+aDm/8v32Q6gPi/h2S2MSSCNhRFTYjdJF83LvBysw8s3FJOQ0YPY+kOaFcC2ErKbB5yU7U10etby2kWGoNHUevDjR3/I4ZseWVjJjUlRPUid3GpVqXox+aybFSydQK3HCN/JhiYhiWvq8fLJyDwBgS+VRdC3OT0KNvE29SS987lWlyecl9XDrmdYZ+Lw4n6SO+/RpjxMCnxmzkRnNi+rzFuUzZel6W0aagAxaHCuB2np24UVEqoZKa/8O6fsTshI6l0KEIsZrU01f/lPCNl6NjfuiCwkvKUOjw66G2YjzSQvgE0hSpK9MKk5MjgOCwCxAN5qNTOQAUql8VPPS+P24hsOu2s/mWdYgXYj5sQHGg5IoioroLseqlVT0fkYkBYU1q7BoXsb+a1XCNt6754X1ychslCI0OuxqRBuZkJV5QqXJbMSPE3Z6AdGOhSX1f6zDN9MPqeZ5UXwPhRsHXKOBlWdZg3RBajb697LE2bKUq19ZgL4dWsS/p8pgzvozUuTnWkb6zvHAHypt6jK2QpoXm3HLFhh32M20R/MSVUNLBhwKlWbmp0PHVde9UuLELRMEdjNQo9nITPblxNqv2HlIpiEIRaQCi0wUTjiWHC4TkQov7y3eqVv20PF6fLlhb/x76vi86JiNJPtSRVizSsjsc/fh7SPNS4oQ07xo5nkxo3lRRBvRAKPPe4t34vX5P2Lb/mMAgC/HDtEt78z9FBoEaAbNSySmebFHePnXUrl2QDqASq0e1IzYkJqNeEmVwVzvVxw6Xh//nCq/1yr1JjUvvHePHHZ9it6g49Yjjdk6tXxeeNtadPiTal6AOp0Xg/oOYMJHa+KCCwCs26OenC6GE/eMS/PS8DjtCpVWIrW/G5Wn9pNIrUq+JlZYJuDjLumG/p1bGBd0Eb12IX3XUkTRZBmzjst+DJUm4cUEei+K2wKpY6HSoqjrDEY+L/yYuWP5ORkoyNFWmApgz/USsWQ2Mi4Tjkh9Xhq3U0th40S9s5qXDi2aoDA30/Q1kgFrv0KalyhmNS9+9Hkhs5EJvPyiaEYbWQyHFdEYjk3Yg5l2ZBRNJAgCszAStpCkjqXqUqdho0UDPfxKucYJg5Xi9WC5n1EtnQdGIT0Y20Wq+PhYxWyeF36zkanL2AoJLybQe1Hc7oS1oo2sOmWKor7w4vbv9iNm7llA0Lc3C2DvWBqfrz2h0kpkPi8yQZgaCwumnS8ZiUWmeRnWO+DlCWUyYcnzogZ/nhf32w2ZjUyg96K4/QppRRvx9lEiEn+LrvDCd3oC5oQXQdDvNgSBPey40WzEXw+WwSLE4/NCLSjpCBw5gdyCQqXZEUURC344YPJYvvJeaDckvJjAyxrKPE3hha+1iaKI2ev3SrcYOOw23pR5m/dh5tpyruulI+bMRvodhwCB2efl2637G87plM+LuvBCZiNvYLfZaM6D59l2rhjk88LO1xsr8fTnG00dyzt58IDsQsKLGfTNRu6+RHnZ6pZA3tm1CMiEF0OzUbyciJvfWoLRU5dj35FavosShkSFUB2zkYFZScqugycAAOvLq2XbP7rzHMNjWdq5PM+L5Fi18xmejbAbAXzLSRjRqVUT284Vg7U79fKEMll8vbHS9LHcw5YHVC8kvJiAxd7vFnlZ6pqXi04pQXZGAOee3JLpPMrGbBQqHY6IeH7WJny3tVFtWXWijula6cSKnYfw6CdrcehYnWmfFzvKSNl58Ljse5/2zfHayL66x7Bk8JWWkWtevPv+pBO+MBsxlvNyn5wsrGjReCeaXmg25LBrAl2fF5ffIS3hpTA3E6sfH4asYACdxs8wPI/yNz75v/VY9ZN23pLpy3fj/SU78X/YylfhNOPnLy8AALRumo0h3VpzHx/1eNEWIu0yBRidg8lsJPN50S9LAk3yseKwmxEQHHcoBtjbBZmN+JQhyvv68PQ1jl3LKUjzYgKWWadb5GWpy6MComHUrCYFZWegJ7gA0ZT4BDuhiGhK1R0Q9AXkaKi0+XrFr2PQMzA57Gr4vKjh3TfKfsIRESPfXIynPlvvaj14EhoqydLI5G037NFGjlbDF/A8SqtDmAdkF9K8mEEvGs3tqIkmGpoXXkmZN29CZpDkYCV6HUTr/GxT2gYj4VOAPYscGl2Hzeel8UWhPC+NLPzhAOZv2Y/5W/ZDFIFHLu3haLr1zKCgmv/DipYuKyMgWzXcKVjfEcrzAq7nYVVTRcsD+BQvm41ybRJeePuCDNUB0/0G7lXUQtFZEAT945JlNmIZLCjPizpSoe6Nb7dh3H9WO3at8cO74/EreqruE+LrYPGTlaTJCnuodHq1KeXv3Xek1nDlcdnxFq/vhZ6dhBcTeFXKFwSdPC+cze1ITYirPGleEtEdF0TRVIfLIpjYYjYyOAdvqLRykc9EvPlOOYHyGfIMOrw0y8vUbjMCYPa1TZrZiKKNEqipD2PoC/Nw/4cr49tmra/gOod1zYulw22BRhwTuJmk7uSippr7cjODuh0VDxsU4bNGZAQ90Jo9hl7/IBrs18JIqBDAvjyA/nWMzEbG5+DyeUmjwUcvPNluDYKgM22x4rDrNZ8XL/sh2s28zfvw475j+GjF7vg23gm1dZ8X9/t7El5MoNdOnH6HsiWdRvsWebJ9uZlBTYnY6aaWYeThScgQRbNmIwOfF4MkdqwYCy98ZiNDnxf2qvkevVtrd/+h1x541sFSkjyzEdsNSSezkVoSSl7hhTQvBnzzzTe4/PLL0aZNGwiCgI8//li3/Ny5cxtyD8j/Nm40lzXQKXST1CWxG578S3kujpzMoPYsy+HWlpWhf/79R2vxz0U7UF1T72g9/IIoirq5KbRm54Kg31HblXjMSJPGG210TOJMmD7DjDp6GZDtvjfR5SQ02pKF82Z7TPOSTqHSau83v/BirQ4pL7wcO3YMp512Gl566SWu4zZt2oTy8vL4X5cuXRyqoTm88qK0b5mHNoU58e8ZQe2kU25rXm5+awn+8PFaPOSgc6KfMHLY1RJAAoKg36GbcNh98speCdvUHbAbYVm8NixxTJ21rtEmryZ8ab1Se6trcMc/l2JBw1IGqYBeNJj9ZiNovvxWzIue83nRiAAVRRFrd1ehNuR8ZFSysEd4sah58YDZyNFQ6eHDh2P48OHcxxUVFaFZs2b2V8gm3Iw2kvY3AdVF+LQHPScxmqmv2xP1oZm5js+xLFURRf22khkQoJafmMXnhVfL9rPT2yRsMxJG2cxGjZ9/3H+s8Vi182mIZBOmr8FXGyvxxbq92P70pYbX9AN676L9mhdA0DipIJgfxJIlvLDeEa3f8c9FO/DoJ+swpGtr/OPXZ9lZMdeQTixEUYQgCNyaFMs+L+7LLt70eenTpw9KS0sxdOhQzJkzR7dsbW0tqqurZX9OYyTlOml/lUq8AUFIkMK17duOVSl6fkZJ3ANtPmnomRCjDrva+zM0fAoECAZJ6vijjdQGUyPTE8ugJ9W8VJ2oRzgSjbCqrK5JKKt1ut2HTxhex2/YuJyQIXqh8wLMR056LVRaqz2+/d12AFEn11RB+m7G8vfwCqFWxygSXhSUlpbitddew7Rp0zB9+nR069YNQ4cOxTfffKN5zMSJE1FYWBj/Kysrc7yeulEkouio9kXaaNQ6Jr2OyklYXx4vJDeyk62VRzH8b/Px+ZrEVbR1kxmK+t5RmRqaLBafF14tm1p5Q58Xnd8WIySxLYkiUH2iHn+dvRnPzdrMVb9UQz/ayN5rBfTWL7KgecnOUE/JYDfM0UY6ZqNUQ9p+YjmDQix2XAmWfV48MA31VIbdbt26oVu3bvHvAwYMwK5du/Dcc89h8ODBqseMHz8eY8eOjX+vrq52XIAx1Lw4eG1pkwkIQkLno9mkHG5rap2gWqcZjogY++FKPPnzXppLGfiJ3/1nFTaUV+O37y5P2GclPFjLbGPk8yIIgq5PRVZGIGF1cLXnZOzzwuewCwAHj9fhxa/V175KwTFGE32zkf03QtsPTjCtecn0mM9LWEOa9nv+l1CDVCbVxMqFl+gP5A0Vp2ijJNC/f39s2bJFc392djYKCgpkf06j11CivgzJeWMCgpAgcet1VE7C8zJMX7Ebr8770cHaJI9jtdrJ/Ix8o/QGKi3NRyAAXelYgL6cOum60xO2qWkC7PF5kZc5fFx7lfF0yrqb3FBpnWgjQVtjYUTSzEaM7UJLCPNzu4pERFz4wjwMeXau7PfJhJeY2SjJDrtewPNT3xUrVqC0tNTtasjQ67iNokjsJKBQ+0YHLu2Oykl4Zzj7jvItwe5VgjqDvN7AIEJ/YUatjMUCjDQv/M9a1efFMFTa+LzKAeXQMe0wea1XKgX6WFcRoDeh0dZYGGGUGsEumDUvKdh+jtWFsP1AdMHbyiM1KC3MBSDv42OaGd4Vvq077LqvenFUeDl69Ci2bm1UE2/btg0rV65EixYt0L59e4wfPx67d+/GO++8AwCYNGkSOnbsiJ49e6Kurg5Tp07FtGnTMG3aNCeryY3uoGQQRWIngpC4LL1bodK82qZkOi06iZ55xVDzouewqxkqbeTzYpRcTv2cSjKNzEYMnaWybR6r41tyAvD3zNkMTiSp07qFZqJUYiRrORDmPC9amhcfNx/ppKI+JOJf3+/CmZ1ayN6J+ohZh11rdfNC9+2o8LJ06VKcf/758e8x35Sbb74ZU6ZMQXl5OXbu3BnfX1dXhwcffBC7d+9Gbm4uevbsic8++wwjRoxwsprcuLq2kUI6Ub60etk0nYR3Aud06LbTLN1+EP9dtQdrdldpljFaRkKvFWlFG2VlBAzzvEjJz8nA6WXNMH/L/obrqvkm8UcbKf1m1FDO6utCkQZtYWLZ77cfRK+2hYbnTAX0l42wt2+JLhehsU8wn1Y/acIL66rSfpZSNJC+llMX78Br30RN7Z/cdW58e7hB5ZSOGXYdFV7OO+883cY3ZcoU2fdx48Zh3LhxTlbJFvTNRiI+W7PHsWsr20yi5sUtsxGv5sUDrd8kuw+fwDWTFxqW07NDG2letKKNJl51Kq546Tvd46Sn/e7hC5CXGcTJj3xuWF8pRj4vdQzOEsq2WReOIC8rA0dV/IT++N/1uOWcjp5QR7uJ/dFG2kKyAH5fiRjJyvNSF4pg894j6FLUVLdtaA3efvbtkFZ98baDjdslZeobJgjJT1LnPp532PUielJ+RVUN7v9wVdLqkqB50SjnfKg0X3k/Cy/bJQnX9NBdAwv6IfVqmo/XR/XDyUX5XFFKWcGAKYHAyOeFTfOiEF5CEeRmaYfYqq555N+xB0B0BWClU7fRgp12Igj6fnBGg56W1iZZmpff/Wc1hv31G/xr6S7dcpoOuz5uP9Kqy9cGa/wcMql5SQWfFxJeTKDXUPYf1Y6osANlm2H2eXG4rfGkfAeAJPV9rmLs86J9bKaK5iM229U7b0ZA7tAbHbzk12XBKFSaJd26mvCSpye8sFXNN4iiiFMf/wI9H/uCOT29/ZGKOnlekJhqIUbPNgVo0SQLfTs0V91v5BNlN29+u013f0pGG0mejdZzqm/QgNoVKt21uCnT8e6LLiS8mMIow2kykTZafWnYO6HSgP76LqmC0WxIVwhR0XywhKcq72tQL0mZDlrCS+xcZjQvtaEIcnSSm6VaQrHaUCSeAbWiqjGrsFHmZTtRCq9SAjqal/+OGYjFE4aiICdTdX+y8rywovUu+blJSesudR+T/qTY8+MPlVbf/sw1pzEd7wHFCwkvZnDVYVcBa12clhW05j1a+NlsxIqu2cggw66aw24sPJWn9QUE/rWOAP2FIQE24SXB58XgGLX75Z03zRpS041+hm57r6snvAqCoBliHAgIyAwGkJOpLmwmy2zESqwfPFYbwr+X7sLBY1ENuK/bj1R4kZmNGreHTPq8aE0UjDSujbjff3urBfoENz3blU2GtdE6Hm3E6a+QBooXi2YjNc1Lw0DC0fzMarg0Hb8b/mdx2FXy5Ya9BloHXw81zOj+SrsddgP6z9Joxp6dqRH1ZuATlWxiffIfPlmL3/1nNX719hIAPte8QF1gkTaSepM+L1rvL+sw4YW5JwkvJnBTvW0khGg65zlRGQlqA7XeXQp6ofU7jG60EYwWZlTJehuMaV4M2p9BiLYVYpqXes61VABgY8URbN57VHO/usOuj0cfDYyiFe0kmmFXa5/xoKeleUneqtKMNPyM/62KrjG26qdoCgPpvf7ZS99iU8WRpFfNLDKzkZbmJSa8cL4nl0yar7qdVSPuhd7bYy3QH+i97154qGp4zWHXC97qTmO4jITOsWpmm5iq3s3x3EuPraY+jItemIfff7zG7arYhjMLM2pNaATDQS/XI2Yjo/vC4nO36qcqjJ66zKYaOY/0F8mEF8l2s2YjLUjzkuK4mTvAbJtxfG0jFS2k3iwyLXxedDUvRqHSia+mVu6XZOLkc+N9r2asKceWyqOYumincWEXkP4c6W3T+5V29ywBHYddQTA2G40ecpLqdq0kiqz84bJTLB2vRHN5CcX3Qzrra3kNebRR43ZVzYtNwgvr++0FhSgJLyawI09D02xz+QGbN8nS3Dfu4m6a+wxyjllG1WykOzg7WBmPoO+wqy/cqckpsQHDSr9h1QzjpMzJWzUP+c2rovV89R127f1RAYNoMyOBsXV+NgZ3bZ2w3arPyyW9SiwdryT+KxTVSpUkdVLhRJbnxW7NC2M5L9xVzy/M6EWMHDFZKMrPRp/2jWnbWRnUpRXKmuehR2l+wr7hp5bKQjKlNM/TFnrsQM9hd7/KIozpECpttDyAXn+jdn9imhejDtnJjsXJp6ZWb7VtB47WOt6e7UD6mH46dAItmmShSXYG9J6Q3c9Ob6FOQQCG9ijG8p2H9c+hss2q2chun7eUDJWWtAaZ8CIpE/M9s0tIYzXne+G+psH81370HpyyET04rKt6QcGc30dAEPDo5afg2n5l6qfVOGXLpvZ09ndfcLLqdlWfl4bXrN+TXybsSwezke5sSBQT7pnUz0UtZDE7qJ0jJVk4+dxUtQ6KTUu2HUTfJ7/EXe8td6wediGt+g2vL0L/iV9Ftxto5OwkIAg6TvwCbh/UGb+/tAf3ea067PJqgo1uiygCew6fSAjH97XDt6bmRX27HbC/3u7fVxJeTKAn5Sobk5bNWIC5cOFT2hTwHwQgWyc5GCuDurRCp1ZNVPd9tbGS61x+jTY6dKwOL3291bgg9JepV9sjbQ9Kh90bzipDYV40YZir/bGTZiOGMrHF6T5fW+FZ5/gYyoHzSI3xqtp2RxvpmY0EISqEXHFaG4M6JeI1zQsAnPP01wnb3B9izSOtu9SxWraqdDzk2Z77ST4vKY6esKsUbLQXShS4ZrF//vmpeGNUP5zRXj1dd/y8zGc0B0+f46VMxHYx9l8rsfDHA0xl9WZFopjYVqTtQSq89GpbgIlX9easqTM4qnnhTx3jabSevm6/b/OgENTL8xLbbOKRWhVelO3or9edZul8mhNKDwyyZpE57Mo8diUfbf59yqagNcH2wm0l4cUEepoXtX0X9yxO2BbVvLD3Gjee3R4XnpJ4nmRjV9SSX81GPD5KusKLSrSRVGCxMjN10izBnoGzkZNaq2vrlKhpHbzQSZpFSxhL7sKMOnleGvYYvYvqPi/W3l+pT5cgWO8PWGUXP/U6MvOQRqi03Q7JyuegFVXmBXMcCS+ciAYJOpQDlgDgb9f3SfDYDwgCc8SN3nvdr2HhtCEN51eLRvrP6AFsF7JYFyV+z7D7w76j+G6rXFjheWUN87wohReZ5qWxcTgd5s6D1rIBerRtnof+nVsYluPtD2euq+CuSzLRjjay7vDPip7ZKPYojZ6oWpXMtAOt49UWIeVFaxD3W7RRfTiCUW8twYtfbZGbjSRJIeXJ6+y9vrKtaC3A6YW7SsILB/XhCC6ZNB/jpq3WLLPox4Oy74IQzVJ57kktE7azdgB6pV4d2RdP/KwnXry+D4CoOnf9ExfLhIN+HY0HDif47gdtLYXVzs9JYoPL0Ofn4aY3FmP9nuqEfTznUWP+lv2YPO8H2TZpxyHNsMs7KbXLb+LuC05G22a5OL9bo+Bt5rkJYBPA6iOR+Jo02jT+ttnr93LXxWmO14Xw3db9CIUj3vALEHSybseEFxNajwyLAkdA0daNNC9G752mic4Dz4CHz9dW4JvN+/DC7M2y36zl8xITzuxSZCvPo615sed6ViDhhYMVOw9j016+9NJ6HQNrp6FXrmXTbIwa0DHuzAkAeVkZyMuyPwpe4Fzk7+nPN2ru82qodG0ojEsmzceD/14V37ahvFrnCG30zEZrdldhS6U8Vb703ko7c7fu1APDuuHbh85Hq6bZ8W1m1PuCwCZQXTt5Ic7402xsrWx8x7ygnubh9neW4qY3FifMnKXo/aJkOuzGWpaZ9qUc03hfZ6V/l9XBV9tsJCq+e5ua+nD8s3ael8btMeHFrtdE2b9rmYm9cB9JeOHATjcNQRCY/QfMXNapTp+3LloZPI0Gweqaemyt1F4HxynmbtqHTXuP4D/LflLdz3NXeVW60vbQokmjMLrBxvVYRIh497azmcsLisFPbc0lu9hx4DgAYPry3Y5dw2m+2xp15n5vyS7NdzCZodJGeV6k//OgfH958+7IzEbBgA0+L/z32kt8vqYcL8zapNnBaJrF4p2MPT9UOSRpaVq9MKmgJHUc2OlkGhDYnTK95NvKW5cb31ikut1Ibhvw569wrC6Mz+4ZiJ5tCvkuagGjdOk87yyvvT0jKGDqrWejNhSWmU+UuSuscu7JrZCVETB1XjOOxAJ471vjZ/e7SHNkBAQdzYv6nj2HT+DjlfYKbnp9VmOwEb/JRjmoTfnVWVz1krajjIB1ry6tVZL90n5++25i3iJp/xHSSFIX22yb5gUCZtwzCCNejC7caMZBP1mQ8MKBnX4a0RkRq+bF3gYkCOYbO29dlD5ArByri6pP523el1zhReW+mO0XjAQhJQFBwMAurQAA0zQ0PyywzOwzAgLMrPJixtwnCNoDuRr6phPvdqZSggFB+zlobD//ubmotVlQDURtdqrE+h/BhP5dKhSd1LoJTm3H945Ku77MYMBwUmTUfjTvm1+kFxWk7Uf+OdHnxa6fGRDkucS03ncPKF7IbMSDnUKoAPZoI7v7ayuns0sL5IXGr4ad0QlhzlNJb63T2jYeDYpUYDUzE+M+wqNtg4eMoMDtv2K34AJE+yytWgiK/3mQTuT2Vicu/2GEdOKWEeTzpVOjtl793iXk3bJ0leTCIvvGfp5t/ZbSYVfT58X9l5SEFw7sNBvZFW1k7trmz2hXXVibfrLDhNXqZda+y9uhSJ9LiFfyYSRWpaBJ3xWzDrs8fZ373aJ1gjoaDyd/3/BeJSgpyIl/D+hogKxFGzUec7TWOHOwHkEWs5HBTasNhVW3+7ktafY7Ukdemx12le+3l6NCSXjhwFbhheN89Rr2XD1iZhc1zLZHAamvebHTEY3XbCTlRL3287MDs0nwzHZmPDO1iEZkhZ8I6vm8OPibognfGr8HBG0NUExkMDNrDwQE9GwwL2gtGcJKRsAGzYuG1soLjqVmYfGZsttspHwKWpmUsyxmWLYD92vgI2zIpRSHZ3kAM2Ngh5Z5AIDeKrZoa9qMZGtCktv52NnX8WteGj9bEV5YLms2VN2c8MJ3jEwtnvD8/TEY6fm8ONmmBcgFgYCOf1usmBkhOygIePPmM3HLOR3x5s39zFQ1TkYgYDyhMtivKbyYq5InYImgin22S0hj1by0aJKtuj2ZkPDCgZ2LifGYjcww6brTMWpAB/zz1yphsWY1Lzphl7x4wWaqhq0+L5wKM5nwoqM50+I6jZXG1eDxXZHWy1SSOk4HcTseQdWJejwzcyM2c+ZlssIP+xpD+/V8XhxVBgjySRbLBKlJNn/cRiAgoKQwB49f0ROdWzflPl5KZpAhz4uR2UhD2Pex4kVb+JXmeYnYnedF/l2rn2jZlC803gko2ogDO98DAc4KL33aN0cfjUUcja6aERBUV0QWRRt9XjzaqdgZbcQ7G5JqxGo4NS83nd0ef/pZr+h19erU8D9P27MsvBjUSYlVAbKmPozT/jgLAPDy3B+w/elLLZ2Phd9/vAZTF+2Mfw8GApZStwcDgm6SQz1kCQ51HHZjtzkzGMCKP1yEl+duxevztzHXzyqt87Ox70gthvUsYdYGL9txSHU7b9i/KEbXFvNqskyAzWE3Hipt0+ikNN9pPWfevD5OQJoXDuwccKNmI7ay2Rn2PiajWc6Chy/QOdael92jsou9Pi+c55K2B1az0bPX9MYlPUvwh8tO4eqIzQ4+pvK8WGgyZh7Hq/N+NH9Bk0gFF6Ahz4uW2p/hfFb80gIys5FePRq3N2+ShXbN85ivY0f+j8/uHoj/u6EPfjO4M3Oo9NWvLFDdzxOpJYoifvnmYlzx929NC4jJQFvz0rjjr19uxrHaECI2Baopn8OgLq1Vy7VUWUMv2ZDmhQM7TR0C2KX+5649zbbrAtHlA2rqtbN8FBXkoGWTLBxQrDMjCDZ6vHhU9aJaK5NV5Q6VlvQcvds1A7DD8Jhr+5XhWg5z0WkNPlB8QkhjWdMOuzatCcXC9gPHLB1vB1o+L4t/PIAZq8sNjxd4Q7Qkx0kfbVR4YTv2ujPLMHv9XpzXTX3AkmJH8EJRQQ4uP60NAOuTIq1oIyWCICAiNmZC3rb/GE4usmb2YmVr5RF8uqoctw3qhIKcTMPymmZHxfe3vt1m29gUe64Lx1+ADeXVaNssD3/7akt8f2lhDsqrajCEoY04DWleOLBLugWijeS2gZ3RqmkWfn1uJ92yuZlB+y4M4K1bzkT7Fnm487yTNMsYpRS3ijdFF3s1L/xmo0Z+3qetbfUAop3RJ3ediy7F+QAsaF5MmY14k9RZwwsRJlrC4XWvLcKHS3eZPt6IRM2LziCo2JyTGcTU287GbYM6G9fPZnOL8mxndlQ3eWuhledFDWn7SGb28gtf+AYvfrUFz32xiak8i88LEF1Kxb5Q6ej/pYW5uKB7ccL9+eqBIfj+kQtRLAnHdwsSXjiw1clUiNp8l0y4EI9efopu0dwse4WX08ua4Ztx5+Oy3m24j039UGmVbSafO7dK2qJvSQy1wbu0MBenlTUzdX47HHZ50De3GZ8sWU0rHBFx81tL8OT/1ifsywiyazzUMG02UoZK60Y9mcduVxGpwDX3wfPQuRWfNqSecWYpiu6HCkhXqddD+z2Qbxc4tGtGGPke5WVloHW++5FGAAkvXNjq89LwP4vpKCfTmceUlWFuFm0HXpgdq2GnCZxXeEmm6yCPECL1bzC9qrTJaKPE49xrNyfqwliy7WD8uS768QDmbd6HN75NdHKNrm1kvq5mzTIC5O+o3nmsvIN2+b41nq/xs9pK2EZ15dGKu931FOYam4wAds1L1MBol8OuLadJCiS8uARPI8nOsFfzEiMraOK8PmrcZrAzVJr3VHYPCHrwOFxmSGJv1Y5jOVUyzUZO8aspS/CLVxfijflRh2C9cPZgIGBN82JStaFsQ3p5Xlirl4zBXlprMykZeN5bt3UvBYzCixbK2sf8eOyAhJcGvvnmG1x++eVo06YNBEHAxx9/bHjMvHnz0LdvX+Tk5KBz586YPHmyk1Xkws6XmGdm5ZTmJdOE5sWuLMNe6jhl17PxXFaijQBnO5JsDj+qTMlSAmrLCmQZRMMJ4FO9WNXKOdVmYouMvrckGlmklk4gRjDgjllGeZjeophm7tP0O8/BrPsH8x9oQKLgzncDuIQXXc2eM0jbdEEOW5wMa92imk17HXb9gKPCy7Fjx3DaaafhpZdeYiq/bds2jBgxAoMGDcKKFSswYcIE3HPPPZg2bZqT1WTG1mgjT2he+B6/gNTI83K0NoRnZm7E2t1VCfvUOgGzdQ1zHqg0ydmZFFHJn39+KtoU5uDJK3sZlpWmCFerk2E70sk1oobban0jYh283oCZEQhYGlBM+zwpHHQDugMbW/3aNsuNfz6jfXN0bXD6VqN5XlSrwBtkIG1WSr8dFng0D3/87zq+k9tAdU3j+k/5DJFGALujtbm4NHX8I7o4HCo9fPhwDB8+nLn85MmT0b59e0yaNAkA0KNHDyxduhTPPfccrr76aodqyY6d/hA8viM5Nkcbxcg0kT8mFaKNnvtiE6Ys2K6awMzKekRKuP11Ffc2oBEuO/XWs/HQtNWYeNWpqudhGTNPLmqKBeOHMtUrQ6p5URlVsjKCALQX5+NtMlaFF+Xhoiii+kQIhXnW1PUxYs9JX/PCF2GlchWTRynNRtr1aMmY4v3h4d1REwrjmr7tDMu+/5v+eHbmJowd1hWXvvgt0/lj9YyhDPe2E0EQ8P4S42gvu5EuXskqmGo1L6VQIwj2jU1+0rx4Ks/LwoULMWzYMNm2iy++GG+++Sbq6+uRmZnY+dTW1qK2tnFJ9upqNk9uM9jpZMraRq7p284x727pjPnSU0ux69BxjB/eQ/eYVHDYXV+u3UbUamW2plopy1kJBAConGJgl1b4TieRoN1I/VxUhReDFap5oyGkGg3WdiKKoqbP0ISP1uD9Jbvw3m1n45yTW7FXRINYBx/W8RLN0InyUUP5O3nGkKbZGfHBUU0AVqvHjWe3R3PGRGPNm2Thb9f3YSrbvaQAb95yJlNZKTLNCxL7GY8r4wyRTopYTVwsaxsB/GZZPZTtx8uijKccdisqKlBcXCzbVlxcjFAohP3796seM3HiRBQWFsb/ysrYE3bxkuwX6JZzOtqeoE6KVHjpUZqPT8cMxICTWjZsUW+2PhLMNdGb+NipXdtYwbeuTkJqbo/c7AwDsxGL/wyPydXMI5D7McjPEJtp//XLzSbOnEjsDoR0shBGhTz2X6JsdzxP/oPf9NfcF31cifX4w6X66RmSjVWHXdPX5bwObwTh3uoa3Pj6InyxriK+TU9jJ0WrlHKJBL0lIHhJZtCAVTwlvACJNy/WEWnd1PHjx6Oqqir+t2uXcyrBZDvsZhrMaC3XQTKKs75QqeDzoqe2tdPnhRdlrcyqcEub2ZtASurzohYFY+TzIgCYMEJfoydFJojo7JNtZzgvaxs3gsnnhTPPixnNy/xx5+PHP49Ar7aFjccpygQEIcGh+o4hnW3PHWUVaf8uaOh3WfOjOMWCrfvR67Ev8K/v2ceYJ/63Hgt+OIAnP9sQ38ZqmtZqP1MWbJd9F2BvlKRf8JTwUlJSgoqKCtm2yspKZGRkoGXLlqrHZGdno6CgQPbnHPY1kMPH5an3z+7UIqFMJqdDrRWYZxQuCua7Dh7H4h8PWD4Pr1CQrI4hEFB+N3ezJ151Ki7sUYSpt6qsKG4CqRCtFiptZMMXBOCck1qhU6smTNczY1JkOcaudWxYfV54LpcQ/srwopW1yEtoI8qmHQwIGNylNYZ0laRz9+A4p3TYVU5Wdxw4jhEvzuc655GaesMyRs0mHBExbdlPuOKlb3HjG4txoj6McdNWM9eh6nhiHdiFaMYJpY1J6vyEp4SXAQMGYPbs2bJts2bNQr9+/VT9XZKNnQ1k1U/ySJdXR/ZNKGMUgmonLC+UIGjNifgxE7k16Jk5uO61RbIooeN12o6iWkiFl2nLfsKgZ77G5r1RE4+aoJI04YVxRVcjSgtz8cbNZ2JgF+v+HYA8z4tanQZ11b9O7AjWCBS5zwvTIWyaF97FpjSIDax6wlBQ4EtSZ1cbS3TYjZr9/vHrs2y/lp0ozUZ28PD0NZaO/2JdBbr/4XM88O9VWK3or1lR054r242W4M08n7TP5UX13F7F0dHx6NGjWLlyJVauXAkgGgq9cuVK7NwZzZMwfvx4jBo1Kl5+9OjR2LFjB8aOHYsNGzbgrbfewptvvokHH3zQyWoy4+QCpM1Ulhg/pdRJLZKcUJgtRaUXlgdY0yC8LN95CKc8+gUe/5Qv9FE6/j7w71XYdfAE7n5vBQD1Z2xnBJIedpmN7EYabSSt0xuj+uHJK3vh/gu7YtVjw3DPBSerHl95JOpQb8d8U+uW6JmaYtg1aMfaj77PC1+SujWKwdHso09wuFQ5UUYSNbqsBJRmIxua/mcMC2Dqccc/l6HeosCrpj2vV/S1rJl0tRAgeFIgdRpHW/HSpUvRp08f9OkT9VQfO3Ys+vTpg0cffRQAUF5eHhdkAKBTp06YMWMG5s6di9NPPx1/+tOf8OKLL3oiTBpwPkLml/3bAwDuGNwZE686FRedUmxwhH0waV5go8+LlWMbDn5+VnSBM6UN2Ag1oWDf0VrZuaXYNGE3JkHzkqTrGiAzG0k+d2iZh1/274CczCAKczPRWmOxthMNUVfskUPq2/ccPoHqE+qaNpmWQ+N4u31edDUvAT4B/ZrJC2Xfzb5neoP+7y7uhq7FTXHHYOOFF+1AupZWjIEnt8LzKkEICWYjT8e5sKMmvLy7eCd2HTwe/67VTFjfFyfnOBlKW7aHcDRU+rzzztN9AFOmTEnYNmTIECxfvtzBWpnH6THsTz/rhXuHdnVl4SsWfwAR9nmjW5EDYwOVWZ8gtd9QfSJqm1Y1G0VE/GPBdnQp5lssjrteiu+e0bxomI0S/C00jueNmtKaRZ7z9Ndc50k4r23CS/R/vSSEAqfZSO14u7nr/JNx1/nq2jEnmDZ6AAb+ZQ4qqmvi26bepu6HJRVWBHjbXMFDhkbQxaQvt+D5X0SFOM2QaMZrCHDObNShZR4u7FGMLzfsRd8OzZ25iEk8lefF6zitmRMEwbUVO5mjjewyG2m8mpXVNShiXG7drPCidljs96t1JN/9sB9zN+0zdS0e1JPUuY9seQBJnbIVPlla1Y0JOXb6ryQcI1O8qJ/BLs0Li8+LKIquOFF6yXkzIxjAh3f0x8/+/h0OqziuahH1rUsNtDQXUsd3bc0L+3WcMhsJgoA3bu6Hmvowd0Z2pyHhhQO3F/RyEhafFzs7FOm7dvh4HR7/dB32VNVgybaDeIQxrNbsy6S/0m7itu37j5m6Di/KWplOEW8z0g5YeuuUmZ+1hpzYVub3h0EQMYPt0UY69kSrY4lpnxdrl7WdDi2b4I1R/RLMYkoSktR57YeYJEtj/TipRkZL8GAVSOzM86KFU1nerUDCCw+pK7uwJ05y4B48/flGfLxyT/z7UzM26JRurANvHpx9R2qxU2JrVkPtNvCuUWQWtRWBvYC0o5U6MLJ2aLHbx3ob91bXxDPmMmtrGMqFdDLi8sCSYVdkrJMWdjns+hG1UGm7UGpWnfZj1NK8SLXGWlVgF17IYZcwIElBJ0nlqj5tAQC3D0qOE58aPx06wVU+9hh4zEbvLNyOM5/6Ele/sgCfr23MJZSfLZff1X1euKpnGqWw4hXNi/Q+14Uab0YOo9koJvyxvj5LdxzCxM83ctVRqqHR6sft0ryw+Lwo68SLWcOJFw0uXYq0F3JUw65oIzWUT8TME+KZNGn5vLCcgydUOpUn1lqQ8MJBKpqNnv/FaVj3x4vRQxGW7fQMTjrjMXstnoUlH/1EPZxaeQ61J5ysWU3CqtJeFF7CjYstKUNutWobu388s9zXvvmRvYJQLg+gXsY2s1HDL9XTVooiaV5iFOZlYsmEoVj9+DDjwgDgYLSR0mlbFIGdB45j7qZK5nPwmKu1JlgZdmpeYHURUH9CZiMOUlEzJwgCmmQnvxmwmwO0C9rhQKbsItUiUuwa9Hgr4xWHXZnZKKQXYaO+PXb7TDni2ljOPodd4IMlO/HqPG0BS7Q41fHGk7cPVid8wNm1japr5KH2IkQMfnYOAOC928/GOScZJ3bkmTRpaVgyJRMTLSGFJ1SazEaELrzN426NpF2/OrcjPrtnoPUKuYBdNmJLZxFjodLWezhlJ6lmCkie5kXOkG7RlO7N8tzNLl0iGXjqdBy7tWbLESvSCyPJXB4gIAiG2Vujmpfkh0oLALIz/d2t25lPygjpI1qx8zAA4KdDxzFzrXaCO55JBZPmReNYxryhjoZKexnSvHDA2xk9MKyb6vbHLu9pR3WSjp2zISsv27zN+zFyQEeb1n6S/yg1zUvSFC+K+3v/hV3RoUWTuBDjFm2a5WLyL/uiICcD7y7eaXyAgpjQ4ORtTLbmhQUrlzOfpE5AXqa/u3VBcM7nRYlaPzTwL3N0j+Gx5mr1UXKHXWvRRiLYXRrOOakltu0/hvKqGuPCHsffInqSSUPh1jGsKNW/3LAXe6trZB3Ad1v3y9Y8YkXZEanNzpNlNlJqLnIyg7jx7PZo2yw3KdfX45JeJTjn5Fa6mhetEdeMz0sMM9FGWm3LviR1xqNX9Le6IL0AyMv2XlgrD4EkZtg1p1Vle/4TP9+AGWvUNThSrbF2nhe2ukVEURZUoCdcvXd7f+YFUr2Ov0X0JON0WJ2XcLrrsHorK6trZQtX3vTGYgDAkglDue3rUlTNRkkSXjyciTuONNpIiVabsfasTeSG0ThES/Pyf19tQSgi4v6LujJdilUrYMlh1+xxApCX5T/hRZ7nJXmaFzMo6/bX2ZuRERBw99Au8W3fbNmv6xMlS1Kn6bDLVh/l8cGAgEjS1jRxDx90l94hjWQXX6Dm8/LlBvaoASBxhqfqsOtStJEX0RVeNEacuObFkRqh4dx8Z99x4Bg+/H4nDh+vw/OzN+NvX23B4eN1qmWP1NTLJi5MmhdY+71W8pzkZfl7Thpd2yg5mHm1AwLwr6W7cPUrC/DDvqP421db8PzszaiuacwifKBhrTQtDp+ox44DxxrqYM1sFImIsnNIoxQzPBKx6AT+buVJxsoY1q2EL9eBV7FrABIBLPzhAP4+Zyu2HzCXwVZtsK8LhWXfDxytRUEuu8Ormw67hS475rKgXBFXilY3Gc/z4qjDLl/5Ic/OBQBsPyBZIE/lHEu3H8Q1kxfi+jPL4ttYxgOrodJm25wAwZealwSSpHoxY74WIGDcf1YDiGrtYtTWRwBGpe+kL7dg0pdbsPT3F2qu/fX3OT8wnSsiyrU00cR40fc0GBASNI5e1mrxQMILB1b63iFdW+PZa3on5FPxF/Z67N7w+iJrp1B5ItItP+w7iqHPz0Ovttr3XCmsqI3NTiepe+JnPfHpyj34w6WnOHshG7i6bzss3XEIvdsVMh8TiTvsOie9iBqflby3eCduPLt9/PvCHw7EPwdVNHl/axicPvh+V3wbi1ZEhGjJzLzXpENlQAByfSi8yBZmdEnz8t7inUyRfdLHL5ULzGRw3rL3qObEdkN5NdM5IqI8MH9Yz2JMX74bQNQxuFZHW+pnSHjhwGro47X9yowLepAz2jfD8p2HZbNPqzDn79AoWBMKY90e/Zf7k4YlB9bu1i6374hcveuG2ejnfdpi1ICOjl7DLq7rV4YuRU3RXUUIN8zz4nKoNABM+GiNTHiRCT2M9WPWvLCdTpVjdWHjQioIAvwfbZREn5etlUfjn3cfPoFHPlpreIy0atIcWXprXWlhVcgF5IuAPjy8O04vaxYXXryS6NIJ/N3KkwyrA1Xr/Gy8eH0fZyuTRN7/TX/8dOgETmrdFIt+PGB8AANWB7JfvLpQ9RzSbXprz2ihpq53OtpIa/0TLxIICOjXsYXqPm3hxbzZyMxK1Fyr8Wo821A4gl//Yynmb9mfsI/VH8WtVaUHdmmJv36Z/GvbhZDEaKMH/r2K+xjp828i0XJJTarMwpdFIReQm42GdG2N4xLBV83nxQ++dSyQ8MIFWzP75nfn+1J1q0V2RhAntW5q6RyCwBbOyorWwCDdbCavhxs+2akyO9LqFK0M4lY1dEZE5I0yztcbK/HN5n2qx7A8LZ7cG3YiAOjboQU++E1/lLXIS/r1zeKnVaWl5iGpf5GZSY4I6z51EUlYviAoHHZtSOTpVUh44YBZreyfibQmWp2H2fcsIyDIViR2alYqVcGaCXF2Y7acKhEBmgszxnxeOG/ut1v24+Ax9QggJSKz14viOA2B+kS9ttmGLc8LVzVsp3/nlu5d3CKCIHhmRXU1jtU2to08idnIdP9msZ1INS8BQZD1J37S6vKSur/MAVjbWCo3GLMoO3zWe2llBsaaXluKG7PlgJd7ahvgXVU6xi/fXMx9rR/3HeUKl9cyN+nNotleb5eWcU2BpiQgmqDRqxytbVwfKVuSa8qMw65V36joORr9ZgTINS9q6SQev8L7gQEs0CjLgZ0OfemGmnaBxVzCqwmRljeljnVxtpyqiBZ8XtgvEv3vwhfm8R0mqRSryZHFZ8BqqLRZ/OrPIDMbCUBBjvfTBgDyZyzVvDAnMoRouZ1Eo41i15VrXtT62JOL8vHVA0OsXdQDkPDCAetcykqCKa9w79BottGrzmgr2272pym1C6LofP4UXhv0jgPHTNeJBFa9JHXR/53URUz4aA3qwxFL6wkxmxwZo43cWOk3BboeCIKAglx/eDRIn7Epnxcb2klEbGyvSp8XzbWVUsA64I8W4hGStUCfF7jx7PY49+SWKGsud/qz4vMiO48NMw41pAMkb4jzkGfn4mentzF13XRJya1HD418FVaijVj5ckOlLBcLK9IBR1o9vbbDEUiSdPwqu3Ru1RQtmmTF86z4UfMSMmGntsM1SqZ5gVx40dJuq+U08hskvHCQTmsbAUCHlvYt4JXwEolAl6Km2CLJs2AHMrNREh12oz496dU+lHQpzsf7t/dHcUE2Lni+0XyTjFWlAWBfNX9iNy0hxep6Vnbk7zCDXzUvWRkBLJ4wNO4bx5MV202kWpN6k23Gep4XxF+ugCDI/AszNDQvWll9/YT/dUeEL1AKLyLAtLrp87M34/FP1zFfx61Q6VQJd7bKgJNaorMirD6eYdeDsp0s2ojRYZfld9jhiJluZAYD8ffIL5qXiFXNi2iPz0tMiBIEuYk+U0vzkgL9FQkvHHix8/ULSklfFEVmM9yUBdtRdaLeuCCsa17M2p9ToTNwisbH4L0XSPq8pSZHPcGXxbdBjP+TXPzqsKukSbZ3o42kyDQvJoQXO/yi5GYjeZi5Vp6XVEjPQMILBy4FP6YEag67fPk42MqyDkA6JzAFCS9yftGvXfxzMnxeAHOPTitJnV5dWXypopoXF8LuU6QZNs/LcrsKTIgy4YX/eYcj9jjsxk4hCPKJolZOolRIz0DCCwekebG3Q+aRLczMKM2sSWT296XCTMZOnrnmtPjnuPDi8DVZH7dUIydNzcHqsMui0XPKId2QFPBlAKKD6xuj+rldDUOkTeF4XWP+F9b+Khyx3k5EhdlIGvU3uGtr1WNSYbJFwgsHsYaamxnEHYM7o2uxtZT5fsSsLfpITUj2XYQzDtBuOexmZ/hDze0GjQszekP6r5dILLI8L4w+L6xCsd0/VxCANoU5+mXsvaSr+CG1vVRr8tC0NVi24xAA9klQOGJ9OhiJNAregiDIBJOr+rRVPYYcdtOMWEd3VqcWGD+ih2YMfSrTs00BRg85CX+6shfXcb3bFaK9ZK2VaH4DjhOYeNfM5F0wq8LNyUy/tsCK0wtbxmAdBqSr/2oJInp1Zvo5DjjsbnjiElzTt51umVRaU41lGQa3UTafJz9bz3V8WLQelRaReIcHBKB5Xib6dmiOM9o3Q+v8bHx2z0CUFubguWsbtaE+uLWGUI/LQayJxQTbZHXKXkIQBDw8vDtG9u/AdVx2RhBfPzAEd553EoAGtTrXddnKSTsCs0mjzJCXRVkHjPDK2yITXmRmI7a2w2Y2sl/TlJMZxMgBHZGfk4Ebzmov2/fIiB44q2ML7vfSy/hhgP3bV1tk3+O+J8xmo4hNq0qL8esKgoD/jB6Aab89B4IgoGebQiwcP1Qm+GZJJt5Xmsxt5TbU4/IQd4qKNkyPaMF9QUCI5hyIdUjR9Ok8Drv85cz5vJgjL4VmvE7hlfeFxWykp4FjijYSnXHXbZ2fjZWPDkvwWbh9cGfcPrizA1d0Dz9GTvEKrOGIHXlepMsDxP7Xv3eBgIBVjw5DWBTxzMyNlq7vFqR54SDWRGLNwo30334lpgKWdkhOrLwqc7pMquaFhBcjnPZ5YT29tF1IBVzp4bqh0izRRhz14SUVnC1Z8OPPFAHM27wP9324kql8OBKxZ20jicMuK4V5mWjRJAsjB0S1dRd0L7JWkSSTFOHl5ZdfRqdOnZCTk4O+ffti/vz5mmXnzp0LQRAS/jZudF86lIajAeZm9ulKbCmNRs2LyCX88c5jF/14AIu3HeQ6BjA/wKaSr4FTeCVUWpqPQ7Y8gKSCegnH2B3BqX+whA+FFwC4+a0lzGVDdjjsSvwHzWirerYpxKpHh/kiukuK42ajDz/8EPfddx9efvllnHvuuXj11VcxfPhwrF+/Hu3bt9c8btOmTSgoKIh/b91aPeQrmTT2WWQ24kWIa16i8M5MmcNgRRFHa0O4/rVFXPWLX8fUUUBuJllg3aY+xJYkTOrzEpEJL5Jz6eTsYBG6uR3SiQRaNc12uwrc8I4JERtCpaWTaLPaqsI8f2Q0luK45uWFF17Arbfeittuuw09evTApEmTUFZWhldeeUX3uKKiIpSUlMT/gkH3Z7Zxs1FDAyGzETvxd0qi1+TTvLAx6cst2FBezXzehOuYfKaZPgjrTDalDWG93YqjCzY6/bbUMWY43Xe0Nv5Zyzykdy6WXGR2m40GdWll38l8QtfifDx62SluV4MLXj1KKMKngVZDKoAb+bqkEo4KL3V1dVi2bBmGDRsm2z5s2DAsWLBA99g+ffqgtLQUQ4cOxZw5czTL1dbWorq6WvbnFI2e5FFIeGFHGfbIu/YLj1Dx3BebOM6suI7J49LFD4GHD37TH7ec0xFv3hJVRzvt81LHqHm5dvLC+Gctvyg9LQ5TtJHIG0+nTdtmuZjyq7NsOZff+PXATm5XgQveJm5HkjppG06nXshR4WX//v0Ih8MoLi6WbS8uLkZFRYXqMaWlpXjttdcwbdo0TJ8+Hd26dcPQoUPxzTffqJafOHEiCgsL439lZWW2/44YjaHS0SYS4V/KIm2Jje2NZiO+/AZOhFWrXsdkR0IZdhPp0LIJHr+iJ9o1j+b38YrmRYrMYVcWbaRzDHOoNHd1VGnRJCutheO2zXLdroJjRJPUmWsoahYAP+TGsYukGOqVqixRFDXVW926dUO3bt3i3wcMGIBdu3bhueeew+DBgxPKjx8/HmPHjo1/r66udk6AUXh0p7vmJTsjgNpQBLPvH4yL/qouXMaIRxvJQqXZr8Vzr62EWJp9pqmwVojTOP26mFlbRr60kbrzrhK9NpKXFcTxujAgAi99vZW7PkQiM+4dhB/3HcXPX9bX1nsB3jYesqB5CQoCQqIoN32mUTfkqOalVatWCAaDCVqWysrKBG2MHv3798eWLVtU92VnZ6OgoED25xTKWPp0F16+//2F+PqBIejS4NOgh6AIlRbB6fPCcavdmHyQ5sUYpxcq1IsQYoE1z4vevocu6R7/vGnvEUv1IaIU5maiT/vmbleDCd4WbmYJkxixCZPMbJRG3ZCjwktWVhb69u2L2bNny7bPnj0b55xzDvN5VqxYgdLSUrurx02socUG4HSPJijIyUTn1mzrO8XNRg3/b608yrXqc7LkRNOzoIC5V+k2n9n0reD0MzS1irgE6dFmzUZxzaKNghqtZp+6WHHYja1PJG2PZDaykbFjx2LkyJHo168fBgwYgNdeew07d+7E6NGjAUTNPrt378Y777wDAJg0aRI6duyInj17oq6uDlOnTsW0adMwbdo0p6tqiCJS2jMLzfmBgCJUeglnDhantDQJxyZxVekuRU3xyKU9TF3Pjzj9tsxev9fS8bJsuzrl9ASbuE+XjT+Wuhn/wDsmRBPMmbtWzA8qlKYOu44LL9dddx0OHDiAJ554AuXl5ejVqxdmzJiBDh2iWf3Ky8uxc+fOePm6ujo8+OCD2L17N3Jzc9GzZ0989tlnGDFihNNVNSQx2ki+/3cXd/NdlsJkoUxSxwuP8GLFnGfWCduMQ2WzvMy0Cm30kwLB9PIAJpYOueGsMpQU5OKvX25W3U/CS+piJUldrMs5UhOKb0un7iQpDrt33nkn7rzzTtV9U6ZMkX0fN24cxo0bl4Ra8aOMNlJ2Ynedf3KSa+Qf4j4vJt8uJxLaqR5roisZPeQkU5qXtBJc4H3zh7R2ejNotmgj9t+aEQjQquQpgrlQaXPvRVZGtM1I81qlk9mI3hgOlOtHpLvDLg9W/Vmdikyycp0YYy44GRlB/lcpmEYdDeB9DYLMYVdHA7f78AnNfWbMRt1K8tNqxkw0Eragecky0eekEun96zkIhSP4x8Lt0c8NMy+vd8ZewuoKsckyG5k5UgBgph+JzZzSBe+/LlKfF2u5N1joVpyPCSO644az2uvOmL1/34gYvO3GiuYlOzMx63w6CcG0IAsjYVHEroPRGVdtfXRaRpoXdpTRRrw4sZSA6rEmnqkgmIs2yk434cXj7wtrkjo9pKkAjOjVthC/GXwSQ728fd+IRrjXNrLgsKvWf5DZiEggQzI4hRt0yiy2byKKMs8LLzy32spjSWaGXdK8eAtWnxc9pEkYWctGP8vbz6gBHUxdn3AX3gmtJbORSv+RPqILaV6YkUaTxBJ50oSIHWWGXV64lhJIutlIMBVtlHbCi8ffF3s0L/GzcZSVf177x4vRJCuIdxbuMFcJwjV4szxHRPPvhZrmJZ2CANKr97SJ2OBIZiN2lGsb8cJzp6057JozG5nRvGRnuL9SejIZe1FXt6vAjNV3m1/z0vi5aXZGWg1CqURNfZirvCiaT1KnNvlJp0TfJLyYIGYuIuGFnVgq66TkebGQJd7sjNuM5iXdfF7uvuBkzLxvkNvV0ES+tpG5c5ht33qHUTfjH3iFFyurSqtNftJJ6E2v3tMmGoUXlyviI6y+UzwCSbKjjQBzwku6mY0EQUD3EufWHrNKrNks23EQ8zbvM3UOHoddqf+X3sKeXs+PQzRSE+KbOUVE88+XQqUJbshRl5+AZYfd5CwPYObgaLSR/Hc9dvkphselm+ZFyellzdyugozYo7/+tUXmT8KxdIjMbMRQr3Tnwh7ez15exym8iKJoesaUbpMfJen9600Spt4kgVPbFurutxoqzYMVzcuqn6q4jxEgoFuJfGXtX51rvOBius+clAw7hX2leSeIzYB5nS5jnNG+WVwIWbbjkGF52buQRup+szz/i9Nxzkkt3a6GrUREe6ON0on0/vUmia0uLZ1tr358mFvV8QT/umMAWjbJ0txvNf9AspLUmUEQgO4lBehRymcSSffOR/mUHr+ipyv1iGGl2WQFA/jgNwPi36sl682woPZ2tM7PBgBaL62BwtxMjOyfWiHkYdF8f5Xumtv0/vUmiWle3rvtbLRtlou3bzkTBTmZLtfKXXKzgji1nbb2JdY5m3Uo47HUuaUYG8o5yKRr53PDWe0BJEYfNcnyb+aG9i3zkJUR4GrfUj8utcP+d/dAPHNNb9zvoygtp0k1BZWVJHXpPvnxb2/hIuGGTufszi3x3cMXuFsZn9CYpM4cnta8NPzPqwDOSrNQ6Rh//nkvPHRJNzTLk2vqhBToi3nat7S9qPmCFRfk4Bf9ymyoVerQsmm221WwFZHTbJSTGUBNQ4b3dEu1oCQFuovkEyGHXW6sJ6ljL5vsx8My2z6/W+uEbenq8iIIQoLgArif2twOmZfnJ0ivl075OazQr0Nz3DO0C/52/eluV8UWeNc2GtSlsR8xk1sqlUjT7tMa5LDLj+UkdT7QvGhxUusmePtXZyVsP3CszpkK+RS3+2I7QpJ5hBepkJ1q5hCnEAQBYy/qip+d3tbtqthChCPYqDA3E+Tj3QgJLyYgzQs/enksWPhx3zHmsl6TLbWqUx/yWEVdxs+al7hPF4d4LhXIra66TsgZ0NkfUUmiyK55yQwKsnck3VsMCS8mIM2LOnq3JfbOmXXYHTdtNXNZN6KNWLjvwi4AgO4l+ejdrhC/7N/ewVr5D7dnksl+q2X9SLqPRDZjJmmkG/Bk2M0MBii8XgI57JqAktTxY9XnhYfkCy9sP+q+C7ti1ICOaKETUp7OuK95cdFsZPnK6UvHlnnYfuC4bJtVTW+y4FmYUSm8+OMXOgdpXkxAZiN+rPq88ODlx0OCizYswsvP+6SGrwMg70fSaU0auxnYpVXCtqBPbidPkrqMoICqE/Xx73VhC4u4pQAkvJiAzEb8xG36Seik7ZhB24rHquNVWCbL1/Zth0yHRiY7HhOPECLV4KZr5JkdqOUHcluLx4ooAmt3s2X1blOYK8v+fLyWLxFiqkGvjAnSXODVhKW/SGXNi9dkJr/BNPALzjm32hIqzVFWOgka3qsUJxc1xQ1nUV4XXvLUhBefmI2W7jiIv321RbfMae0Kce7JLfGXa3rL2tfRWr4VrFMN8nkxQbJ9KvyC3m2JKUdT0efFCG/Vxt84O6M2/6QEE4pFqYYwJzOI2fcPJvORCZpkJyZrC/rkPsYSzulxQfdi3Nvg7C/t2o7W1msckR6Q5sUEIVK9qNI8zxtLJJBPUuriqOiS5GajdPwnwcUcuVkqwotPNC8sSE2KUm3dEc71s1INEl5MQGOjOhMu7aGZXyH2ziUjn4XHFC+EjThpDrDF54WjfZtcvJpQ0LdD84RtfjEbsSAVaqUC71HyeSF48ZpZwisU5efg/d/01y2TlmYjj9XHzwjxf7wJV6g0zYIsMX/c+fj36AHoXpK4mrtfoo1YkJpKpX3J0dqQ61mp3YR8XkxAeV7Mk8oOu4TzCIJzbSjpDrvUUC1R1iIPZS3yVPelkuZF+lOkTeZoTQiZwQBqQ+npxkCaFxNQp6PPqAEdZP9LScad85rmhbAPJ/1CrGjIYuYiviR11E6dwi8OuywENMxGYy44GVkZjUP47YM6JbVebkPCiwkoz4s+j13eE/+7eyB+f+kpCfuSIfjR40ldnBySfjp0woazsNeQhBfn8EueFxakWiRpmxnZvwOuPzMaWp+TGcDDw3skvW5uQmYjE1Cfo08wIKBX20LZTDb2KRnCi9cGBW/Vxt84OSg98O9VyM6MpmBPRhMiDa5zBFJoWi43G8mzMj94cTf07dACA05qmVIRViyk0CMmvIaair8+CWHmNCSkLoLgrNP3K3N/sKTd4akbRRs5SeODGKSyfICf0DIbAUB2RhCX9CpBYa430lQkExJeOHjmmt4AgMm/7OtyTfxLWmpevFUdX+O0OcDs7DWepI7jGIpCcw5pMxk1oKNr9bADaZOkJtMICS8c/KJfGbY8NRyX9Cpxuyq+I/bShcjnhdDh8tPauHr9gCBYcgo2u7YR4RwZnHHT7/z6LIdqYg6pzwv5WzaSFOHl5ZdfRqdOnZCTk4O+ffti/vz5uuXnzZuHvn37IicnB507d8bkyZOTUU0mMmkFNUukU4fds000/8SVKbQSstP83w198NKNfeLfO7VqItsfEARHEx0GA9bOTqHS3oM38kjZ5txGqm30mlbZTRwfiT/88EPcd999eOSRR7BixQoMGjQIw4cPx86dO1XLb9u2DSNGjMCgQYOwYsUKTJgwAffccw+mTZvmdFWJJJAKSys8dvkpuHVgYliislt57/b+ePPmfrj7gpOTU7EUIUMy07z6DLng53QQiVWfRwqV9gbSfiaD86F6zfFV5rDr/+7TNhwXXl544QXceuutuO2229CjRw9MmjQJZWVleOWVV1TLT548Ge3bt8ekSZPQo0cP3Hbbbfj1r3+N5557zumqEg4SW5gxGWYjp2EdcwpzMzG0RzFp63R4cFhXAMA9EgFPOtNUmmGiZh3n6iM4fH4pKfAqeBZp4jZeYcRrwovW8gDpjqOh0nV1dVi2bBkefvhh2fZhw4ZhwYIFqscsXLgQw4YNk227+OKL8eabb6K+vh6ZmXKv6traWtTW1sa/V1dX21R7wglS5eXzVvfmX+46/2T8/Ix2aFOYE98mF17k5Z0WLIJxs1RiO83LCuJ4XVj3eFoewBvUSlZr5vV58ZrwQmYjdRydEu7fvx/hcBjFxcWy7cXFxaioqFA9pqKiQrV8KBTC/v37E8pPnDgRhYWF8b+ysjL7fgBhO/UpEB8qIjlrNKUDgiCgbbNc2exSOngoo4ucvu16A5ee+SFWfx6PGRqInKMm1ChkBjmTvvCamZxGqrglebeRpOizlapfURR1vfLVyqttB4Dx48ejqqoq/rdr1y4bakw4RThFjLZOpqlPd6S3VjmOOH3fAwHtxZOYzH9ceV5oJHKK08uaxT/zOuyS5sUfOCq8tGrVCsFgMEHLUllZmaBdiVFSUqJaPiMjAy1btkwon52djYKCAtkf4V3s9nlxapZ0SU8Kh3cL6eCh1GQ4uTAjEBWWtM7PMqjx1C1F5HhPcUH3Iqx+fBgKchrdC3iz7XpNeCGfF3UcFV6ysrLQt29fzJ49W7Z99uzZOOecc1SPGTBgQEL5WbNmoV+/fgn+LoR/iE0Y7H75Foy/wNbzxRh3STf87+6Bmvu91b2lFno+L1bzsBihN0tnEl4oz4urZAYFmeAC8JnyAO8JL/JoI2ozMRw3G40dOxZvvPEG3nrrLWzYsAH3338/du7cidGjRwOImn1GjRoVLz969Gjs2LEDY8eOxYYNG/DWW2/hzTffxIMPPuh0VYkkYLfPi1M5PwKCoJnNVRRFVemlddNsR+qSbkjvO6/Py//d0MeghD560UZWW5ryvGQ2sp/YLZXeWV7NS4bHFkYis5E6ji/MeN111+HAgQN44oknUF5ejl69emHGjBno0KEDAKC8vFyW86VTp06YMWMG7r//fvz9739HmzZt8OKLL+Lqq692uqpEErDb58WpSbjRGjpqQtNN/dtjY0U1zutW5Eyl0gTpTFNN86LFh7/pj7M7t8TJRU0x/G/6iTC10HNrYdGq6JXICAgy4Z1m0c4hXXqBd0kJjyleFMKLixXxGElZVfrOO+/EnXfeqbpvypQpCduGDBmC5cuXO1wrwg1aNLFXO+FUPyNAP9+H2r7sjCCeueY0h2qUPuhGG+k8k7M7R33iOrc2nyE1mmFX/SJ6E/LYEXr1ywgEUB9ujIIhzUty4O0jBEFAMCB4xqwnFaaozTTiLf0YkfLcO7QLercrtO18Tvk/RB1DPTYFSxOkzzQnM4Cpt54t2ee0w6620Moyg9drM8p8I6R5SQ5m+ggv+b3INC/UZuKQ8EIkhZgatzAvE6+N7GfbeR3TvOiYjTRcXgibkA4cOZlB2aDPZrox/3T0BBQm4UWniDLUmsYh+ymRJDuMYUYO8VKul4Aszws1mhgkvBBJx85+wTmfF/0hkNK8OIe0feRkBhVmJOPjrTwbvVBplvPqFVHO5mkgso+3bzkTV5zWBg8M65awj9fnBeDPDeMk5POiDgkvRNKx09TjlGlHgHY9RYhkUnIQaWedqxBeBJizG7Vvkcd27YB2KLbVJ66czXvFpyIVOL97EV68oQ8KcxPTaRQXJGpjrj9TPxN7kHNJAScxI3ylAyS8EI6SlRFtYoO6tI5vs1Xz4lALNoo2IpxDJrxkBWWzYBafF7X9rOvb6A0U+lnBdS6uUQdabTw55GYF8ccresq2tWmWq3uMp8xG1BGpkpRoIyJ9WTx+KPZUnUDPNo1OuvZqXpwhoGM2EkUSbJxEauPPzQzK9jGZbiw8nKAgyMJsZec1fdYomZIf9vm9g9C9JN/iGQktlI+wuEAe5Wgkm7gtMJQU5KCiuqahLq5WxbOQ5oVwlOZNsmSCC2C3z0vyzUax/YQzSDUtOZkBecZdhjuvVoL1eQUCgsp60pLzGJ1I52Cp+atHaQGtj+UgouJBKIWZgEEnxLSOlYPITKWSdvKbwZ0BANf1owWISfNCJB0/aF70XCtEgFQvDiIPlQ4iHGnMjeL0LDQYSBzoGusVbRN6wo3evj7tm2FL5VELtSPswkizkumyz4tUeJF+fuiS7hjeqwS92tqXbsKvkPBCJB1fRBsZJKkjnEN633MzgzhRF5bsMxeuzOoaGxCEhFl7/LxocObViRLSCyD6/WWnoHV+Ni4/rQ1jbQinMIomclvzkqERYRcMCOjTvrkLNfIeJLwQSccP0UYBnSR1lOfFWerDjUtI5GYFIRyXd+RG7cdK+xKgr3kJCEBYYx+gH/5ckJOJ313c3XTdCPMon4pRE4kFGriF3GzkYkU8DPm8EEnHF5oXnUyrTl6XAOpCjcJLTkZQvtaRWY0Yo+olIuprTyhE3h8YpdAxyqDrtuZFWj9KB6QOCS9E0pHam4vyszF/3Pn46M5zXKxRIsbhuDSIOUXzvKz4Z2XeFbOh8az9v57mRGCI06Zxxhv066hvWjHyecly22wk8bmhdEDqkNmISDrSfiMYEFDWIg/H6kKWz2UnlOfFPcpa5OGlG/ugRYMQI9e8mEMr/DmhHBIjVaTXNgw2ommyJ+jboQXeu/3seHJC3mgj181Gsqy61KbUIOGFSDpSrUVMPWo2HbdzGXa1M62KEEmwcZjLejc6tUpnyXr5d/Rg7f5FUdSc6bIItDTMeIdzTmoV/3xB9yLZPiPTtZeijWgxRnXIbEQkHWnHEfOqN+tkaXRYk6ygfgGt8wb0Z9kkuyQPOwRF1slrJCI3WyXUxeDJk+bFm+RmBbHujxfjwh7FeObq3p6PNpIKL2FqU6qQ8EIkHdlMuuElNevEa+ibYlqjo20Xpwy7yUX6HMx246yq94gooml2VODt0FK+HlIs2kiNmFBD44x3aZKdgTdu7odfnFlm7PPistlI2o5I8aIOmY2IpCPzeWn4YuT9r30ug7BZU2c1jjYikke2ZCAx7/PCVi4iNg4WvdoUYseB45Jra5sSea9DuIuhz4vNmpffXdwNBbmZqD5Rj2e/2GRYXtqMyGykDmleiKQjHQCCcc2LeQ2JtQJR5jx4HiaMaMzBoXTOvOEseTpuSu2ePFo2zcad552Eey44GU2yMxy991Gfl+hgoRzgWBaFpGHGHxj7vNg7NLZokoWR/TvIBHG9CZvU/JidScO0GqR5IVwlGPd5MXe80XGsp+3Uqgku6F6EP8/YCKBBmJJGuUhNF6JIwkuSGXeJXLBU0qqptp8KwBlt1FBU6bPJFCpNqhdfYKTptdtsFLtaTX1jisOgICCsIe6KACaM6I7Ne49iQOeWttYlVSDhhXCVmMOuY2YjDiFDtgCgIsOu8iwku3iHzKCAr8aep1uGJ8+LqKV5gbwdPHb5Kfjjf9ebug7hLkb9glMOu7WSBIx6VRBF4DeDT3KkDqkC6aMIVwk12HPdXoIeSBSgBEH9c3R5APfrS0Tp2aYQhXmZumXM+LwoI1KiodKN2351bqf457bNcrmuQ7iLYbRRhr3vd+xyUs2LXp9HzcgYEl4IV6muqQfAJ7wU5Wczl+WRiaR1iIiiTDwhYcW7sHT0PNFGsbJq2kBle3r3trMx4tQSPHFlT47aEG5jpOjNtlnzEus/pJoXVp8XQh0SXghXqToeE17Yj+ERSHhEjrbNctG7XSHO7NgcuZlBeVp6qeaFsw6EvSTce4aOnidJXaxsgiYOiTP2c09uhZdv6oui/BzWqhAewCjayKrZ6JERPZCfLfHKUNG8KNvx9WfKgwIIfUh4IVyluia6LIBTZiMun5eAgI/vPBf/umNANFRap+yQrq0BAHkmk+AR9sEiL7AKFaLYOOtNNCMKhr5ZJLv4A6P+JtOiw25JYQ5+dW7HhO23DeoMALjy9DYJdZA6CZMQbAw57BKeQG0mdFLrJvhh3zFL5+UViaT1kPm8SMsIQI/SAnz1wBC05jBhEXYhf6psJiEes1H0s3JwEWDsWE6Djj8wUqywaF7yczJwpEZ9TTZBADKCifmJuhbnY90fL0ZeVhB9n/xS89xa62sRjZDmhXCFWPjf2Z1aAFA3G2lpTXj8T6wodKTXEQGMHnISOrTMw8gBHQEAJ7VuioIcfUdRwnlYBAY+h12NhRkF4xk7DTr+wEgjy7K00cpHh2nuCwiCTACSXi+Wq0hPDo5EtPcRUUjzQrjC3286A9OW/YQr+7QFoD4o2GFIClnJTqmowMPDu+Ph4d3VyxKuwSS8MJ9LjJ9P2SQFGJuNKBmqPzASQllSN+iVESBf3FGtpFKAIjc6Pkh4IVyhRZMs3D64c/w7S2SHGQ43OATrMem601W3S6tEHYt3ULYLe6ONdMoKLGYjkl78gNH7bOTQa3h+heZF9Rp6eV4sXT09ILMR4Vm0zEN2+/ae2WC6SrwOiSx+gEVgYDcbiZplBRhHxXVq1YTtQoSrGL3aRnlgjAgIUJiN+K5BQrAxJLwQnsBKtNEbo/rZWJNGpDXKcCjjJpEcmJcHEPXLZgT020Hvds00NXmEdzDym7Ma/SgIAjKkZiMGnz75EiSWLp8WUI9MeIIMC2ajC08pxpNX9rK5RvLr59DiaJ7BRJoXruUBtPxWBIHNnHBln7bo3a6Q8YqEGxj1LVbNRgFBvjK1mrBkIAcTBtDtIzxBICDgm9+dj8m/7GtY1m5jjtb5pB1Odgblc/EKiT4v9kkvol60EQTDENvGsoSXMXo+VhWtgmAcbq1rNiKvF0McFV4OHTqEkSNHorCwEIWFhRg5ciQOHz6se8wtt9wSTRAm+evfv7+T1SQ8QvuWeTiltCD+3W2fE9K8+AM7NS+1oTDCDaoX5WxZEDh8IRTlSgtzGGtAJAUjzUsSzEa6axuR7GKIoz3yjTfeiJUrV2LmzJmYOXMmVq5ciZEjRxoed8kll6C8vDz+N2PGDCerSXgIu5eiZ6FVU+NEczmZpHnxKiz9fP/OLXFSa2Nn2u+3H0IoIqIoPxut8rNk+zKCAebVz6WlLj21FP+89Wym44jkoOXz8vtLewCwQXiB3GykWkZxiV5tG02NrNFx6YxjodIbNmzAzJkzsWjRIpx9dvTFff311zFgwABs2rQJ3bp10zw2OzsbJSUlTlWN8DDyFNlaycLs0cg8cFFX/HpgJ02BSXqZbBeEKoINFmfcZ6/pjaY5GTheG8ZpT8wyLH9WpxYJWpasoHGeFzX+ftMZ3McQzqLWhax/4mLkZUWHRDPPWUpAECAYdBnSa9wxpDOu6tMWD/57laXrphOO9cgLFy5EYWFhXHABgP79+6OwsBALFizQPXbu3LkoKipC165dcfvtt6OyslKzbG1tLaqrq2V/hH+RCgn1YWfTTOZkBtEkW1t+l87OSPPiXVjmqM2bZCEzGEBBLtt8LRgQEhx3MwIB5hk5Rdl7G+njefzyUzD7/sFxwQXgWyhW9fwKE6PahEvalkb27yBzEia9izGOCS8VFRUoKipK2F5UVISKigrN44YPH453330XX3/9NZ5//nl8//33uOCCC1BbW6tafuLEiXGfmsLCQpSV0cqcfkaqBalzWHgxcoqTa15IePEKCSp/jp6eVWsXDAgJ7SMzw5zZiPAe0nZw49kd0KU4X7bfqtkoIso1K0YZdhPaFUkvhnALL48//niCQ63yb+nSpQDUOwpRFHU7kOuuuw6XXnopevXqhcsvvxyff/45Nm/ejM8++0y1/Pjx41FVVRX/27VrF+9PIjyENGS6LuSs8GKUyl3aSslh17s44R8QFIQEp8ksHp8XUr14GunjUXukVs1GkYjI4LAr/SwvQD4vxnD7vIwZMwbXX3+9bpmOHTti9erV2Lt3b8K+ffv2obi4mPl6paWl6NChA7Zs2aK6Pzs7G9nZtLJvqiDt9HmEFzOvulH/IK0LaV68Q3FBNiqqaxy9RjAgJPjSZGWw+7yQ6OJtpM9H1aRjUXgJR0Su9ZOUZUl0MYZbeGnVqhVatWplWG7AgAGoqqrCkiVLcNZZZwEAFi9ejKqqKpxzzjnM1ztw4AB27dqF0tJS3qoSPqc+7OwrbGQ2kvZfpHnxDi/e0AcTPlqD77YeAOBMRx8IJGpeMoMBy2njCW9gpHmxbjYSZdmYjTL6KoViUrwY41iP3KNHD1xyySW4/fbbsWjRIixatAi33347LrvsMlmkUffu3fHRRx8BAI4ePYoHH3wQCxcuxPbt2zF37lxcfvnlaNWqFX7+8587VVXCozhtNjJCOiPLzSLNi1fo0LIJ3r2tMfeTU2YjNYdddrOR7VUibEXfmdaqkJrg82KYV0b+nZLUGePoqtLvvvsu7rnnHgwbNgwAcMUVV+Cll16Sldm0aROqqqoAAMFgEGvWrME777yDw4cPo7S0FOeffz4+/PBD5OfnJ5yfSG20HHbtGhhYxrzr+pXh4PE6dCum9udVnJilqjvs8piNSHrxMsbLA1g7f0QUDR125dcjzQsvjgovLVq0wNSpU3XLSO3Kubm5+OKLL5ysEkHEYckP8pdreiehJoQVnOjoAyqal6xggN0XgmQXT2MoTFicIYUjIpfTb4LPCwkvhpAhn/AdYaMwIUaogyC0CAaQ0EB4fF5IdvE2RquDW442EkVZ5KRRsyFfKn5IeCE8S36OumKwVs0XxoQkQrJLasCiQeMloJakLiiorn5O+I+ebQowoHNLXHVGW9X9djjsygUgtbQhkus1jMT3Du0CAHjiZz0tXT8dcNRsRBBWuPqMdpiyYHvC9pr6sC3nJ81LamD0GE8va8Z9zqCQ6PPCYzaiibS3CQQEvP8b7QV/rcqo4Qivw260wP0XdcUdQzrLsv0S6pDmhfAc/x49ALcP6oSHh3dX3a+qeTEBefSnBnpCaKdWTfDGzf24z5kREFCUL18Jms9sRNKLn7HbbGR4PUm7IsGFDRJeCM9xZscWeOTSUzTXE7LL58Wm0xAuEVtK4owOzTTLjOzfgWnVcCWBgIAbzmqPG89uH9+WGQxgYBfjHFcAZNlVCf9h1WzUr0NzmZbO6GykqeOHRDwibRnaPXHtLcI/zLx3EKYv343bBnWy/dxBQUBWRgB//vmpeG/xTgBRgeSy3qXIzgigV9tC3eMfv6Inrn9tEX475CTb60Y4j1nhZeWjF2H/0Tp0bt0Uh4/Xxber5SKSan5pOQl+SHgh0pLZ9w9OWIyN8BedWzfFgxd3My5oAjXfllZNsyAIAob1LDE8/qTWTbFkwlAalHyKWbNRs7wsNMvLAiBvQw6vMZuWkPBCpAS8FiASXAg9pIPXxKtOxbo9VTivK5+mjgQX/xK0waFC6vMSpugA2yHhhSAIQoHUgfKGs9rrlCRSEas+L4BcAI6oONiRPGMNctglCIJQYHVVYcLf2CK8CFKzEUkqdkPCC+FbsjOo+RLOQMFC6Y3VUGnlOchsZD/U+xO+hYQXwinsGLwI/2KH5k3q86RmNgqFSaCxAvX+hO/4zeDOAIDHLqcU2oQzkNkovbH78atZjQ4cq7X3ImkGCS+Ep/nsnoG44rQ2OLmoaXzb+OHd8f0jF+Lqvu3i20grS0gZeHI0mdxlp5WaOp7WMEpv7F4oUc1stP9onUpJghUSXghP07NNIV68oQ86tMiLbxMEAa3z+bOmEunDP289Cxv/dElCin9W7HDYJPyL3Zq3MCV6sR0KlSZ8gdmxpKxFLnYdPIE7BndGt5J8nF7WDC2aZNlbOcJzCIKgubwEC+Tzkt7YLbxSsJH9kPBC+ARzncnMewdj9+ET6EpJ6QgOSHhJb+w2G6ktD/DARV3x/OzN+MvVp9p6rXSBhBfCF5jtS5pkZ5DgQnBDZqP0JmCzQ4VanpcxF5yM689qTyZwk5DPC+ELaCJMJBPSvKQ3dguvag675LtnDRJeCF/ws9PbAgA6t2qiul+kcCPCIj1KC+KfSfOS3tgtvKrleSGsQWYjwhcM71WCT8eci86tmxoXJghOnrm6N04ra4aLJ30DgDQv6Y7tmhcKNrId0rwQvkAQBPRu1wxNs9Xl7b4dWiS5RkQq8Yszy2RRaCS7pDdqwmssOaYW88edr7mPlgewHxJeiJTg1HaFmPbbc1CYm+l2VQifkpPZ2B3SQnrpjVJ2eeuWfpgwoofuMWWSXFQxchvC9Yd0bWVb3YgoZDYiUoa+HZqjtDAHVSfq3a4K4UOyMxrzwoRIeElrpOsSTfvtANOa3YXjL8BPh06gV9tCu6pGNEDCC5FSkKMlYZZMyVLS9eSkQDTQLM98UstmeVmWjie0IbMRkVI8e23v+Oe7LzjZxZoQfkM626YVf4k/XdkL913YBSdRkIAnIc0LkVL0bFOIbRNHYN+RWsqhQJgmFCHNS7ozsn8Ht6tA6EDCC5FyCIKAogJzC/IRBADUk+aFIDwNmY0IgiAaOKl1NAni+d2LXK4JQRB6kOaFIIi04saz2+O9xTtV931+72AcrQ3RyuME4XFI80IQRFrx1JW9sGj8UJyqEr6alREgwYUwJCtIQ6fbOPoEnnrqKZxzzjnIy8tDs2bNmI4RRRGPP/442rRpg9zcXJx33nlYt26dk9UkCCKNEAQBJYU5CFAaXYKT92/vjx6lBXj/N/3drkra46jwUldXh2uvvRa//e1vmY955pln8MILL+Cll17C999/j5KSElx00UU4cuSIgzUlCCLdGHhySwDyzLoEoceAk1ri83sHoW+H5gCAW87pCAC4pGeJi7VKTwQxCcvxTpkyBffddx8OHz6sW04URbRp0wb33XcfHnroIQBAbW0tiouL8Ze//AV33HGH4bWqq6tRWFiIqqoqFBQUGJYnCCI9qakP4z/LfsKQrq1VU7sThBF1oQiWbDuIfh2bIyczaHwAoQvP+O2pKce2bdtQUVGBYcOGxbdlZ2djyJAhWLBggeoxtbW1qK6ulv0RBEEYkZMZxC/7dyDBhTBNVkYAA7u0IsHFBTwlvFRUVAAAiouLZduLi4vj+5RMnDgRhYWF8b+ysjLH60kQBEEQhHtwCy+PP/44BEHQ/Vu6dKmlSgmK9WlEUUzYFmP8+PGoqqqK/+3atcvStQmCIAiC8DbceV7GjBmD66+/XrdMx44dTVWmpCTq9FRRUYHS0tL49srKygRtTIzs7GxkZ1MaeIIgCIJIF7iFl1atWqFVq1ZO1AWdOnVCSUkJZs+ejT59+gCIRizNmzcPf/nLXxy5JkEQBEEQ/sJRn5edO3di5cqV2LlzJ8LhMFauXImVK1fi6NGj8TLdu3fHRx99BCBqLrrvvvvw5z//GR999BHWrl2LW265BXl5ebjxxhudrCpBEARBED7B0eUBHn30UfzjH/+If49pU+bMmYPzzjsPALBp0yZUVVXFy4wbNw4nTpzAnXfeiUOHDuHss8/GrFmzkJ+f72RVCYIgCILwCUnJ85JMKM8LQRAEQfgP3+Z5IQiCIAiCMIKEF4IgCIIgfAUJLwRBEARB+AoSXgiCIAiC8BUkvBAEQRAE4StIeCEIgiAIwlc4mufFDWKR37S6NEEQBEH4h9i4zZLBJeWElyNHjgAArS5NEARBED7kyJEjKCws1C2TcknqIpEI9uzZg/z8fM2VqM1SXV2NsrIy7Nq1ixLgOQjd5+RB9zo50H1ODnSfk4NT91kURRw5cgRt2rRBIKDv1ZJympdAIIB27do5eo2CggJ6MZIA3efkQfc6OdB9Tg50n5ODE/fZSOMSgxx2CYIgCILwFSS8EARBEAThK0h44SA7OxuPPfYYsrOz3a5KSkP3OXnQvU4OdJ+TA93n5OCF+5xyDrsEQRAEQaQ2pHkhCIIgCMJXkPBCEARBEISvIOGFIAiCIAhfQcILQRAEQRC+goQXRl5++WV06tQJOTk56Nu3L+bPn+92lXzFxIkTceaZZyI/Px9FRUW48sorsWnTJlkZURTx+OOPo02bNsjNzcV5552HdevWycrU1tbi7rvvRqtWrdCkSRNcccUV+Omnn5L5U3zFxIkTIQgC7rvvvvg2us/2sXv3bvzyl79Ey5YtkZeXh9NPPx3Lli2L76d7bZ1QKITf//736NSpE3Jzc9G5c2c88cQTiEQi8TJ0n/n55ptvcPnll6NNmzYQBAEff/yxbL9d9/TQoUMYOXIkCgsLUVhYiJEjR+Lw4cPWf4BIGPLBBx+ImZmZ4uuvvy6uX79evPfee8UmTZqIO3bscLtqvuHiiy8W3377bXHt2rXiypUrxUsvvVRs3769ePTo0XiZp59+WszPzxenTZsmrlmzRrzuuuvE0tJSsbq6Ol5m9OjRYtu2bcXZs2eLy5cvF88//3zxtNNOE0OhkBs/y9MsWbJE7Nixo9i7d2/x3nvvjW+n+2wPBw8eFDt06CDecsst4uLFi8Vt27aJX375pbh169Z4GbrX1nnyySfFli1biv/73//Ebdu2if/+97/Fpk2bipMmTYqXofvMz4wZM8RHHnlEnDZtmghA/Oijj2T77bqnl1xyidirVy9xwYIF4oIFC8RevXqJl112meX6k/DCwFlnnSWOHj1atq179+7iww8/7FKN/E9lZaUIQJw3b54oiqIYiUTEkpIS8emnn46XqampEQsLC8XJkyeLoiiKhw8fFjMzM8UPPvggXmb37t1iIBAQZ86cmdwf4HGOHDkidunSRZw9e7Y4ZMiQuPBC99k+HnroIXHgwIGa++le28Oll14q/vrXv5Ztu+qqq8Rf/vKXoijSfbYDpfBi1z1dv369CEBctGhRvMzChQtFAOLGjRst1ZnMRgbU1dVh2bJlGDZsmGz7sGHDsGDBApdq5X+qqqoAAC1atAAAbNu2DRUVFbL7nJ2djSFDhsTv87Jly1BfXy8r06ZNG/Tq1YuehYK77roLl156KS688ELZdrrP9vHpp5+iX79+uPbaa1FUVIQ+ffrg9ddfj++ne20PAwcOxFdffYXNmzcDAFatWoVvv/0WI0aMAED32QnsuqcLFy5EYWEhzj777HiZ/v37o7Cw0PJ9T7mFGe1m//79CIfDKC4ulm0vLi5GRUWFS7XyN6IoYuzYsRg4cCB69eoFAPF7qXafd+zYES+TlZWF5s2bJ5ShZ9HIBx98gOXLl+P7779P2Ef32T5+/PFHvPLKKxg7diwmTJiAJUuW4J577kF2djZGjRpF99omHnroIVRVVaF79+4IBoMIh8N46qmncMMNNwCgNu0Edt3TiooKFBUVJZy/qKjI8n0n4YURQRBk30VRTNhGsDFmzBisXr0a3377bcI+M/eZnkUju3btwr333otZs2YhJydHsxzdZ+tEIhH069cPf/7znwEAffr0wbp16/DKK69g1KhR8XJ0r63x4YcfYurUqXjvvffQs2dPrFy5Evfddx/atGmDm2++OV6O7rP92HFP1crbcd/JbGRAq1atEAwGE6TEysrKBKmUMObuu+/Gp59+ijlz5qBdu3bx7SUlJQCge59LSkpQV1eHQ4cOaZZJd5YtW4bKykr07dsXGRkZyMjIwLx58/Diiy8iIyMjfp/oPluntLQUp5xyimxbjx49sHPnTgDUpu3id7/7HR5++GFcf/31OPXUUzFy5Ejcf//9mDhxIgC6z05g1z0tKSnB3r17E86/b98+y/edhBcDsrKy0LdvX8yePVu2ffbs2TjnnHNcqpX/EEURY8aMwfTp0/H111+jU6dOsv2dOnVCSUmJ7D7X1dVh3rx58fvct29fZGZmysqUl5dj7dq19CwaGDp0KNasWYOVK1fG//r164ebbroJK1euROfOnek+28S5556bEO6/efNmdOjQAQC1abs4fvw4AgH5UBUMBuOh0nSf7ceuezpgwABUVVVhyZIl8TKLFy9GVVWV9ftuyd03TYiFSr/55pvi+vXrxfvuu09s0qSJuH37drer5ht++9vfioWFheLcuXPF8vLy+N/x48fjZZ5++mmxsLBQnD59urhmzRrxhhtuUA3Na9eunfjll1+Ky5cvFy+44IK0DndkQRptJIp0n+1iyZIlYkZGhvjUU0+JW7ZsEd99910xLy9PnDp1arwM3Wvr3HzzzWLbtm3jodLTp08XW7VqJY4bNy5ehu4zP0eOHBFXrFghrlixQgQgvvDCC+KKFSviKUDsuqeXXHKJ2Lt3b3HhwoXiwoULxVNPPZVCpZPJ3//+d7FDhw5iVlaWeMYZZ8RDfAk2AKj+vf322/EykUhEfOyxx8SSkhIxOztbHDx4sLhmzRrZeU6cOCGOGTNGbNGihZibmytedtll4s6dO5P8a/yFUnih+2wf//3vf8VevXqJ2dnZYvfu3cXXXntNtp/utXWqq6vFe++9V2zfvr2Yk5Mjdu7cWXzkkUfE2traeBm6z/zMmTNHtU+++eabRVG0754eOHBAvOmmm8T8/HwxPz9fvOmmm8RDhw5Zrr8giqJoTXdDEARBEASRPMjnhSAIgiAIX0HCC0EQBEEQvoKEF4IgCIIgfAUJLwRBEARB+AoSXgiCIAiC8BUkvBAEQRAE4StIeCEIgiAIwleQ8EIQBEEQhK8g4YUgCIIgCF9BwgtBEARBEL6ChBeCIAiCIHwFCS8EQRAEQfiK/wcat9n4ZCz9TwAAAABJRU5ErkJggg==\n",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"filenames": {
|
|
"image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/chapter13_3_0.png"
|
|
}
|
|
},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Metal device set to: Apple M1\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Model: \"sequential\"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Layer (type) Output Shape Param # \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" simple_rnn (SimpleRNN) (None, 32) 1184 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense (Dense) (None, 8) 264 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense_1 (Dense) (None, 1) 9 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Total params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Trainable params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Non-trainable params: 0\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 1/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2023-10-15 21:49:35.228942: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 3s - loss: 0.5276 - 3s/epoch - 66ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 2/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4234 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 3/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4043 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 4/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4010 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 5/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3979 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 6/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3967 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 7/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3962 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 8/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3957 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 9/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3929 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 10/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3920 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 11/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3918 - 454ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 12/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3898 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 13/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3921 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 14/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3911 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 15/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3888 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 16/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3871 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 17/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3894 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 18/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3873 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 19/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3855 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 20/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3871 - 453ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 21/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3802 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 22/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3856 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 23/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3804 - 453ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 24/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3842 - 453ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 25/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3815 - 452ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 26/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3782 - 454ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 27/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3798 - 454ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 28/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3804 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 29/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3812 - 452ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 30/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3780 - 454ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 31/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3800 - 453ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 32/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3767 - 467ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 33/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3787 - 493ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 34/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3758 - 464ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 35/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3784 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 36/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3766 - 461ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 37/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3733 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 38/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3749 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 39/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3756 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 40/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3737 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 41/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3743 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 42/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3730 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 43/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3706 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 44/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3724 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 45/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3716 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 46/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3713 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 47/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3705 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 48/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3703 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 49/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3701 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 50/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3676 - 470ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 51/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3679 - 464ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 52/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3689 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 53/100\n"
|
|
]
|
|
},
|
|
{
|
|
"ename": "KeyboardInterrupt",
|
|
"evalue": "",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
|
|
"Input \u001b[0;32mIn [1]\u001b[0m, in \u001b[0;36m<cell line: 58>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 55\u001b[0m model\u001b[38;5;241m.\u001b[39mcompile(loss\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmean_squared_error\u001b[39m\u001b[38;5;124m'\u001b[39m, optimizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrmsprop\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 56\u001b[0m model\u001b[38;5;241m.\u001b[39msummary()\n\u001b[0;32m---> 58\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrainX\u001b[49m\u001b[43m,\u001b[49m\u001b[43mtrainY\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m16\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 59\u001b[0m trainPredict \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(trainX)\n\u001b[1;32m 60\u001b[0m testPredict\u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(testX)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/utils/traceback_utils.py:64\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 62\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 63\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 64\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 65\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e: \u001b[38;5;66;03m# pylint: disable=broad-except\u001b[39;00m\n\u001b[1;32m 66\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/engine/training.py:1384\u001b[0m, in \u001b[0;36mModel.fit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 1377\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m tf\u001b[38;5;241m.\u001b[39mprofiler\u001b[38;5;241m.\u001b[39mexperimental\u001b[38;5;241m.\u001b[39mTrace(\n\u001b[1;32m 1378\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtrain\u001b[39m\u001b[38;5;124m'\u001b[39m,\n\u001b[1;32m 1379\u001b[0m epoch_num\u001b[38;5;241m=\u001b[39mepoch,\n\u001b[1;32m 1380\u001b[0m step_num\u001b[38;5;241m=\u001b[39mstep,\n\u001b[1;32m 1381\u001b[0m batch_size\u001b[38;5;241m=\u001b[39mbatch_size,\n\u001b[1;32m 1382\u001b[0m _r\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m):\n\u001b[1;32m 1383\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_begin(step)\n\u001b[0;32m-> 1384\u001b[0m tmp_logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1385\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m data_handler\u001b[38;5;241m.\u001b[39mshould_sync:\n\u001b[1;32m 1386\u001b[0m context\u001b[38;5;241m.\u001b[39masync_wait()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 150\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 152\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:915\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 912\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 914\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[0;32m--> 915\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 917\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[1;32m 918\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:947\u001b[0m, in \u001b[0;36mFunction._call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 944\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m 945\u001b[0m \u001b[38;5;66;03m# In this case we have created variables on the first call, so we run the\u001b[39;00m\n\u001b[1;32m 946\u001b[0m \u001b[38;5;66;03m# defunned version which is guaranteed to never create variables.\u001b[39;00m\n\u001b[0;32m--> 947\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_stateless_fn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# pylint: disable=not-callable\u001b[39;00m\n\u001b[1;32m 948\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_stateful_fn \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 949\u001b[0m \u001b[38;5;66;03m# Release the lock early so that multiple threads can perform the call\u001b[39;00m\n\u001b[1;32m 950\u001b[0m \u001b[38;5;66;03m# in parallel.\u001b[39;00m\n\u001b[1;32m 951\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:2956\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2953\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock:\n\u001b[1;32m 2954\u001b[0m (graph_function,\n\u001b[1;32m 2955\u001b[0m filtered_flat_args) \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_maybe_define_function(args, kwargs)\n\u001b[0;32m-> 2956\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 2957\u001b[0m \u001b[43m \u001b[49m\u001b[43mfiltered_flat_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1853\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1849\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[1;32m 1850\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[1;32m 1851\u001b[0m \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[1;32m 1852\u001b[0m \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[0;32m-> 1853\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_build_call_outputs(\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1854\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcancellation_manager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcancellation_manager\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m 1855\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[1;32m 1856\u001b[0m args,\n\u001b[1;32m 1857\u001b[0m possible_gradient_type,\n\u001b[1;32m 1858\u001b[0m executing_eagerly)\n\u001b[1;32m 1859\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:499\u001b[0m, in \u001b[0;36m_EagerDefinedFunction.call\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 497\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m _InterpolateFunctionError(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m 498\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cancellation_manager \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 499\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 500\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mstr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msignature\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 501\u001b[0m \u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_num_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 502\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 503\u001b[0m \u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 504\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mctx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 505\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 506\u001b[0m outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[1;32m 507\u001b[0m \u001b[38;5;28mstr\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msignature\u001b[38;5;241m.\u001b[39mname),\n\u001b[1;32m 508\u001b[0m num_outputs\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_num_outputs,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 511\u001b[0m ctx\u001b[38;5;241m=\u001b[39mctx,\n\u001b[1;32m 512\u001b[0m cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_manager)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:54\u001b[0m, in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 52\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m 53\u001b[0m ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[0;32m---> 54\u001b[0m tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 55\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 56\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 57\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Start importing packages\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras.models import Model, Sequential \n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"from tensorflow.keras import optimizers \n",
|
|
"from tensorflow.keras import regularizers \n",
|
|
"from tensorflow.keras.utils import to_categorical \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"# convert into dataset matrix\n",
|
|
"def convertToMatrix(data, step):\n",
|
|
" X, Y =[], []\n",
|
|
" for i in range(len(data)-step):\n",
|
|
" d=i+step \n",
|
|
" X.append(data[i:d,])\n",
|
|
" Y.append(data[d,])\n",
|
|
" return np.array(X), np.array(Y)\n",
|
|
"\n",
|
|
"step = 4\n",
|
|
"N = 1000 \n",
|
|
"Tp = 800 \n",
|
|
"\n",
|
|
"t=np.arange(0,N)\n",
|
|
"x=np.sin(0.02*t)+2*np.random.rand(N)\n",
|
|
"df = pd.DataFrame(x)\n",
|
|
"df.head()\n",
|
|
"\n",
|
|
"plt.plot(df)\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"values=df.values\n",
|
|
"train,test = values[0:Tp,:], values[Tp:N,:]\n",
|
|
"\n",
|
|
"# add step elements into train and test\n",
|
|
"test = np.append(test,np.repeat(test[-1,],step))\n",
|
|
"train = np.append(train,np.repeat(train[-1,],step))\n",
|
|
" \n",
|
|
"trainX,trainY =convertToMatrix(train,step)\n",
|
|
"testX,testY =convertToMatrix(test,step)\n",
|
|
"trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
|
|
"testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(SimpleRNN(units=32, input_shape=(1,step), activation=\"relu\"))\n",
|
|
"model.add(Dense(8, activation=\"relu\")) \n",
|
|
"model.add(Dense(1))\n",
|
|
"model.compile(loss='mean_squared_error', optimizer='rmsprop')\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)\n",
|
|
"trainPredict = model.predict(trainX)\n",
|
|
"testPredict= model.predict(testX)\n",
|
|
"predicted=np.concatenate((trainPredict,testPredict),axis=0)\n",
|
|
"\n",
|
|
"trainScore = model.evaluate(trainX, trainY, verbose=0)\n",
|
|
"print(trainScore)\n",
|
|
"\n",
|
|
"index = df.index.values\n",
|
|
"plt.plot(index,df)\n",
|
|
"plt.plot(index,predicted)\n",
|
|
"plt.axvline(df.index[Tp], c=\"r\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "897a47c7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## An extrapolation example\n",
|
|
"\n",
|
|
"The following code provides an example of how recurrent neural\n",
|
|
"networks can be used to extrapolate to unknown values of physics data\n",
|
|
"sets. Specifically, the data sets used in this program come from\n",
|
|
"a quantum mechanical many-body calculation of energies as functions of the number of particles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6776ae2a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"# For matrices and calculations\n",
|
|
"import numpy as np\n",
|
|
"# For machine learning (backend for keras)\n",
|
|
"import tensorflow as tf\n",
|
|
"# User-friendly machine learning library\n",
|
|
"# Front end for TensorFlow\n",
|
|
"import tensorflow.keras\n",
|
|
"# Different methods from Keras needed to create an RNN\n",
|
|
"# This is not necessary but it shortened function calls \n",
|
|
"# that need to be used in the code.\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras import regularizers\n",
|
|
"from tensorflow.keras.models import Model, Sequential\n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"# For timing the code\n",
|
|
"from timeit import default_timer as timer\n",
|
|
"# For plotting\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"\n",
|
|
"# The data set\n",
|
|
"datatype='VaryDimension'\n",
|
|
"X_tot = np.arange(2, 42, 2)\n",
|
|
"y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,\n",
|
|
"\t-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, \n",
|
|
"\t-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35227d36",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The way the recurrent neural networks are trained in this program\n",
|
|
"differs from how machine learning algorithms are usually trained.\n",
|
|
"Typically a machine learning algorithm is trained by learning the\n",
|
|
"relationship between the x data and the y data. In this program, the\n",
|
|
"recurrent neural network will be trained to recognize the relationship\n",
|
|
"in a sequence of y values. This is type of data formatting is\n",
|
|
"typically used time series forcasting, but it can also be used in any\n",
|
|
"extrapolation (time series forecasting is just a specific type of\n",
|
|
"extrapolation along the time axis). This method of data formatting\n",
|
|
"does not use the x data and assumes that the y data are evenly spaced.\n",
|
|
"\n",
|
|
"For a standard machine learning algorithm, the training data has the\n",
|
|
"form of (x,y) so the machine learning algorithm learns to assiciate a\n",
|
|
"y value with a given x value. This is useful when the test data has x\n",
|
|
"values within the same range as the training data. However, for this\n",
|
|
"application, the x values of the test data are outside of the x values\n",
|
|
"of the training data and the traditional method of training a machine\n",
|
|
"learning algorithm does not work as well. For this reason, the\n",
|
|
"recurrent neural network is trained on sequences of y values of the\n",
|
|
"form ((y1, y2), y3), so that the network is concerned with learning\n",
|
|
"the pattern of the y data and not the relation between the x and y\n",
|
|
"data. As long as the pattern of y data outside of the training region\n",
|
|
"stays relatively stable compared to what was inside the training\n",
|
|
"region, this method of training can produce accurate extrapolations to\n",
|
|
"y values far removed from the training data set."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "7dc577d1",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# FORMAT_DATA\n",
|
|
"def format_data(data, length_of_sequence = 2): \n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" data(a numpy array): the data that will be the inputs to the recurrent neural\n",
|
|
" network\n",
|
|
" length_of_sequence (an int): the number of elements in one iteration of the\n",
|
|
" sequence patter. For a function approximator use length_of_sequence = 2.\n",
|
|
" Returns:\n",
|
|
" rnn_input (a 3D numpy array): the input data for the recurrent neural network. Its\n",
|
|
" dimensions are length of data - length of sequence, length of sequence, \n",
|
|
" dimnsion of data\n",
|
|
" rnn_output (a numpy array): the training data for the neural network\n",
|
|
" Formats data to be used in a recurrent neural network.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" X, Y = [], []\n",
|
|
" for i in range(len(data)-length_of_sequence):\n",
|
|
" # Get the next length_of_sequence elements\n",
|
|
" a = data[i:i+length_of_sequence]\n",
|
|
" # Get the element that immediately follows that\n",
|
|
" b = data[i+length_of_sequence]\n",
|
|
" # Reshape so that each data point is contained in its own array\n",
|
|
" a = np.reshape (a, (len(a), 1))\n",
|
|
" X.append(a)\n",
|
|
" Y.append(b)\n",
|
|
" rnn_input = np.array(X)\n",
|
|
" rnn_output = np.array(Y)\n",
|
|
"\n",
|
|
" return rnn_input, rnn_output\n",
|
|
"\n",
|
|
"\n",
|
|
"# ## Defining the Recurrent Neural Network Using Keras\n",
|
|
"# \n",
|
|
"# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.\n",
|
|
"\n",
|
|
"def rnn(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with one hidden layer and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer\n",
|
|
" hidden_neurons = 200\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to \n",
|
|
" # the network immediately after the input layer\n",
|
|
" rnn = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\")(inp)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1978b6a1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Predicting New Points With A Trained Recurrent Neural Network"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "d4ad417a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_rnn (x1, y_test, plot_min, plot_max):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" x1 (a list or numpy array): The complete x component of the data set\n",
|
|
" y_test (a list or numpy array): The complete y component of the data set\n",
|
|
" plot_min (an int or float): the smallest x value used in the training data\n",
|
|
" plot_max (an int or float): the largest x valye used in the training data\n",
|
|
" Returns:\n",
|
|
" None.\n",
|
|
" Uses a trained recurrent neural network model to predict future points in the \n",
|
|
" series. Computes the MSE of the predicted data set from the true data set, saves\n",
|
|
" the predicted data set to a csv file, and plots the predicted and true data sets w\n",
|
|
" while also displaying the data range used for training.\n",
|
|
" \"\"\"\n",
|
|
" # Add the training data as the first dim points in the predicted data array as these\n",
|
|
" # are known values.\n",
|
|
" y_pred = y_test[:dim].tolist()\n",
|
|
" # Generate the first input to the trained recurrent neural network using the last two \n",
|
|
" # points of the training data. Based on how the network was trained this means that it\n",
|
|
" # will predict the first point in the data set after the training data. All of the \n",
|
|
" # brackets are necessary for Tensorflow.\n",
|
|
" next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])\n",
|
|
" # Save the very last point in the training data set. This will be used later.\n",
|
|
" last = [y_test[dim-1]]\n",
|
|
"\n",
|
|
" # Iterate until the complete data set is created.\n",
|
|
" for i in range (dim, len(y_test)):\n",
|
|
" # Predict the next point in the data set using the previous two points.\n",
|
|
" next = model.predict(next_input)\n",
|
|
" # Append just the number of the predicted data set\n",
|
|
" y_pred.append(next[0][0])\n",
|
|
" # Create the input that will be used to predict the next data point in the data set.\n",
|
|
" next_input = np.array([[last, next[0]]], dtype=np.float64)\n",
|
|
" last = next\n",
|
|
"\n",
|
|
" # Print the mean squared error between the known data set and the predicted data set.\n",
|
|
" print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())\n",
|
|
" # Save the predicted data set as a csv file for later use\n",
|
|
" name = datatype + 'Predicted'+str(dim)+'.csv'\n",
|
|
" np.savetxt(name, y_pred, delimiter=',')\n",
|
|
" # Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
" # for the training data.\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.plot(x1, y_test, label=\"true\", linewidth=3)\n",
|
|
" ax.plot(x1, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
" ax.legend()\n",
|
|
" # Created a red region to represent the points used in the training data.\n",
|
|
" ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn(length_of_sequences = rnn_input.shape[1])\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2cad4fc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Changing the size of the recurrent neural network and its parameters\n",
|
|
"can drastically change the results you get from the model. The below\n",
|
|
"code takes the simple recurrent neural network from above and adds a\n",
|
|
"second hidden layer, changes the number of neurons in the hidden\n",
|
|
"layer, and explicitly declares the activation function of the hidden\n",
|
|
"layers to be a sigmoid function. The loss function and optimizer can\n",
|
|
"also be changed but are kept the same as the above network. These\n",
|
|
"parameters can be tuned to provide the optimal result from the\n",
|
|
"network. For some ideas on how to improve the performance of a\n",
|
|
"[recurrent neural network](https://danijar.com/tips-for-training-recurrent-neural-networks)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c39f1516",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer, increased from the first network\n",
|
|
" hidden_neurons = 500\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Create two hidden layers instead of one hidden layer. Explicitly set the activation\n",
|
|
" # function to be the sigmoid function (the default value is hyperbolic tangent)\n",
|
|
" rnn1 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=True, # This needs to be True if another hidden layer is to follow\n",
|
|
" stateful = stateful, activation = 'sigmoid',\n",
|
|
" name=\"RNN1\")(inp)\n",
|
|
" rnn2 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False, activation = 'sigmoid',\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN2\")(rnn1)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn2)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn_2layers(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "842c7602",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Other Types of Recurrent Neural Networks\n",
|
|
"\n",
|
|
"Besides a simple recurrent neural network layer, there are two other\n",
|
|
"commonly used types of recurrent neural network layers: Long Short\n",
|
|
"Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short\n",
|
|
"introduction to these layers see <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>\n",
|
|
"and <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>.\n",
|
|
"\n",
|
|
"The first network created below is similar to the previous network,\n",
|
|
"but it replaces the SimpleRNN layers with LSTM layers. The second\n",
|
|
"network below has two hidden layers made up of GRUs, which are\n",
|
|
"preceeded by two dense (feeddorward) neural network layers. These\n",
|
|
"dense layers \"preprocess\" the data before it reaches the recurrent\n",
|
|
"layers. This architecture has been shown to improve the performance\n",
|
|
"of recurrent neural networks (see the link above and also\n",
|
|
"<https://arxiv.org/pdf/1807.02857.pdf>."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "6f0e9b62",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons on the input/output layer and the number of neurons in the hidden layer\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input Layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)\n",
|
|
" rnn= LSTM(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True, activation='tanh')(inp)\n",
|
|
" rnn1 = LSTM(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True, activation='tanh')(rnn)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn1)\n",
|
|
" # Define the midel\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the model\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with four hidden layers (two dense followed by\n",
|
|
" two GRU layers) and returns the model.\n",
|
|
" \"\"\" \n",
|
|
" # Number of neurons on the input/output layers and hidden layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden Dense (feedforward) layers\n",
|
|
" dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)\n",
|
|
" dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)\n",
|
|
" # Hidden GRU layers\n",
|
|
" rnn1 = GRU(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True)(dnn1)\n",
|
|
" rnn = GRU(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True)(rnn1)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Define the model\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the mdoel\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Change the method name to reflect which network you want to use\n",
|
|
"model = dnn2_gru2(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)\n",
|
|
"\n",
|
|
"\n",
|
|
"# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)\n",
|
|
"# \n",
|
|
"# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"# Reshape the data for Keras specifications\n",
|
|
"X_train = X_train.reshape((dim, 1))\n",
|
|
"y_train = y_train.reshape((dim, 1))\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Set the sequence length to 1 for regular data formatting \n",
|
|
"model = rnn(length_of_sequences = 1)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(X_train, y_train, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict the remaining data points\n",
|
|
"X_pred = X_tot[dim:]\n",
|
|
"X_pred = X_pred.reshape((len(X_pred), 1))\n",
|
|
"y_model = model.predict(X_pred)\n",
|
|
"y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))\n",
|
|
"\n",
|
|
"# Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
"# for the training data.\n",
|
|
"fig, ax = plt.subplots()\n",
|
|
"ax.plot(X_tot, y_tot, label=\"true\", linewidth=3)\n",
|
|
"ax.plot(X_tot, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
"ax.legend()\n",
|
|
"# Created a red region to represent the points used in the training data.\n",
|
|
"ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0752ba7f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Generative Models\n",
|
|
"\n",
|
|
"**Generative models** describe a class of statistical models that are a contrast\n",
|
|
"to **discriminative models**. Informally we say that generative models can\n",
|
|
"generate new data instances while discriminative models discriminate between\n",
|
|
"different kinds of data instances. A generative model could generate new photos\n",
|
|
"of animals that look like 'real' animals while a discriminative model could tell\n",
|
|
"a dog from a cat. More formally, given a data set $x$ and a set of labels /\n",
|
|
"targets $y$. Generative models capture the joint probability $p(x, y)$, or\n",
|
|
"just $p(x)$ if there are no labels, while discriminative models capture the\n",
|
|
"conditional probability $p(y | x)$. Discriminative models generally try to draw\n",
|
|
"boundaries in the data space (often high dimensional), while generative models\n",
|
|
"try to model how data is placed throughout the space.\n",
|
|
"\n",
|
|
"**Note**: this material is thanks to Linus Ekstrøm."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "784138f8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Generative Adversarial Networks\n",
|
|
"\n",
|
|
"**Generative Adversarial Networks** are a type of unsupervised machine learning\n",
|
|
"algorithm proposed by [Goodfellow et. al](https://arxiv.org/pdf/1406.2661.pdf)\n",
|
|
"in 2014 (short and good article).\n",
|
|
"\n",
|
|
"The simplest formulation of\n",
|
|
"the model is based on a game theoretic approach, *zero sum game*, where we pit\n",
|
|
"two neural networks against one another. We define two rival networks, one\n",
|
|
"generator $g$, and one discriminator $d$. The generator directly produces\n",
|
|
"samples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a42f89ee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" x = g(z; \\theta^{(g)})\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abe7212f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The discriminator attempts to distinguish between samples drawn from the\n",
|
|
"training data and samples drawn from the generator. In other words, it tries to\n",
|
|
"tell the difference between the fake data produced by $g$ and the actual data\n",
|
|
"samples we want to do prediction on. The discriminator outputs a probability\n",
|
|
"value given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0821eaee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" d(x; \\theta^{(d)})\n",
|
|
"\\label{_auto2} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55e0ccf6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"indicating the probability that $x$ is a real training example rather than a\n",
|
|
"fake sample the generator has generated. The simplest way to formulate the\n",
|
|
"learning process in a generative adversarial network is a zero-sum game, in\n",
|
|
"which a function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f37ece14",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto3} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d6d6d5fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"determines the reward for the discriminator, while the generator gets the\n",
|
|
"conjugate reward"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c74b45c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" -v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto4} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "605ac8c9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"During learning both of the networks maximize their own reward function, so that\n",
|
|
"the generator gets better and better at tricking the discriminator, while the\n",
|
|
"discriminator gets better and better at telling the difference between the fake\n",
|
|
"and real data. The generator and discriminator alternate on which one trains at\n",
|
|
"one time (i.e. for one epoch). In other words, we keep the generator constant\n",
|
|
"and train the discriminator, then we keep the discriminator constant to train\n",
|
|
"the generator and repeat. It is this back and forth dynamic which lets GANs\n",
|
|
"tackle otherwise intractable generative problems. As the generator improves with\n",
|
|
" training, the discriminator's performance gets worse because it cannot easily\n",
|
|
" tell the difference between real and fake. If the generator ends up succeeding\n",
|
|
" perfectly, the the discriminator will do no better than random guessing i.e.\n",
|
|
" 50\\%. This progression in the training poses a problem for the convergence\n",
|
|
" criteria for GANs. The discriminator feedback gets less meaningful over time,\n",
|
|
" if we continue training after this point then the generator is effectively\n",
|
|
" training on junk data which can undo the learning up to that point. Therefore,\n",
|
|
" we stop training when the discriminator starts outputting $1/2$ everywhere.\n",
|
|
"\n",
|
|
"At convergence we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfec7462",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto5\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" g^* = \\underset{g}{\\mathrm{argmin}}\\hspace{2pt}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto5} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37c65d4c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The default choice for $v$ is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c868a092",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto6\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)}) = \\mathbb{E}_{x\\sim p_\\mathrm{data}}\\log d(x)\n",
|
|
" + \\mathbb{E}_{x\\sim p_\\mathrm{model}}\n",
|
|
" \\log (1 - d(x))\n",
|
|
"\\label{_auto6} \\tag{6}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ad465af3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The main motivation for the design of GANs is that the learning process requires\n",
|
|
"neither approximate inference (variational autoencoders for example) nor\n",
|
|
"approximation of a partition function. In the case where"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "27858a4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto7\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto7} \\tag{7}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86006023",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"is convex in $\\theta^{(g)} then the procedure is guaranteed to converge and is\n",
|
|
"asymptotically consistent\n",
|
|
"( [Seth Lloyd on QuGANs](https://arxiv.org/pdf/1804.09139.pdf) ).\n",
|
|
"\n",
|
|
"This is in\n",
|
|
"general not the case and it is possible to get situations where the training\n",
|
|
"process never converges because the generator and discriminator chase one\n",
|
|
"another around in the parameter space indefinitely. A much deeper discussion on\n",
|
|
"the currently open research problem of GAN convergence is available\n",
|
|
"[here](https://www.deeplearningbook.org/contents/generative_models.html). To\n",
|
|
"anyone interested in learning more about GANs it is a highly recommended read.\n",
|
|
"Direct quote: \"In this best-performing formulation, the generator aims to\n",
|
|
"increase the log probability that the discriminator makes a mistake, rather than\n",
|
|
"aiming to decrease the log probability that the discriminator makes the correct\n",
|
|
"prediction.\" [Another interesting read](https://arxiv.org/abs/1701.00160)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2fee38bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Writing Our First Generative Adversarial Network\n",
|
|
"Let us now move on to actually implementing a GAN in tensorflow. We will study\n",
|
|
"the performance of our GAN on the MNIST dataset. This code is based on and\n",
|
|
"adapted from the\n",
|
|
"[google tutorial](https://www.tensorflow.org/tutorials/generative/dcgan)\n",
|
|
"\n",
|
|
"First we import our libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "004a0b53",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import time\n",
|
|
"import numpy as np\n",
|
|
"import tensorflow as tf\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from tensorflow.keras import layers\n",
|
|
"from tensorflow.keras.utils import plot_model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af161",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define our hyperparameters and import our data the usual way"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "8cbaf16a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"BUFFER_SIZE = 60000\n",
|
|
"BATCH_SIZE = 256\n",
|
|
"EPOCHS = 30\n",
|
|
"\n",
|
|
"data = tf.keras.datasets.mnist.load_data()\n",
|
|
"(train_images, train_labels), (test_images, test_labels) = data\n",
|
|
"train_images = np.reshape(train_images, (train_images.shape[0],\n",
|
|
" 28,\n",
|
|
" 28,\n",
|
|
" 1)).astype('float32')\n",
|
|
"\n",
|
|
"# we normalize between -1 and 1\n",
|
|
"train_images = (train_images - 127.5) / 127.5\n",
|
|
"training_dataset = tf.data.Dataset.from_tensor_slices(\n",
|
|
" train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "822b8cc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### MNIST and GANs\n",
|
|
"\n",
|
|
"Let's have a quick look"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "52b5965c",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.imshow(train_images[0], cmap='Greys')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "21c5199c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our two models. This is where the 'magic' happens. There are a\n",
|
|
"huge amount of possible formulations for both models. A lot of engineering and\n",
|
|
"trial and error can be done here to try to produce better performing models. For\n",
|
|
"more advanced GANs this is by far the step where you can 'make or break' a\n",
|
|
"model.\n",
|
|
"\n",
|
|
"We start with the generator. As stated in the introductory text the generator\n",
|
|
"$g$ upsamples from a random sample to the shape of what we want to predict. In\n",
|
|
"our case we are trying to predict MNIST images ($28\\times 28$ pixels)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "356759c7",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_model():\n",
|
|
" \"\"\"\n",
|
|
" The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to\n",
|
|
" produce an image from a random seed. We start with a Dense layer taking this\n",
|
|
" random sample as an input and subsequently upsample through multiple\n",
|
|
" convolutional layers.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
"\n",
|
|
"\n",
|
|
" # adding our input layer. Dense means that every neuron is connected and\n",
|
|
" # the input shape is the shape of our random noise. The units need to match\n",
|
|
" # in some sense the upsampling strides to reach our desired output shape.\n",
|
|
" # we are using 100 random numbers as our seed\n",
|
|
" model.add(layers.Dense(units=7*7*BATCH_SIZE,\n",
|
|
" use_bias=False,\n",
|
|
" input_shape=(100, )))\n",
|
|
" # we normalize the output form the Dense layer\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" # and add an activation function to our 'layer'. LeakyReLU avoids vanishing\n",
|
|
" # gradient problem\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" model.add(layers.Reshape((7, 7, BATCH_SIZE)))\n",
|
|
" assert model.output_shape == (None, 7, 7, BATCH_SIZE)\n",
|
|
" # even though we just added four keras layers we think of everything above\n",
|
|
" # as 'one' layer\n",
|
|
"\n",
|
|
" # next we add our upscaling convolutional layers\n",
|
|
" model.add(layers.Conv2DTranspose(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(1, 1),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 7, 7, 128)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 14, 14, 64)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=1,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False,\n",
|
|
" activation='tanh'))\n",
|
|
" assert model.output_shape == (None, 28, 28, 1)\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "854bcd6b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"And there we have our 'simple' generator model. Now we move on to defining our\n",
|
|
"discriminator model $d$, which is a convolutional neural network based image\n",
|
|
"classifier."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "41473304",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_model():\n",
|
|
" \"\"\"\n",
|
|
" The discriminator is a convolutional neural network based image classifier\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
" model.add(layers.Conv2D(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" input_shape=[28, 28, 1]))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
"\n",
|
|
" model.add(layers.Conv2D(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same'))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
" model.add(layers.Flatten())\n",
|
|
" model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af567",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Let us take a look at our models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "f899d4e3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generator = generator_model()\n",
|
|
"plot_model(generator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "87ef384b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"discriminator = discriminator_model()\n",
|
|
"plot_model(discriminator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2bef82d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we need a few helper objects we will use in training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "e397847a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
|
|
"generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n",
|
|
"discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "db3396cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The first object, *cross_entropy* is our loss function and the two others are\n",
|
|
"our optimizers. Notice we use the same learning rate for both $g$ and $d$. This\n",
|
|
"is because they need to improve their accuracy at approximately equal speeds to\n",
|
|
"get convergence (not necessarily exactly equal). Now we define our loss\n",
|
|
"functions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "931eaced",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_loss(fake_output):\n",
|
|
" loss = cross_entropy(tf.ones_like(fake_output), fake_output)\n",
|
|
"\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "0c4a44bb",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_loss(real_output, fake_output):\n",
|
|
" real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n",
|
|
" fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)\n",
|
|
" total_loss = real_loss + fake_loss\n",
|
|
"\n",
|
|
" return total_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fcf8f066",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a kind of seed to help us compare the learning process over\n",
|
|
"multiple training epochs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "eea2bbee",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"noise_dimension = 100\n",
|
|
"n_examples_to_generate = 16\n",
|
|
"seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94a6e341",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have everything we need to define our training step, which we will apply\n",
|
|
"for every step in our training loop. Notice the @tf.function flag signifying\n",
|
|
"that the function is tensorflow 'compiled'. Removing this flag doubles the\n",
|
|
"computation time."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "8d48470b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tf.function\n",
|
|
"def train_step(images):\n",
|
|
" noise = tf.random.normal([BATCH_SIZE, noise_dimension])\n",
|
|
"\n",
|
|
" with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n",
|
|
" generated_images = generator(noise, training=True)\n",
|
|
"\n",
|
|
" real_output = discriminator(images, training=True)\n",
|
|
" fake_output = discriminator(generated_images, training=True)\n",
|
|
"\n",
|
|
" gen_loss = generator_loss(fake_output)\n",
|
|
" disc_loss = discriminator_loss(real_output, fake_output)\n",
|
|
"\n",
|
|
" gradients_of_generator = gen_tape.gradient(gen_loss,\n",
|
|
" generator.trainable_variables)\n",
|
|
" gradients_of_discriminator = disc_tape.gradient(disc_loss,\n",
|
|
" discriminator.trainable_variables)\n",
|
|
" generator_optimizer.apply_gradients(zip(gradients_of_generator,\n",
|
|
" generator.trainable_variables))\n",
|
|
" discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,\n",
|
|
" discriminator.trainable_variables))\n",
|
|
"\n",
|
|
" return gen_loss, disc_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "62015b88",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a helper function to produce an output over our training epochs\n",
|
|
"to see the predictive progression of our generator model. **Note**: I am including\n",
|
|
"this code here, but comment it out in the training loop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "b189ed96",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_and_save_images(model, epoch, test_input):\n",
|
|
" # we're making inferences here\n",
|
|
" predictions = model(test_input, training=False)\n",
|
|
"\n",
|
|
" fig = plt.figure(figsize=(4, 4))\n",
|
|
"\n",
|
|
" for i in range(predictions.shape[0]):\n",
|
|
" plt.subplot(4, 4, i+1)\n",
|
|
" plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n",
|
|
" plt.axis('off')\n",
|
|
"\n",
|
|
" plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')\n",
|
|
" plt.close()\n",
|
|
" #plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ba2c82d5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Setting up checkpoints to periodically save our model during training so that\n",
|
|
"everything is not lost even if the program were to somehow terminate while\n",
|
|
"training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "a0e2fc8a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setting up checkpoints to save model during training\n",
|
|
"checkpoint_dir = './training_checkpoints'\n",
|
|
"checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')\n",
|
|
"checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n",
|
|
" discriminator_optimizer=discriminator_optimizer,\n",
|
|
" generator=generator,\n",
|
|
" discriminator=discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d93a0f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our training loop"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "a1275556",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def train(dataset, epochs):\n",
|
|
" generator_loss_list = []\n",
|
|
" discriminator_loss_list = []\n",
|
|
"\n",
|
|
" for epoch in range(epochs):\n",
|
|
" start = time.time()\n",
|
|
"\n",
|
|
" for image_batch in dataset:\n",
|
|
" gen_loss, disc_loss = train_step(image_batch)\n",
|
|
" generator_loss_list.append(gen_loss.numpy())\n",
|
|
" discriminator_loss_list.append(disc_loss.numpy())\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epoch + 1, seed_images)\n",
|
|
"\n",
|
|
" if (epoch + 1) % 15 == 0:\n",
|
|
" checkpoint.save(file_prefix=checkpoint_prefix)\n",
|
|
"\n",
|
|
" print(f'Time for epoch {epoch} is {time.time() - start}')\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epochs, seed_images)\n",
|
|
"\n",
|
|
" loss_file = './data/lossfile.txt'\n",
|
|
" with open(loss_file, 'w') as outfile:\n",
|
|
" outfile.write(str(generator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write(str(discriminator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ff3a75a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"To train simply call this function. **Warning**: this might take a long time so\n",
|
|
"there is a folder of a pretrained network already included in the repository."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"id": "371ed41a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"train(train_dataset, EPOCHS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "654399f1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now to avoid having to train and everything, which will take a while depending\n",
|
|
"on your computer setup we now load in the model which produced the above gif."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"id": "dec4b560",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))\n",
|
|
"restored_generator = checkpoint.generator\n",
|
|
"restored_discriminator = checkpoint.discriminator\n",
|
|
"\n",
|
|
"print(restored_generator)\n",
|
|
"print(restored_discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "296bfa5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We have successfully loaded in our latest model. Let us now play around a bit\n",
|
|
"and see what kind of things we can learn about this model. Our generator takes\n",
|
|
"an array of 100 numbers. One idea can be to try to systematically change our\n",
|
|
"input. Let us try and see what we get"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"id": "eecfbb1f",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_latent_points(number=100, scale_means=1, scale_stds=1):\n",
|
|
" latent_dim = 100\n",
|
|
" means = scale_means * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" latent_space_value_range = tf.random.normal([number, latent_dim],\n",
|
|
" means,\n",
|
|
" stds,\n",
|
|
" dtype=tf.float64)\n",
|
|
"\n",
|
|
" return latent_space_value_range\n",
|
|
"\n",
|
|
"def generate_images(latent_points):\n",
|
|
" # notice we set training to false because we are making inferences\n",
|
|
" generated_images = restored_generator.predict(latent_points)\n",
|
|
"\n",
|
|
" return generated_images"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"id": "333a593d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def plot_result(generated_images, number=100):\n",
|
|
" # obviously this assumes sqrt number is an int\n",
|
|
" fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),\n",
|
|
" figsize=(10, 10))\n",
|
|
"\n",
|
|
" for i in range(int(np.sqrt(number))):\n",
|
|
" for j in range(int(np.sqrt(number))):\n",
|
|
" axs[i, j].imshow(generated_images[i*j], cmap='Greys')\n",
|
|
" axs[i, j].axis('off')\n",
|
|
"\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"id": "2f5f0154",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generated_images = generate_images(generate_latent_points())\n",
|
|
"plot_result(generated_images)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff581bf2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We see that the generator generates images that look like MNIST\n",
|
|
"numbers: $1, 4, 7, 9$. Let's try to tweak it a bit more to see if we are able\n",
|
|
"to generate a similar plot where we generate every MNIST number. Let us now try\n",
|
|
"to 'move' a bit around in the latent space. **Note**: decrease the plot number if\n",
|
|
"these following cells take too long to run on your computer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"id": "d3617ad8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 225\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=-5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=5))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a074f93",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Again, we have found something interesting. *Moving* around using our means\n",
|
|
"takes us from digit to digit, while *moving* around using our standard\n",
|
|
"deviations seem to increase the number of different digits! In the last image\n",
|
|
"above, we can barely make out every MNIST digit. Let us make on last plot using\n",
|
|
"this information by upping the standard deviation of our Gaussian noises."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"id": "4ef8937d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 400\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=10))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "385a2d0a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"A pretty cool result! We see that our generator indeed has learned a\n",
|
|
"distribution which qualitatively looks a whole lot like the MNIST dataset.\n",
|
|
"\n",
|
|
"Another interesting way to explore the latent space of our generator model is by\n",
|
|
"interpolating between the MNIST digits. This section is largely based on\n",
|
|
"[this excellent blogpost](https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/)\n",
|
|
"by Jason Brownlee.\n",
|
|
"\n",
|
|
"So let us start by defining a function to interpolate between two points in the\n",
|
|
"latent space."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"id": "57de87b8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def interpolation(point_1, point_2, n_steps=10):\n",
|
|
" ratios = np.linspace(0, 1, num=n_steps)\n",
|
|
" vectors = []\n",
|
|
" for i, ratio in enumerate(ratios):\n",
|
|
" vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))\n",
|
|
"\n",
|
|
" return tf.stack(vectors)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfb76bb6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have all we need to do our interpolation analysis."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"id": "e25decef",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 100\n",
|
|
"latent_points = generate_latent_points(number=plot_number)\n",
|
|
"results = None\n",
|
|
"for i in range(0, 2*np.sqrt(plot_number), 2):\n",
|
|
" interpolated = interpolation(latent_points[i], latent_points[i+1])\n",
|
|
" generated_images = generate_images(interpolated)\n",
|
|
"\n",
|
|
" if results is None:\n",
|
|
" results = generated_images\n",
|
|
" else:\n",
|
|
" results = tf.stack((results, generated_images))\n",
|
|
"\n",
|
|
"plot_results(results, plot_number)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |