Morten Hjorth-Jensen
2775 lines
150 KiB
Plaintext
2775 lines
150 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d2db899",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html chapter13.do.txt -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4908a97",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Recurrent neural networks: Overarching view\n",
|
|
"\n",
|
|
"Till now our focus has been, including convolutional neural networks\n",
|
|
"as well, on feedforward neural networks. The output or the activations\n",
|
|
"flow only in one direction, from the input layer to the output layer.\n",
|
|
"\n",
|
|
"A recurrent neural network (RNN) looks very much like a feedforward\n",
|
|
"neural network, except that it also has connections pointing\n",
|
|
"backward. \n",
|
|
"\n",
|
|
"RNNs are used to analyze time series data such as stock prices, and\n",
|
|
"tell you when to buy or sell. In autonomous driving systems, they can\n",
|
|
"anticipate car trajectories and help avoid accidents. More generally,\n",
|
|
"they can work on sequences of arbitrary lengths, rather than on\n",
|
|
"fixed-sized inputs like all the nets we have discussed so far. For\n",
|
|
"example, they can take sentences, documents, or audio samples as\n",
|
|
"input, making them extremely useful for natural language processing\n",
|
|
"systems such as automatic translation and speech-to-text.\n",
|
|
"\n",
|
|
"More to text to be added"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43cb4913",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A simple example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "cf6b9dab",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAACYY0lEQVR4nO2dd5wU5f3HP7NXKXdHvTvKAYdU6R4qKAqKIrZoNMaKGjUJKjZCUDQxRmPwF40hVtSoGNFoFGtUBJUiAiK9o0g74Y6j3h3t2s7vj2X3pjwz88zsM2V3v29fJ7tTn53nmef5Pt/2SLIsyyAIgiAIgkgQQn4XgCAIgiAIwg4kvBAEQRAEkVCQ8EIQBEEQREJBwgtBEARBEAkFCS8EQRAEQSQUJLwQBEEQBJFQkPBCEARBEERCQcILQRAEQRAJRbrfBRBNOBzGrl27kJOTA0mS/C4OQRAEQRAcyLKM6upqtG/fHqGQuW4l6YSXXbt2oaioyO9iEARBEAThgNLSUnTs2NH0mKQTXnJycgBEfnxubq7PpSEIgiAIgoeqqioUFRXFxnEzkk54iZqKcnNzSXghCIIgiASDx+WDHHYJgiAIgkgoSHghCIIgCCKhIOGFIAiCIIiEgoQXgiAIgiASChJeCIIgCIJIKEh4IQiCIAgioSDhhSAIgiCIhIKEF4IgCIIgEgpXhZfnn38e/fv3jyWMGzp0KD777DPTc+bNm4eSkhJkZ2eja9eumDp1qptFJAiCIAgiwXBVeOnYsSMee+wxLF26FEuXLsXZZ5+NSy65BOvWrWMev3XrVlxwwQU444wzsGLFCtx///248847MWPGDDeLSRAEQRBEAiHJsix7ecNWrVrh8ccfx80336zbd++99+Kjjz7Chg0bYtvGjh2LVatWYdGiRVzXr6qqQl5eHiorK2l5AIIgCIJIEOyM3575vDQ0NOCtt97C4cOHMXToUOYxixYtwqhRo1TbzjvvPCxduhR1dXXMc2pqalBVVaX6IwiCIAgieXFdeFmzZg2aN2+OrKwsjB07Fu+//z5OPPFE5rHl5eUoKChQbSsoKEB9fT327t3LPGfy5MnIy8uL/RUVFQn/DQRBEATwwYqdmLOpwu9iEIT7wkvPnj2xcuVKLF68GLfeeituuOEGrF+/3vB47WqSUauW0SqTkyZNQmVlZeyvtLRUXOEJgiAIAEDp/iO4++2V+NWr3/ldFIJAuts3yMzMRLdu3QAAgwcPxnfffYd//vOfeOGFF3THFhYWory8XLWtoqIC6enpaN26NfP6WVlZyMrKEl9wgiAIIsbeQzV+F4EgYnie50WWZdTUsF+CoUOHYvbs2apts2bNwuDBg5GRkeFF8ZKG6mNsHyGCIAgnKLXf4bCncR4EocNV4eX+++/H119/jW3btmHNmjV44IEHMHfuXFx77bUAIiaf66+/Pnb82LFjsX37dowfPx4bNmzAK6+8gpdffhkTJkxws5hJx7NzNqPfQ7Pw/oqf/C4KQRBJgtJwX0/CC+EzrpqNdu/ejTFjxqCsrAx5eXno378/Zs6ciXPPPRcAUFZWhh07dsSOLy4uxqeffop77rkHzz77LNq3b4+nnnoKl19+uZvFTDoe/3wTAODed9fg54M6+lwagiCSAaXbYdjbDBsEocNV4eXll1823T9t2jTdtuHDh2P58uUulSjFYPs4EwRB2EZSdCikeSH8htY2IgiCICxRal4aGkh4IfyFhJckhhQvBEGIYHPFIVz94uLY9/pw2MfSEAQJL0mNQWocIkX4bE0Zhvz1SyzbfsDvohAJzq3Tl6G6pj72vYHMRilLXUMYtfX+C68kvASU0v1H8Lv/rsLGclrugHDGrW8sR3nVMdz46hK/i0IkOBXV6vQW5POSmsiyjBGPz8WQyV+irsFfAYaElwChbAy/fX0ZZiz/CRc/vcDw+E3l1fjZMwsw1yBdt0SGIwJAvYl/wvzv9+CcJ+dh+Q7SzhDAC/N+xMi/z8WeavOEdKR5SU1q6sPYefAo9h+uxc4DR30tCwkvAeGDFTvR/YHP8MnqMgCIaVzqTAae37y+FKt/qsSNBum6yWxEWHH9K0uwueIQrnlpsfXBRNIz+bON+HHPYfzzy+9NjyPNS2qiHFP8bgMkvASEu99eCQC4/c1ImLjRWk5KDh6hLLqENTKsO5ljdf7bsIngYKatA4AGcthNSZTpffzO9UPCS0ARoTQhxQtBEGbMXFuGP324FvU2/Rf8nnUT/mMl4LqN6wszEs4QYfLh0d4QBJG6jJ0e0fT2aZ+HX55cxH2e3wMX4Q9B0ryQ8JLAkGxCEIQIKqqPme6XNQMVOeymHtv2HlY59vutfSPhJYFhyS4LN+813U+kHrQMDWGF3Tbi98BFeM+IJ+aqvvvt90Q+LwHFaZjzNf/6VnkRIoV4b/lPOONvX2FTebXwa+/YdwTLth9AmAatlMBKmCHNC+G36ZCEl6DiouBBHU9yMv6/q1C6/yh+985Kodd95qsfcObjc3D58wvxrwVbhF6bCAZWJmhtj0HLAxANFG1EsHAr2ugPH6zBwIdnWdq4icRFdOruJ2Y15vx4ffF2odcmEhOaABF+twESXhIYq2gi1v7pi3eg+lg9/r2QBiHCPpS1OTkxmkQbmQnJ54Xwuw2Q8BJQeCKJ4hlGeBKXEYSWEMkuKcO7y37CgD/Pwrdb9un2NVCodMrjt/8bRRv5TENYRtVRdzLlmglAFIFCOIFyByUn2mqVJGDCO6sAAFe+qF86wm9/B8J//Na8kPDiM1e/tBhLtu7XbVeq51eVHkT/jnm2Bw6zo6nrSV60ph2Rdb1172HUNYSRkUZK21TGb38Hwn/8bgPUA/kMS3AB1DOhS579BuP+swL7D9fauraZsEMTJ8KIhrCM299cjmfnbGbun/bNNm8LRLiOtj+w6h9Ys+59h2rw0EfrsKGsSmDJiKDit+aFhJcE4ZPVZRj1j3mqbfFo8MnnhTDi6x/24JPVZXj8802Y8oV+deFl2w8wziISmb/P/t5wIsWClaDs0U83YNrCbTj/n1+LLBoRUPz2eSHhJaCw5JK9h7SaF4toI7OdJLsQBihXmJ7yxQ+6/eT2kpz88oVF3MeyEpRt3XtYZHGIgEOaF8I1TB12vSsG4TG6erdZ2SScJAd1DWF8uWE3Ko+IDwiI+jscrqnHnE0VqK0Po3WzLOH3IYKL35oXctgNKG5HdWgXWiOIKCHL/EEeFYSIixfnb8Hjn29Cr8IczLz7TKHXjs66x725HHM27cFvh3dFm+aZQu9BBBvSvBBMeMYH60HE+AAKFkgdahvC+O/SUu6ZEskmycFHK3cBADa6sNZVfUPEtDhn0x4AwFtLStFaIbzU1DcIvycRLGhhRkIYWm2KVrj5aNUuD0tDBImJ767Gh6t2ch0bsugVKMtuYpCR7ryerBz6tXJwTnY6crMzYt/dMFURwaKOFmZMXTZXmMyIHGTYNbMErSw9iDv/syL2/aNVu/Dusp+sb0IkDatKK7mOszRZkuySEKRbSaFxENZ0NjkKwSWy37VbEwGB8rykMOc8OV/o9bQdinKM+bHikGrfnuoaTHhnFQ7YzB1D+EtDWMbCH/ficE29a/ew9Hlx7c6ESDLSzGtKpN9bbna6SldDqRgSF952Ueez2YgcdgNGdNxw4vOiTdmt3G/UHI/WNaAld+kIv3l+7mY8Met7nFLcCv/97VBb5/J2SlZtj5YISAyMNC/3v78GlUfq0CwrzfG1tROl3CYZKs0vxQMkJrX1YVz41NfoWZiDZ645yfRYv9e3IuElYESHBScDhFmHoe1soqSFJByqqcdrC7fh/L6F6Nq2ue37Et7xnyWlAIwzM4tAq3kJSWQGSETSGZqXmvoGvPntDstzrfyaWD4vSm0LNZfEZOGPe/FDxSH8UHEIz1xjfmwdmY0IUejNRtaqF0kCJn+6AY9/vgln/30e+yAioTASfHm7Gu3pWmGG9C6JQSZj/alKQYvAhmVZFVGUq/F5oVQMyY/feV5IeAkoVoqXQzX12F1Vo9qmdaBSm43YDS0kSVi6jdK9pwK7Dh7Fo5+sx86DR02PsxJeiMSAtXhm1VExvlKyHDExRMlMD5HZKMXw26/JVeFl8uTJOPnkk5GTk4P8/Hxceuml2LRpk+k5c+fOhSRJur+NGze6WdTAIEkStu87jJo6c2eo8/6hd/Y1E4SNOhMallKHLzZU4KWvt+KGV5aYHqc1GWhlF5JlEgOW2ajqGJ/mxWpgkmVZNVmiNpF6+C2guiq8zJs3D7fffjsWL16M2bNno76+HqNGjcLhw9ZrYGzatAllZWWxv+7du7tZ1MDQEJYx/PG5OFpnnuSJNXvWqvGU/YlRO6MJUuqxWRN5piXkUFjZU11jfRDhGWzNiyizkVrTG5IklanI74GNcB+/q9hVh92ZM2eqvr/66qvIz8/HsmXLcOaZ5umq8/Pz0aJFCxdLl7go1bVRKqqP6RZLU/o+GHUmsuy/+o8QS7yT4FBI67Br7fPy8oKteOR/6zFhVA+MOzs1JhpBhxUqXXVMjNkorNG8AOo+hvqUxMROoIjfAqqnPi+VlZEkWa1atbI8dtCgQWjXrh1GjhyJOXPmuF20hGJ31THdtlMe/RJXTFWvChuWZXy1cTf2VNcYRhtRJ0No0XZfPD4vj/xvPQDgiVnfu1AiwgnpLmpeZFmdmiEyCVLvN6KuIexqniLCG/weOzwLlZZlGePHj8ewYcPQt29fw+PatWuHF198ESUlJaipqcHrr7+OkSNHYu7cuUxtTU1NDWpqGtXVVVVVrpQ/SJQzhBdAb0oqqzyGm6YtRV6TDEw4ryf7YiS7EBq0sy+9zws5OCQCGVr7HyBMaJBlWaXp1Q5khmZqWcZ5/5iPqmN1+Hri2WiS6TzXDOEzPo8dngkv48aNw+rVq7FgwQLT43r27ImePRsH2qFDh6K0tBRPPPEEU3iZPHky/vznPwsvb5A5Umtv0bPKo3WGUyGSXQgr7EYbVR+r06WLJ9xlY3kV1u6swuUndYgJl0rNiyzLkCRJ2ErAYVmTmkHWmI0M+pudB49iy96Iz+OuyqM4gfJKJSx+jx2emI3uuOMOfPTRR5gzZw46duxo+/whQ4bghx9+YO6bNGkSKisrY3+lpaXxFjfwGJmAzDCeCcVXFiJ4xK8YUTcKnQOvxdl//GBtvAUgbDJ6yteY8M4qzFq/O7ZN6bAbFVpE5ebQ+rzIUGtfjO6yYsfBxvK5uPYSIZbXF23TbfM7l4+rrUeWZYwbNw7vvfcevvrqKxQXFzu6zooVK9CuXTvmvqysLOTm5qr+gsi0b7Zi5tpyIddy0mgMHXZ9l5+JoKFtK3Y1L19trBBYGsIOy7c35mzKVDjs1jVEnPx5NS9WXYwMdbSRLMtceV4OHmlcS436nuBh9Kb/8cN1um1+T3xdNRvdfvvtePPNN/Hhhx8iJycH5eWRwTsvLw9NmjQBENGc7Ny5E//+978BAFOmTEGXLl3Qp08f1NbWYvr06ZgxYwZmzJjhZlFdZVN5NR76OOLQuO2xC+O+npP1sAwddqn/ICzQ+bhYyDLUpPxDmWJBaTaqq5eBTGdaWxZhWTZ12N1/uBbH6hqQnRHxaZk670e0aZ6lugb1PYmN39XnqvDy/PPPAwBGjBih2v7qq6/ixhtvBACUlZVhx47GtTZqa2sxYcIE7Ny5E02aNEGfPn3wySef4IILLnCzqK6xbe9hQwdbpzgyGxlqXghCjbZNaM1G7y3fiQ4tmuD2s7phU3k1+nfMM78A4RlKf7g0RcXVHte8aMObjbBStr0wbwv2aDJ8KzuZX76wCG2aZ2HpH87BpvJqPPZZJMnow5f0aTycqyREUPFb+HRVeOExb0ybNk31feLEiZg4caJLJfKW/63ehXFvrkBWuljrnBOztZn3v9+NkGgkHJYx7j/LUZjbBA9efKJuP48FJ16fFx6z0dNfbcbUeT+irkHGoz9XRw+Kmt0T9jmqEF6U/a9d4YWH91bsbLwXY//eQzWqfyNlUh8TDsu6vEJEMIg6eRvuT+blAVKdl77eCgCoYSSViwdnPi9kNkoEvtu2H5+uKccr32z1rQzatmI0ttQdD5X9zxL1KsXUpPxjV+VRHDgc8StRVmPUUVek8KJEazZSokyqqWxb//p6CwY9MhsbypI/vUUiYpWJ2++xg4QXF3HDG/vDlTsdaV6IxOCZOZv9LoJuELLK66JdC8nvTi2VWbHjIAY9Mlu14jPQWCcNLlWODGMNrnLypnQYfuPbHag8WkfRaQFC+aqfy1g/L0iQ8OIibqjP73prJXbsP2L7PBpQgk9DWMbXP+wVcq14BOd424rf6mQCqDxSp6qFw7X1WLer0mXNC/va0UgnAFi/S69lIbMR4QTPktSlIk6ignioqLbvAGzUsZBQExycCrv3vL0SP+5pVPFKkPDMV841ONq2YuVDo91Pbcp/0tNCqnq46sXFqDxah5wsvi7fSR0anaMUXpR+MlFIdklM/M7zQsKLi7jluOgk0RTleUlOftxzCO8zBoS/zxa3xpBVM9aOPSS8+E+aJKne7crjaxpVu7SmkCwb9ySshWSVpJH0kpD4/ZqT2chF3BJenNitjeQdGmiCg926aAjLqoyl9u5lcjO7bUKjeiGBOBh4+W6b3aq2wVx4sZsEkQgGfo8dJLy4iFuOtRZ9ARNDs1GcZSHEoVvczqJ36POnmfhmszMfmZtfW2pSjvjwu1NLNVjtxGsBUpaN672mjoSXREHrfG+G35MUMhu5xLa9h3HwiJjl57U0OHCmMTQbmah7CW/R5cCQgTSTvuRYXZhpMuIZC8xS+GvLYSVE6cxG1rcnBMJywjULXeZhvc3wZfn4fyyUWX9ZkNUouPx70TbsO1TL3Of3JIWEFxfYVF6N86a4F2bmRPPy+OebmNtpoAkuEaHB+55dpwGyOF7vsEutyksM1yuKox7WMaKCzJBlGDYUa+GFpJeg8iBjTaMofr/lJLwI5oV5P2Ly8VTYbiHSl4bGmeDCqhov+nm95sXe+ZSHyFuYmhd4O7iY3e+YlfBCqpeExO+xg3xeBFLfEHZdcAFEZ8k0v9bG8io89NE6VYpvwh30ZiN/egftXa0MizT0+AtL8+K19ivi88K+p1V0JMku/hIOy/jv0lL8uOeQzckR+bwkDXsNbINKrNaL4EGk8GLVx42e8jUAYNrCbRjdpxBTrhoYWymWEIveYdfZdUSPBaTWDzaGmpeAaMCsikGh0v7y7rKfMHHGagDAG7ecyn2e3+2LNC8C2c2xerSIChcqvNg4dua6ct06NoQ4/O4MSo9nbtbOoHsW5pieF68wTsSHscOut4Yj46AA8zOp/fjLsu0HHJ3nd39FwotAynmEFwH3Ebk+id1LRZNdEe7jtdno3uOzL93aRhbn0dDjL0aTGU/zvJhEN1kJUWkkvPiKsn7s1ITfcaokvAikgkvzEn+Fi9W8yKoyHatrwNqdlRQx4gM6XxOPq2DnwaPMgtiNNiK8pZ6ROsE4cNkdzPK8WLVjshr5i9N+xu8hgoQXgRypNfeqB8RoXgxDIx2gbYBXv7QYFz29ADOW6/OHEO6iFRgd17JDaSKaTEw77BnleSCCAXMy4/HAYpbnxQryqQoQNqrC7+ktOewKhKcyZRn4z5IdqKhyHr3jZG0jI47VNeDHPYdj36Pp5t/+bgd+UdJRd7ydDIyEPbS16rXZaP/h2kjSQs1t1+ysND2P2oS/MKON4IPZyEjzYnEuhUr7i2z4xeI80rykFjJkTHpvDf7xhfOF80SajV6cv8XW8X7bOVMJLzoH5aS3tiGM8f9dZf++NPZ4zqGaejw7ZzPKKo8yJzNeO+ya3YnMRolDIvXuJLwIhKfTFxJtJHBU27r3MHM7zaa9R1etXggvmu+s5QaI4DHtm614/PNNGDr5KwPNS5zrA9jEvEuycNgl6cVXlHVnZ2jxeyJLwotA3KpM7cstNkmdPUiocYdDNfX46ycbVNtYZiOe52+nhlhhqqR4CT67KhuDA4wiAD0OlDZ08rcTKl1T34A/fbgWczYZr71FiEU5btkaw8hslDzwSK1O/Bi0ExMvk9QR3vDE55vw9tJS1TYvqoYleFCkWfApatk09rn6WL1uv+dVaKDoYflQaVH2b68v2o7XFm3Hr179Dg1hGfVOFnIjHGNP8+IvJLx4jBPBQ+uN71faeMI9vt9drdvmhRDBCvSwrXkh1YvnKPuAOsYAH3HY9dbnhXU7Ht8bZZ6Xnw4cjX0+98l5GP74XF81zSmBzPxofZrP4xAJLwLhqUxGSgZLtMKLyJfZyOfFzBYQDssor7TOaUPEh199tt0+iUyJ3qN00mUKLxwaD5HIMjtUmifqySjD7pa9h7Hz4FGuzOWEc5TVY0cg8VukJOFFIDz17sTZVms2srtcvRm1DtSyE95ZhSGTv8Qnq8uElSPVYc5aHXYPdjQhbMHD3n1J8+I9SsG2rsEo2sg7DLPrytat2Mphl9qXuygFFnuaF/FlsQMJLwLhqUtHZqOAeeO/dzwi5akvf/C5JEmOw87BVmthmY1s3pfMmN5jZTbyA7YATqHSQcdxnhfRBbEJCS8C4dK8CPB5CQo0aLmLF2YjES1r8Zb9zO0NYRmXP78Qd/xnhYC7EEpkDuElCGsbhQ3MSUqsJmdklvQOO9pe8nlJIngq3onZKKCyCwkvcXDgcC0e/WQ9Fm/Zhw9X7kRNvX5pCedmI/4GI8Jhl8WxugY8+skGLNt+AB+v2iXgioQSpWBbW8/yefE+SZ2Rw64VQZ2cpQrO87z4Cy0PIBAuzQvDPm1FUF9ukl2c84cP1+KT1WV46euthsf49XxF3PfRTzbg9cXb478QwUQ5CTJKUue1w67hcBan2Sig3V9SYqvNkM9L8sBTl6wVYK0IScF5gZXlIM2LfV5buA2z1pVjrcV6QQC7PXnRDkTM2D9erde2HK6pxztLS3HgMC30GC/Kd4+VC8XrV9M8VNqcNM5G/e2WfZi5loIERCMbfLY+z9/+nzQvIuEJlXZkNopYfYMmKlD6BXus3VmJP320DgBQ3KaZ5fHRcNh/fvEDQhJwx8juwsvE8sESMfAdPKLP+jrpvTX4aNUuDOrUAu/fdnr8N0lhlHVU60CbKxqjyEOzzLtRrMyc0dOvfHExAGDe70egc2vr94ewj61QadK8JA98mhdnodJ2/BhEwHM30rzYY/u+I7HPvLV54HAt/vHF9/j77O9xqEafSZWFnZZitCKxG3x03PclunI5YY0sy1i6bT8OHlFrqyzzvMB/h0pATMi2tp+hHFNioVBpwtVoo4BYjVT43XgTjSO1CuGDo0JlWT0wOfGX4rmHfhtVbFCYtX43fjF1Ec55cr5quyrPC9Nh12+lfoQwR7I8q3Jq9wfhdyUTtDAjg8mTJ+Pkk09GTk4O8vPzcemll2LTpk2W582bNw8lJSXIzs5G165dMXXqVDeLKQyuaCMHwouE4Pi8KCHNiz2O1DZGFPE4Ybv9fElICT6fry0HAOw9VKPazrc8gKtF48LEjZebsKbPpH7HTchsBCAihNx+++1YvHgxZs+ejfr6eowaNQqHDxukpAewdetWXHDBBTjjjDOwYsUK3H///bjzzjsxY8YMN4sqBLc0LxGfl+BJL9SJ2OOwQvPCIzhojxD9vKn6EhdlW2D5vHgdKm2ELHO0dbsN0f+flVSoVpVOoGfrqsPuzJkzVd9fffVV5OfnY9myZTjzzDOZ50ydOhWdOnXClClTAAC9e/fG0qVL8cQTT+Dyyy93s7hxw1PvzjLs2i9LvPBoeshh1x5Haho1LzydhFb1z5sjiFdLF5ZlZGeEcKxOPXN3owMjLY8zjJ6aVbRRYEZ4Dp8Xq/1aoT0gvywpSaBIaW99XiorI+GhrVq1Mjxm0aJFGDVqlGrbeeedh6VLl6KuTh/B4CdllUfxyP/WY/u+iCYp1TLs0oBkD6XmhUeLEpbV7UX06roygC6MqA3SqAUHo7pQr23kf4ZdI2QO6cWqnNpmH4TflUw49nlJZrORElmWMX78eAwbNgx9+/Y1PK68vBwFBQWqbQUFBaivr8fevXt1x9fU1KCqqkr15xW/fX0ZXl6wFb98YREA93xeQpIkJo+7YEjzYo+jCp8XvqgzWdVelD4zIjDqfOo9cgwmnKNeHsDIbOQ/POYry/2axkPCtVhUwksC6V48E17GjRuH1atX4z//+Y/lsdqw4GjjZYULT548GXl5ebG/oqIiMQXmYPVPEU3S7qrjznQ8mhcneV5sn+EeyrLsp2RjtrA7w5FldUd91hNzxZbHoME6Cee3vlcwnc6DjlE7UQq1rJXhA+WwK1rzEleJCDNI86LhjjvuwEcffYQ5c+agY8eOpscWFhaivLxcta2iogLp6elo3bq17vhJkyahsrIy9ldaWiq07HZwy+fFD4wchLWl//eiba6XJdGJ1rly8OZpB1qzEcAnyPI6d0ecKfXbnWSBtr5XYrT7oGHs89L4mRltJJud7R08WhJLnxitzwu1JUveX/ET3lv+E9exKoddG/fwuxZcddiVZRl33HEH3n//fcydOxfFxcWW5wwdOhQff/yxatusWbMwePBgZGRk6I7PyspCVlaWsDJbsevgUczZVIHLT9ILYTwvlVOVZ1AnrQ9+uA7XD+3idzECy08HjuD8KV/j6lM7qQQFHg2cDNlV09w9b69kal9cMRsJv2Jqo3bYZZiNAvLEjQRk7TGm+y2+E2oO19TjnrdXAQDOPbEAOdn6cVMEfguRrmpebr/9dkyfPh1vvvkmcnJyUF5ejvLychw9ejR2zKRJk3D99dfHvo8dOxbbt2/H+PHjsWHDBrzyyit4+eWXMWHCBDeLys2FT32NB95fi7/P0uer4alLRwNDgCSXABUlIXh2zo+orqnHi/O3aEISOYQXGTDwxRTCZ2vLmW3WLe0gtR37GLUT9fIAwXbYjdfnRRdtFIQfFmCO1jX6xrH8obSozdn8z9bvWnBVeHn++edRWVmJESNGoF27drG/t99+O3ZMWVkZduzYEfteXFyMTz/9FHPnzsXAgQPxyCOP4KmnngpMmPSB42u2zP9e7zzsltkoyJ1+RlqQSxcslFXPZzaS9WYjwY4jrFLUuWI2En7JlMDYbKTUvOjra9veI3hvxU6XSmUDDs1LvNFIhBplUj+rFbsB54EXfteL62YjK6ZNm6bbNnz4cCxfvtyFEomDNYZwhUr7XeOcSJI+syWL7Iw0D0qTuCjbibLqeToMx4nGbMg3h47p10tyZRkC3+dpCQqHwy5rdj12+jK3SmQLGWLyvKjW36GmZIqyb9FOdsJhGW98ux2DOrVE3w55unPtLQ/gL7S2kcc4VcnbmXDffY6Y1Yd5BC0SXsxRVptyAOcRDAH3Hbx3V+sXuatzI9rI4pKVR+sM85WkMkZCn/J5Bvm58fm8WJiVtNFGfo+aAces3/549S788cN1uOjpBYqtSoddG2ajZPZ5STXcyvNil4LcbCHX4SlrVjo1ITOMNC88guE/v/wBx+rE5nbRwirGU1/+IPw+NYzFA6PsO1SDAX+ehVH/mG94TKpi1Eys1jYKCjKH7uW95TtRuv+I4X6tAER5XsxRTYw0j2p9mXketER6tDTyxIFWG8LlsOuB8JImyC+Cp5MgzYs5yrBl5UyF59nOXr8bL329VbXN79mOUwb8eZahqez3764GAGzda7zmGaFGLbwEt02EOTQv+w7X4szH55hcQz0tDO6vDQZ2J8iJmmHXVZ+XZEeC/ReJ11ygxO4Zonw6WS+B9trpPB5hKYzyeSkfJ69P7AbNTImn+SRSjVRUH8NXGyv8LkZgMda8NH4OtOZF5jNEyDJw6/RlqDyqXwJGBsjnxQZKrS7P03cqGPrtx0bCSxyEJEk1A+KZFTvSvMj8iccAIE2QQMEaYKnjcI7y0fE6bmvrMtlU5lqH4bqGMDLSSCE8//s9qA+HDQeIsMphN8jCC7+28LO15cztlz+/ELPvUS7km1zvgGjiEfRshUqT5iVx0ZmNOM5xMvjYPUPUQo48A2yyDaaiUdZE2KbZCNCHOibb49a21aN1DSkvvNTWh3H9K0sAAEO6shexVbafWhN/oiAQb5NtCMuYeNy0CCTfOyAapSzLp/VSamr48bseUruXiANWvg23ktTJsmzLFOSl2ShBVjvwDVU7cWBbDiWp5mX5jgOoqD6mF14ELz6ZiCiXZ6hmhLIDWrNRcNsET7QRDweONK6jRn2OOcp+mysZpuGXYEOalziQNF4vPDZAp5oXr/0YJMm44edkp8c61WQZTL3AiY1Y63zNet6RulJ/DzqXPbcQALDg3rNU20l4UWM0UKuS1LmQVFAUbvhF+O1rEXTCDE3KzoNH8dL8LahgpEZwuqq03/VAwotDJAQ32khUFlbDkjI0CIdq6rGxrAolnVsKzwKbyKgcdh2MMVrNBMu9IT0kBXr2bYbWp+cICS9cqPO8BLfuw7KYybyyT6H5kjlqzUvk31teW6pz/meRSNFGZDaKAyc+L07yvLAayfl9C21fR8R9AbUvTFTK/8XzC/GLqYvw36X+reoddJzMVDbtrlZfg1Ep6aHkeY2P1rHNJKkEV6buBLGdyJrsuEKuKfRqyYdyghztc8wEF+XznLtpD/d9/K6H5On1PEaS9BFAPO+ok1BplqnALKJIlN7DyCTEcjzdWB4ZZGcsD8B6KgFFxHjDqhOvwtXP6N5G+DW1z+RobXBNIH5gNPAnirlWVClVmaoT5Lf7RVhtB7JE+TxnrmNHfLFPtFEoFyDhRSjWtelkbSNZ1puCzAYsEVYbCRLz10iSpDJ/aE0h1LGoUSepi/96rBl3mmZxTDth9XZwI5uytr0cqSXNC1+EiPPrn1rMjmByA1EOu2SJ5kcZFOJmb+y3zwsJL3HgJIxVlLpXG4UimgWb92LGsp9022VZNs1tQ7KLGnWnG//DYT1frdnIrU7FDV8m7e856vJyCIkAzwQgHs1LpqdLesjC2yP1Meao++fIv25oZ/2uBxJeHCJJ+s7cS4ddU82LoJn3k7O/Z25Xm42M9xHaPC/xX4/HbORWFXhhnUoUXw43UT4BnrWN7CIqiSUPojQvqmv6ba8IGE99+QMe/WR97LtyjDlSW48HP1zrSqCI37VA0UZxoO0CuEKlHTnsyrp7pSlm27nZ6ahS5INwW8WqXvdLNtxHaBdmFKB5YWxL15iN3KoCUckPlWgHYRJe+EJXtQFG2RkhHKvj8xcStfYZDzIEmY0UPWCAI8M9R5bl2CTzmlM7o7hNMzQoHtDTX23G+yvM/RCd1o/fLgKkeXGIBEknvbileWGdoUxC+rdf9Ld9Tadof6P259DQo0apnRMxLrMGd21G2iVb98d/IwZuCC/a9kSKO3AlM9QOHPk5/CvJu21yVhJZVDH+SjXI9ZjyKLuD6LpQynQK32uiFVk4rR+/64GElzjQa16sERUqrfZz0DpsuofW4Vjv89L4nWbRakQ8jRpGKnjPzAAu3Eb7TMjsqB5MjJ6G9jnlNcngvr6nmhc3zEbURmIon0U0waNS88LqL/TXcHpvZ+eJgoSXOHDi8+JIeGF0YcpZsLYvcrNv0pq9jHxeqo/VYejkL3H3WyvcK0wC4EWIp1eh0u5oXsjsWLr/CH5QzJB5noHWdJLbhN8DwHOfF9HXFHy9REbZVo7VR4WXxm019dYO8I6FF2enCYOElzjQJ6lzL1RaO+tV+jl4GUWoNXtpZ4DRTvWT1WWoqK7BByt3eVW0YKLyeXHnFlqfF7dwY8zTDtRO3o9E54y/zcG5/5iPfYdqADiLNsrN5te8eGk2EuVcG80jBZDmRYmyHdQcj9RTLhdRw+kH5QjyeUlMJL3LC5co2iAolbdy9qQPYXWvc9JqjvSamMh3N2bpiYgqz4tLc5U0jzLsulOj5mbIVGJDWWSAVkcbsZ9HPMKLR7IugOPjm3CzkdjrJQvRNAPKPprLbEQ+L6mHzmzEcc4hB0m4mA67JsKBm3KDVnhhOVzKsuz9SpIBJd61jXjISGizkfq7k2i8ZGHf4ajmpXGbcai0+nvzbH6zkZeal+93V2PJNrEO5KnbQvSEVT4vkQ5GLby4aDbyuSIoVDoOlF3Ad9v2c80a91TV2L4PO1Ta2GzkZtdkaTaSZdz46neY9z3/GhnJjMrnxTXNizeDkRtJ6rRmohSWXbDvUC0AZw67dvyevHTYHf/fVcKv6fegGSSUz8Kp5sXxvSnDbmKiXVX65mnfcVXlbsaS5E5IV5mNwPzsBjqzEcNhlwQXNm4NzIns81LfYC4MJzvKCc/+w7XHNzbuN3oe2s12BEsvHXbdwO9BM0go28exqPDCyLBrRqI+TRJe4kLpz8DXUCocaV7020IGwktIklyZIUfRz5TV3yk8Wo2kVr24glerSrthNrLS5CU7yp+/73BU86Lcz+fzYkce8dJs5AYp1kRMUbafGobmhYsENRuR8OIUSdIJDTx16WTtFtZ1VZoXqE1IruZ50cyUrZLWAantx6DErYHZs1BpF3qLO/+jDqVPtaai1LzUH49x5fF50Q5QdrQpXpqN3CDFmog5DLORVptpfQmHDrskvCQuIUn92a1ICVmWddoUo87K7X7JaqbMkvrrUjSf9xfrd+PZOT/Gvrv1rieyz8uO/UdU31NN88L6tayF9XTnxWE2SnDZxf9RM0Cw1pmz+w451Zb7XQskvHDSEJbx0arGnCURDUdjL8CreXECM9pIOWCpPkouRxupBRGWw64WuzOBZOGWfy9VfXdLuNUuD+CEW0ecYHmMF2Neqo1LrN/rutnIoINIFKEm1bRzZqiFl8hnu0vQOH2efqc1IOGFk/pwWKfiVuKmnwl7eQB2tFFktWvXiqJbEE7b8FkvTqoKL1rc6nRFaF4uP6kDzuldYHqMF4NbqvlMKVX20ecrc2hetMKLCFNQopiT/B40g4RK0A3L2HnwKBb8sJfr3Opjdag8Upewz5NCpTnRvthaISEkwTU9mizLOie7kEGSOvejjfQmIHVnS2YjI1zzeRESbSTBSoHjRR+XcmYjxc+NanKV28qr2NGJWiHPjhOuUR8R0cgE//mnmHxrivJ9qQ/LOP2xr7jP7ffQLABA9/zmju7t96tKwgsnWlWrLKu3RcxGLvm8MLYZal6O/+cWrJmxshGz9qfabNoItzRQGQI8aSXJOprIi1pMtaZidwD4btt+vLv0p8aw6uPYmbQY1XNaSALsxxN4TqoJuKZwhNVbsevgUYe39rceSHjhxGpmE3HYdenmrFBpA22LZBJulJOVjuoa+xl+lbAEEeVLw3TYbUh+zcvR2gZIEpCdkWZ4jFudbpoAzYsEDuHFg74qUVXYTmENAGaP4Iqpi5jb7YSxG5kZec2P//3tUJRVHsVdb63kvqdI9lTbTzeRrCi7W6eTI+c+L87OEwX5vNggTZNbRS00SK5VJlPzYjBgGXVi3fObo6hV07jLEvVpUT6LsIXmJdl9Xmrrw+j94EwMfHiWaVi4WxooEaHSkuSuozcvqaalYzvs2n8GdvxVjNoL7yVaNcvA6d3acN9PNC/M3xILK091rCaOdq9hB7/fVFeFl/nz5+Piiy9G+/btIUkSPvjgA9Pj586de7wTVf9t3LjRzWJyw/J7iRIKuadGYy8P0Fh1qjwvElvxcv1pXYQMTtEXJF0lvCheIFa0UZL7vJRVRtSux+rCpv497gkvYl7jIGReNXpEW/cexvZ9h70tjAewfq6TscSOz4uR8GKn/v1uKQeP1vlcgmCgbCp2o4xi16BoIz2HDx/GgAED8Mwzz9g6b9OmTSgrK4v9de/e3aUS2kM5RjBDpT2syzQDs5FZhl23hBfl72aN3XVJrnnhrXeWYCcCEQ67PGYjL+ZarA7xaG0DznpiLoY/PjfpTJBKwT/6+J3MhO3InekGntms+jfOJ+Wv+HLwCAkvgDoBqNP+JVE1L676vJx//vk4//zzbZ+Xn5+PFi1aiC9QnJhqXhR5Xk7v1hrfbN4n7L7sPC+Kcii2G3VifIOTNVHpPtIBRrz7rDQvqWYKANiDcPQ5DOnaCou3iFtpV4TGhCfE3q9oo+hqy0DERCcir01QYD3THyoO2b6OCJ8XpvAiSWhg9EB+K+kOHKm1PijJWfjjXtW78MnqMkfXceyLRz4vegYNGoR27dph5MiRmDNnjumxNTU1qKqqUv25hTY8WZtfJdoGLurfHueeaJ4zww6yrJ/ppGnVQNEySsaxRiL6G0uzUQo67CqrJvoo/m/mJt1x0VlS3/Z5Qu+fIUJ4gWTpN+GF8MJqKso25cb6Sr6iDJU+/tN++/oy25ex0wQyDDR1LHOSkUXSzYhGHrTRVqnGgh/24pqXvjV04LaDY4fduO8cH4ESXtq1a4cXX3wRM2bMwHvvvYeePXti5MiRmD9/vuE5kydPRl5eXuyvqKjItfJpZyxKgUKZI8Ek4McRLMnYaPJp6ngpoONvsHDYZeHUFpsosNaimTrvR91xLGdnEaR5FCrtBcwMzUncfkT5ydkx4xj5SLHaJVuglSD5PHIcSHHhZdEWvkR0buK3z0ugQqV79uyJnj17xr4PHToUpaWleOKJJ3DmmWcyz5k0aRLGjx8f+15VVeWaAKPyM4HeXBOtSzfGADsOu8zzDRx57RIVXpTqSqtGnOyaFyVmg1F0YBa9qq+YJHXWCy8a/bY/XNgbf/lkg5AymJnbzMqQqIjq/+0IxEbthdV3GLVVv8Vcctj1H7/fxEBpXlgMGTIEP/zwg+H+rKws5Obmqv7cQvciG/i8iF5fiOnzYuiwa6zSFTFmOtK8JLnDrpLFW/ahwiIrqug07KJWlXbihDmgqAUGFLUQcn+A3ZaU7cfv3BKiUWuanNejLYddAyk1kRx2U9GPjlATKM0LixUrVqBdu3Z+FwMAw2FX+VmSGmeNot9rWT8rUuWcUWwPmZiNRHQ4Vj4vZuekAjdNW2q4L2r+EK15EeWw68TnRXRTZ5uNkldzJ5t80x1r8p7ZMhsZaF74zUb+O+z6bbLwG7d/fseWTfDTAfPMu35XgavCy6FDh7B58+bY961bt2LlypVo1aoVOnXqhEmTJmHnzp3497//DQCYMmUKunTpgj59+qC2thbTp0/HjBkzMGPGDDeLyY0+SZ3S5wUKzYtYhzaWqlzZAWl9b9x02I0OJBEfichM2Up4SXazEe87HHXYFd3xi0pSZ3UZ1u8MCTJHRmG1JWWofbINWSx/KSPMUg6ISFLH2swStCUT7a5X+D1wJjs8EX1+m3BdFV6WLl2Ks846K/Y96ptyww03YNq0aSgrK8OOHTti+2trazFhwgTs3LkTTZo0QZ8+ffDJJ5/gggsucLOY3Gi1rVqNR6PPi9gXm/WiOlnWXkSxokqUtJCEkCQhLMvM3C5Kktnh0g7RMHLhZiMBocMSrNstK3uw6My8rLak8nlJslFLOQBY/bSaeuOFh+z4bBu1F5bmxTD1gs+al1TsUl6c/yO+2bwPL15f4vvzB/wXIF0VXkaMGGHa2UybNk31feLEiZg4caKbRYoL7aATMtC8AGJf7sYYpkaMZ0/GU2ERQlVU8xKSpFiElVVyJBJeIkQHZuEOu6LMRhbXYWlFIqeI+z1WZqOka0k2flBNvfEswU6kmFFoPesarG0RQZf7dq7g96zfD/76aSTT/Acrdvpckgh+Cy+Bd9gNEtpBR7+2EUvMiB+WAKgcaJT7zRx2hUQbNURNH41CUoOFQ26yzZadEh2ERYdKi9G8WJuNWDKoaOf0sAzM3VSB2et3AwB2Vx1DTZ1CeEmypqR8plYDspnwYmdiYidJnbEJmsxGfnG01v2lvxOhzw68w26QUIdKa7UwjZ9Fz0puOK0LPl6lzp6oEl5U5TC+OU+5bj/rBDw7R5+jJEqj02njb05mh0qRxExughuIUdIxu1jN3llaEVEh+FFq6htw46vfAQBeuXGwqQN0MmDLbFRnPGjZqQMjfwaWUGMkFPnusOvv7X3FzUWAoyiFamUCViV+CzikebGB2YxZ7fMi7p73ju6FSef31m03WlvILM07z2yJdwBrNBtZRxM5Tj+dpFi1j9zsdPRuxx/yL255AItoI4fn2aFWoV14Yd4WfRlkGYdq6oXdz29UDrsWx5q9R3aqwFDzYuCcy76f39JL6vYpXjx6pVBtdDu/a4CEFxuoBnZJ6/MixSpcpEr1jO5tkJke0jXYkErzok6fbnR3Hqc+q04ppnlRCC9WPi3J3s/YFc6shI0OLZviV6d14b6eEJ8XGGdtjsKaaZm1NycoI9NYT/U3ry9D3z99js0O1v8JIsrfaNWMzF4zO7VgNEFhKfAM87xw380dkrxLMcWLZ69UphuNCX736yS82EC/PEDj55qGMLbtPRLb7rZ0bKR5MYxCOv6fFVYmjUafl8bfaKV58buRi4Q1gNv9fVbCi12HSKOkY7aQOLRuDOug6LauCgdmPNclWyMLWk5fvF3cTX1E2Z6sfF7M2pmdOjBqLrwOu3bv5wbJrM2tawjjbzM3YvEWcYv7xoNRd+W30zQJLzYwc9hdVXoQOw+aJ/VxglEnoexU9GYj5x2O1TGxcN9Qo+bFKo9LsnQ0m8qrUfKXLzDtm62q7XZtvzyRIbZWCRbg8xJxvHUSbSR2FFOajczajSg/H79R/URLzYvxAXaUb4YZuC3MRiN6tsXgzi3RpXUz381GSdKlMHlt4TY8N/dHXPXiYvYBHjx7ZVszai9+1wE57NpA2V+aNZ/IitNiGpjRdZRJ6lT2yTjzblh1glETkTKpmaXmxXlxAsV9763G/sO1eOjj9bjx9OLYdru/z2rA5wlbViIsVNpSeNFvO1bXINRMWqMQhM3MkSIirPxm7qYKvPrNtth3q3ZkPljYMRuxt7PqX7ll2q9OgSzLvgsuQPL0KSw2lleb7rermXUCj2Didx0kfg/gIdoBxdhEw8+ah0ahS+umlsdpr5lmaDYyvgbXjN8q14difR5+nxe/m7kYWEnaACc+L+b7WaaYJhlphscLMRvBWnBl/c6jdQ1izUYKzYuZUCxqPSc/ufHV7zDv+z3cx4ty2DV6x9lJ6rTa5mA89yTpUpgcNYkqA4D6hrAH0UbKCTH7GL/rgIQXGyhfZLMQUUkCtwSTk53hqENQzpK0odKm5bJxXRbKUOloua0WXvS7kYvCaCy1+/ssNS+QdMdcdYrxSukiVpWWwCG4MvMNie1ClCZIM3Ok6Fw5QSAeId/O0zB6dKz6F20WtENRqyb4w4X6SEvAf38LN7HK4/LQx+tNc/44ZdY9Z8Y+a0Ol2ZDPS8Kg6zBNnGNFYfSSqld11iSpM7g9T6l4O6uQwmxkleclWboZo5mv6GgjieE82z0/Bx+PG+boejzwmBvZSerEqrBrOc1GPGuvJBrxhUrzV4LRsSwZ2E9FC0uIj5EsnQqDI7XWqQDW7KwUft+WTTMV34Lv85J8PYCLKAeJxVv2q5wLldhN3GU24zLKHaOaJamkZOM783RwvJ1VKuZ5EaV54RE2WHJyfm4W89gMQdoP61WlWZFWslBhXflO1aec5oWvL2DB8zQu6t8O/77pFEOBwE60kVcY3T5Z+hQWRxUZpSe8swrrd1XpD3Lh5ytfKWVfZxxt5C8kvNhA+yJvKGM0KtiPwzdrBEaCgfIeaodd4xLw9Pf8mhdlht3UMBuJ8t2xEiJZJpwQQxsTRcRALsuyZd13aqX3zQrLgjUv9UqzUXL7vGiRYf6uxOvzct/5vXBmj7bINNBaMc1GPo4QZpPAZOlTWBxTmI3eXfYTLnr6a0/ua7TkTFD8nLSQ8GID3kHCbtZRJx2WUah0yFT9b3/Gb0RaSOL2eSmvPIbvd5t70CcCRoKkbbORVduQ9OpySZIM258InxcZ5gPgf387lCnchGXZRbORseYlKYUX2dyTw2wfTx1E6693uxyM7lOo289ql35qXsz8sJJYdtE57LK6HTd8fpRjlsrnRXGMMkWB34EYJLzYgP9Fju+F71WYE/tspNRQFkUbbRSXwy6jsxhQ1IJxrUYhycrn5Zk5mzHqH/NRVik+D46XGAkp9s1G5vslMMxGMBZ6lAP5KV1aGc6szQjLsqlwnhaSmB2mLIv18VI66ZqZI9OS1OfF3GwUn89L9BBJkjB1TAmu1jiBj+4bEWg6tmxi67puIZkEH0Qfhd8DqBsc8WDhRRYqTwSZLb28OGZw4zEelMmM5OsBXIT3Pbbt86JpBq/+6mTVXkA/QIRUUrK1io+3TKzzs9NDeOTSvqptaQrhhVfzsLEssbUvRj/TrubFSZK6kCQZqvCVodLNs9PRtwP/ukgxZPNyNYRlDO7cSrddtO8Br9koIwk1L5DNlwAwXx7AGish84S2zbF40kh8MX54bJvvj9mgTcqQMeGdVRj++FwuB9dE4phFqLRbGCY+VRxzerc2sagkv+VGEl5swPsix/u+pxmo71T3YPvrHo820pcgMpu3P2hG76XdGllVms9sFCXRnewMNS82r8MTbaR30DY+T2s2cvKUZQC5TYxzVtaHw/jZgPbMfSIn58r2nmoOu4C5OcAozxAQeefHDOlsem2rRyZJQGFeNrIVOYXM+ozFk0Zi6nUnmV80DiSY+7y8u+wn7Nh/BDPXlrtWBj+oqQ+A8KLcrmg4ygSafmu9SHixAa963G6WW20bUAofxg67RlJynHleGC2CFbKozCdjFW3EKmciYhxtZFPzwrG2ESshotFAIsLnJT0koVUzdjQTEBFQQyEJXds0U20Py7JrC8WZZ9hV3/Whj9bhiqkLLZeqCDIyZNN3xOx5SAAeubQvhvdoa3wBBxVldkphXja65eeYHBE/Rm1e+c75HRHlBdo+xo2+VPkYjSZqSoHS7+6chBcb8Hre232V9MJL4+doI9LNxFXfFS+ySRl53nGW1iaSkE57/0aH3QbONylpNS92fV6soo0MBBUjJ9WstMaZcn3YfABkMaxbG7RunoVWqjwPamICqqYIsuBoIyVmg7U2Od60hdvw3bYDWPDDXncK4wFW9Xbtv7413Nfoz2J8vhPfpK5tm5nud3X2zdBAxu6rPCwJZJd1uyox/PE5+N/qXcz9nPPDuDALAokiKfp9v6UXEl5swK95EXhTQ7ORcUMzTlLXuGPUiQXMY5hmI8YvTwvZ17x48QK6ibHPi73r8MwUWcKikalEOY7XN4Rt9yl3ndMdANCqOYfwoiEi0LkzevC2KyWJnHk1kufF2bnRt9GsJvTNR72B1SyvGFyEu8/pjrd/M4R5TTedS63MRrHjNAWXZRnb9h7mEqxWlh7E32dt8s3PJMptbyzH9n1HMO7NFcz9Xkz81KnDFH6UimNI85Kg2HHYjes+is+s/rtP+1zDhEJmi0IqyzXlqoF49hq9vZo1PkoSa42Txuvx+rz439zjw1jzYtdsZL6f5Z+k1HTprqfYXtcQtj0CRg9v3cxYeIlGlGlL4KbmxQyjZ57IJgT5+H9OUEYSGR9jZa7U708LSbj7nB44tWtr5jmHXXSWNdJAAurBtaLqGPYfro19f/h/6zHiibmYOm8LAGD1Twfx8+e+wdJt+3XXufTZb/D0V5vx3NwfBZfeHodrzIUnbd/jRk+qDgJp3K6tgpjihXxeEgfesMF415RW3oclTHw8bpjqmIGdWpgez7pu08x0jOydrzvGcLFJE7MR76wgwa1G4hx2Lc1GLOHF+Hi18GJ/+It2Qtkmiz8qVxNX4qbPCy9KDU0iO/LGp3lR/2t2jOF+xgGFudmm53Rr29ziqnFiZDZSPKe/fLIBlz+/MPY9ulL3/83cCAC4+sXFWLHjIH4xdZHhbdbvqoy7qPFgJQhod7shOIRCErrlR+pzZK/82GTmzO6NflSR4I3jDrvCS2AP4/ACQgd3v2iz/9Q2RAnAjad1wQ8V1bEZj/KSUYfPVQ+OQuXROnRo0ZiXQRnCbFUso8gi/Ta9ySLNgcNuopuNjMovfG0jSDrHaVNfBsW++nBYJzpfUdIR7yz7yfB85e9aNOlsDJ38le4Yo7WEZIjJBZIWkmyZiZSPXOmkm9DCC5wPCDx1YLkgqGL3u2OHovJoHdor+hYW+bnZ+PJ3w7G78hiuMfHJAYDmWek4VMOvqTE1G2m+b9172PA6hzlMW2Zh+V5g1Yd45S/45i2n4pM1ZbjspI7ISg+h6mgdpn+7I7ZfGYzi92SUhBcb8KqkJcTXoUsS8NDP+lgel9c0A3lNM3TnGjUqc6ffCKzOPyQxol+UodLcwktiSy+Gsx27Pi8cA6y2/Zi1J2W7rG+Qka5RoPzx4hNNhRfl72qXxx6sRvaKaOmYZiPDK/MTkgA7XgdK/ZJKeJEk7D1Ug9zsDGSmB0exfKimHtv2Hkaf9rmGdSnLzt8RHoddOxU1uIs+p48RJ7RtzrW+1ms3nYzLnzfWfmhhaSCj8Gge7HTBTvyrRKK8f2TyoS6PtnhuJQ/Mz83Gr04vjn3PzkgzESDJbJQw8KYkt9uwtE2AZXTiNllJkmmYmxLehdiMfTAin3lf/Dv+s8J3O2k8GGte7F3HUghmmo3MhJfGz7UNYdszIqvDz+rZFukGmhdRAqldXxW15qXxS1nlMQz+yxcYPWW+kHKJ4uKnF+Cipxfgiw0VpsfFazYyk1Csm53zATErw3ooyUwzNkuykGCsReZ5Tnaepd8h9lZl1faby7YfcLE0aox9XjwrAhMSXmzA84IC5upOFrpGEIdQHZJMYvQZTres81nnmQ2mdmYtSse6RMPod9qdgVj6vIC9PIARWs2LLvTeojxWAog6VFK9T5TDbjzmHuXAM3v9bgDAFhMzgh9EzRofrWKHwkaQHduNou+2WV3YMRvZhWdJCid1bFQmVptlTYyWbNU76bLwW3ixNht5VBAGWqE22tb8noaS8GKDLK0+3gC7nYB28IunEwlJErNRRRyttNv4NS/avkm5MKOd2beTDmzZ9gO4//01OHjEX8HH6Hfa1rxYvHUyGJoXzvw9rFBpbT1rNYhW5Veeru3IZFkWsrZRPJoX5ZICPCY5PzErnSw7V8XHzEY27q2bUTu6cwSeiV2GzWSKSudQLSxZg/V6PvTROq578Zq+3cLq9n5qrLWvVHpIQocWTVS+ln5AwosNsjht6PF25vGcHZIk44bOcWHm0gIMzUvEHh35bOfFd2Jbvvz5hXjz2x14+OP1ts8ViaHLiwdrG5m1KWWd1TIcD7Vnzp94Fv551cDYd61QdsfZ3Qyvry16WJDmxb7A34hyJeqAyy6mv1OGCLORs3vz7DeDR/PiRLC0pXmxffVGvHLY/e/SUvxy6iKdBtpKaPVTttLWW0FuNr6572zMmTDCnwIdh4QXG5iFkiqRbNqNzJYHsI1k3AHyDZqMS0p6jYkyqsmOQMKbjZfF9xXVWPDDXlsRCzys+akSq0oPWqqOxa0qzVEPDt/M+nDYUphq36IJLhnYoXGD5vDfjeqJlQ+e21gWk+L65/Ni4LAbdOnFBFl27gLJ47Crfcb6qnP+7Ix8oszuz4NRX8iaMLHaPe8tvTIbTXx3NZZs249/zP5etT1scftED3ZwAxJebMCveYkP1vlWL2Gr4zH5o04sYHaAHVs25SoXa3bEdNgNNWazsSO8vL5oOy577hsccOD7snZnFa57+Vvc8MoS2+caX7MSFz+zAJc8+w1++YJ5JIRxnhexmhfW844yYVQP09wb/AkDG2GVv4ViqQCz8kZCpW3fUoddmUNZ4rp6RZ4Xlf9PYq1zFNG8OBZfjv8/uMKb3ZJJkvGvaWCM9vEM7163Fe0ELCih0ixcCmyKGxJebMCreYGJrZbrdAenzr7nTEy/+VT8bEB71eqzr910Cv540Yk47YTWXNc1yv2iFWokCY40L09/tRnLdxzEs3M2c5+jRaSn/eIt+2KfV+w4aHqs8cKM9u5ppR2I+Lyw9407uzteu+kUw3OdRBvlZmeY7teubaJElmVheV5sofR5MdC81NQHT3ix9nlxRvRn26kKoygSt7B7fTMhniWkxzO+ByrPC+Mnv2uS6sBtgioQU54XG7jl88ITKm1F6+ZZGNY9S3e9M7q1ia00y3NdViSMBEm3PWI2imyrt9J5MgjiwGKGLMsqx1Aldu3RVtFGAENdztkkWIKkUfH+9ov+2LLnMEo6t7Qoi/E+UXle7KcXYJuNQhrhxWShbGEcq2uwYVI212I5z/NyXPMSn8VZKNqcU3b7NeUESQu7nTs3Gznpw+JBWyyrPuRvMze5VhYrSPOSBHCHSpu8dCzMVpU228ZzPeV5PH4UhmsbhbTHKTPs8pctSqL5JizYbLxasV1VP09d8gg4LBrCMv56WT/ddtbj/uXgItx3fi9LwUGledHsC8uyP2YjVZ6Xxgao1Dq6vdhefUMY17y0GP0e+hxLt+3n8sUy17zEESptsi8zLYQrSjpaCljxLrL4wpgSlbN362aZuGVYcWMZHbQTQ7MRM1Ta/vWjODG3pgpB7apdFV7mz5+Piy++GO3bt4ckSfjggw8sz5k3bx5KSkqQnZ2Nrl27YurUqW4W0RbZvKHStq8s9sVRrQiq6jE4SsY4RJYNfF6Obwo7cIXnTfgXFL7aaJxczLbmhcdhVxdtZM6QrpGMqBcPaI+Szi3x9cSzVPvjyTZrNuiIioKw7bCr+KwUXpQmJLe1ezsPHsXCH/ehrkHGL6YuQt8/fY6KqmPmJ1n8TDuPUxXCfvwzaxD+51UD8fgVA0yLMrJXPnq3y7Vxdz3n9SnE70b1RFGrSAjt2b3yccNpXRxfz67mJR6/EL/zvASZoJqNXBVeDh8+jAEDBuCZZ57hOn7r1q244IILcMYZZ2DFihW4//77ceedd2LGjBluFpMbXs0LIF4Fawfe5QGYxzBKHpZl/fIAUuNvdJIjIY0z58OsdeW2r+0G5jl+bCap41oewNYl8cJ1g/H3KwZg8nGtS7ri+cqybLg2EQ9mSerCgcjz0vhFOXjX1LureWE1+8+PJ8lzip3xN02lEYt8Xr5D7w9mFOGn3PryjScL04a+O/Y0/OXSvvjTxX1UZjxHmhcbPi8D/zwbW/YcMr3e2p2VzHxRnvu8eDhAxFutQTUbuerzcv755+P888/nPn7q1Kno1KkTpkyZAgDo3bs3li5diieeeAKXX365S6XkJ8fCsTGKbfs9h9konuvFrstxbk62vkmEGZqXNEXuF5bnvxU8a6EAwG9eX2b72nbgrSszzYXw5QFgPydGXtMMXF7SMfa9MDcbQ7q2QmZ6GppnpSMrPYRqe8WMwZrhxxCU58VuaLiRz4vSd+FYnbuzaZa5MByW8fzcH9EtvznOPbHA5vXsRa6FIum0ATTWAaster1uT0FuNq4b0hkAUHm0zvF1lKZpLazfVNsQ1vmGaK9w0dMLkJOdjjUPnafa7rXPi5eEJOMlY3hwax2leAmUw+6iRYswatQo1bbzzjsPL7/8Murq6pCRoRceampqUFNTE/teVVXlWvlKOpk7NkbR1vWAohbYuucQqo7x5Sdhrm1kQ1Q3aqhmg2Z6SMKvTu+CPu31qmOm5kVhNnIyaUk0nxczZ21X8rzoIkHsOjtK+M+vh8TOe/rqk3DDq0vwx4tOtHWdSFnMnUxF1GR8axuFFZ8VmheXfV5YfLN5L2Yd175se+xC3X6z91iGfmkHM1h+UUyBytfsrErNi768TTPTDH1tJINzAGNhI51Do1vN6If9jjZyk3hlj6D21IFy2C0vL0dBgXq2UlBQgPr6euzdy3aYnDx5MvLy8mJ/RUVFrpUvFJLw3LUnWR6nrewPbz8dZxyP+GGhVemzGtv5fQsBAMVtmlne33htI+NzxgztjAcuPJHZWYRlWddRRmZFzjUviebzYiq8CF7bCHCW0EuLsi6HntAa6/58HsYcnxHbuo7qs7pcYUa4UbYN82oUuw7KyieuNCHUe+DzMu2brbjx1SVMzc62feZrKllFbtlpScpXyOy6Ru4cXryBKq2dg5ONugkjWSPR+hUv4Jn4/u+OYYb7gvpIAyW8AOw8EqztUSZNmoTKysrYX2lpqbvl4z1Oc6Dypfrt8K4AgEd/3hdAxEvf6h7jR/XAP68aiHfGDrW8t1EHaB7pYL5P+3tCUuMFnaileX1e4mHBD3uxsVyMJk6k2UiyeusYZjoROPV7sVyYUdOyuPMhKe8RR7iRUnis9yDa6KGP12Pupj1467sdun2Ha8zvafUO2olcUz4zs/YSlARnzChKk3NDpg67RpqXwA1pTEyX+xB9L44LNs00fmeDajYKVE0XFhaivFztoFlRUYH09HS0bt2aeU5WVhZyc3NVf27C0w2wBnOlqeCyQR2x8ZHRuPbUyCx4QFELfDyuUfJlNZas9DRcMrAD2jTnSFxh6LBrorI26eDYDruNsyInwguvz4tTtu49jOte/hajp3xt67zP15Xjyw16p0uztVvshkrzaV6M9ylP/9PF9s1AdlFWlbZYrFDpJk6EF7uyi/KzynlXETbt8ph9iGF+OFxrbho21bzYNRsxnGFZpxtFA7ohIJvdQ4KEs3oaa6BZ59pZmBEIpubFa58jLVypGUyeW0Bll2AJL0OHDsXs2bNV22bNmoXBgwcz/V38gGcW08CIwFAOWGkh89lpvG3FKOTRvOM0pr7BwOfleEn3VNewTjPFyu9j5tpyWwsxvrO0FEMnf4n1uyKalq17zaMOjPjt68tw82tLdQnpzDQv8fi8DOiYpz9A4g+VjgrAbmIu9OrL5kx4ce7zojIhqQYKdwcN1tWPWGheTK8nxx9txDrfKNrojrO7oTA3G3eN7G6rnHZQmRwl4JUbT1bvN6l3CcZ9llE6/zU7KzXX5ymlNZsrDuHDlTttT1TW76rCiQ/OjCujeLzwvFumwovIwgjEVeHl0KFDWLlyJVauXAkgEgq9cuVK7NgRUbdOmjQJ119/fez4sWPHYvv27Rg/fjw2bNiAV155BS+//DImTJjgZjFtwdN2WZK20pGM1ZjUuVmclS1KYV42vvzdcHz3wDmq7VeURPyBBnVqob+/ye9qCMv6PC8Kle6qnyoZZ5lj5Vg3dvoyvPLNVu7r/f7d1SirPIb73lsNIP7cBFoh1Ux42W2V20OD8lH+6vRi5jG8g7kXsyLlLX4xWO1TFmYsD+DEbGQ/Qk9hNlJUlXJQc9tawhrIlHlmhj8+x/YaXrajjY5jmovHYOafn5uNRZPOxj3n9uC+p12U7ThifpZU2hGzWo/keTHSvLB/07pd9szEGYp+yCxf1TlPzsNdb63E5yapG15ZsBUXPf21qs4f+ngdaurDePxzTRSUhxIBz63SzTThAVW9uCq8LF26FIMGDcKgQYMAAOPHj8egQYPw4IMPAgDKyspiggwAFBcX49NPP8XcuXMxcOBAPPLII3jqqacCESYdhddspK1vpWRrpXUQYWM8oW1ztM1Rm5hObJ+L7x44B+/8Vu83Y9Zp1oXDOrV+miLayAluRxvxls3oMDu/bfJnG/kPBodgJXP4xcSu5T7KAejaUzrhXYXfFavVOHHYjctspAqblpnH+MH2fUfwrwVbVNvMo43i0bwYo1xkU4vb/gwq4eV4jfD+RGOjUXyr0ytRagmPcvhIrSw1nqg9/L/1WLuzSqVlCcKwz1PHiah5cTVUesSIEaZqtmnTpum2DR8+HMuXL3exVPHBozZkmZaUkq0XtmYjtAJNFLOfxTIbSYo8L05wyzYdjQpSlu3SZ7/Ba786BXlN+U2P2uchchbPZYPWmo0MzvHCmU5ZVaGQhMFdWsW+R5LUqXGiebErzKrMRkrNi4PIt9r6MNbsPIgBHVvYcvjkaRLaZGqm1SUg2kjZP4Uk4JpTO+GCfu1sXFUwcTRPSTI204tK55+ZngYg4qd0uLYezbLiHxKVQpCT11N0lBxPGcz6Yz/HKzMC5fOSCPCZjfQNxkrz4mNAQOT+JvvqGsLMJHXxYKqmjINoJlxl8VaWHsRz89g2Z97HLrJ6WLNRs2PM8KJbYQlIA4taAAAu7t9eiMNuPEKYUdg07zs18d1VuPz5RTrVvuV9Oa6vPcTaYdeZ2QgMn5eLB7THXy7t52tOJeWto2VTmfxMz5UMzUPiIqgar2MVKQYAixSr0MeD6Br55eCOhvt47mUW/RlAH2gAJLzYhsth1yLaKIgJ2qx8XpjLA8Qx4LglzGcyNC8AcMzmonN6zYs48cXyp0v8z8eLSRFLkHr1xpPxt1/0x18v6ycmVDoOs5GysuoUmhde/5EPVu4CALwwf4vFkSZlMEDvR2Hu/GynlbGijZQOy35PiID4+wij5HG8S5JY3V35jA5zLKy5qvQgfthtnqtaWTLluyGyD1HSsWUTnN6tjeF+njQEZpPRgCpeSHixC0/7iwg4Gk0FZ04G/zD+YUyH3VB8LrFudaxRs1G8Pi92E8/ZQVk2o+egjzZq/K7MyeCF2ah1c73PRMtmmfjl4CI0z0pnJKnzItqIPXuvq/du8OYzIdu4Hpz7vDQu1dF4AT/zu0RR1qqs+dfyXMnYDCgq/Fj5jKqO8S1l8P1uZ5GMAx+ebX2QAyTJ/P3h0ryY+rwEcbwK2PIAiQCvw64WZdtINM1LXTisW3smXgHMrW6V5fMiAuXz6VHQPK5rWQocsrkmomPLppgwqgf3WltO+edVAzFr/W7cPKzY9Djtz/Eiw64SI58Xu23MbhG4NC82BAhZdm42in5S/X7/ZRfVM41qoXjLZWY2MgqVtovy8vsO2YsMM0L5+5S/P551nswISebBEzx9oakPYvCGKwAkvNiG12ykbS9K6ZXVUfcszAEA5Bs41LqNpcOu1uclFJ8A4pYKNebzEud1tMVT1rvaWdSd32El4I47273cHFEuGdgBlwzsYHmcCIdd24KDQR3Es0aN7TbD4/OiaR+SFNk24Z3VzMvZMhsxMh8rB+NgaF7iMRtJhvUpSvOirJ+9h+znqzK4auyTV2ZdU80LT4BAAjrskvBiF453prhNMyzZtl+1TVn/LF/V7Iw0bHh4NNfCYm5gGirNSlIXIM2LsgPKOj7rt51uXntN3T3Y++IdHwxXAA9oh8FCW1YnDru2o40MWpA6z4u9yrGda4ajFWvHWAnA4i37MWP5T8zjv7fwp1Ci1rzoyx4E4cUKZR31aZ+rytMiwVjDwh0qbVGnyss4FV7qG8K4+qXFjs4VgSRZCd7W7doqWWAQIeHFJmYd1vPXnoQGWcaAohb471L1GkvKBmDUUTcxWV/CbcwddsM6YUCSpPi0DgL7VWVoYaPZiO9co3dW+9uMihvvz+A9P6qZCyLaR+ikHYvKsFsXx4zcjU5aK0BIEnDEYAmBFTsOYtybK7ivrYzqZj0+n7PSA1And2zVzDjfDMDw85KMHXN5NS9Wdaqsn73V9s1GtfVh/G3mRny37UBsm8psZFACkXMTCWrho21OFvYfro09o3i9FII6jyLhxSZm4/WAohZo36KJ5TWCqIYz6wrqG/QOu/GGSot0iFUuwNfYWarLZyh8GOzQa15ky89O4Dl/7PATDPPzBBGzFbiNiMffRFSG3XhMV0boNS/i3n2rdzAIipe0kIR5vx+B+rDMzKGiLKJ2kA1JkqHmhTfayArlVQ4csS+8/G3mRvxrwVYhZXGKpPF5CUnqfiXe4aa4TbP4LuASJLzYxOyVMVV9K1pQIjrsajvKeH+CyI5VmRQqWk7R8qFs8NmL2W3HltYCsZ/oHXY9TlKn2G6nPg4eqVXldrErWHDleZFllO4/Yvs8HpSzbVZ7d8sfyy6dW/MNflrThQSgWz5b4yjqpyk1L078aFhLmBg57LKoqW+I+ek5JaSJNpIgqd6DeCfLgzq1xN+vGIAubZrGdR3RUKi0TcxeGrM2ojIbBVHzYvLD6htkfbRRSIqrAxHZrarDQ49fn+FrAES0NOPfXonP1pRFthuajbQb2J/dDKlm3TqIaAd9s3WgjLBtNlI8FaO2a1U3f/lkA974tnF5ErtKEZ66l2Xg/H/aW9mcF3Wel8T0eVGilV8lCTi9W2s8ccUA1+4Zr4Mzq63ztAsJEqZ9sxU9/zATs9frV7G3Q2T1bcW1GRqseLm8pCNKOreyPtBDSHixiVkD5030E68zqRuYmo3C+mijuB12BfarMrMDYt/glW+24r0VO3HrGxZLUMjarzJzV9wOu1wHBXsQ0jYFJ2Yju69EeeUxS82C1WP7cY86X4dbPi+HFMnPRM5b1HleWPcWdy+RtM/Ljn1W1pHe50WCJEn4RUlHFOQ6M5taPm9V32H/mvFoTR76eD0A4K63+P2cjFBOLrXP0a8gELch4cUmZu07ZKLGDWqin5G98gEAN5zWxfQ4nc9LnC3HLY2FkeYlSkUVX0SBtnxGobkBlys8Z9qvTkamg8ZhVxj+96LtuP/9tQBM/JYs6kZ7R7uCxefrrGfMrGgjUU1GOWAlUrTRtJtOYW7XJ2Z0j4c+WgdZllXPSJjmRWU2sv4V8dZTRPNifJ+MeDvrgEI+L3YxaWhKjUq7vGD7KUR56frBqDxah5ZWkQCsaKM4umG3NC9RwUL0rNPoemQ2Ug/6LZtmorb+mO1rZDjQ1vxnyQ5MvqyfsDpwY4Khz/Mi0GGXsTyA+t7CbiWUHgVsPxbRiTDNmLZwG87qlR+/8OJQMFAl74uznkIh9fW0jy1ZhZfk/FUuYtbQlOP7zcOKcUVJR7x0/WAAwQ03C4UkS8GFRZoUHJ8X5eAV7YB4nRWNqqX6mDqc1S2zEV+iszjv4TLaQd+JmjojDlMqb8SYFp2DqAvvKGtAFOVIq3bS1OOFYO2URy7pAwB4+upBsW3aduR2n1l9rE71hJw47LJMpMqrcP2EOKtJgqRqy1qhz4kPWiKQnL/KRcw6HuVMKDsjDY9fMQDnnlgAILiJfpxid1bUp30uLjupA/p2yI1siLMD/7+ZG7H/cCS0Ue10d/zynNcxOu6Mv81RHyezP3uhmg+q+j+Ktik4WTE8PY7ZodHTMXtXP1y5E2t2Vqq2uePzot8mzGykmm4z7i0mg74rjBnaBd//5Xycc7x/BPSaF7eFFwmS5l22fw1rs5H1NeI3G6knztp7ZpLPCwHw+7xoCarmhYeJo3vqttmdKPcqzMWTvxyITq0i4XbxduDPz/0R986IpFiXGapfkQO+LMuoqGo0hciQUdcQxn0zVuO95TvjuzbjSZj52wQRbVMwXSfFgHhU23Y1L8u278ddb61Ebb16dHfDTOGm4JnGyLB7Uf92ntxbBNqBn+WwG/vsULQ0O0tb3doVwLfsOYTxb6/E5grjrMdW7dbo/mqzUbyqF7XPi/Y5JqvZiHxebGJuNjITXhJTehl/bg/cNqKbbrvdUOloPxt9yUT0q8u3H4hcS7HNItjIEQ98sBZvKkJqZRlYvGUf3vqu1OQsPrhyhcR9F29xkscoI47ZoV3zyI8Vh5nbq2vq8cnqMpzft1BYRKAuZF9gN8AKEHjiigH43+oy5r2DjraPNAuA4GX5joPcx2qFiOtfWYKfDhzF3O/3GJ7DaifK9sjnsMtdRHYZJI3Pi2Z/sgovyfmrXMRMFR3ACOi4KczNZm4P2XTYjb1cx/+NPsctew7hlQVbUVPfwD6RA5YZh7dkZlUWLaNScInez6mjHou3fjNE2LX8QNtBOzEBOTE1RTF8JY22m1T67W8ux8erdzkuixbd8gCIz1dMSYgxYCkTBAZd86JFl+fF4/tr14D86cBRAIiZp1lotTV+oFtVWms2Ypi2fj7IesHVoEPCi00uVKhltQQxc65b2B2fohqX6BOSEREOznlyHh7+33pM+2Ybvt2yDxc9/TWW7zhgszTumI1qDVKTi0QGMKRra9U2rYo8KJlSjRBhNnIjF4WRcG11p+80i6rGA2ttI1GoU8InTqi0EfGGrseLk/esjtVHKH1e4igPLyEJpmYj1kTr0Z/3db1cbkPCi03a5TXBhFE9mPtMV+ZMILnmb7/o3/hFUe4Xx5Q0brYZbaSdWIdl4Iqpi2Iq02XbD+DKFxdj7c4q/HLqIq5rRtc3YTnd8ZTtgffX4PuKQ4b7tT4RStwcFhLO50XRRmQ4E+LjEfwNM+waPDcrVb5INbubE3Mrs0rAm40OszwvbnSf2mvaWezxX19vwfDH52Dnce2MElW0kWHBxf0iCRJTCxdFa5Jt0zwLTTMT32Mk8X+BD+RkZ9g+J6hJ6lj8cnARJr4bcYZVlrp188aQavtLHBzXvBw/b8e+w1i6vVHDonymvIuuVR6tw8byKlWnJ9swG72hMQdpMRJeZFk2XBnYDYIc8grohQEng78bodJGWN1JpElQl+clslXItbXr2WgJgEXDFlr/Ea8XsOV9XjIiS0sEBsl8nSurUOlEmlgrIc2LA5z0s4naQLIUNnRlZxLxeeEnemr0Cg2aTt1pLoLJn25Ua17CkQHjh93GEQK8GJmNqo/V46ZpS+O+PsDpsJtgg5CRFuWCfoUm57gQKm2w3epWIvNiuClAmPk5AME3N2rRr23kbacp6nmpr+P+b9CGSvNGG73561PRo6A53vntUDeL5xokvDjAyUvV0yCrZFC5c2R3nN6tNUb3aRxwlIOS3bEmFm10/F9tp151tM5JMSFJatt+WJbxzFebhcyMjDQv1TXeaF26Hl+KflQf40E/aMiybOjz8ty1JbGkjVqcKjvmbKwwKQt7u5UW1E3Ny78WbMW6XVVCrm1lNko0n5eBRS1V39VZY8ULAdpLOklSx+KDlbssHXlFR52ZXc9IeDnthDaYdc9wDO4SrAUXeSGzkQOcNLwRPdvi/y7vh97tcsUXyAXGn6v369FqXuyoXnQOu5qO9cARY49+8+vqfV7+Pvt74+Nt1J2Zz4sooiahmXefgdFT1KsPz7z7TFQerUPbHGeL0vmFmfOtkdbSqeblV9O+w93ndGfuc2puc7JUgRGsMezprzYLubZFjrpAJ6lTsvC+s1FedQyVR9QTGK+V1bzCHk+5Zq3fjdF9Cz3RuEs6s5H6pskaSELCiwOc2GIlScKVJ3dyoTTeodK8OAyVjr5Y2o71wBFnmhdAk7pf4GyzxgPhpVXTiB9Rr0K9UJuZHko4wQUwD3s2enXiyfNiV8i0en2dREsZ4ab2I2QyYAFA86zE6N7bt2iC9i2aYO4mtRbNfZ8X9fXt+LxYUXWsjnEHd4gszKj8rt5/6Jh3/nleQmYjBySpIGuJslNPC9nzxYh2RNEraDv1g041L5qoJ5FDhZuh0lOuHIjrh3bGeQlkEuJBkiTTmZ6RySae2eGxOiPHavbxj3++yfR6YrMzC7uUDiPNy7PXnIS+HXLVUYMJgD7Drrv302XY5awsnsO8HCIkSTI1Ieq12ollTjQiMUTzgJGo2XLjRRkN4PgZHD9N67B74EgtJCn+zl7kwOOm2ejSQR1wqUGiqARzVVAhyzJTi6JNUqglHm3HMYMEh9HH+PqibXh2zo+YfsupCEmNyceM4I1242FPdY2wa2kxyu1xYf92pvmogorZ8gBewOvzwtPHxCZrHvwEbYZd7XM86NCfMOiQ5sUB2vZYkJtlaHdPJpTh0XZDpRujjY4Hi2re/2N1YUeDNsvnRRRe+LywaJYg6n4WHVo2YWpR0jSaN93+OKKNagw0L28s3g5ZlvHHD9ehvOoY/vjBWhyusc7kXK9NtRoHmwREvRlhthhfIqKPNmJ/FoW2v+Htf56f+6PlMXE0Z9tIMF9hvDpJzUaJ20v6iFayfWHMYAwsauFPYTxE7/PCT8xh9/glRPqmKH1eEkXzYkbLpvbzCPnNF+OH41BNPfJzsnG0Vi8gRN8ZIz+GeDLsGmleNpZXq9alaQjLXH5aIjUvbuJ1HhS30a9t5O79tH1QVPOysbzKcFkUAFhfZh0t1mgmZ/8IkT9NbzZSX/3PP+uDa15anDDtmhfSvDhAt3S7P8XwnJAmVNqOANK4MGMEUZPbSKh043eRQpEXywOwaNE00/qggNEtv3lMgGcJIo0O2+zztWajV25kh1SzqKkz1qZs3q3OoMzTPOp9qne7KPuhZJBjtL/B7cSez8xRR32FZRlrd1Zi9JSvMWTyl0Lu4ZXZKE3lj6i+6SnFrbD2z+e5XxCPIeHFAdqXKhk6Dh6UpiLWrM9spqQdvMRpSCSVwCIyPNQvzUuLBNS8KGH5r1jNRLUd7qnFrZnHsTBy2AWARz9V5/vhaXeJMkNVhccmwRTKzGHXjT5Wm28nLMuYd1xTd6wuHNc9vdSKRZzkG7+zTPrKBTuTBRJeHOD1DCEoKGd6LL8Gs8ykknbwEjg+KC8l1Gzk0wy8ZQJqXpSwHC2tTADaRFp2Ov9jJpoXLTytQ6TPi5skvc+L6rP7P1Ars8ZzRytNo9gkdZocXCkyqnvyM5977jkUFxcjOzsbJSUl+Prrrw2PnTt3LiRJ0v1t3LjRi6JykWy2Zl7SVDM9/UBglplU+zKLymapjVASOWn2K716hxZNfLmvm0RNjkY+J1ph2M4rZuTzokWGzGc2SpDsbsoBPRl6JK3Q63e0UTz9vJWmUSSRhRmNzUZaEjmaUYnrDrtvv/027r77bjz33HM4/fTT8cILL+D888/H+vXr0amTcdK2TZs2ITe3MXFX27Zt3S4qNzrNSzL0HBxoXwrtS5CVkQYYeLZrHXbFJu9yJ0md19aDRy7ti41lVTi7V763N/YAq4FAq+q2JbyYmI30WFdqXYOMW6cvQ9PMdPz9lwNsXNtb1JqXxO+EtGOu9wszqttGPOZDSffBPUIhfTBFKuC68PLkk0/i5ptvxi233AIAmDJlCj7//HM8//zzmDx5suF5+fn5aNGihdvFc0SqNA4tylkE67XOMjUbNV4FECcY6EOlRQov3kovY4Z09vR+XmJlNkrTOPm6YTaSIHHNOkv3H8GCzXsBAI/+vG9g/QUkjSY00dFrXry9v8jXvdFMbrBfYI1JkqQKpkjW5QC0uGo2qq2txbJlyzBq1CjV9lGjRmHhwoWm5w4aNAjt2rXDyJEjMWfOHMPjampqUFVVpfpzG7N8BEmN5nfqzEYmwot+YUaRodKNiNSWJIjfZkJgJYxkaAz1dl4pq6RzUZZs24+tew9bHqc0H9QFIPLoSQPtj9sOrV5j6vPiwe8TZcoGvM3CLsE6mCIZcVV42bt3LxoaGlBQUKDaXlBQgPLycuY57dq1w4svvogZM2bgvffeQ8+ePTFy5EjMnz+fefzkyZORl5cX+ysqKhL+O7TEk1ArkVFmTmWtVJqVbjxD1S7MKCqiQ+/zIjK1O0kvTvn9eT1xXp/G997KrKGdLbrVAf/+3dW2jq8LgPPuZSd1RKdWTU2PSQ6zkb+aF5F9R2OGXQ98XiRJ5aSrfI7xrBkWdDxJUqetQFmWDSu1Z8+e6NmzZ+z70KFDUVpaiieeeAJnnnmm7vhJkyZh/Pjxse9VVVWuCzDaPBapEm3UNDMd947uhfqGMFo1y9TpWc2jjdT/zlckD4sXZacjcqZstaw9YcztZ3UDAHS57xMAgIkvNwDGO+XjK9XgUnuKB5ajc5OAmrOcoq1zs6yxblBTH8acjRXWB3Jg1X5FRxupNS/A1OtK8McP1+LpqweJu1HAcFV4adOmDdLS0nRaloqKCp02xowhQ4Zg+vTpzH1ZWVnIyvJ25V2dijs1ZBcAwK0jTjDcZ+7z4o73vdaPQWSYK8ku4ogOREaT265tmqm++6lJUAosfuX60cIKgOrQsgnGn9sj4fMCRWmaqR6O/GgDS7cfEHId7UK0bhLSZdgFRvctxHl9CpJCI2eEq/aPzMxMlJSUYPbs2arts2fPxmmnncZ9nRUrVqBdu+AsNBakWaKf2AqVjv4r+FlJknpWamWOsiM8ee2wm8yYmYH+fsUA5DXRD8CXDmzvZpEMUQkvPmtexp/bAwDbhNkQlnHnyO64fmgXj0vlDrnZWuFF+TnBOlmLPC9CbyWps583bmffPFl6NdfNRuPHj8eYMWMwePBgDB06FC+++CJ27NiBsWPHAoiYfXbu3Il///vfACLRSF26dEGfPn1QW1uL6dOnY8aMGZgxY4bbReUmme2IdtD2p2br0zQuzCgWSZNwRmSODpJdxGHWibfLy2Z2tI9d3h8frNzlYqnY1NU3VrxfmpcT2+XimWsGofi4RorVFEU6mAaBXI0Am8i9rJXPlgS9v55TJEgqn7FUcWNwXXi58sorsW/fPjz88MMoKytD37598emnn6Jz50hYaFlZGXbs2BE7vra2FhMmTMDOnTvRpEkT9OnTB5988gkuuOACt4vKTbouMiI1GosWrR3e7IVtzPMi/lkpSyHWbJRcg4OfmHbmUrAGKqXmxU+fl65tm8c+s9pisgkv8WRZDhratdzYx0gq/6p47sVaEiDZ8cRh97bbbsNtt93G3Ddt2jTV94kTJ2LixIkelMo5ZDayj1uhgzqfF4EdepKNDb7wwAW98einG/C3X/Q3PEaCxHyH/HqvagPo88Ia45JduE7kdCU8gldIAvgXtjC/l3I+zbNyejKQmjG/caKdISTwO+YZVmt9xIPK50VktJEsY84mvuiD9nnZePOWU5n7mmd5MkcIJL8+syt+ePR8DOkaWWiR1a1KElsj55dGUykTHKkVMbzED+u5JcoCkk7p37FF7HOi9rFmmmZRWmiJU/NyQtuICXLUifzBMkGGhBcHaFfNTVXNi3bip3Uq7NO+cXkHt6KNIKk1JCLzcoTDMn716nd8xZAknNatDXNft/zmzO2pAisnkBLDLKQ+vVdKc8z1ryzBmp8qPS+DthWzHHaTOZQ/LSRhZG/FMhkJ1sfKMvDO0lIs3b6fuV+SJNuaJaPjJU20kRFv/WYo/u/yfnjw4hPt3TigkPDiAKvOmAA+uP10/PGixpfEVc2LomMX6Qdgx98hVQVYERjNQP16pFpzzOTPNnheBq2wwmrWyah5eeCC3ijMzcZXvxueeBFGCpZtP4Dfv7sapfuNsz/bnchpHZpj1zGINtLSNicLV57cSReSnqjQKOwAvfCSuC9ZPGgng8rOZmBRC9Vz0mbYFYUm2EhotFGtDS2OqT9qajYPboyej1+DVxBlApZ/SzJqXn59ZlcsmnQ2OrduZn2wINzwrfnHF98Lv+8/rhwo5DrJAgkvDiCHXT7Uq96q/xVFfYOsyoop0mxkx1kzkSMj/MbQbORpKRoJgiOs3iSrPyYZNS+Ae0LrXy7ty9zu17tr9749CnLwwpgSy+ukSvQrCS8OiGcRuWTCqutUL9Me+Vd0xzRzXTn+vWh77LtIs1FtA7+zZqq2AREETe4za0NerXeljRhhhkoHQMjyChFN5PKTOjK385hc3MBuuw9Jen/LyPaAvUAeQcKLA8ySsaUy2o5dvTaJN+myReblUCYrs8JMKEuhMcYSdqZY9rF+9clm5hjf6pJx34YALBqZSBi1J7/MLnaFpjRJ8k3QCiIkvDhAbzZKzQZlNQtVp/eOfnCvPIBYVfqeQzXcx6ZmCxCDkXnOP58XE+HFw3JY3TeVNC8iMNJQ+GVmsXtXSZKYIdHaTZTnhTCEzEZ8qBcLcylUWoNIs9FXNlaYTVH5VQh2zHNeEARXEq1cohSorh/aGc0y0/DrM7p6XCr/ECHIGocax31pR9g196SF1MsARMnJTo6FOe1CwosDtKo7GrjYsJa0T9ZnZWo28rAciUhQsthGMdNoeOfzokYpvDx8SV+s+tMotG/RxJOyBAHl23XLsGJH1wiab4hdgSwkqf0I7z6nO047oTV+c2bqCLFKSHghXEMZUe7Wwox2cUulavS7ljwwkpxeLKgJmPCiFVCUY4xXNak1D+gXQU2trntQpxaxz2kOfQ6DJLtIkn1fm5BG83LdkM5489dDUjaDd2q9AS6RKqFpWnThnJr9UgppXoxmdfk52R6XJNiwBv/AaV4CYDfS+tWluvz7x4tOxF0ju+OL8WdaHstapsNoCQrAvwmVXU1QSJNJ1zjFQJJ2shpIeBFAqjhIabH63coXLWpqS9YXK1mFMi8YWNTC7yKoMJNdvBIitJqVVO1jouRkZ+Cec3ugW36Oo/OtBIWp153k6LrxYFfzkqZZUiBVA0WipKa+STDH6oI1c/QKq45c9aK5WxRuklV4SkSWPDASFVU16F7gbEDyA6+EiIwQaV6McPIOm50hSRJG920HSfLuOUdWUrepeQlpJoQp3pWR8CKAo3XBipYICqrZTjTayOcXrqbenboym9kVt2mGVT4s7hdEBh3XsuRkpyM/JzvhzGpeDW7aqJIgZP1NZMzeT2UWB6+e8ivfbLV9jt5slNrSCwkvAuiQQl7/dgixMuz6VBYAOHikFm98u8OVa5sJZX+6uA8+WLnLlfsmGi2aZmLFH89Fk8w0v4sSaLTrpwXADScwOJkA8ZwTkqRAC4lpkqT+HQa/KVVMjCS8OOSdsUMxe/1uXHZSB7TNyfK7OL5g9YqozUYuLW5kgy828OdtsYvZz2rZLNNTlXTQadks0+8iBB5WPg8igpMnY+rzEmfX1LVNM2zZe9jZyTaIRCiR2SgKCS8OOblLK5zcpZXfxfAVq5wX6iR1x/91s0A+ErQcEoRYDtXUoyEsI9OjEOUMWoJEKDwDfcQHxf4MoyA3Gzv2H3F9oUxJkhAKqb+nMiS8EI5pbpHZUfluhfxXvLgqOKV2N5LcyDLQ90+fAwDm/X6EJ/dMD1EgqBHOzEZ8Pi9O8LKqKFS6EXpDCMc8+csBpvvTGM5lSftipfgsKJlRJtEb/vhcT+6pzfOS1yQ1U8B7iVPtaUiSPPMyCakmhKnd55DwQjjmhLbNVd+1ViTlyxV1IkvW9y3V7c/JzLLtBzy/Z7qmQb1+8ykY0DEPb/1miOdlCRpuhEpH/nVWHi/9kySGKT5VIeGFiItpvzoZOdnpeP5afZIn1szAr/fN7ZlRivcjhGC0Ser6d2yBD8cNw5CurX0qUXCwilSz+65HuymnmgzWSs8iuXhAe9xxdjcApG1RQj4vRFyM6JmPVQ+OQigk4a3vSlX7JEX/G9XK+PXuybK790515zlCLFrNC9HI9UM7Y9b63VhVelDodZ0+cbff/aevHtR4L1fvlFiQ5oWIG+0q21GUM5JwTHjx5/VzO/cBjTWESLR5XohGcrIz8OHtp+OSge35T7KOlI7DbOTsPB5+dXoX1ffm2Y36BiMtDOV5IYg4Yfm8+IXbOVaS1hGZ8AWtwy6hp75B/1LPuHUoahwu18Izseqe3xw/VBxSbXPLlLPmoVG6FaPbNM/CX3/eD1npIWSms6WmopZNXSlP0CDxnhDGhFE9AQA3nV4MgD2T8c1s5PYNaKwhBEJmI2vqw2ohpXWzTJR0Zufe4nmaPI/8ypOLGOe5U1c52RlMgeqaUzvh8pKOuu3Tbz4VV51chLvP7eFKeYIGaV4IYfTrmIeNj4xGdkbEoU6leYmajXwa5d32eaGxhnDKP64cgHveXqXapnXYJfSwNC+AE4ddSfUvz7HqbTZv6BLDurfBsO5t/C6GZ9AbQgglKrgA6hDCaDZe/150l7NfWghlY4Z0BgCM6NnW1XIQicfZPQt024Z1S51ByCl1NjLa8iSp48rCy9hGEUD+QJoXwjVYnYFfr3nYmRmcG6v+64ELe+Osnvk4pTi1l5Qg9Ch9F87ulY9bzijGaSeQ8GJFfYP6pTZdvohrH4/mRb+NtK7+QMIL4RqSymE3us2fssiQXTVZWf2urPQ0nNUr37X7E4mLUnhp1SyTBBdORK8lRJqXxILMRoQnBMHnxU2oAyOcojSv1jW4rCJMIrSal3j1ujyvMNvnRbJcpJYQDwkvhCf47fNCXQuRCBg5oRJ6xGleIp0SzwTEK7NRr8Ic8RdNMjwRXp577jkUFxcjOzsbJSUl+Prrr02PnzdvHkpKSpCdnY2uXbti6tSpXhSTcBG/u2TX87yQ5oUQQC1pXripsyHoma9tZH2M2XVEa12funoQ/nfHMKHXTEZcF17efvtt3H333XjggQewYsUKnHHGGTj//POxY8cO5vFbt27FBRdcgDPOOAMrVqzA/fffjzvvvBMzZsxwu6iEi8gByLBLodKEEaed0BpnB8AnicxG/DSE+R12zWjMsMtlN9JtCgkeRXOy0ylUngPXn9CTTz6Jm2++Gbfccgt69+6NKVOmoKioCM8//zzz+KlTp6JTp06YMmUKevfujVtuuQU33XQTnnjiCbeLSnhAkkZKx35XBmVGjRs/5Nt+HfLw+/N6en9jDSS88GNkYuuW31y3jS+Hi/U9mRGUkiR0UkY9CB+uCi+1tbVYtmwZRo0apdo+atQoLFy4kHnOokWLdMefd955WLp0Kerq6nTH19TUoKqqSvVHBA+/o43CLtuNop1XGqlgEpLM9FAgko3V1fttYE0c6rSal+P/FuRm44ahnZn7zODyeWFcSfQrTyZoPlwVXvbu3YuGhgYUFKiTMBUUFKC8vJx5Tnl5OfP4+vp67N27V3f85MmTkZeXF/srKtKnbyb8J+aw69f9Xb5+tAPLEK1DTkH8aCMZaaFArE+lHZAJY8ycm3u1y+W+TsznxZnVSLjPi/+tMDHwpKfVSpKyLJtnPGQcz9oOAJMmTUJlZWXsr7S0VECJCdH47vPi0eJGtKBeYpKRFhDNC5mNuDmnt3qSq6w/7fvOZxLi0bw4O88OlHaBD1eT1LVp0wZpaWk6LUtFRYVOuxKlsLCQeXx6ejpat26tOz4rKwtZWVniCk24SrKGSkd/VxppXuLGDwE3Iy0IehcKlbbD/Rf0Rt8Oubh3xhrdPjur2EdrnivayIPFZkl24cPVnjYzMxMlJSWYPXu2avvs2bNx2mmnMc8ZOnSo7vhZs2Zh8ODByMjIcK2shLtEOxPfzEYW2r54iZmNSPMSN348wayA+LxQqDQ/TTLTcOXJneK+ji2zEdPnhcxGfuD6NHH8+PH417/+hVdeeQUbNmzAPffcgx07dmDs2LEAImaf66+/Pnb82LFjsX37dowfPx4bNmzAK6+8gpdffhkTJkxwu6iEi4R99th1XfMCctgVhR+6h4y0EIIwbJDmxTlKwcKJmZgvVFq/Sfgr738zTAhcX9voyiuvxL59+/Dwww+jrKwMffv2xaefforOnSPe4GVlZaqcL8XFxfj0009xzz334Nlnn0X79u3x1FNP4fLLL3e7qIQHJG2odEzzQmajeGkQvGYND+TzklzoW1D8q0p/NO50bCqv1m0Xr3kJQENMADxZmPG2227Dbbfdxtw3bdo03bbhw4dj+fLlLpeK8JJGh11/7h+WZcZaKOKIdmDppHlxzIntcrG+zJ9UBxnpIV+HjGtP7YQ3vt2B343yP9dMohJv32IlNORkZ3iytlEQhOhEgFaVJjyh0efFnzfzs7Xl+GwtOzxfCMd/FmXGdM4LY0pw6xvLcPOwYs/vnZkW8jW/xl8u7YtxZ3dDu7wmvpUhqdAIE2ZVG613q+qXYBRtZK9oVlC0ER8kvBCe4LfmxW2aZqQBIM1LPBS1aor/3XGGL/fu0qapL0s8RH2kJEkiwUUgWj0IXySR+VGSxF4KQLjZiLoQLmiaSHhKsr6XzbIi8wDK85J49CrMQa/CXF+0guTgLY54n6RVVUjH/7N7nl2oRfBBwgvhCY2JBn0uiEs0y4poXijDbuJxUf92APxpm6Spcwc7Lii8odKSZJTnhTQvfkA9LeEqxW2aAQBG9y0EkDie9Jk2fVeaZpLmJVHx2tdl1ImNCTpJeBGHsh6NHGinXDnQ8Hyn5h/xzYfaBA8kvBCuMvPuM7B40kh0y8+JbEiU99JmOZsfNxud16cwtu2fVw0UWKDUxgvzilcyzGUndYh9ptD6+DmrZ1sAwI2ndYlt0/m8HK/bSwd1wC9KOjL3WVV/RPPiQZK6ROkjfYYcdglXyUpPQ2FeWux7sr6XTTMjv/G6IZ3RLi8bgzq1RNscWrYiXt6/7TSs21WFuZsq8MWGClfv5ZUGpl6Rx4Z8XuJn6pgSbCyrRr8OebFtZmYjoydu7bDL1huTz4s/kNhPeEoiL/f+u3N7GO6Lal7SQhJG9SkkwUUQgzq1xHVDOqu2Lbj3LFfu5VXLVCbhI7NR/GSlp2FAUQuEFM9SH23UuE/bBY3sVcDcrkUyOCaR+7REhoQXgjDgthEnqL5fplE3K2maRUpMN1HOpDu2bOrKPbwag1TCC5mNPEFZt0pB5pKB7XHf+b0AWJt/JMmjtY1IGOKC3hzCUxLptZw4uhfO6N4m9t2s7M2z0kz2EomAV87kpHlxH7OMt0rZ4KqTOyH7eI4mS58XSEwBl6rQH0h4ITwl0SYVylmV2QwrGm1EJB5eh/GrNS8J9kIkKMqnrKznkOozj+ZFT0iSfFlMNNUh4YXwlEQTXlTqZpOyNyezkatkZbjfVXnm8yIrHXapC/Yepf+LxNpseBbb50VMqTiLQRyH3hzCUxIlz0sU5WzMrOTRaCPCHbLTPXi+fmheyObgClqr0b7DtbHPRhMSy6rwKFSa4IOEF8JTEu09lwy/qGlGmhdXycpwX3jx0uclMz3S9Z7erY3F0YQTZI0hp6a+cUV5lQlJ9dnCbGRwhPBQ6QTrI/2CelyCMMEoSkFLVjrNA9zEi+frpc/Ll+OHY86mCvxycJE3N01x7j6ne+yzWvOi8GmzaGKGSepIe+YLJLwQnpIoYYDRUirLa1b0RPldiUoTD8xyXqn/T2jbHEWtmuL6oV08uV8qojQbLfvDOWjVLDP23Sjni7Xmha18pXffH0h4ITzF69c8Jzsd1cfqHZ+vVTHP//1ZOPPxOXGXi7CHFz4vbrbNJQ+MRNnBY1i3qwojjqezJ9zjpM4tY59bN1cnjDTy0bVemJFCpYMECS+Ep3g9Sbn8pI6YtnCb7fNYoY+SJCEnm14ZP8j2ItrIxbaZn5ON/JxsDChq4d5NiBgnd2mF128+BV1aN9PtU1azyiHfKlQa7DYiPEldggU1+AUZ6glPSfc4NDTeaA7tLI00xP6Q7dBh187imDRoJBdndG+Lolb6bMxGpmCrrsI4w65YrR31MXyQ8EJ4Sm4TbzUXGQ4dPaP9h9J2Lkls34sL+7VzdA+CH6eaF1v+CDRopBwq/xeeY2lto8BAwgvhKXlNMjy9X4ZAg7QECVnpafjsrjNi2zq2bIJnrhkk7B4EG6eaFzvQGJQaGOd5sc5SZ5Rhl/AeEl4IT8nNNhdeLh7QXuj94l34TtUvHf/cu11ubFNudgbNvDzAC18jqsXUwDDaiMdsxExSx/aRI9yFhBfCU/KamgsvrP7jN2d2dXy/DIGr9pKM4h9ndm+LM7q3wa/PKHbtHiSEph5qQcZZqDRpXvyBQicIT2lusYAhqx/o3zHP8f0y4lz4Ttm5USflH+lpIbx+86mu3oNqNzUw0rZY+rxIErMPEL62ETVELkjzQniKMhvl1acUYWSvfPz5Z31i25hJoOIYVpxGGzEXYGMc17m1PpqBCA52ap8GjdTAKLcLh8uLo1DpoV1b4z+/HsJdPoIPEl4Iz4mu6/KbM0/AyzeejB4FObF9StVtj4LmmHLlQMvrPXJJH0gSUJCbpdvnNNqIhbKPemfsUFw2qAMeubSvsOsT/hCNKKNQ6dTAyElXu5gj6zwnZqP0NAlDT2hto4QED2Q2Ijznu/vPwb7DNShuE0kglakQMJT9wKx7hgMAPl1TZnq9KwYX4cqTO+HRT9bjtUXbVfsyHOaViQ1oBunDT+7SCid3aeXo2kQwIc1LaqDK82LnPINwI8qw6w8kvBCek9c0Q+W4q1x0jzX75ekbMtNDTIe79Dh9XlTloE6KIBIeI7ORVcSQUZI6cvT2BzIbEb6jTEDmpB+Iqm1Z5zoNlY5ei/ql1IEcslMElZ+LvTpn+sKJdtgl8yUXJLwQvpOZ1piA7PRuEduwMkqIJ/8CwB584l0egHUfInHgqTMSVFMLo6y6bvm82IXaIR8kvBC+k6XQvJzQtjnmTBiBpX84V3GE+dsc07ww9qUJzrBLJC/x1O5bvzGOJskS6DROxI/kUPMiQTJMUkd4D71VhO8oO/ewDBS3aaZaRoAnhBFQh2FHiXthRoNsnERwmHpdCa45tVPc13Hqu5CRJhnmIrrp9GJ8cueweIpFCIaRNPs4jaqXb+47W3+eZBwqbaW1IcTjqvBy4MABjBkzBnl5ecjLy8OYMWNw8OBB03NuvPFGSJKk+hsyhGLkk5ms9EazUV1D2PRYlibFTO0vVvNCBJHRfQvx15/3c3x+Y6i0c1hauWaZaXjw4hPRLT+HcQbhFzzrGbHWRDPKsEtJ6vzBVeHlmmuuwcqVKzFz5kzMnDkTK1euxJgxYyzPGz16NMrKymJ/n376qZvFJHxGGSpdW68XXpTvchPGAn3RGTOrI3K6PEBsMIrDuY9ILJTVO7CohWrf787twX1uFJGCMyEOI22qUnvC0uJGJtP668Xj89I8Sx3wO6RrK/QgYZcL10KlN2zYgJkzZ2Lx4sU49dRIWu+XXnoJQ4cOxaZNm9CzZ0/Dc7OyslBYWOhW0YiAoezkaxmaF6XQ0CQzDYdq6pnX0XYhmWkhoQv60VCUeNjxU1Ll/9Cc1q5FE9v3FrmuFiEOHlkjjbUMAIwWZnTeM7wwpgTX/uvb2Pe3fjPU8bVSDdferkWLFiEvLy8muADAkCFDkJeXh4ULF5qeO3fuXOTn56NHjx749a9/jYqKCsNja2pqUFVVpfojEpc6huZFiTKsWouyE/nzz/pg9UOjkO40SR0j6wMpXlIXMyWKLBuF6VODCSLKWlFqWJRvvFFINDvaKL7ytM3RZwYnrHFNeCkvL0d+fr5ue35+PsrLyw3PO//88/HGG2/gq6++wt///nd89913OPvss1FTU8M8fvLkyTGfmry8PBQVFQn7DYT3NGUs3KjsG8wy5io7kaaZacjOSIND2YUJmY1SF6vZNUvD41RwJlzGIMOurLAbsd71qA+m/duZn8Na1oSwxvbb9dBDD+kcarV/S5cuBcCuNFmWTSvzyiuvxIUXXoi+ffvi4osvxmeffYbvv/8en3zyCfP4SZMmobKyMvZXWlpq9ycRAWDyZf0wZkjnWJ4XJcrmYupHoFL7G/vBEIQV2lbDm2tISbwrmhPuwJNh16i+WZsrj9bFVZZLB3YAAOSTBsYWth0Cxo0bh6uuusr0mC5dumD16tXYvXu3bt+ePXtQUFDAfb927dqhc+fO+OGHH5j7s7KykJVFlZ7oXH2Kcagrr/Ci3BX97FSlG5tJUwhk0vGbM7vispM6YPSUrw2P0U6wrJxvWXudZncm3MVovTLVMRznRtl3uDau8vzq9GK0zcmitdJsYlt4adOmDdq0aWN53NChQ1FZWYklS5bglFNOAQB8++23qKysxGmnncZ9v3379qG0tBTt2rWzW1QiCTFzglRqWRqXDIhv9ru54lBc5xP+YqQR6VWYa+s6lmYjl7M7E+JQCizKKmrTvHESbNRvsISdYd2sx0Mz0kISLjmufSH4cW1q0Lt3b4wePRq//vWvsXjxYixevBi//vWvcdFFF6kijXr16oX3338fAHDo0CFMmDABixYtwrZt2zB37lxcfPHFaNOmDX7+85+7VVQi4Cg7DDNVvLIjiuV+ifPeDZR9KpA0y4yEzDfN1IfOW6EVRFg1rBU8TB12wW5nFG0UTFTVr/jcsyCHtdnw3Nn3nIlv7jsbXdo0E1o+gg9X36433ngD/fr1w6hRozBq1Cj0798fr7/+uuqYTZs2obKyEgCQlpaGNWvW4JJLLkGPHj1www03oEePHli0aBFycij2PWVRdBhmqnjJBZ+X/7u8H/p3zMO7YymEMUj8d+xQDO/RFv/9rf164dHGnd1LHWxgrXnRb6Noo2Cill0av53Zo23jdo6qa5qVjg4OQugJMbiW5wUAWrVqhenTp5seo/TwbtKkCT7//HM3i0QkIKpoI5MBQW3LjuBUeImeVtK5FT4aR+ndg0af9nl47aZTHJ2rc8ZlHJOeFsKdZ3fDU19tBmDejozyf5hFxhHBQFltPQtz8PrNp6Bl00xjXxil/52IYACSbx1DbxeRUKSZhkqzfF7Yx/ZuZ8/ngUgeeAVapTnJiRxCmpdgoooq0uw7o3tb9O2QZxJtxPaXIbyHhBci8ChntWZOkKrkU9FoI4PjrdS95OqSeEy/+VT0KGiOE9o2+iCwap93why2yPsRxaipiFgskhDPwSONoc25igVgeVAKsbzBACTjuIOrZiOCEIHy5TcTXkIqn5foNqOjSTpJNoZ1b4NZ9wzH7/67Cj/uOWx4HM+MWYI6yZydAei5a09C17bNVA6gRHCoqD4W+2zkVE2al+BDmhcioTCL4FD5vFCSOsIA3hnzTacXo0dBc4w/t4etkPsWTTLQqzCXMjIHlD3V7GztSox8XpTLhij7luuGkJbNa0h4IQKPcgww8yOQbPi8tLcwG9G4k7xo69ZIB5fXNAOz7hmOO0d2t6V5sWuKILyFS3gxqHDVytOKgx65pC9WPTgKnVo1jbd4BCckvBCBRzkLMlsvRpXnJbaN3Qv9bpTxquZEYqOcHbOqn0cbpz3E7JRoxOTtZ52AK0o6ok97cgYPMleeHFn/ThkarcWoupXCi6Txf8lrSkKrl5DPCxF4lAOHeZI6heYlpN8W5dYRJyCPZscpixNfBSMzgpLfn9fLQWkIr7nljK4YUNQCAzq2MDzGyORnZDZyCk+7ItiQ8EIkFNkZxhlV1ZoXSbetcR+RyjgZMMiMmDykhSQM6apfAFaJkYCrNhsJLBRhGzIbEYFH2Ue0bJppcqA+2og1gyIn3tTGSfVTi0kteJytefsR6m7cgYQXIvgoXv5WzY2FF/Wq0iaaF+pMUhqegUl3hMkpJAynDiqfF0a1y5SCwTNIeCECj7LDaN3MTHhh5XnR9zA01CQ5qvGDpXmzf0mWqem5a09C62aZeP3mU+1fkEhIRPu8EM4hnxci8FQfa8yIaWY2UmfYNcnzQp1OSuNk0GGdckG/dji/byHlc0khjEKlnUJNxzmkeSECT9XR+tjn7Ay+tY0afV70x1F/kdqI9HkhwSV1EeGw27dDXvwXSVFIeCECT5VC85JmtrYRM9pIf3xdQxgA8LMB7Y2vZbeQRCDp11E/OEQFjl6FkfT9F/ZvZ3kdElIIQLOoY5xtYt2fz0PzLDJ+OIWEFyLw/HxQB7RsmoHLTupgKryoV5VW/6vkaF0DAOBvv+gvtJxEMFAOMB1aNMGXvxuu2h9tEx/fMQxL/3AOTmjb3PKaJLukHq/cOFi3TY5jxVat1rgZCS5xQcILEXhaN8/Cdw+cgyd/OdA0wy7v2kZHayPCi1nOmF7tKEtqsnBC2+b43x3DYt+jWrmMtBDaNM/iugbJLqnH2b0KVCuUA86Wc422nVl3Dzc9jrAHiX5EQpB+fEHGrm2bIT8nCxWM9UlYmhfWjDmqeQGAy0/qiBnLf8JTVw9CmiShZdMMfLx6F+4c2V3sDyB8RelbwLWqtKbhkOaFAGCZmdtMMdOpdVNcfUoR/rOkVHCpUhMSXoiEIiMthG/uOxv/99lG/GvBVtU+9QKOoePbjDUvAPDEFf1x7+ieyM/Njm07rVsbwaUmvKSFxRozzgQRkl5SkWN1YdX3E9o2xx8u7M2tsQPUfnpxWJ0IDSS8EAlHRloIIcb0WSmotDIJqVZqXiRJUgkuROJz98ge2Lr3MH4+qANzvxNHS9K8pCZVR+t02245oyvXuX+86ES8NH8L/njRibFtJLyIg4QXImmorW+cJbVoZjz7VmpeiOQjr2kGpv3qFMP9jlaVjrdQREJSXVNvfZABNw8rxk2nd1EJy5SBVxzksEskJCyv/0rFLCnHxJP/CAkvKY0TQYRCpQknaNsNaV7EQcILkZCwOoHKI7Wxz2aDzbE6El5SkcGdWwIAzu6Vb3nsyV1aqb53aNHElTIRqQXJLuIgsxGRkFx2Ukf8a8FW9FNEkRxk2KdZFLdpZn0QkXT897dDUdsQNg2R/+6Bc7C76hh6a0Ll2+ZkYcatQ/Hnj9dj9U+VbheVCBhZ6WLm+aR5EQdpXoiE5MT2uVjywEi8d9tpsW2/HFwEADjthNaG5w0saoHJl/VzvXxE8AiFJFPBBYgIKUYp20s6t0LvQsr/k0pcebxP+f15PbmOtxJOyOdFHKR5IRKW/Bx1lFDfDnlY8sBItG5mHMb42OX9KLqIcMy5Jxbg7aWlyM2mrjMV+Otl/XDzGcXonm+dhZmHK0qK8N7ynejPWLaCsAe9gURSoRVotJDaloiHkb3z8d/fDkU3QYMZEWzSQhJ6FOQIu97QE1rj64lnoYAmUHFDwguRUpDwQsSDJEk4pbiV9YEEYUBRq6Z+FyEpIJ8XIqUwW9iRIAiCSAxIeCFSih4FpO4nCIJIdEh4IVKGv13en5KNEQThGjnkyO0ZJLwQKcMJ+ZTfhSAI93j66kE4sV0uXhxT4ndRkh5JZuVZT2CqqqqQl5eHyspK5OZSTgYCWFl6ENv2HsalBgv1EQRBEP5jZ/x2VfPy6KOP4rTTTkPTpk3RokULrnNkWcZDDz2E9u3bo0mTJhgxYgTWrVvnZjGJJGdgUQsSXAiCIJIIV4WX2tpaXHHFFbj11lu5z/nb3/6GJ598Es888wy+++47FBYW4txzz0V1dbWLJSUIgiAIIlFwVXj585//jHvuuQf9+vGlY5dlGVOmTMEDDzyAyy67DH379sVrr72GI0eO4M0333SzqARBEARBJAiBctjdunUrysvLMWrUqNi2rKwsDB8+HAsXLmSeU1NTg6qqKtUfQRAEQRDJS6CEl/LycgBAQUGBantBQUFsn5bJkycjLy8v9ldUVOR6OQmCIAiC8A/bwstDDz0ESZJM/5YuXRpXobS5OGRZNszPMWnSJFRWVsb+SktL47o3QRAEQRDBxnZGnXHjxuGqq64yPaZLly6OClNYWAggooFp165dbHtFRYVOGxMlKysLWVnGqwgTBEEQBJFc2BZe2rRpgzZt2rhRFhQXF6OwsBCzZ8/GoEGDAEQilubNm4f/+7//c+WeBEEQBEEkFq76vOzYsQMrV67Ejh070NDQgJUrV2LlypU4dOhQ7JhevXrh/fffBxAxF919993461//ivfffx9r167FjTfeiKZNm+Kaa65xs6gEQRAEQSQIri7E8OCDD+K1116LfY9qU+bMmYMRI0YAADZt2oTKysrYMRMnTsTRo0dx22234cCBAzj11FMxa9Ys5OTkuFlUgiAIgiASBFoegCAIgiAI3wnM8gAEQRAEQRCiIeGFIAiCIIiEgoQXgiAIgiASClcddv0g6sJDywQQBEEQROIQHbd5XHGTTniJrj5NywQQBEEQROJRXV2NvLw802OSLtooHA5j165dyMnJMVxSwClVVVUoKipCaWkpRTK5CD1n76Bn7Q30nL2BnrM3uPWcZVlGdXU12rdvj1DI3Ksl6TQvoVAIHTt2dPUeubm59GJ4AD1n76Bn7Q30nL2BnrM3uPGcrTQuUchhlyAIgiCIhIKEF4IgCIIgEgoSXmyQlZWFP/3pT7SKtcvQc/YOetbeQM/ZG+g5e0MQnnPSOewSBEEQBJHckOaFIAiCIIiEgoQXgiAIgiASChJeCIIgCIJIKEh4IQiCIAgioSDhhZPnnnsOxcXFyM7ORklJCb7++mu/i5RQTJ48GSeffDJycnKQn5+PSy+9FJs2bVIdI8syHnroIbRv3x5NmjTBiBEjsG7dOtUxNTU1uOOOO9CmTRs0a9YMP/vZz/DTTz95+VMSismTJ0OSJNx9992xbfScxbFz505cd911aN26NZo2bYqBAwdi2bJlsf30rOOnvr4ef/jDH1BcXIwmTZqga9euePjhhxEOh2PH0HO2z/z583HxxRejffv2kCQJH3zwgWq/qGd64MABjBkzBnl5ecjLy8OYMWNw8ODB+H+ATFjy1ltvyRkZGfJLL70kr1+/Xr7rrrvkZs2aydu3b/e7aAnDeeedJ7/66qvy2rVr5ZUrV8oXXnih3KlTJ/nQoUOxYx577DE5JydHnjFjhrxmzRr5yiuvlNu1aydXVVXFjhk7dqzcoUMHefbs2fLy5cvls846Sx4wYIBcX1/vx88KNEuWLJG7dOki9+/fX77rrrti2+k5i2H//v1y586d5RtvvFH+9ttv5a1bt8pffPGFvHnz5tgx9Kzj5y9/+YvcunVr+X//+5+8detW+Z133pGbN28uT5kyJXYMPWf7fPrpp/IDDzwgz5gxQwYgv//++6r9op7p6NGj5b59+8oLFy6UFy5cKPft21e+6KKL4i4/CS8cnHLKKfLYsWNV23r16iXfd999PpUo8amoqJAByPPmzZNlWZbD4bBcWFgoP/bYY7Fjjh07Jufl5clTp06VZVmWDx48KGdkZMhvvfVW7JidO3fKoVBInjlzprc/IOBUV1fL3bt3l2fPni0PHz48JrzQcxbHvffeKw8bNsxwPz1rMVx44YXyTTfdpNp22WWXydddd50sy/ScRaAVXkQ90/Xr18sA5MWLF8eOWbRokQxA3rhxY1xlJrORBbW1tVi2bBlGjRql2j5q1CgsXLjQp1IlPpWVlQCAVq1aAQC2bt2K8vJy1XPOysrC8OHDY8952bJlqKurUx3Tvn179O3bl+pCw+23344LL7wQ55xzjmo7PWdxfPTRRxg8eDCuuOIK5OfnY9CgQXjppZdi++lZi2HYsGH48ssv8f333wMAVq1ahQULFuCCCy4AQM/ZDUQ900WLFiEvLw+nnnpq7JghQ4YgLy8v7ueedAszimbv3r1oaGhAQUGBantBQQHKy8t9KlViI8syxo8fj2HDhqFv374AEHuWrOe8ffv22DGZmZlo2bKl7hiqi0beeustLF++HN99951uHz1ncWzZsgXPP/88xo8fj/vvvx9LlizBnXfeiaysLFx//fX0rAVx7733orKyEr169UJaWhoaGhrw6KOP4uqrrwZAbdoNRD3T8vJy5Ofn666fn58f93Mn4YUTSZJU32VZ1m0j+Bg3bhxWr16NBQsW6PY5ec5UF42UlpbirrvuwqxZs5CdnW14HD3n+AmHwxg8eDD++te/AgAGDRqEdevW4fnnn8f1118fO46edXy8/fbbmD59Ot5880306dMHK1euxN1334327dvjhhtuiB1Hz1k8Ip4p63gRz53MRha0adMGaWlpOimxoqJCJ5US1txxxx346KOPMGfOHHTs2DG2vbCwEABMn3NhYSFqa2tx4MABw2NSnWXLlqGiogIlJSVIT09Heno65s2bh6eeegrp6emx50TPOX7atWuHE088UbWtd+/e2LFjBwBq06L4/e9/j/vuuw9XXXUV+vXrhzFjxuCee+7B5MmTAdBzdgNRz7SwsBC7d+/WXX/Pnj1xP3cSXizIzMxESUkJZs+erdo+e/ZsnHbaaT6VKvGQZRnjxo3De++9h6+++grFxcWq/cXFxSgsLFQ959raWsybNy/2nEtKSpCRkaE6pqysDGvXrqW6OM7IkSOxZs0arFy5MvY3ePBgXHvttVi5ciW6du1Kz1kQp59+ui7c//vvv0fnzp0BUJsWxZEjRxAKqYeqtLS0WKg0PWfxiHqmQ4cORWVlJZYsWRI75ttvv0VlZWX8zz0ud98UIRoq/fLLL8vr16+X7777brlZs2bytm3b/C5awnDrrbfKeXl58ty5c+WysrLY35EjR2LHPPbYY3JeXp783nvvyWvWrJGvvvpqZmhex44d5S+++EJevny5fPbZZ6d0uCMPymgjWabnLIolS5bI6enp8qOPPir/8MMP8htvvCE3bdpUnj59euwYetbxc8MNN8gdOnSIhUq/9957cps2beSJEyfGjqHnbJ/q6mp5xYoV8ooVK2QA8pNPPimvWLEilgJE1DMdPXq03L9/f3nRokXyokWL5H79+lGotJc8++yzcufOneXMzEz5pJNOioX4EnwAYP69+uqrsWPC4bD8pz/9SS4sLJSzsrLkM888U16zZo3qOkePHpXHjRsnt2rVSm7SpIl80UUXyTt27PD41yQWWuGFnrM4Pv74Y7lv375yVlaW3KtXL/nFF19U7adnHT9VVVXyXXfdJXfq1EnOzs6Wu3btKj/wwANyTU1N7Bh6zvaZM2cOs0++4YYbZFkW90z37dsnX3vttXJOTo6ck5MjX3vttfKBAwfiLr8ky7Icn+6GIAiCIAjCO8jnhSAIgiCIhIKEF4IgCIIgEgoSXgiCIAiCSChIeCEIgiAIIqEg4YUgCIIgiISChBeCIAiCIBIKEl4IgiAIgkgoSHghCIIgCCKhIOGFIAiCIIiEgoQXgiAIgiASChJeCIIgCIJIKEh4IQiCIAgiofh/z0jLNNXbIJgAAAAASUVORK5CYII=\n",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"filenames": {
|
|
"image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/chapter13_3_0.png"
|
|
}
|
|
},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Metal device set to: Apple M1\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Model: \"sequential\"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Layer (type) Output Shape Param # \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" simple_rnn (SimpleRNN) (None, 32) 1184 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense (Dense) (None, 8) 264 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense_1 (Dense) (None, 1) 9 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Total params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Trainable params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Non-trainable params: 0\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 1/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2023-10-25 15:32:11.077734: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 3s - loss: 1.4222 - 3s/epoch - 66ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 2/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.5274 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 3/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4426 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 4/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4375 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 5/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4336 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 6/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4310 - 461ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 7/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4287 - 454ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 8/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4277 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 9/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4266 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 10/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4253 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 11/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4236 - 461ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 12/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4221 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 13/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4208 - 461ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 14/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4199 - 462ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 15/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4203 - 468ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 16/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4181 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 17/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4177 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 18/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4162 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 19/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4148 - 463ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 20/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4146 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 21/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4137 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 22/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4128 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 23/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4130 - 462ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 24/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4106 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 25/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4099 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 26/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4100 - 463ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 27/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4086 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 28/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4088 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 29/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4078 - 462ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 30/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4063 - 465ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 31/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4054 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 32/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4054 - 464ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 33/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4048 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 34/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4051 - 454ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 35/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4027 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 36/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4008 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 37/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4011 - 463ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 38/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4024 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 39/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4014 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 40/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4001 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 41/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3992 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 42/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3990 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 43/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3990 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 44/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3975 - 453ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 45/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3972 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 46/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3948 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 47/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3954 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 48/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3931 - 462ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 49/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3946 - 461ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 50/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3940 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 51/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3935 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 52/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3932 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 53/100\n"
|
|
]
|
|
},
|
|
{
|
|
"ename": "KeyboardInterrupt",
|
|
"evalue": "",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
|
|
"Input \u001b[0;32mIn [1]\u001b[0m, in \u001b[0;36m<cell line: 58>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 55\u001b[0m model\u001b[38;5;241m.\u001b[39mcompile(loss\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmean_squared_error\u001b[39m\u001b[38;5;124m'\u001b[39m, optimizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrmsprop\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 56\u001b[0m model\u001b[38;5;241m.\u001b[39msummary()\n\u001b[0;32m---> 58\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrainX\u001b[49m\u001b[43m,\u001b[49m\u001b[43mtrainY\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m16\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 59\u001b[0m trainPredict \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(trainX)\n\u001b[1;32m 60\u001b[0m testPredict\u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(testX)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/utils/traceback_utils.py:64\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 62\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 63\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 64\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 65\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e: \u001b[38;5;66;03m# pylint: disable=broad-except\u001b[39;00m\n\u001b[1;32m 66\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/engine/training.py:1384\u001b[0m, in \u001b[0;36mModel.fit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 1377\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m tf\u001b[38;5;241m.\u001b[39mprofiler\u001b[38;5;241m.\u001b[39mexperimental\u001b[38;5;241m.\u001b[39mTrace(\n\u001b[1;32m 1378\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtrain\u001b[39m\u001b[38;5;124m'\u001b[39m,\n\u001b[1;32m 1379\u001b[0m epoch_num\u001b[38;5;241m=\u001b[39mepoch,\n\u001b[1;32m 1380\u001b[0m step_num\u001b[38;5;241m=\u001b[39mstep,\n\u001b[1;32m 1381\u001b[0m batch_size\u001b[38;5;241m=\u001b[39mbatch_size,\n\u001b[1;32m 1382\u001b[0m _r\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m):\n\u001b[1;32m 1383\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_begin(step)\n\u001b[0;32m-> 1384\u001b[0m tmp_logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1385\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m data_handler\u001b[38;5;241m.\u001b[39mshould_sync:\n\u001b[1;32m 1386\u001b[0m context\u001b[38;5;241m.\u001b[39masync_wait()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 150\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 152\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:915\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 912\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 914\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[0;32m--> 915\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 917\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[1;32m 918\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:947\u001b[0m, in \u001b[0;36mFunction._call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 944\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m 945\u001b[0m \u001b[38;5;66;03m# In this case we have created variables on the first call, so we run the\u001b[39;00m\n\u001b[1;32m 946\u001b[0m \u001b[38;5;66;03m# defunned version which is guaranteed to never create variables.\u001b[39;00m\n\u001b[0;32m--> 947\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_stateless_fn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# pylint: disable=not-callable\u001b[39;00m\n\u001b[1;32m 948\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_stateful_fn \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 949\u001b[0m \u001b[38;5;66;03m# Release the lock early so that multiple threads can perform the call\u001b[39;00m\n\u001b[1;32m 950\u001b[0m \u001b[38;5;66;03m# in parallel.\u001b[39;00m\n\u001b[1;32m 951\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:2956\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2953\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock:\n\u001b[1;32m 2954\u001b[0m (graph_function,\n\u001b[1;32m 2955\u001b[0m filtered_flat_args) \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_maybe_define_function(args, kwargs)\n\u001b[0;32m-> 2956\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 2957\u001b[0m \u001b[43m \u001b[49m\u001b[43mfiltered_flat_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1853\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1849\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[1;32m 1850\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[1;32m 1851\u001b[0m \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[1;32m 1852\u001b[0m \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[0;32m-> 1853\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_build_call_outputs(\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1854\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcancellation_manager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcancellation_manager\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m 1855\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[1;32m 1856\u001b[0m args,\n\u001b[1;32m 1857\u001b[0m possible_gradient_type,\n\u001b[1;32m 1858\u001b[0m executing_eagerly)\n\u001b[1;32m 1859\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:499\u001b[0m, in \u001b[0;36m_EagerDefinedFunction.call\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 497\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m _InterpolateFunctionError(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m 498\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cancellation_manager \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 499\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 500\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mstr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msignature\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 501\u001b[0m \u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_num_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 502\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 503\u001b[0m \u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 504\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mctx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 505\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 506\u001b[0m outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[1;32m 507\u001b[0m \u001b[38;5;28mstr\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msignature\u001b[38;5;241m.\u001b[39mname),\n\u001b[1;32m 508\u001b[0m num_outputs\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_num_outputs,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 511\u001b[0m ctx\u001b[38;5;241m=\u001b[39mctx,\n\u001b[1;32m 512\u001b[0m cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_manager)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:54\u001b[0m, in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 52\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m 53\u001b[0m ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[0;32m---> 54\u001b[0m tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 55\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 56\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 57\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Start importing packages\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras.models import Model, Sequential \n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"from tensorflow.keras import optimizers \n",
|
|
"from tensorflow.keras import regularizers \n",
|
|
"from tensorflow.keras.utils import to_categorical \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"# convert into dataset matrix\n",
|
|
"def convertToMatrix(data, step):\n",
|
|
" X, Y =[], []\n",
|
|
" for i in range(len(data)-step):\n",
|
|
" d=i+step \n",
|
|
" X.append(data[i:d,])\n",
|
|
" Y.append(data[d,])\n",
|
|
" return np.array(X), np.array(Y)\n",
|
|
"\n",
|
|
"step = 4\n",
|
|
"N = 1000 \n",
|
|
"Tp = 800 \n",
|
|
"\n",
|
|
"t=np.arange(0,N)\n",
|
|
"x=np.sin(0.02*t)+2*np.random.rand(N)\n",
|
|
"df = pd.DataFrame(x)\n",
|
|
"df.head()\n",
|
|
"\n",
|
|
"plt.plot(df)\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"values=df.values\n",
|
|
"train,test = values[0:Tp,:], values[Tp:N,:]\n",
|
|
"\n",
|
|
"# add step elements into train and test\n",
|
|
"test = np.append(test,np.repeat(test[-1,],step))\n",
|
|
"train = np.append(train,np.repeat(train[-1,],step))\n",
|
|
" \n",
|
|
"trainX,trainY =convertToMatrix(train,step)\n",
|
|
"testX,testY =convertToMatrix(test,step)\n",
|
|
"trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
|
|
"testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(SimpleRNN(units=32, input_shape=(1,step), activation=\"relu\"))\n",
|
|
"model.add(Dense(8, activation=\"relu\")) \n",
|
|
"model.add(Dense(1))\n",
|
|
"model.compile(loss='mean_squared_error', optimizer='rmsprop')\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)\n",
|
|
"trainPredict = model.predict(trainX)\n",
|
|
"testPredict= model.predict(testX)\n",
|
|
"predicted=np.concatenate((trainPredict,testPredict),axis=0)\n",
|
|
"\n",
|
|
"trainScore = model.evaluate(trainX, trainY, verbose=0)\n",
|
|
"print(trainScore)\n",
|
|
"\n",
|
|
"index = df.index.values\n",
|
|
"plt.plot(index,df)\n",
|
|
"plt.plot(index,predicted)\n",
|
|
"plt.axvline(df.index[Tp], c=\"r\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "897a47c7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## An extrapolation example\n",
|
|
"\n",
|
|
"The following code provides an example of how recurrent neural\n",
|
|
"networks can be used to extrapolate to unknown values of physics data\n",
|
|
"sets. Specifically, the data sets used in this program come from\n",
|
|
"a quantum mechanical many-body calculation of energies as functions of the number of particles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6776ae2a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"# For matrices and calculations\n",
|
|
"import numpy as np\n",
|
|
"# For machine learning (backend for keras)\n",
|
|
"import tensorflow as tf\n",
|
|
"# User-friendly machine learning library\n",
|
|
"# Front end for TensorFlow\n",
|
|
"import tensorflow.keras\n",
|
|
"# Different methods from Keras needed to create an RNN\n",
|
|
"# This is not necessary but it shortened function calls \n",
|
|
"# that need to be used in the code.\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras import regularizers\n",
|
|
"from tensorflow.keras.models import Model, Sequential\n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"# For timing the code\n",
|
|
"from timeit import default_timer as timer\n",
|
|
"# For plotting\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"\n",
|
|
"# The data set\n",
|
|
"datatype='VaryDimension'\n",
|
|
"X_tot = np.arange(2, 42, 2)\n",
|
|
"y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,\n",
|
|
"\t-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, \n",
|
|
"\t-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35227d36",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The way the recurrent neural networks are trained in this program\n",
|
|
"differs from how machine learning algorithms are usually trained.\n",
|
|
"Typically a machine learning algorithm is trained by learning the\n",
|
|
"relationship between the x data and the y data. In this program, the\n",
|
|
"recurrent neural network will be trained to recognize the relationship\n",
|
|
"in a sequence of y values. This is type of data formatting is\n",
|
|
"typically used time series forcasting, but it can also be used in any\n",
|
|
"extrapolation (time series forecasting is just a specific type of\n",
|
|
"extrapolation along the time axis). This method of data formatting\n",
|
|
"does not use the x data and assumes that the y data are evenly spaced.\n",
|
|
"\n",
|
|
"For a standard machine learning algorithm, the training data has the\n",
|
|
"form of (x,y) so the machine learning algorithm learns to assiciate a\n",
|
|
"y value with a given x value. This is useful when the test data has x\n",
|
|
"values within the same range as the training data. However, for this\n",
|
|
"application, the x values of the test data are outside of the x values\n",
|
|
"of the training data and the traditional method of training a machine\n",
|
|
"learning algorithm does not work as well. For this reason, the\n",
|
|
"recurrent neural network is trained on sequences of y values of the\n",
|
|
"form ((y1, y2), y3), so that the network is concerned with learning\n",
|
|
"the pattern of the y data and not the relation between the x and y\n",
|
|
"data. As long as the pattern of y data outside of the training region\n",
|
|
"stays relatively stable compared to what was inside the training\n",
|
|
"region, this method of training can produce accurate extrapolations to\n",
|
|
"y values far removed from the training data set."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "7dc577d1",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# FORMAT_DATA\n",
|
|
"def format_data(data, length_of_sequence = 2): \n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" data(a numpy array): the data that will be the inputs to the recurrent neural\n",
|
|
" network\n",
|
|
" length_of_sequence (an int): the number of elements in one iteration of the\n",
|
|
" sequence patter. For a function approximator use length_of_sequence = 2.\n",
|
|
" Returns:\n",
|
|
" rnn_input (a 3D numpy array): the input data for the recurrent neural network. Its\n",
|
|
" dimensions are length of data - length of sequence, length of sequence, \n",
|
|
" dimnsion of data\n",
|
|
" rnn_output (a numpy array): the training data for the neural network\n",
|
|
" Formats data to be used in a recurrent neural network.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" X, Y = [], []\n",
|
|
" for i in range(len(data)-length_of_sequence):\n",
|
|
" # Get the next length_of_sequence elements\n",
|
|
" a = data[i:i+length_of_sequence]\n",
|
|
" # Get the element that immediately follows that\n",
|
|
" b = data[i+length_of_sequence]\n",
|
|
" # Reshape so that each data point is contained in its own array\n",
|
|
" a = np.reshape (a, (len(a), 1))\n",
|
|
" X.append(a)\n",
|
|
" Y.append(b)\n",
|
|
" rnn_input = np.array(X)\n",
|
|
" rnn_output = np.array(Y)\n",
|
|
"\n",
|
|
" return rnn_input, rnn_output\n",
|
|
"\n",
|
|
"\n",
|
|
"# ## Defining the Recurrent Neural Network Using Keras\n",
|
|
"# \n",
|
|
"# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.\n",
|
|
"\n",
|
|
"def rnn(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with one hidden layer and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer\n",
|
|
" hidden_neurons = 200\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to \n",
|
|
" # the network immediately after the input layer\n",
|
|
" rnn = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\")(inp)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1978b6a1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Predicting New Points With A Trained Recurrent Neural Network"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "d4ad417a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_rnn (x1, y_test, plot_min, plot_max):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" x1 (a list or numpy array): The complete x component of the data set\n",
|
|
" y_test (a list or numpy array): The complete y component of the data set\n",
|
|
" plot_min (an int or float): the smallest x value used in the training data\n",
|
|
" plot_max (an int or float): the largest x valye used in the training data\n",
|
|
" Returns:\n",
|
|
" None.\n",
|
|
" Uses a trained recurrent neural network model to predict future points in the \n",
|
|
" series. Computes the MSE of the predicted data set from the true data set, saves\n",
|
|
" the predicted data set to a csv file, and plots the predicted and true data sets w\n",
|
|
" while also displaying the data range used for training.\n",
|
|
" \"\"\"\n",
|
|
" # Add the training data as the first dim points in the predicted data array as these\n",
|
|
" # are known values.\n",
|
|
" y_pred = y_test[:dim].tolist()\n",
|
|
" # Generate the first input to the trained recurrent neural network using the last two \n",
|
|
" # points of the training data. Based on how the network was trained this means that it\n",
|
|
" # will predict the first point in the data set after the training data. All of the \n",
|
|
" # brackets are necessary for Tensorflow.\n",
|
|
" next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])\n",
|
|
" # Save the very last point in the training data set. This will be used later.\n",
|
|
" last = [y_test[dim-1]]\n",
|
|
"\n",
|
|
" # Iterate until the complete data set is created.\n",
|
|
" for i in range (dim, len(y_test)):\n",
|
|
" # Predict the next point in the data set using the previous two points.\n",
|
|
" next = model.predict(next_input)\n",
|
|
" # Append just the number of the predicted data set\n",
|
|
" y_pred.append(next[0][0])\n",
|
|
" # Create the input that will be used to predict the next data point in the data set.\n",
|
|
" next_input = np.array([[last, next[0]]], dtype=np.float64)\n",
|
|
" last = next\n",
|
|
"\n",
|
|
" # Print the mean squared error between the known data set and the predicted data set.\n",
|
|
" print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())\n",
|
|
" # Save the predicted data set as a csv file for later use\n",
|
|
" name = datatype + 'Predicted'+str(dim)+'.csv'\n",
|
|
" np.savetxt(name, y_pred, delimiter=',')\n",
|
|
" # Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
" # for the training data.\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.plot(x1, y_test, label=\"true\", linewidth=3)\n",
|
|
" ax.plot(x1, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
" ax.legend()\n",
|
|
" # Created a red region to represent the points used in the training data.\n",
|
|
" ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn(length_of_sequences = rnn_input.shape[1])\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2cad4fc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Changing the size of the recurrent neural network and its parameters\n",
|
|
"can drastically change the results you get from the model. The below\n",
|
|
"code takes the simple recurrent neural network from above and adds a\n",
|
|
"second hidden layer, changes the number of neurons in the hidden\n",
|
|
"layer, and explicitly declares the activation function of the hidden\n",
|
|
"layers to be a sigmoid function. The loss function and optimizer can\n",
|
|
"also be changed but are kept the same as the above network. These\n",
|
|
"parameters can be tuned to provide the optimal result from the\n",
|
|
"network. For some ideas on how to improve the performance of a\n",
|
|
"[recurrent neural network](https://danijar.com/tips-for-training-recurrent-neural-networks)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c39f1516",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer, increased from the first network\n",
|
|
" hidden_neurons = 500\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Create two hidden layers instead of one hidden layer. Explicitly set the activation\n",
|
|
" # function to be the sigmoid function (the default value is hyperbolic tangent)\n",
|
|
" rnn1 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=True, # This needs to be True if another hidden layer is to follow\n",
|
|
" stateful = stateful, activation = 'sigmoid',\n",
|
|
" name=\"RNN1\")(inp)\n",
|
|
" rnn2 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False, activation = 'sigmoid',\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN2\")(rnn1)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn2)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn_2layers(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "842c7602",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Other Types of Recurrent Neural Networks\n",
|
|
"\n",
|
|
"Besides a simple recurrent neural network layer, there are two other\n",
|
|
"commonly used types of recurrent neural network layers: Long Short\n",
|
|
"Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short\n",
|
|
"introduction to these layers see <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>\n",
|
|
"and <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>.\n",
|
|
"\n",
|
|
"The first network created below is similar to the previous network,\n",
|
|
"but it replaces the SimpleRNN layers with LSTM layers. The second\n",
|
|
"network below has two hidden layers made up of GRUs, which are\n",
|
|
"preceeded by two dense (feeddorward) neural network layers. These\n",
|
|
"dense layers \"preprocess\" the data before it reaches the recurrent\n",
|
|
"layers. This architecture has been shown to improve the performance\n",
|
|
"of recurrent neural networks (see the link above and also\n",
|
|
"<https://arxiv.org/pdf/1807.02857.pdf>."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "6f0e9b62",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons on the input/output layer and the number of neurons in the hidden layer\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input Layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)\n",
|
|
" rnn= LSTM(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True, activation='tanh')(inp)\n",
|
|
" rnn1 = LSTM(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True, activation='tanh')(rnn)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn1)\n",
|
|
" # Define the midel\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the model\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with four hidden layers (two dense followed by\n",
|
|
" two GRU layers) and returns the model.\n",
|
|
" \"\"\" \n",
|
|
" # Number of neurons on the input/output layers and hidden layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden Dense (feedforward) layers\n",
|
|
" dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)\n",
|
|
" dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)\n",
|
|
" # Hidden GRU layers\n",
|
|
" rnn1 = GRU(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True)(dnn1)\n",
|
|
" rnn = GRU(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True)(rnn1)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Define the model\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the mdoel\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Change the method name to reflect which network you want to use\n",
|
|
"model = dnn2_gru2(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)\n",
|
|
"\n",
|
|
"\n",
|
|
"# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)\n",
|
|
"# \n",
|
|
"# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"# Reshape the data for Keras specifications\n",
|
|
"X_train = X_train.reshape((dim, 1))\n",
|
|
"y_train = y_train.reshape((dim, 1))\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Set the sequence length to 1 for regular data formatting \n",
|
|
"model = rnn(length_of_sequences = 1)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(X_train, y_train, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict the remaining data points\n",
|
|
"X_pred = X_tot[dim:]\n",
|
|
"X_pred = X_pred.reshape((len(X_pred), 1))\n",
|
|
"y_model = model.predict(X_pred)\n",
|
|
"y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))\n",
|
|
"\n",
|
|
"# Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
"# for the training data.\n",
|
|
"fig, ax = plt.subplots()\n",
|
|
"ax.plot(X_tot, y_tot, label=\"true\", linewidth=3)\n",
|
|
"ax.plot(X_tot, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
"ax.legend()\n",
|
|
"# Created a red region to represent the points used in the training data.\n",
|
|
"ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0752ba7f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Generative Models\n",
|
|
"\n",
|
|
"**Generative models** describe a class of statistical models that are a contrast\n",
|
|
"to **discriminative models**. Informally we say that generative models can\n",
|
|
"generate new data instances while discriminative models discriminate between\n",
|
|
"different kinds of data instances. A generative model could generate new photos\n",
|
|
"of animals that look like 'real' animals while a discriminative model could tell\n",
|
|
"a dog from a cat. More formally, given a data set $x$ and a set of labels /\n",
|
|
"targets $y$. Generative models capture the joint probability $p(x, y)$, or\n",
|
|
"just $p(x)$ if there are no labels, while discriminative models capture the\n",
|
|
"conditional probability $p(y | x)$. Discriminative models generally try to draw\n",
|
|
"boundaries in the data space (often high dimensional), while generative models\n",
|
|
"try to model how data is placed throughout the space.\n",
|
|
"\n",
|
|
"**Note**: this material is thanks to Linus Ekstrøm."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "784138f8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Generative Adversarial Networks\n",
|
|
"\n",
|
|
"**Generative Adversarial Networks** are a type of unsupervised machine learning\n",
|
|
"algorithm proposed by [Goodfellow et. al](https://arxiv.org/pdf/1406.2661.pdf)\n",
|
|
"in 2014 (short and good article).\n",
|
|
"\n",
|
|
"The simplest formulation of\n",
|
|
"the model is based on a game theoretic approach, *zero sum game*, where we pit\n",
|
|
"two neural networks against one another. We define two rival networks, one\n",
|
|
"generator $g$, and one discriminator $d$. The generator directly produces\n",
|
|
"samples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a42f89ee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" x = g(z; \\theta^{(g)})\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abe7212f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The discriminator attempts to distinguish between samples drawn from the\n",
|
|
"training data and samples drawn from the generator. In other words, it tries to\n",
|
|
"tell the difference between the fake data produced by $g$ and the actual data\n",
|
|
"samples we want to do prediction on. The discriminator outputs a probability\n",
|
|
"value given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0821eaee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" d(x; \\theta^{(d)})\n",
|
|
"\\label{_auto2} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55e0ccf6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"indicating the probability that $x$ is a real training example rather than a\n",
|
|
"fake sample the generator has generated. The simplest way to formulate the\n",
|
|
"learning process in a generative adversarial network is a zero-sum game, in\n",
|
|
"which a function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f37ece14",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto3} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d6d6d5fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"determines the reward for the discriminator, while the generator gets the\n",
|
|
"conjugate reward"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c74b45c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" -v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto4} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "605ac8c9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"During learning both of the networks maximize their own reward function, so that\n",
|
|
"the generator gets better and better at tricking the discriminator, while the\n",
|
|
"discriminator gets better and better at telling the difference between the fake\n",
|
|
"and real data. The generator and discriminator alternate on which one trains at\n",
|
|
"one time (i.e. for one epoch). In other words, we keep the generator constant\n",
|
|
"and train the discriminator, then we keep the discriminator constant to train\n",
|
|
"the generator and repeat. It is this back and forth dynamic which lets GANs\n",
|
|
"tackle otherwise intractable generative problems. As the generator improves with\n",
|
|
" training, the discriminator's performance gets worse because it cannot easily\n",
|
|
" tell the difference between real and fake. If the generator ends up succeeding\n",
|
|
" perfectly, the the discriminator will do no better than random guessing i.e.\n",
|
|
" 50\\%. This progression in the training poses a problem for the convergence\n",
|
|
" criteria for GANs. The discriminator feedback gets less meaningful over time,\n",
|
|
" if we continue training after this point then the generator is effectively\n",
|
|
" training on junk data which can undo the learning up to that point. Therefore,\n",
|
|
" we stop training when the discriminator starts outputting $1/2$ everywhere.\n",
|
|
"\n",
|
|
"At convergence we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfec7462",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto5\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" g^* = \\underset{g}{\\mathrm{argmin}}\\hspace{2pt}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto5} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37c65d4c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The default choice for $v$ is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c868a092",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto6\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)}) = \\mathbb{E}_{x\\sim p_\\mathrm{data}}\\log d(x)\n",
|
|
" + \\mathbb{E}_{x\\sim p_\\mathrm{model}}\n",
|
|
" \\log (1 - d(x))\n",
|
|
"\\label{_auto6} \\tag{6}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ad465af3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The main motivation for the design of GANs is that the learning process requires\n",
|
|
"neither approximate inference (variational autoencoders for example) nor\n",
|
|
"approximation of a partition function. In the case where"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "27858a4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto7\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto7} \\tag{7}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86006023",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"is convex in $\\theta^{(g)} then the procedure is guaranteed to converge and is\n",
|
|
"asymptotically consistent\n",
|
|
"( [Seth Lloyd on QuGANs](https://arxiv.org/pdf/1804.09139.pdf) ).\n",
|
|
"\n",
|
|
"This is in\n",
|
|
"general not the case and it is possible to get situations where the training\n",
|
|
"process never converges because the generator and discriminator chase one\n",
|
|
"another around in the parameter space indefinitely. A much deeper discussion on\n",
|
|
"the currently open research problem of GAN convergence is available\n",
|
|
"[here](https://www.deeplearningbook.org/contents/generative_models.html). To\n",
|
|
"anyone interested in learning more about GANs it is a highly recommended read.\n",
|
|
"Direct quote: \"In this best-performing formulation, the generator aims to\n",
|
|
"increase the log probability that the discriminator makes a mistake, rather than\n",
|
|
"aiming to decrease the log probability that the discriminator makes the correct\n",
|
|
"prediction.\" [Another interesting read](https://arxiv.org/abs/1701.00160)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2fee38bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Writing Our First Generative Adversarial Network\n",
|
|
"Let us now move on to actually implementing a GAN in tensorflow. We will study\n",
|
|
"the performance of our GAN on the MNIST dataset. This code is based on and\n",
|
|
"adapted from the\n",
|
|
"[google tutorial](https://www.tensorflow.org/tutorials/generative/dcgan)\n",
|
|
"\n",
|
|
"First we import our libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "004a0b53",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import time\n",
|
|
"import numpy as np\n",
|
|
"import tensorflow as tf\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from tensorflow.keras import layers\n",
|
|
"from tensorflow.keras.utils import plot_model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af161",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define our hyperparameters and import our data the usual way"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "8cbaf16a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"BUFFER_SIZE = 60000\n",
|
|
"BATCH_SIZE = 256\n",
|
|
"EPOCHS = 30\n",
|
|
"\n",
|
|
"data = tf.keras.datasets.mnist.load_data()\n",
|
|
"(train_images, train_labels), (test_images, test_labels) = data\n",
|
|
"train_images = np.reshape(train_images, (train_images.shape[0],\n",
|
|
" 28,\n",
|
|
" 28,\n",
|
|
" 1)).astype('float32')\n",
|
|
"\n",
|
|
"# we normalize between -1 and 1\n",
|
|
"train_images = (train_images - 127.5) / 127.5\n",
|
|
"training_dataset = tf.data.Dataset.from_tensor_slices(\n",
|
|
" train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "822b8cc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### MNIST and GANs\n",
|
|
"\n",
|
|
"Let's have a quick look"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "52b5965c",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.imshow(train_images[0], cmap='Greys')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "21c5199c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our two models. This is where the 'magic' happens. There are a\n",
|
|
"huge amount of possible formulations for both models. A lot of engineering and\n",
|
|
"trial and error can be done here to try to produce better performing models. For\n",
|
|
"more advanced GANs this is by far the step where you can 'make or break' a\n",
|
|
"model.\n",
|
|
"\n",
|
|
"We start with the generator. As stated in the introductory text the generator\n",
|
|
"$g$ upsamples from a random sample to the shape of what we want to predict. In\n",
|
|
"our case we are trying to predict MNIST images ($28\\times 28$ pixels)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "356759c7",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_model():\n",
|
|
" \"\"\"\n",
|
|
" The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to\n",
|
|
" produce an image from a random seed. We start with a Dense layer taking this\n",
|
|
" random sample as an input and subsequently upsample through multiple\n",
|
|
" convolutional layers.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
"\n",
|
|
"\n",
|
|
" # adding our input layer. Dense means that every neuron is connected and\n",
|
|
" # the input shape is the shape of our random noise. The units need to match\n",
|
|
" # in some sense the upsampling strides to reach our desired output shape.\n",
|
|
" # we are using 100 random numbers as our seed\n",
|
|
" model.add(layers.Dense(units=7*7*BATCH_SIZE,\n",
|
|
" use_bias=False,\n",
|
|
" input_shape=(100, )))\n",
|
|
" # we normalize the output form the Dense layer\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" # and add an activation function to our 'layer'. LeakyReLU avoids vanishing\n",
|
|
" # gradient problem\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" model.add(layers.Reshape((7, 7, BATCH_SIZE)))\n",
|
|
" assert model.output_shape == (None, 7, 7, BATCH_SIZE)\n",
|
|
" # even though we just added four keras layers we think of everything above\n",
|
|
" # as 'one' layer\n",
|
|
"\n",
|
|
" # next we add our upscaling convolutional layers\n",
|
|
" model.add(layers.Conv2DTranspose(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(1, 1),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 7, 7, 128)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 14, 14, 64)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=1,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False,\n",
|
|
" activation='tanh'))\n",
|
|
" assert model.output_shape == (None, 28, 28, 1)\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "854bcd6b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"And there we have our 'simple' generator model. Now we move on to defining our\n",
|
|
"discriminator model $d$, which is a convolutional neural network based image\n",
|
|
"classifier."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "41473304",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_model():\n",
|
|
" \"\"\"\n",
|
|
" The discriminator is a convolutional neural network based image classifier\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
" model.add(layers.Conv2D(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" input_shape=[28, 28, 1]))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
"\n",
|
|
" model.add(layers.Conv2D(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same'))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
" model.add(layers.Flatten())\n",
|
|
" model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af567",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Let us take a look at our models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "f899d4e3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generator = generator_model()\n",
|
|
"plot_model(generator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "87ef384b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"discriminator = discriminator_model()\n",
|
|
"plot_model(discriminator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2bef82d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we need a few helper objects we will use in training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "e397847a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
|
|
"generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n",
|
|
"discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "db3396cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The first object, *cross_entropy* is our loss function and the two others are\n",
|
|
"our optimizers. Notice we use the same learning rate for both $g$ and $d$. This\n",
|
|
"is because they need to improve their accuracy at approximately equal speeds to\n",
|
|
"get convergence (not necessarily exactly equal). Now we define our loss\n",
|
|
"functions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "931eaced",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_loss(fake_output):\n",
|
|
" loss = cross_entropy(tf.ones_like(fake_output), fake_output)\n",
|
|
"\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "0c4a44bb",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_loss(real_output, fake_output):\n",
|
|
" real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n",
|
|
" fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)\n",
|
|
" total_loss = real_loss + fake_loss\n",
|
|
"\n",
|
|
" return total_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fcf8f066",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a kind of seed to help us compare the learning process over\n",
|
|
"multiple training epochs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "eea2bbee",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"noise_dimension = 100\n",
|
|
"n_examples_to_generate = 16\n",
|
|
"seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94a6e341",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have everything we need to define our training step, which we will apply\n",
|
|
"for every step in our training loop. Notice the @tf.function flag signifying\n",
|
|
"that the function is tensorflow 'compiled'. Removing this flag doubles the\n",
|
|
"computation time."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "8d48470b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tf.function\n",
|
|
"def train_step(images):\n",
|
|
" noise = tf.random.normal([BATCH_SIZE, noise_dimension])\n",
|
|
"\n",
|
|
" with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n",
|
|
" generated_images = generator(noise, training=True)\n",
|
|
"\n",
|
|
" real_output = discriminator(images, training=True)\n",
|
|
" fake_output = discriminator(generated_images, training=True)\n",
|
|
"\n",
|
|
" gen_loss = generator_loss(fake_output)\n",
|
|
" disc_loss = discriminator_loss(real_output, fake_output)\n",
|
|
"\n",
|
|
" gradients_of_generator = gen_tape.gradient(gen_loss,\n",
|
|
" generator.trainable_variables)\n",
|
|
" gradients_of_discriminator = disc_tape.gradient(disc_loss,\n",
|
|
" discriminator.trainable_variables)\n",
|
|
" generator_optimizer.apply_gradients(zip(gradients_of_generator,\n",
|
|
" generator.trainable_variables))\n",
|
|
" discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,\n",
|
|
" discriminator.trainable_variables))\n",
|
|
"\n",
|
|
" return gen_loss, disc_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "62015b88",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a helper function to produce an output over our training epochs\n",
|
|
"to see the predictive progression of our generator model. **Note**: I am including\n",
|
|
"this code here, but comment it out in the training loop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "b189ed96",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_and_save_images(model, epoch, test_input):\n",
|
|
" # we're making inferences here\n",
|
|
" predictions = model(test_input, training=False)\n",
|
|
"\n",
|
|
" fig = plt.figure(figsize=(4, 4))\n",
|
|
"\n",
|
|
" for i in range(predictions.shape[0]):\n",
|
|
" plt.subplot(4, 4, i+1)\n",
|
|
" plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n",
|
|
" plt.axis('off')\n",
|
|
"\n",
|
|
" plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')\n",
|
|
" plt.close()\n",
|
|
" #plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ba2c82d5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Setting up checkpoints to periodically save our model during training so that\n",
|
|
"everything is not lost even if the program were to somehow terminate while\n",
|
|
"training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "a0e2fc8a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setting up checkpoints to save model during training\n",
|
|
"checkpoint_dir = './training_checkpoints'\n",
|
|
"checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')\n",
|
|
"checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n",
|
|
" discriminator_optimizer=discriminator_optimizer,\n",
|
|
" generator=generator,\n",
|
|
" discriminator=discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d93a0f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our training loop"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "a1275556",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def train(dataset, epochs):\n",
|
|
" generator_loss_list = []\n",
|
|
" discriminator_loss_list = []\n",
|
|
"\n",
|
|
" for epoch in range(epochs):\n",
|
|
" start = time.time()\n",
|
|
"\n",
|
|
" for image_batch in dataset:\n",
|
|
" gen_loss, disc_loss = train_step(image_batch)\n",
|
|
" generator_loss_list.append(gen_loss.numpy())\n",
|
|
" discriminator_loss_list.append(disc_loss.numpy())\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epoch + 1, seed_images)\n",
|
|
"\n",
|
|
" if (epoch + 1) % 15 == 0:\n",
|
|
" checkpoint.save(file_prefix=checkpoint_prefix)\n",
|
|
"\n",
|
|
" print(f'Time for epoch {epoch} is {time.time() - start}')\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epochs, seed_images)\n",
|
|
"\n",
|
|
" loss_file = './data/lossfile.txt'\n",
|
|
" with open(loss_file, 'w') as outfile:\n",
|
|
" outfile.write(str(generator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write(str(discriminator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ff3a75a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"To train simply call this function. **Warning**: this might take a long time so\n",
|
|
"there is a folder of a pretrained network already included in the repository."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"id": "371ed41a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"train(train_dataset, EPOCHS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "654399f1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now to avoid having to train and everything, which will take a while depending\n",
|
|
"on your computer setup we now load in the model which produced the above gif."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"id": "dec4b560",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))\n",
|
|
"restored_generator = checkpoint.generator\n",
|
|
"restored_discriminator = checkpoint.discriminator\n",
|
|
"\n",
|
|
"print(restored_generator)\n",
|
|
"print(restored_discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "296bfa5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We have successfully loaded in our latest model. Let us now play around a bit\n",
|
|
"and see what kind of things we can learn about this model. Our generator takes\n",
|
|
"an array of 100 numbers. One idea can be to try to systematically change our\n",
|
|
"input. Let us try and see what we get"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"id": "eecfbb1f",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_latent_points(number=100, scale_means=1, scale_stds=1):\n",
|
|
" latent_dim = 100\n",
|
|
" means = scale_means * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" latent_space_value_range = tf.random.normal([number, latent_dim],\n",
|
|
" means,\n",
|
|
" stds,\n",
|
|
" dtype=tf.float64)\n",
|
|
"\n",
|
|
" return latent_space_value_range\n",
|
|
"\n",
|
|
"def generate_images(latent_points):\n",
|
|
" # notice we set training to false because we are making inferences\n",
|
|
" generated_images = restored_generator.predict(latent_points)\n",
|
|
"\n",
|
|
" return generated_images"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"id": "333a593d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def plot_result(generated_images, number=100):\n",
|
|
" # obviously this assumes sqrt number is an int\n",
|
|
" fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),\n",
|
|
" figsize=(10, 10))\n",
|
|
"\n",
|
|
" for i in range(int(np.sqrt(number))):\n",
|
|
" for j in range(int(np.sqrt(number))):\n",
|
|
" axs[i, j].imshow(generated_images[i*j], cmap='Greys')\n",
|
|
" axs[i, j].axis('off')\n",
|
|
"\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"id": "2f5f0154",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generated_images = generate_images(generate_latent_points())\n",
|
|
"plot_result(generated_images)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff581bf2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We see that the generator generates images that look like MNIST\n",
|
|
"numbers: $1, 4, 7, 9$. Let's try to tweak it a bit more to see if we are able\n",
|
|
"to generate a similar plot where we generate every MNIST number. Let us now try\n",
|
|
"to 'move' a bit around in the latent space. **Note**: decrease the plot number if\n",
|
|
"these following cells take too long to run on your computer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"id": "d3617ad8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 225\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=-5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=5))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a074f93",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Again, we have found something interesting. *Moving* around using our means\n",
|
|
"takes us from digit to digit, while *moving* around using our standard\n",
|
|
"deviations seem to increase the number of different digits! In the last image\n",
|
|
"above, we can barely make out every MNIST digit. Let us make on last plot using\n",
|
|
"this information by upping the standard deviation of our Gaussian noises."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"id": "4ef8937d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 400\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=10))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "385a2d0a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"A pretty cool result! We see that our generator indeed has learned a\n",
|
|
"distribution which qualitatively looks a whole lot like the MNIST dataset.\n",
|
|
"\n",
|
|
"Another interesting way to explore the latent space of our generator model is by\n",
|
|
"interpolating between the MNIST digits. This section is largely based on\n",
|
|
"[this excellent blogpost](https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/)\n",
|
|
"by Jason Brownlee.\n",
|
|
"\n",
|
|
"So let us start by defining a function to interpolate between two points in the\n",
|
|
"latent space."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"id": "57de87b8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def interpolation(point_1, point_2, n_steps=10):\n",
|
|
" ratios = np.linspace(0, 1, num=n_steps)\n",
|
|
" vectors = []\n",
|
|
" for i, ratio in enumerate(ratios):\n",
|
|
" vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))\n",
|
|
"\n",
|
|
" return tf.stack(vectors)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfb76bb6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have all we need to do our interpolation analysis."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"id": "e25decef",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 100\n",
|
|
"latent_points = generate_latent_points(number=plot_number)\n",
|
|
"results = None\n",
|
|
"for i in range(0, 2*np.sqrt(plot_number), 2):\n",
|
|
" interpolated = interpolation(latent_points[i], latent_points[i+1])\n",
|
|
" generated_images = generate_images(interpolated)\n",
|
|
"\n",
|
|
" if results is None:\n",
|
|
" results = generated_images\n",
|
|
" else:\n",
|
|
" results = tf.stack((results, generated_images))\n",
|
|
"\n",
|
|
"plot_results(results, plot_number)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |