Morten Hjorth-Jensen
2775 lines
150 KiB
Plaintext
2775 lines
150 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d2db899",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html chapter13.do.txt -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4908a97",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Recurrent neural networks: Overarching view\n",
|
|
"\n",
|
|
"Till now our focus has been, including convolutional neural networks\n",
|
|
"as well, on feedforward neural networks. The output or the activations\n",
|
|
"flow only in one direction, from the input layer to the output layer.\n",
|
|
"\n",
|
|
"A recurrent neural network (RNN) looks very much like a feedforward\n",
|
|
"neural network, except that it also has connections pointing\n",
|
|
"backward. \n",
|
|
"\n",
|
|
"RNNs are used to analyze time series data such as stock prices, and\n",
|
|
"tell you when to buy or sell. In autonomous driving systems, they can\n",
|
|
"anticipate car trajectories and help avoid accidents. More generally,\n",
|
|
"they can work on sequences of arbitrary lengths, rather than on\n",
|
|
"fixed-sized inputs like all the nets we have discussed so far. For\n",
|
|
"example, they can take sentences, documents, or audio samples as\n",
|
|
"input, making them extremely useful for natural language processing\n",
|
|
"systems such as automatic translation and speech-to-text.\n",
|
|
"\n",
|
|
"More to text to be added"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43cb4913",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A simple example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "cf6b9dab",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAACYAElEQVR4nO2dd5wU5f3HP7NX9g44jnLcHcjRRAFBkKIUpSgKiJpYYqyoMRYUKzEYNDHGhjHEHzEWYiWKRqPYRQSVZijSexepd3Tu4ODqzu+Pvd2bmZ155nmm7Mzuft++eHk7O+XZmWee5/t8qyTLsgyCIAiCIIgEIeB1AwiCIAiCIEQg4YUgCIIgiISChBeCIAiCIBIKEl4IgiAIgkgoSHghCIIgCCKhIOGFIAiCIIiEgoQXgiAIgiASChJeCIIgCIJIKNK9boDThEIh7N27Fzk5OZAkyevmEARBEATBgSzLOHbsGFq1aoVAgK1bSTrhZe/evSgqKvK6GQRBEARBWGDXrl1o3bo1c5+kE15ycnIAhH9848aNPW4NQRAEQRA8lJWVoaioKDqPs0g64SViKmrcuDEJLwRBEASRYPC4fJDDLkEQBEEQCQUJLwRBEARBJBQkvBAEQRAEkVCQ8EIQBEEQREJBwgtBEARBEAkFCS8EQRAEQSQUrgovr7zyCrp37x4NW+7fvz++/vpr5jFz585F7969kZWVhQ4dOmDy5MluNpEgCIIgiATDVeGldevWePbZZ7F06VIsXboUF1xwAX75y19i3bp1uvtv374dI0eOxMCBA7FixQo88sgjuO+++zBt2jQ3m0kQBEEQRAIhybIsx/OCzZo1w9/+9jf89re/jfnu4Ycfxueff44NGzZEt40ePRqrVq3CwoULuc5fVlaG3NxclJaWUpI6giAIgkgQRObvuPm81NbW4v3330d5eTn69++vu8/ChQsxbNgw1bbhw4dj6dKlqK6u1j2msrISZWVlqn8EQRAEQSQvrgsva9asQaNGjRAMBjF69Gh88sknOOOMM3T3LSkpQUFBgWpbQUEBampqcPDgQd1jJkyYgNzc3Og/KspIEARBEMmN68JLp06dsHLlSixatAh33XUXbr75Zqxfv95wf21Ng4hVy6jWwfjx41FaWhr9t2vXLucaTxAEQRCE73C9MGNmZiY6duwIAOjTpw+WLFmCf/zjH/jXv/4Vs29hYSFKSkpU2/bv34/09HQ0b95c9/zBYBDBYND5hhMEQRAxbCo5hvlbDuCm/u2QmU7ZNghviHtVaVmWUVlZqftd//798cUXX6i2zZw5E3369EFGRkY8mkcQBEEwGD5pHgAgJMu4Y9CpHreGSFVcFZsfeeQRzJ8/Hz///DPWrFmDRx99FHPmzMENN9wAIGzyuemmm6L7jx49Gjt27MDYsWOxYcMGvPnmm3jjjTfw0EMPudlMgiAIQpDVu0u9bgKRwriqedm3bx9GjRqF4uJi5Obmonv37pgxYwYuuugiAEBxcTF27twZ3b99+/aYPn06HnzwQbz00kto1aoVXnjhBVx11VVuNpMgCIIQJGDgh0gQ8cBV4eWNN95gfj9lypSYbYMHD8by5ctdahFBEAThBAGSXQgPIW8rgiAIQhijCFCCiAckvBAEQRDCkOxCeAkJLwRBEIQwEkh6IbyDhBeCIAhCGPJ5IbyEhBeCIIgU42/fbMSbP2y3dQ4yGxFeQsILQRBECrF1/3G8NHsbnvjSuEyLkp2HTuCbdSXRUi0RyGyUmtSGZBw9UYXSk/rFkuMFCS8EQRApxLEKsUln0N9m4853luHbDftV2wM0e6QkJWUVOOuJWTj7qW89bQd1P4IgiBSiNiSb76TD4p8OabaQ5iUVqS+W7G07SHhJYk5U1WDLvmNeN4NwmW/WlWDy3G1eN4NIEKwKL6//sB3HK2uin8lhNzWJWA+9zrBMwksSEArJePLL9fhy9V7V9hGT5uOi/5uHhdu0KyYimbjznWV49uuNWLnrqNdNIRIAq8ILAHy/sd505PXkRXhDiDQvhFXW7inF377ZiBNV4VXQjHUleOOH7bjnvRWq/XYePgEAmL6mOO5tJOKD0onSawc6IjGola0LL8r+5vXkRXiDXzQvrtY2Itzh0n/+AACorpXxyMguOHi80uMWEV5RWROK/p2dkab6btFPh5AWkHB2u2bxbhbhY5SaF1mWLaf593ryIrzBL5oXEl4SmA3FZVz7ed3JCPcoV/ggZGXUK1KPVVTj2lcXAQA2P3UxMtNJyUqECclK4YXGB0KMiOzrdbehEY0gEpjyylrd7ccq6oWau6Yuw8fLdwuf+0h5FVaRH03SUVNrx2xU/zdpXlKVcCcIeOyxTcJLAkNVXYnyqnohRWUOUOzz3cb9GPvfVcLnHvTcbPzypf9hwdaDdppI+AyjfiIKRRulBrIs4+8zN0UDQkI+8Xkh4SUFoDEmeVGajUI2HDH1OFZ3bmWECZH41MpqnxcRZJDDbqqx8KdD+Of3W6MBIVGfFy8bBRJefEVlTS1qakPmO9Yxb/MBrN1T6mKLCL+jzLuh7DpeDyyEf7GjeRE1G5WerBYWkAh/UXpCHcUYeZxea/5JePEJFdW1OOsvs3Dh83OFjotEHrHwupMR7nGiqt7nxU7+DjNKT1Rj5roSVNXwC9eEP9E67FrGZFhZuO0QevxlJsZ/vMbGRQgvmb1xP+56d7lqW6T/eG02JOHFJ2wqOYaT1bX4+dAJ2+da9NMh/Lj9MNe+sizjy9V7sfuI/esS8UcpvLi5wr3+9UW4451lePH7La5dg3CHFTuP4M53lmJn3dgSL4fd//t2MwDg/SW7LF+P8JbfTFmi+lxTG1JoXjxokAIKlfYJdjqC8tCTVbXREFke/rt0Fx6etgZZGQFsfPJi640gPCGk0LaofBkcvs66veGw/E9X7sXYYZ0cPjvhJle9sgAhGdh+sBwzHxysMRtZ7ylmK++Qi5pAwhsqakK+SVJHmhefoCwvL+5EV0+Vgc/Mmt2l2KxT52jG2hIAQEU1mQMSEeXk45bZSDlGlZ6sJj+rBCPSLTbvOw5A67Ardi7l7pKJ3chOJl/Cn1RU1yrMRiS8EFBPELbeeZ1jS09W47IXf8Cw/5sX893h8iobFyP8hNPRRnqUnqzGpf/8ASt2HnH9WoQ72NGIKI81m7tI85J8KIUXryHhxYeEZBmlJ6x56et1rP3HKqJ/a8+5ajetohMZ5eOMZ7TR/yj3S8KQkabuDeryAOxjteOFUpti1sdI85J8VFSHouvjgMfSAwkvPmTOpgPo8cRMPPLJWq79lYOInvCiNknVby8prYjZl0gslE875JLPy2vzt8dsowi2xKFZw0zV5xoBjchDH65Wfa5VaV5MzEZkiU46KqprowItmY0IAGoV7HPfbAQA/OfHnap9Kqr1U8ErMRuXlF//dPA4b/MIn6IURs3U9Jv3HcOYd5fr+j4RyUnpyWo0baAWXtRCLrvPTNOUlVAKL2aTF5mNko/KmpBvMuxStJFPUGpH9N75+VsOYNQbP2LsRaczz6NnalIOUOHvw9dKV+j9WjfNFmwx4TfM1PTX/GshjpyoxoJtB7HisWG2rkWKF//z/cZ9uHXK0pjtSo2IqGWnlsPn5Uh5FdLSJDIbJSGV1bVITwvPG14PASS8+ASzyeDROhPS87M2x3z3p8/WRf/WGzBUZiPF9ppQ/SjWpEEGZ0sJPyESbXSkLlPmEU3GTCuYRZoQ3vP0Vxt0t9cq3ntR8UKptdELlT5ZVYueT84CALTPayh4dsLvVNTUokHdg/d6AUNmI5+gjjZia09YmJqNVGYG/e1E4qB6nvQQCQVGPilqzYtYn6kx8XlRJrt0M+Mz4Q0V1SEKlSbUGGlHRDGzM6tW6oqBi8aZxETlsBtHB0nluHWsohovfr8F2w+Wx68BhClGU0utjY5iZjaSDfYlkoPKmtroQybNCxGD3mKId4FkHvoIrN9bhkv/OR+zFdWCqXhagiLrC6Px5Ikv1mPizM0YPik2jxDhHXorY1mWbWViFnHCrYmnNE3EBVkGOeymKhtLytCmWQM0yFTfejOzES88poPb316KPUdPYu2eMsvXIfyH3eiOjSVl2H6gHBef2dJ0X6W/w5Kfw3W0qGijv9CbW0Iyn8NuKCRj8/7YqLQagRwxpHlJPsLCS/i5ep0uwVXNy4QJE3D22WcjJycH+fn5uPzyy7Fp0ybmMXPmzIEkSTH/Nm7c6GZT48L3G/dhxKT5+MWL/wMQHiDe+GE7Vu46qhoI9F55XnnGTHiRZf2suqR4SUxUanpV2nfxBzpi0nzc9e5yruy55LDrf/Qml5pQSD1GGHSTiTM3YcSk+THb3128U2dvxekU5xPJJ0MkBjLqu4zXI4CrwsvcuXMxZswYLFq0CLNmzUJNTQ2GDRuG8nJz2/imTZtQXFwc/Xfaaae52dS48MmKvQCArfvD+VW+WL0XT365Hpe/9D/VgGLH8dI8z4us6/xrp0Ab4Ty8WhSRPC+819t7lJIXJgN6k0ttSOaqKv3ynG262w8er4z+bSYg19qoXk34E1mW6x12PXY6cdVsNGPGDNXnt956C/n5+Vi2bBkGDRrEPDY/Px9NmjRxsXXes2VffZI4oyggUXg0L/rHWb8m4SzvLt6Bv369Ee/8ti96FDVh7iurhF7ldrFrHlJo41rkBE33V5k5xS5FxAk9rX5NSBZKUsdC1zfPICCASAz2H6vAV6uLcWWv1sjNjk2fIaN+zPHa5yWuslNpabiOTrNmzUz37dmzJ1q2bImhQ4di9uzZhvtVVlairKxM9S9RSAsoI4ycedF1ywNoJhp9h2AaaPzCo5+sRVlFDe5/f4Xpvk5FdyhLRaSRG39SoDe5aLUhdl57M/M2mY0SjxteW4y/fLEef5i22nCfyDNOarORElmWMXbsWJx33nno1q2b4X4tW7bEq6++imnTpuHjjz9Gp06dMHToUMybpx/JMGHCBOTm5kb/FRUVufUTHEclvKhWzXbU/+zvZVlfTNp2oJwqBfsM0ZWrakUt2IX2lSmLd5rv77WzHmGOkeZFiR3xghx2k48tdS4N323Yr7+DItrI6zEgbsLLPffcg9WrV+M///kPc79OnTrh9ttvR69evdC/f3+8/PLLuOSSSzBx4kTd/cePH4/S0tLov127drnRfFcwEl7svPN6mpeYgn0G57/i5QXWL0w4Do/5UF1V2ro5oKpWLOsqiS7+R+8ZaccHOwul7QePY/Q7y7BGUZneqD8S/uTlOVvx36X8c6YMhc+Lx4NAXEKl7733Xnz++eeYN28eWrduLXx8v379MHXqVN3vgsEggkFzG70fMTIb6TrUcg4yertRJt3EhGfwN442sn4t6iNJgm60kXMP99OV4QCEWRv2YdszIx07LxEfth04judmhKN/f92Hz2IhywqzUTLneZFlGffeey8++eQTzJkzB+3bt7d0nhUrVqBlS/PcE4lGuqHZKHZf3iHHTPMCmSKLEgUes5FsQ2BRIhrhJklAeWUNrn9tEXYcOmF+ABF3dKONtD4vDlzHjsaP8I6yk4waZwZyidph1/k2ieCq8DJmzBi89957+Oyzz5CTk4OSkhIAQG5uLrKzw1WMx48fjz179uDtt98GAEyaNAnt2rVD165dUVVVhalTp2LatGmYNm2am031BKVDnfKV33+sMnZnTvQmPG10Aa2sEwPR0Gf1JCKGFZ+rD5bswiqFyYDwF3qTS00o5JjAqweNLcnNa/N+whU9TwGQ5JqXV155BQAwZMgQ1fa33noLt9xyCwCguLgYO3fWJz6qqqrCQw89hD179iA7Oxtdu3bFV199hZEjk08tmZ5W//CthjjH7he7ozYrJo0viYGow67a9GPD2ZdjfwlqPxnCf+hNLrUhtW6ENCWpi5Un/9PBcvx91mYA3vu9uW42MmPKlCmqz+PGjcO4ceNcapE3yLKMcR+txher9qq2Gzns2kFvsR6ysSInvIPL50Xl6G39OYv6vIQzXwtehIgLn6/ai7xGmbqTS01Io3mlASFlsTvneJ3nhWobxYHlO4/gw2W7Y7anKR5+jckq9lA5nylJz9SgTl4mC63KdxwqR2Z6AC1zs7mPIZyBx2ykSgpmw+lW7RZlfrAkUYkAP7L9YDnu+084P9DZ7ZrGfE8RQIRTeJ1hl9JRxYHyylrd7UrNSzUjlfaSnw8zv1eiNzZpfSF4h6/Sk9UY/Lc56D/he84jCCcpr6o1FTSNQ+zZx5WeqEZ5ZQ3+u2QXDh2v5Kp3o0SCfh6RBVsPmgrihHsokw3qCZc1IXV5EBJlCD14liVeL15IeIkDRv4sauHFeMB/63/bua+lN9lpk5fxrsr3HDnJfV3CHR74YCXze+WjFIkYevTTNfjTp2sxbtpq3PLWErV2juP4krIK/GveTzHbr399MSZ9u4W7HYSzKJ109YTL2lDINLLRDnrna5CZ5uxFCF/gtdmYhJc4YDRAKIUXp5wf9TQvNRZDGVVlBSiMIC5o7/NnK/ca7BmLiNloyc+H8eXqYgDAmj2lOgIu+wQvzd6GAwZRcVMX7+BsMeE0AcWYoptht9Zdh90/f742ZluzhpmOXoNwCnvP3mufFxJe4oDRilgZDVBV44zwohehUiu6rK5DLbzYaBTBxbIdR3DOM98JHWOcYZdNSJPvx8lweuor3mE2ndRqHHadeFaZimJYy3cejfme+oP/sbI49TrPCwkvccDIR07ZYVhmIxHbop6gVBMSS/2ud13RJGaEOHe8vdRQm2GEoQBi8rjCjtv1n0MarY3d5712TynOnzgHM9aW2DoPIYZyQaT3CN0olpiRxh6fSGvrf8Z9tBrfb9wndIzXeV5IeIkDRhOBcjNLeBFBb6DYdfik4nv+cyn7JgUpuI+VPmAcKs1+YCGNgKJVztl53LIs4463l2L7wXKMnrrMxpkIUQIm2tKwds5Zh910kzLkNHb4nw+X7catU5YKHUOalxTAaOWhnDyqOKOJzDCb/4R8XhR/k+bF/9QK1LCSZVk1qai1NrLt532iWj/CjnAXddZuPS2s1mxk/73OqBNejM6lbMdPB45jxKR5MTmvCH/Bo1QhzUuSUnqiGldPXoCJ32ximI3q/3bM58VkmWNV80L4H5FyAtpdtZHStnxeoBZ8H/1kjfWTEUKYaUvdiDaKmI2MzqVsx7iPVmNjyTHcW5eLhkhcvJ4eSHhxiWU7D2PJz0fw4uytWL+3THcf5erWTbOR6nuhs5HPi9+RDUxFZo9L+zxrNdKL3cetXJW9u3gnY0/CSZSaF33/N9nxCKOI5oXHPH68ssbRaxPWSfQMuyS8uESNwgx0qLxKdx9l36l2MdpIdU2BHqvsm4cNfgPhLUaraLMJStsNYqKN7ExwJOd6hkrzoqN6cSPDbqRGm9GZlWOOkalh1+ETeOyztdhJFcrjht2eQBl2kxSVM6TBgKF8qZ3K82I2OB0pZ5RBZ3DeX2dbOo5wF9nob0HNS6wwY6tZhA/QzflU63yodKTMidG5eC5x4xuL8fbCHbjpzcX2G0TEBcqwm6QoZREjbYjK54UlvAj0ETPh5bIXf+A+l9c2zWTn4PFKvDJnG/YfqzDf2QCjaCOR4wC1gL33aIUqzbxwm0B9xyvMtG9uVJWOmA+M+h9Pv9xRp3H5mTQvCYPXPpFUmNEllALLRzpFGQH1yiheDrsieG3TTHZGv7MMS3ccwYx1JfhszLmG+1VU1yIrgyPFusCKWjuhKLvNHz+NzZIqCnUd7wnpDCmx0Ub2r2P2rMldLvHg0ap4PT+Q5sUlRCsCO/WCO+lYSxOQuyzdcQQAsGrXUQDG6vVb3vrR8BxWi+xpu4mZr5QI1G28w0wTVxsKOV6YUZIkyLKMtXtKdb8nZ39/Yt8p35l2WIWEF5fg0YAYpXa3g5u+CpQp0xsW/XTY8DujnB3mSeq0Pi/OPdvwmUiE8QKzLMtuZNiVAMxavw+/mrxQfwcaNhIOHnMiaV6SFJ6VrHLCcGp14qTZSNskp5yKCeew6rCr/ZpWx4nBrsMn8O7iHais0U8CaObPUhsualW/jwPPPRAAPmMknaO+lZx4rXkhnxeX4DMbKfb3odlIe6aqmhCC6VTe3lcYrLTNegHL58V+k+SYgS28jbQxdrng73NQXStjX1klxl50esz3sonmJdZh1z4BSVIVZ9RCkWvJCUUbJSk8mhelgCOSHZVFjUNlBvRwyqmYUJPmUJEQbYp/Fqw8L25Ai29nqK57vxduO6j7vVHJhwhhh11nfe0ksIszOp0Uj3AfPofdODSEdX1vL5+8iGtenPJ5cdJspD4XmY3cgbVqZVFVE8IL32+NfrazonZKeI5cWzuu0fTlLMaTi5jmRe/JnKgSy4IrSVI0y64eVrtW6QlrOakIPswXOP5+a0l4cQkexzj1Ksl4PxEB11GfF81n0ry4A2vVakRVTQj/XvCzeqONR++0ap8sRC5jcH/Noo14xqVnpm8Qbg5LeLHSLz9evhs9npiJ/5u1WfxgwhHKq9jFVZ0qaWMVEl5cgi/aiF/Nz4voJMRS/cU47JLw4gqZ6WKvYVVNCL2fmoWnNZOMndB7ZzV2etv8vYpLNIxeW7MFEU9hxvlb9E1SRshgC+BW+tYjdcU8//HdFuFjCT7Mnkrfp79lfl/p8XxAwotL8KxwrGZHZSF6Hra/hfpcXnfWZIW5atXh50PlOFYRq9pXP3qxfuC2bEGii7MYabbMwuVrOBx2RfuCLMvMPqw8nZlCLiABm0qOoaK6fqwxKmxLuIuZ5sXrxSwJLy5wvLIGz3690XQ/5TDCNBsJ6OBFzUYi5yafF3eIal5szvB2Mqc6a26UY3wySPHiLEY+Lyo/Op3XtbbW3GFX1MFWltkCuMiCKiQDwyfNU22jekfOYhRmL4rX8wEJLy4wfXUx135mkQFWEE1CxdS7kNkoLohqXoywkznV6WijmFBp0r04Tigk46Y3f8TYD1bWbzMxRWs1L460Q5aZpk+7Xevgcapo7xTPfr0Rnf44A2v3lDKfy8lqcwGnspqEl6QjwBlD5nSNEUA8aoSVJZEcduODaLTRweOVuttDNvqTs3le+LYR1pEkYOuB45i3+QA+XrGnXlAx9XnRxhrF7mSl76SbjHnk8+QPJs/dBgB47ptNts9VSZqX5IM3/l25SnJKbS9ao4Ycdr0nQ8Bh98vVe3H9a/pqdDvzA00uiYUkAQ99uCr6OZL/xSz9Qk1Mht3Yczvt82LlnIS7ZDiQpMXr+YCEFxewUvOBpbYXOZuo+p9XSwR4b+NMVoICmpcJ01m+VNaj1xw3Gzl6NkJLeWUtVu+uL4QYCVtVmY10jospzOjAYw/JsqkATiUC/IUTiTGrHPKdsQoJLy7AKxCoMuwavNvCk5CjZiP1ubyO609WAg69hSLlAbQ4+WhlxDqC09zlLHuOnlR9jrybRoU6I4Qz7Cr20TUbiQq+QJrJgs3u4yfNoLM44WfndfQpCS8uwCvUmql4AfFB/7uN+4X2Z405NF4kFqoQWMFnN235bkfbooUcdp3lwDG131NEha+8y3qOrrUa4QWIFQysOHuLVjEXZeqiHbaOJ9SkBSTb7ySZjZIQXrMRT54X0Zf+pwPlQvszNS8xg1zsPvvKKrD9oNg1CTXR+2pTk2uW48NLSBB2lyods5Ee4Wij+n2e/HI9zvvrbFup+GWZo4q5zef/Dgkvtig9Ua26h+lpkm11mNduBFRV2gWsOOwaWXvcrsjKdNiNqYIS25i+z3wHAFj2xwvRvFHQ0bYRYkT6SlVNyHdZ4XzWnKQj4rBrdqNrNYVbF/10GAAwdfEOjDm/o6Vr/3yoHIfL2eHMesLLsYpqXPfaIozoWmh6DRJ+7XHf+yswd/OB6OeMQMD2O5nUodITJkzA2WefjZycHOTn5+Pyyy/Hpk3mIVpz585F7969kZWVhQ4dOmDy5MluNtNxuDUvyr/r3s72eQ1V+2wqOYbiUrV920lEnItZAwhpX9h8tnIPXpq9Vfc7p8ZlGcC2A8dx+h+/xp8+W+vQWa01hGobxZeoz4tJb9L6vESwU2laloHnTWoQnfPMt1iwVV124J1FO7B2TxkmzqT6RU4jy+pkhErBBQDS0iTbAqHXmhdXhZe5c+dizJgxWLRoEWbNmoWamhoMGzYM5eXGE9327dsxcuRIDBw4ECtWrMAjjzyC++67D9OmTXOzqY7Cm7VW2bkiodLaQy978Qcs+fmIY23TYtdsRPBx//sr8bdvNmHtntKY78pOVuOSF+brpvwXQZblqIC0TdB86DbkcOkuEf8Dvay6SsLRRrEoxyw3TI7HKmpw/euLVeObiM8E9R5+ZFnGta8uwjX/WmT43mU44PPiZFZuK7hqNpoxY4bq81tvvYX8/HwsW7YMgwYN0j1m8uTJaNOmDSZNmgQA6NKlC5YuXYqJEyfiqquucrO5XJysqsX1ry/C4NNb4IELT7d1Lj2fFzOvfacRiZhjdVVaafNRVhH2LVAOKhtLjnnVHFfQjWDxoB2pRL3mhU1NSIZZoEm85EySZ92h9GQ1Fm8PmwMPHK9Efk5WzD7paQHbLgkTrjzT3glsEleH3dLS8KqzWbNmhvssXLgQw4YNU20bPnw4li5diurqWKeyyspKlJWVqf65yccrdmPFzqOY9K1xtVNeiVTP58VKjhg7sLREZlEJBB/K8PVIiKIbt1KWjWvexJuY8gDUdVwlmqTO5EaHtXvsffz4qGjscZb0gGTrno4efCquO6eNgy0SJ27CiyzLGDt2LM477zx069bNcL+SkhIUFBSothUUFKCmpgYHD8aWap8wYQJyc3Oj/4qKihxvuxKePCq8EUKqUOm684okjXMCZqh0jMMu80xONCcpUdqGI2nU3RiKZci+0IDpClE6P7iktAL3/mcFlu04HJ+GJTC1IRkVjHoz9Unq2OdZuesodh9h+9DFTfPi0r5EPb/77yp8tCw2DUJ6mmTrnjZtkGHjaGeIW7TRPffcg9WrV+OHH34w3Tc2wZWsux0Axo8fj7Fjx0Y/l5WVuSrApHMk9+HRvMiyrNG81AkvcZ58hDIt0ghiCWVCsXrNi/M308zfIZ7wFGb8/UerMH/LQXyxai9+fvaSOLUsMRkxaR52HD5h+H29gGzer/TMlOrn5cMX3YdNSgTmbzmI+VtiF/1pgYCtexpvC4EecRFe7r33Xnz++eeYN28eWrduzdy3sLAQJSUlqm379+9Heno6mjdvHrN/MBhEMBi/EF2zAmQAn+YlJEO3iFq8O4WQwy6NIMKs2V2Ky16sF9jT09zVvCQSPx/yl1Oxn9my/zjz++qa2Ay7RGrC0wdkjsSCLHwgu7hrNpJlGffccw8+/vhjfP/992jfvr3pMf3798esWbNU22bOnIk+ffogI8N7VRVPWmUe4UWreZE90rywzUZqJs7cjNv+vTSqWSI7NBtZllWCi/o7a+c0y4jsgzFFd0ikruIuVZxmIyMkSKiorsWtU5boZub1Guo+/PDcq5CsHzKfSLgqvIwZMwZTp07Fe++9h5ycHJSUlKCkpAQnT9ar0cePH4+bbrop+nn06NHYsWMHxo4diw0bNuDNN9/EG2+8gYceesjNpnLDI7zwhL8fOF6p6jyRatBx93lhfKcVTg4cq8S3G/bhf3X5GpQDpR8kcb/x7YbYUg2RW+qGlkSGf56DthkJPk76Ht48Lyw+XLYb3wuWF4kXtFDih2fxXBuyt6Dwg9nIVeHllVdeQWlpKYYMGYKWLVtG/33wwQfRfYqLi7Fz587o5/bt22P69OmYM2cOzjrrLDz55JN44YUXfBEmDdSr/QFj3xaeznP5S/9TTf4RfwVfmY0MtkfrqNCAwqSkrCJmm6vVdX3yOGRZNvRbI8zZeegE7pq6DCt3HeU+promog21dk1JAo7bzDPEA3UD9+FzW7C3fPKB7OKuzwvPgDVlypSYbYMHD8by5ctdaJF9MhTCS3VtCGmBtJh9eCKS9pVVqlZJCeGwq0H5K33Ql/2HTv+XZeDoiSrc+58Vzl/OL9KLDrqJ0ajX6HLXu8uwbm8Zvl5bwu3IzFPbKD8niP2ago5K4tF/1hfXp7J44TvjdBNa/Nuz/QePgBgKybYWFEmveUlG0gP1t8woPXItZ6dQl68P/9/PeV7qj2F/T4QxkmGfn7VZNwLALn7J88Lr8+KD8c+X7DhUH1XEmzOqmsNW3YQR3kqPInngMhslgeaFhBdBlNFG1QbprXmd5lTlASIZduOsemFfjv1DXDWBJAF6K5uQLOPgcePVrxmsQSMk+yPPC0CToR2U9677499wmY9qoknqjPfJzojVEquv69+nRkMNPzzzT8igxhUvfugpJLzYoLpW/+nzmI0ATZK6qNnIR0nqBDo3bz2nVCLeUTe+Gt858rwQBijuXXlVLcZ9tMr0kMj9ZS0oshjCiyT5+xn5uW1+g8ccVCvbMxv5Ybwn4cUGRoXFrJQHiPzpgz4RxehXkNmID71u4GaIoq+fh5/b5jOsDAHRKDaW5iWTrXnxM77u2z6Dy+dFtvdK+mGeIuFFEOUDN/J54S4PoAyVDnljNrKiKo4cQ6shNnorGzfvmF9CpfXyzeg77BJ6aFe1PO9oRFBm9S+W2cjPJiNCDK5oI5tmI3LYTUCUDzziJKfVwPAKL6pQ6QQ0G3lcET0hcTVkWJbhV5FA32HXn231Gu1t4blNPGajRNa87D5yEu//uNN8R4JrXK4N2cywa/lI5yDhRRDlA6+uDeG9xTtx+h+/xvcb90W38ySpi5wt+lc02siBRgogkqSO9X3kPCerarFfJ79JKqI3kdiVXVgr5OpaGdsOsNPIx4uYPC+kpbOMXi0iLdF+xbjNwXS2z4vf+cPHa7xuQkLAW56GNC8phlbz8sgn4Rfqrqn1eWm4NS8KIYdVfNJVLCSpi8yfqjwvddvOeeZbnPPMdyguZVeuTQX0uoGb2qr1xWX4cXviVGj2fvjzJ9Z8XuqijRjSizJHFZG88Gh37Sap88PLS8KLICqfl5r6T5UK0xFvtJFSyImGSsfbbMT4zuwd0Pv+WF2WzkU/HbLeqCRBP9ooNTQQMT4vqfGzHcHKAiZyf1lDj9lqmZ5RcsBtNrITbWT5SOeIS1XpZMUoMRRvkroKpcATMRs5IE42aZCBoyeq7Z/IDBrsmOiajZAak8TJ6lrV5xT4yY5hZWKIOuwybnQ6wya992gFpi7aYeHKhN/g0fx/vmpvzDsqApmNEhCltGokvPBqXk5W1dcScdJs1LVVY+592VWlKUmdFeprP8V+lyr3bPcRtdkwVTROTmBlCIi8q6x3No1hNnrzf9sNoyf9xMkq6xNuqhDSeYx6fWrW+n2xGznxgexCwosoyqHBruZFKflGQ6Ud6BVpAuobKwl2JZ2vKdQyzO/+uwqd/vQ1dh85ob+DXYfdBL3Nuq9Egv4W93HHbMTSvCQKN7/5o9dN8D3xWCCR5iURUYU36+/C65SpzNAbOcaJPC8igxSztpHJsazVdKoutKct3w1ZBt5euENXA2fVYVeWZXy0bLeq7k2i4/3w508saV44stS54U93fqcWjp+TxY8/J45DulfopiVw+Bo+kF1IeBFFrxK0Fl6zkTITbyhqNrLRuDqEhBfGd6YOu9xXSU30uoFVH//vNuzHQx+ap4n3K1W1IVs1nVIJS9FGmv/rEXBB8/LWb85BZpo308ieoycx5t3lWLbjiCfX9yt6Y4zTUax+yNFEwosgellxtfCWB6hR7OdkVel0l0MiIx137Z7S6DbK46FGgv49sap52VBcZq9BHnPpCz+gz1Pf4ueD5dFtfhgA/YiV2xJZ/LAWTm6Zjbx6jA+8vwJfrSnGVa8s8KYBPiVVkoeS8CKIUniRZf0Bgbfz1Co8qyICjxPji5DPiw2H3VveWlK/b4q8MCLoal5ka5ktE32ej/h3zVhX4nFL/I8V/7Go1Yixj8i4IIJX/g/bFYIwUY+eRSDBhw9dSHixQUiWkZkeewt5HaaUnvORY+Lu88Lo1kY/w6hmz67DJxT7cDchOZGgexOs3he7WopebZrYOt4p0iQJoZCMl2Zvxdb9/sgG7DesaV7C/7caKm0HrwTrlB9jDNAbnxN98aMHCS+CKLtFbUhGUEd44TUbbTtQv3Koryptv5cJrYQ0ux49UYVBz83Gs19vNFzF3fLWEizcpk5CJ8syBj43W6yhSY5ukjqPzGvxLvhphCQBX6zei799s8nrpvgWaz4vdWYjlsOuS33AK81LqqQdECUeZiM/jCYkvAiilGplGbqaF95QaSX1hRmtty2CSE4N7eUueeEH7Dx8ApPnbmOeZ9Qbi9XXFGlgiuBkbSO784MfQhsB4KeD5bj//ZVeN8PXWFrAcPQrt4QXr7pWqvh2mPHYZ2tx//srUFUTwoSvN8QsLIHkTGVBGXYFUWleZFm32JmVhFxOlgeorOFPNqW93J6jfDWJajQjBy2C1EiQHK1tZHfw8Yvw8t5iqgzsBhFBmfUeJpvmhRIfhh20314YzozcKJiOd43eL3+8/o5CmhdBlO9LSJbRMJiu+C78Ja/ZSO+8TpiNRIQXFqxfEdtMGkiUSJKztY3sdgu/mI0Ic6zleQn/n2VKcdrnZdyITgCc0RZbgWQX9RiTaj5kJLwIo8jNEpLRJDsj+vl4ZTjdv50s206sYkTSfDNX9KxVnE9W8n5GbyKxWore7t12I8cH4Q62HHYZ+zgpwBY1y8bdQzoC8FDz4slV/YVyMcR6DMn49pPwYoOQDAQz6m/hkfJwMUQ76kwn8j3xJskDrIdKawcsWgWpkQCD0dUjzUsyjl5JiqVQaZibjdzK/+Sdz0v9j7Wi7U4GlL+a1W+Sca1JwosgWrOR8vPhE1UArDnsRnBiFfOr3q2592UKL4yfoU0Zod01NYcSNXr3wCufFzIbJQ5umY3cyvPiVbJB5U/tP+E73PefFZ60w0uU9yAZBRQWJLwIog2VVg4Wh8sro9utYle937ppNnKy+P2wWZOisvaSFtK8mKOnAUv1aCPCHeZuPoCS0grmPm6Zer2Si5Vj7/5jlfh81V5vGuIhSu0422zkdHkAR09nCRJeBNFm2FV+jpiN7OQfsDvASJJzHauyxrj8vLadVQ45CScLRg67VvvG6t2l5jsxIOElcbDypLYfLEe/Cd8xTdbJF23kyWV9hUrzQmYjgoVS0q3VpHqPaFxCthx2rR8LhDuwiJTN6tSsqCXtcTdq8r4Q+oOrDGsmNburSpcsBoQLiJhhtAIJS+nrZLSRcozxzmGXpBclqeawS3leBNH6vCgFlXHTVqNBMC3q8yJJ4qsDu/bjgATHemplNUPzYjIQpnoOBgmSQZI6b+4LaV78xfQ1xShoHETvts1ivhN5UgEJUL6lzDwvSeewG7utpjaEFbuO4sxTcpGVEZuDK9lI5WGW1mM2CIVkbDugjq2/570VUV8HKysdu5NMQBKzbrKEJZbmhSZDNka3x6vBhhx2/cPGkjLc/e5yXPXKQv0dBB5VjO8ZQxuRLLWN9pWFfXv0FgLPz9qMqycvxNj/roxvozyCV/uUjBXcSXgRRNlV/vHdFuw/VhmzT32qfyvCi9WW1SGJdVTWnkzhhSZDU/QLWHpU2ygJB69EZcu++gXPJS/Mx5HyKtX3YpoXfrORew678e1bfZ/5Dh8u3aX7W1+b/xMAYPqa1Khero42Sq13nIQXQZQTklE0TmSzJc2LTaHAQasR02xEsos5eoOrHX8oO5Cw6R+U1eTX7S3Di7O3qr634/PCUu0lk8PuhK83xv2afoR3KZSMbz8JLy4QMRtZGSzsjgOSJAmdw6rDrtkqLoVNsVH0tCxe3RfSvPiH8qoa1ec3ftiOMe8ut+QPpX2srDMkU5K67BTwZ1GyYNtBPPHFelRoFpSqDLusEzj8jPxQ6NFV4WXevHm47LLL0KpVK0iShE8//ZS5/5w5c+omX/W/jRv9I2XzjC8Rs5EV4cW+z4vY/hKAl67vpfud9kVRHUeTIRMJ+n1Fm9gwXiRitFG1nTobPuZEVex79dWaYmzadwyAXbNR/JPUeaF5aZCpL7xof/6Jqhq8+cN27Dp8Ig6tco/rX1uMN/+3Ha/O+0m1XZVhN8WijVwd0srLy9GjRw+8+OKLQsdt2rQJxcXF0X+nnXaaSy0Uh8dnoSaqeRG/vfEPlZZwTvvYiAcA+HddtVI9yAHUBEnSNRsdPF6JE5qVdzxQTjDBdH9KMscra6LF5V6b9xPOeGwGlu047HGrnOekjvAC1KdaEJEFtO8hszyAa2YjV07LJNtAeNEyYfpGPPHlelzywnyXWxQfftIEiKjzvKQWroZKX3zxxbj44ouFj8vPz0eTJk2cb1CciExOVgYLHsFjwKnNsWDbIf3jBZPUSbCm9jX9aWQ3gt5NeG7GJg/aoZ7kerVpioU/6fcfLxnytzk4eLwSn9w9AE9P3wAAeHjaGnw7drDHLXMWPc0LoKgsLzANxbi8MPd1yWzEaG9aQHKl7pCR2Uh7pR+2HgQAlFXEf8HgBjXae8npsJuMmnJfLsF69uyJli1bYujQoZg9ezZz38rKSpSVlan+uQmPyr/sZPhFcdLnRXkuZs0h4VBpaxI7OYCyMTIbeYVy4sr0oebln99twcHj4ci9Wev3RbcnYzc7Wa0/kf7ypf8BEFtMiJiNvPB5cUvbY2Q2spPdPBHQCoKq8gDxbozH+GoUa9myJV599VVMmzYNH3/8MTp16oShQ4di3rx5hsdMmDABubm50X9FRUWutpHn3ThWES4T4KTPi/JUTNumhXAjK1I55Xkxx08DqbIvZviwxPTfZ22O/q28a8nYz4w0L1Y0FDH3hxUq7UG0UUaaO1OMkdnIR6+cK2g1L7y/NwlfI39l2O3UqRM6deoU/dy/f3/s2rULEydOxKBBg3SPGT9+PMaOHRv9XFZW5qoAw9NXjldaNxsZHRIeIMyvHtak1J/kxet7YuBpLdDjLzONjrAksVP0ijkuaMsto5y4/O6vpM1dsW5vKY6eqMa5HfO8a5SDlFcaO8LLsmwrVNoLsxHLtc8tbU92Bt/UlWyZvmM1L/WQw67P6NevH7Zs2WL4fTAYROPGjVX/3ITnZYjkf9GbJNo2b4BmDTMNjzUyxyjPZVY9VPl9MD0NudkZxvsL+sjwtAGguiOS5C/Ni/J5pbu0GnYKrSr8khd+wA2vL074iJEItYxkP7IsNtFo30O9SubRfQXOK3JdllCU7lKEk5HZKNmJ1byoxBfD45z2efHD2tXfoxiAFStWoGXLll43I4rIdKQnvGSmBZgCkFEn4101BTQ+LGban7CVSbwniqzeF2w9iI+X7xa+RsLjH9lFpSlzyw/BMVSal/q/dx1JDuGFhQx7Pi+sLufWhMOaGN0yUbpljvITNbWhmMg0reDLq3lJRlw1Gx0/fhxbt9Znj9y+fTtWrlyJZs2aoU2bNhg/fjz27NmDt99+GwAwadIktGvXDl27dkVVVRWmTp2KadOmYdq0aW420zWMJniWOcFoXuHtmJHcOGZtUJ3XUrQR/0HXvx6uON21VS46FeaIXywBkSD5SXZJLLOR4u9k9HlhTfZhsxH/ubTPkuU341ZiMdZZve5rfnoHRRnxj/nYuv84Vv15WHRbjNmI1+fFyYb5BFeFl6VLl+L888+Pfo74ptx8882YMmUKiouLsXPnzuj3VVVVeOihh7Bnzx5kZ2eja9eu+OqrrzBy5Eg3mymGwNtgtMJlaV6MHXYVZiOmelAt6PAMHpZCpS0MSnuPnkwd4cVnZiNl/8nwecY65fvhcznLEpU1DJ8XiAkZ2nfXi8R+rGeUmQIaEreI5Dxasr0+19Ginw5j/Mer8cwVZ0KSJG7zvH9GIudwVXgZMmQIc6KeMmWK6vO4ceMwbtw4N5tkGxFfDiPBge1Up79dOUix2qBVpJhqXiyux8wmFR/N23FD29f9dA+U/cAtJ0o3SLb8FF+tLsb/trJz7AhpXjQ7MzUvLt1KpWB8YZcC7D16EuuLwykr3OprqexT958fd+H6c9rizNa5ahOrd03yBBKLBRGZkPQEB7OgIR6fF5a9N6CpbcRjNrIyQViJNkqWAWf/sQrsK6uI2c7IH+U56lBpa6/9DX3bAADO79TCkTYZoXzHkknzIssyxry33GQfe+UBjIrFuomyDed1bI5GWfVrYrccdlOdk3WlW5RP20/jTTygniWIXYddgG1OYOV5GXP+qfhV79boXGgcURU+XMznxZrmxYLwovjZ//h2C8Z/vDrhQhmra0M45+nv0PeZ72LS/Gufq1/NRlb9EJ6+4kxseGIEzjwl16lm6ZKMPi8vfLcFfZ761nQ/GbKQikTEbOSew67yb7Um1y2HXd5Xy0evoGX0nltkbEllnxcSXgQReRmMVh3WzEYSfj+8MyZe3UMoVNpMQ6Ldnxe7C6r/+3Yz/vPjrmgxukShvLJeYOn7zHeq72LqjPho4FT2KzvRRtmZaa6bcpT3cemOI65eK148P2szDpVXme4nqnnRCqJVjErwbj03pYAZ0Pjc+T0sP1E5VlGD5TuPcC+QnB6K/CAMUc9yESOnVrP0/vrb+Y7XalJMV9mStSgEsxUx62VRaltqPFBz20FpGjpWYax5kSR/mcmc9HlxWxviJ42VF9gJla4ycdh97qruwu3JymBPE6omSGovOpag3CIniI75jYTbA1Afuf3tpbjy5QX4cKk3KSi6tnJX+8oDCS+CiExIhtFGLIdbI80Lp4AREAyVZl2Thdl531+yC1+vKVZti4w3Srt8IjmPAuxIMe1Xfh1fmzcM2jre7cU0Kxon2RH2edG8h6x7JwHIyzFOkGnEIyO74Jx2+pXnAW0kpBqWf5UE6z5NPMLLhuIyHDlhru1KZD5eUS+8sG6Jk8Uxp983EG2aN3DsfFYh4UUQuw67gFmeF3PNCwut5sWthGRmKuhVu47irnfVzomRn620y/s+YZoGpkZJmRnWZ74akiTh37eegxev72k7nDZNYzOMOPI6RSXD9JHsyBArD6B9fdhmI2v9smmDTDx/TQ/medXXqP9stjixqsXjGYcv/sf8GO1ossE7HzkpvJzRyt0s9rz4qrZRIiDksGv0YloQXngHHe1uPPlYLGleLBwT0VooTUWJFo3AWvH5qZaRFgnA4NPDUUIrdx0FADQKpkfrcImgXUw7ne3USHhxK8ma3xDyedGajVjCi9U6ZgGJKWRoc1CphBeX3u9UNxtF4L0PNYySFIlKYs0cfkDgpUkzmOHZ0UYG2xVPip3nRTt4cAgvLvi8sKhWvEheZ+AUhvH4tc/VT8oXZVvOKmqCL+89DzMf1C92aoZW8+L07zSagP3kQ+QWsliwUcx7yNJaSZK191abfgFQC1jaivfK8YQVbWS1PYA1X7kaDxL4WUXW+M8Z7weu/ZzUvPgFEl4EEekCeoKDWdp4rtpGpg67Su9/k8FBcLCMwCt06PmIKM0WibaAYo0Bbvi8dMhraP8kOnQ7JReNGQU7WWjnI6cdeHkG2u0HyzHuo1XYfrDc0Wt7jd0MuyyH3bCPifizEtG8iEQbSZAsRy1uqcs+K8LA52YnTGoG/lBwWfG38X6NgslnZCHhxUUMM+xa0bxwF2aUhGzOgLWwt2AGX1VX5U+N/KlcNfl9NV1RXasKj9bTmm3Zdwy3v70U6/aUqrY7MU5aKcOgh95ZrA7kaZoJyWnlGY8q/OY3f8R/l+7G9a8tcvbiHnOsolrohdSOMZXVbO2ClWeVFmA3SdKYjZRkMC5oR/MSMX2KUFxaYRqN5Rd430ze/XLrFirXneOsf5qXkPAiiJDDrsGLyat5GXhaXv12zmtqL8kXbSQ+gPAeocoAWfehKoE0L2c9MRNd//wNKuoyWsYWRpNx4xuLMWv9vmgBSsC5UGkrmYz10HvGWZwCqBZtm+KteXnx+y3YeThcYbq4NDbTcSLTf8L3tjLsMidnCZaLsHJXjpa037GjjeLt2K4UribP3Yb/Lt0V1+vzwp2/hddht27HIS5nx44nJLwIIrJaNda8GB+jPKT/qc2jf/O+49rdIhPNe7f35T6GB9678Nt/L4nZpta8+BdZllFRt5KNTJbaQUWWgX1llQbH22+DY5oXndNkpAXw5b3nCZ9Law51egIyG7gnztzs6PX8hpDPi0iSOrDNP0aEzUbqbeNGdI7+rRRQtAIJS/MrSbHnjRc/HyzHs19vxLiPVnvTABOOKBIassyIvAukiJuhU4shP5B8hjCXEZmPLGWu5fB5YWfolVSTZsTbf8CpeQZHWGsnrxA3Z9MB5VEAtD4v/hVf9JqmVQoYTbSy7Ixg5tTgbnSa0wvEq3xrJ0ynx8NkdC4UQcTnRds/mHleLJpp0jSalw/u6Ie+HeoXVirhRVMeIDPdJMGdcGvsEZnslVF2siwWnu42G0vKMGLSfK591a8KKwdV+LuEC5BgQJoXQUTmWr19zd4RwyR1iu1FzYwTBEmSekLlcYiz8uLakTlUwov107iOXttizEas4x34cY4NNgbP2ErtGbdz8xhFdaZKqLQI2pU0M9oI1gRNrYakeSN1ksPMdKXPi/oaLOHFjs+LVSLvpPK98puw/J/FO1Wf95aeNNyXd4yJ/ESnNLl+gIQXQdzu5jyal2vPLsLowacanEGK2jcBf+VZiBxSoxgsfKx4Uaf719mm9zlCeKVj/8e5PbhbEVxjNC9ONaaOWj93ijgg8ki0z8/s1llz2DXzeVFqXtT9IcjyefFAeImgFF5qfCa8aHn0k7WG3/FqriMap0RLCsqChBdBRMwcds1G6hj++u0ZaQH84eLO0CMgqdvoVg44O0miqmvUZqOa2hAOHdf3G/ESfbNRrM+L0bF+0rw4OWTF+rzE7tO5UNwcFcFvK+F4I/LMRbqHZOJ4a9we9limFF4CkjoVBFPzYrEorBMox1m/CS8iz0gvIEKPqObFR+Yxu5Dw4jGFjbNUn41DpfnOF5AkKAMO3NK8fLNun/AxkXerWjFYVNaEcM2ri9D7qW+xqcRfFab1nOFio42MjnVGS+dctJEjpwEQOwDqmXPyGlmvn2SozfK1kdEbRAQdi8FGCEhqR19tX1IKKGGztf53RueOJ7pmowQrDqtEuVBlaSxDdQ8liRQvJLyIsnj7Ye59eVbeo/q3VX3mSlLHIMbnxUedNVqYUaF5ufSfP2DZjiMAgI+W+StsUVfzonEpYDrsOqB6cWpsd9JfhEfzYodU17yIIDL5W3bY1Yk2UqINh1b2+0yT0hHxrg4SEYCVi4LqBE6dr3xTWK9NZJwin5cUZe/Rk5i1XlzjIEJAAnq1aQIA+EWPVtGid2OHnc51vCTVS9nhz/7rrEZ1NvymvuUxGxkJLyHZGT2BY2YjB7uB1Tbdd0FH9Kzr2yxSXXgRkXlFJiOrodIBiV0VKTNNqZVRRzuykllqs4HHg0jbnvpqfXRbIve3kMp/kKF5qfvKf7OBdShUWoBIrg8n0Y4lgYCED0cPwPHKGuRmZ+Cpy7vh98M7oUkDvlL2kiQJFQiMpypejoZK61/Tb4OIrtlIM0Dc894Kg2OT1+dF2yapbpvZ88vJysBdg0/FHe8sY+6X4v66Qu+kWfc4Lb+RKpW+1VIgyuO0p9DmeVG2n6V5sRr9ZAcZ4XFmpmIR6rdFkwjKd4VdNDb8nQ/XspYhzYuLGPWl0/IbAQDyGmXGrDwCUniwiKRzliSJW3ABgJxgulC0hhca02qDLKB+G0T0zUbqjXM3H4jdqe5gZ/K86I82vzyrldB5XNW8SBI+v+dc0+OqakNcK3+j/psqodJOZPGOkKusXyU5I7xo0UYbKccUdqi0NU2Q0yRSwUYtyq7CWjzU9ynv77dTkPDiAW/ecjauO6cNPrizf2y1Vpsv80PDO8VMsCziGZYaDZU20rz4zHFOFSotRbbxHRvWvNj/PUaal/wc6w6xdjHSvJhRWV3LNXmy+q9I305URLqN2Xih0phY9HkJaMw72mtmpKujjZTvDTvaKP4+ebIsx7yXflg0HS6virZL5BEpfwu7aKy4w25Rs2z+nT2AhBcXMVLjFTVrgAlXnolTWzSKkYPtrEQ65jdCXqMgBnRsjg55DTHyzELzNsbxxY1cyaj+it8c5/TuDK9pyymZMGgw+KcJejo2yHTOQhwjvHBOipU19jQvZt8lC06ajfq0axb9OywsWPN5UV5He02VzwvU741R/43sHPdoI8S+116bqxf/dAi9npxlaIJmoQ6VNn9vRBbHT19+Ju69oKNwm+IFCS8uojfQmnUeJ1YiwfQ0fDt2MF6+obfpvqKTgRNJjozUtF4PIlp4HHYNj4XsiAATTA/oFlMTzYzbu21Tw+/eu70vLu5WiKYNMgz3URKreeGrUVNZE+JaVbL6gd/6iBsImY0MbvwrN/TC87/ugV/1bh3dFs7zot7vhr5t8PhlZ2DyjfVjxQvX9Yy5RnpaAL/u0xoXdytEG02Gb63ZiDfaSEL8Awoqq0N4dd5Pqm1GZux4EWnPV2uKAYiZR5V9hSfPi8jdbtogE78b1kngiPhCwosAIoPK1/cP5NJqxDjsOvQy80YhiE4GToTaGV3SD+pbJXorGV7hJSQ74wwtAzhbsXqOoM3fc0579T4tc9X5g1o1MVYBDzg1D6/c2BstOE1Ren4WPJNQZU0t135G741TAqHfEfmJRvezQ4tGuLJXa9Wz0pppxpx/Kv506Rm45dz2aJdXL5C01QgnkTHpuV/1wCs39o41G6kEFE1tNWaG3fgnqXtp9lb87ZtNqm2JLBArxyPWQtSKSSreYeyi+Lx5icdVvVrjX6N6o0vLxuBx4dBK2XZeZis+FqKZcu0kTTNrn998XvSaK2I2cmqi1dOyaKv1avcYfHoLnMIQWPTgHcP1zEYNMo1DYiOEzUbm5091s5GI9GIkG0SeEcun7sperZFVF8qsXDRpn6WZP1OMw66yPAmrqjTi7/OyfOeRmG1eL5pinxH/sepoo/q/szLUHaNe88J/cr8XcSThRQCeTjXo9DwM7xr2NYm35sXKKyi66nCiQxvledl64Dj++Oka7HIhJN0Kencm3mYjQD9Lspn5TpaBE1U1zH209FdUCmah57DbMjcb9w89TbVda6rq3bYp1+DJ0uIn8iqZFzGfF/37qdc/tHWHjP5uEFT7R5m989rCjMpHlGGyfI+3z4ve+2sUQOAVIndE2VeUi0Nt4kArodJ+iARjQcKLwygfuJX6P3F/mQXNvXZll637j+OZ6RsNv5u6aCduf3upvYs4hK7ZiPd+CZiNLunekvl9ho7To9mEIkPGsQox4cWoXpbZtSNd9sGL1IkUX7upj+rzNX2KuO4JM19FKggvdT9x8o29TP2QjMYLvf7BSlKnzL3UQJNYzmxMiqltpHh+rH7KcvTu0rIx85pW0etaRoup+GFHm13/t1Kw195XKwspEl5SDNFS61r7cbw1daJqeDuaF1kG/u/bzab7bVTUODpWUY1v1pWgorrW8nWtonx8kdvEe79k8A8YL13fCzdpykQoydC559qVld44I6oObxjki0iKqW1kMMg1a6jOT8Tyf1Bi9N5ImorpEZwISXeaoyeqsGZ3qaVjFQH6HA7+AsKLRlhQnruypv79yhY0G2UphJ20gLowo6ljedxDpWO3JbI2T9ly5c8wrpFnz2z05b3ncR/vNiS8OIzyefNoXrTdQ8T7fu7vh2DQ6fWRKM0EktlF8MJsZIZywBs9dRnufGcZnvhyPeMId1CpZOv+z7vylwWT1LHuqlZQAWJ9CYLp5j4nPHRo0dB0H7e7gKjmxY9zz4Bnv8dlL/6AHwVqoUVQOlea3Wuj743MikaZcisV9ca0EUJmbejeOheX9WiFoZ3zcU77ZhqfF+MphvXc3OpiLLPRt+v3YdBzs6O11uKFU36Oyr+NhBSRa2n9G9s2b4Bup+SKNdBFSHhxGGWn4dO8sD+zaNu8Id6+9Ry8dcvZOKd9M/z91z34D65D1LRlzyeH71rKldz/th4CAPx3iQdFG3U0L7wTZUiGkBOSkdAqy/pOj9rJqUfdBKI8zgpTbjkHPYqaMPdxW4BlJqmzEb4eT05UhTUZszftFz5WHdbKvtdGzyISFcgKCFD+fWqLRrrblecyIpiehn9e1xNv3HI2GgbT1dFGjGNrakPOlF4XQFd4qbvht729FDsPn8CoNxbj2/X7sL+swvX2fLh0FxZtO6TaJuSwq/hbL6mmFpFza/f123vmqvAyb948XHbZZWjVqhUkScKnn35qeszcuXPRu3dvZGVloUOHDpg8ebKbTRSC59mphBed/WM0LYzjeTm/cz7+e2d/tG1uvmrWIuqsFg/NS5ZOMbd4vjYRFbrKbFTXAm6zkWCoNG/69Qh6Trz/VOTnkAE0rFP/d2/Nv1pq07wBXr6hF3OfWLMR9+m53iHRaCPtoHrgWCUu/ed8vLNoB3/DBFCaWNyg3rnS/MYa7RMRGlgBAUrBpkVOEN//bjB+fHRozDlFIwyV7w0r2sioxpmb6HWtz1buUX0+UVWL295eivMnznG1Lev2luL3H63GsUox3zQlRtFGRv3CTrSRz2QXd4WX8vJy9OjRAy+++CLX/tu3b8fIkSMxcOBArFixAo888gjuu+8+TJs2zc1mOopyTuGLNvLW5yWumhfOS+ll5YyXX8O/F/yMTn+cgTmb9ms8+cP/5zYbCUYbsQYVHrNRzPVl4KO7BuDXfVqrEpDx0EqTI0aLdiXudM0hI/9JGbLu/dfe55fnbMXaPWX406drHW0XAGw/WI5Of5yBhz9a7fi5I0SFF5gLhkZWGSNtCev97dCiEfJzYp+96IJF+a7qCdkRvHCU1RvvvlxdrLtveZW7QuruIyd1t1tN3BdSmY309xEyG6Wy8HLxxRfjqaeewpVXXsm1/+TJk9GmTRtMmjQJXbp0wW233YZbb70VEydOdLOZ3PA8eDOz0Zkam6FbSep4EfV5MZs0WfB2fj3NS7z48+frAAAPfrBSt2Ird6i0LJpszHi7vtlI/erqXatLy8Z47lc9mAnq9K8p4blfdUfXVvoRHzp1GR3FUPMi6/dX7TNxMyIpkg31g6XumTEjc7o2tFkPo/EisgCIyQdkYDZiIfp8leU/WGYjlubFrWHQT/Ov08KA8nxGCwqR20rRRgIsXLgQw4YNU20bPnw4li5diurqat1jKisrUVZWpvrnJWqzUX1v+uaBQRhz/ql49NIuqv1jHXbdbF0swtFGcWhgtofCS4QGmemauiHh/4sIeyLaItZd1Uuxrp2URCqP8/DrPkX46r6ByGsUe15biQo59mGWB9A1G6k/Z5vUcVq47RDueHspdh+xkk/I/elPmZPD7FYbrdIjfaZlbjZu7NcGtw9sj6yMNGtVpQUPOnS8Kvo3y1/Gk7T8Bo/PbVOgPkZRddZQCvEd8xvp7iOi1dE+upTyeRGlpKQEBQUFqm0FBQWoqanBwYMHdY+ZMGECcnNzo/+Kiopcax/Ps1Oq2pQrwE6FOfj98M5onKXJ2+CBdHu+olaO6CrVTnkA3itps0MCden2NQ+goroWk+duw9b9x2L2t0uDzDSDyBaXoo0Yt1Vv9arddkPfNurruzjJagdAvaa7cX0Z+gKhVthpaJLt97rXFmHm+n0xNW7s8vr8n/DXGRttmzjrzUaSqUnOSLBQPqOnLj8Tj15yBgBtqLR5W27q35Y7xD1C6cn6hSZb8+IPsxGgbnOionwPLjYoyst65trUBlqzEQkvJmgHxvqwQf27Pn78eJSWlkb/7doV/6gUI1Vsp8Ic82M1n+PdP0RzgbBWYVxFGzkuZxT2+7sPV6k+v/j9Vjz79UZc+Pw885MKos0yaiXaSMjnhXFf9ZLUKSeUv/2qe6ypzcV+pBUu4yV/y7J+9l2tsKB8dtrvyirqJylmxWNGG4x46qsNeGXONtz97nLh8yqJ9jEOzYvoWkL0Uf3m3PaCR6hh3WMvMtsavb+lJ+IvvLhlNrqxXxvjUGmDY1s3zY55V7QLVZ/JLv4SXgoLC1FSUqLatn//fqSnp6N5c/3U5cFgEI0bN1b9izfDz6iXcpWd5uGLO+OOQR2YiX20fSzeHcRJzYvZQMu7ItXTvADAx8vVUQErdx3lOp8VGmSkqSu2RqKNRBx2Ba5ndOtkWT/FujJxXbwr8+ZkZajCsvW0A0YaAztaCRmygc+L+rNS86LMXwIAFQonTD3nVC2Hy6tw5ztL8f3Gfdzt/HptiflODCLvZEAyd4UW1YQqxyfWo/jndT3xl190Rfs88QjGCJnpAaSnBfDlvefh0ZFdYr6vCYm+JfYxup4Xmhenf7lSY2fssGvcX7Tt0S5UeZNYxgtfCS/9+/fHrFmzVNtmzpyJPn36ICODnSY7HhgNvGkK/wOlqq1xVgYeGdmFmdhHOzzFWzUnnmHX+DuzoZb3SkFOnxc7zsNmNAymqZ5FveaF32FXMNzIEL3fqexnXrjVjR7cof76cdS86N1/7TZlhthyTRiqyo+Jo0c+M30Dvlm3D7dOWRptg37bnHtva6OTkLlgKupUybv/ZT1a4eYB7YTOHeHxy8ImqpevD4fcdzslF0WaStVeYfSYPBFejLqMxfdJ5Ssl6LArSbHtifSVyTf2Rsf8RnjpenYKhXjjqih1/PhxbN26Nfp5+/btWLlyJZo1a4Y2bdpg/Pjx2LNnD95++20AwOjRo/Hiiy9i7NixuP3227Fw4UK88cYb+M9//uNmM7kxWnQrJVRhNa7HTlGiZud4OOwGOW3srDBMu8Q47Nb9nz9UWjDaiDFi6Wkb0jWVfLV0LNB32EtkZPBFGykn/BNVtVDqbFXaNI4HtI8zUZmTKeYj5+J51QTdUSAp9ndrqLnl3Pa49pw2KlNmE5MaTVpcizYy+M0nPSg/4rTWSZncUDRJnQRJx2wU/v+IboUY0U3fh8ZLXNW8LF26FD179kTPnj0BAGPHjkXPnj3x2GOPAQCKi4uxc+fO6P7t27fH9OnTMWfOHJx11ll48skn8cILL+Cqq65ys5ncGHW2NBsqfO3eBY3NVdlOIiosmRVacwJeVbi2bkpZRTVu+/cSfL5qr6XrKiegBplpqpc5cp94zfSysM+L8XedCnNwlibrbbqqz9Vv/+TuAbhv6Gn47Xn2fBXMcDq3Cw+yLBvUNqr/++DxSpW25XiM5kWhTbPSBoOjRDWYzGtEJyHJ9J1qYBJZpSVeT03rg9W3fTP85Rdd8eHo/nFqgT5G450Tj0/UBG90TavvltI/1EqSOjOzkd9wVfMyZMgQpjp1ypQpMdsGDx6M5cvtOby5hVHfVKpi7TzwT+4e4KpdMT0goSYk49yOeZi96QAAZ2sbmf50zkuxTjP073Pwxb3noUFmekwUxEvfb8W3G/bj2w378QuFTwYvJ6rqJ7qsjDTdwoz8A5SYPZ/1mzPSAvh0zLm4/e2lmLU+7HsRVPgFKR2ce7Zpip5tmnJf14yBp7XAJyv2ID8nqG6vykndvM9HdrE7R+iNJ1v2HcflL/0PN/Zri799s0n13QlGojGeCYt3UnMy35pIqHROlth4ofJ5iaO/iSRJls1QTmIovNg87xs/bMcL323B+3f0c60ithmqDLsG+0gS8OfLzsBfvtCpFae5CfHIpm4HX/m8+B2jjq8cYERt0MpqyTzRSXaYO+58PP/rHrhFMYgIO+wyfp+5zwvftVjX2HagPJoRU1tt+VB5ld4h3CgnoPCLG2s44tVUhULOaV4i/H54J2SkSRg9+FR0bNEI1/Qpwk3922Kwojin0zzxy654dGQXfDLmXNV2owJ/RjiR8CpsNord/uina1BcWhEjuACxz0vPCZt9TePjWdexQ9RsBPN3ypbw4rPoESVuafaMhju7PktPfrkepSerMf7jNdzHGLq8OODzYmRRlyTgunPa6H7HMr/6EX+5D/scow6u3CzqhqFMQe12crZTmmTjyl6tVdvEHXbtRBvxXcPsHkYGYKcddpkTXd3f3LWNHCoPoDzF6QU5WP/EiGi5gL/+qjv/BSySk5WB2wd1iNmubC9f5mn7bdl95CQ6toj15WFpV7SaRZU4alPzsr+sAp+v2ourexepfEnsIhIqnZ0haDZSnM8Psku8BSi3rydW58vZxtQLvcb5gcImJb3tjjYlLpDmRQDjrOX1X4iuMI9V1JsqvJB0nTQbmf32kMy32jW7DxF/D9HkWWbECC86f3M77MrOlAfQolfnyAtENS+RZ1rUVKxUgZI/fbpWNxMqswq1QP9eu6cUr87bFq52bIDybDe8vhhPfbUBv/twlaMlCaK+Cxz6B9EhQyW8+Fj14t5Q6LzPy9sLf7Z+sIOoygMYOubyLZQSAX+MhAkCz/gkKrwcr/Q2s6OThRnNXV54zUbs7yMaF63ZyO54p/Vx0QuV5q2EK8O58gB+RLS9kf075ueYVq1mUV4ZK7ywEi1qNWWsZ3LpP3/AM9M3clej3rL/OADg2w37sHaPc2VJahXqfzNEn4Pa58UbpvzmbDQKpuMFRRV0LcrH9MjIzo5d29BsZPFulFVU47HP1lk61thh1xpqXyn9s6QFDDQvjPb4FRJeBOCZ6EUXxnqDcTxxUvNi9tbxvhxTF+3E/mPGIapGmhcnnUFDsqwxG4U/aPOGGJ/LZmPqSAShhkdjqJw0R57Z0vK1Js/dFrONpfWIMRvpPFMtG4rLFPuovzN6rr//aJVhG0QJKXxezBZDdqMbvWBIp3ys/vMwbqf6wafnO3ZtHtO/CLUci5mt+4/jxtcXY/FPh9TXdDxUWukrpU92Zprhd35L/28GCS8CGK7aVOo662YjLxDVdtvSvAi8HA9/tNrwu7Q6pxiucgQCaO+FsrmR78oZ/hWqYwV9XhJr2DAuicGzvx3W7CmN2cbSvLAGZC6fF84n42Sel2i+Do6y0qL31S8OuyKZgZ00IRk77Fo9n/mBd76zFD9sPYhrXl1kes2Vu47acNgN/58VpdYgI8331aJ5IeGFk+raEP746VrT/URDpSP5OIZ3LTDZ0x8wM+ya/HYZ/FEEPx0sN/yuXvPinsNuSJbVZiOIa15EVlaJtupRzqrxijYyguVEzUrCyHPHYzQvBkc5KLtoMqWykQBc2CWsmXh4RGe0ys3C5BuNzXLKx9CqSXxzSokg6lPFi6HmhfP43UdO4K8zNkaTF/IIrcWlfIkOAeDyl/7Hva8WZZ4Xo/ctPS1gKNgof8l7t/e13I54QdFGnIRkGQeP64fiKh+66CDd/9Tm+PHRochrGDTf2UHyGmXi4PEqnCGYkyAe0UYAsOPQCUz4eoPudxGhRZlh1wmHSa3woiKieeEWXsQ0L8rmT7y6Bx6qK0LpV5FG9awF8ry4AWsCYZuN9I+xIkc6KXyq1P+mZiPgX6P64NDxSuQ3zsJdQ0412V/CqseGoToUEk5w5wY8t83JvmN0OV6t8PWvLcbOwyewYNshfDbmXOHCtupr6m+3GiauyrDL2E+vT0mSOsPugFPzLLUhnpDmhRNWKvpMk1TtZuTnZAkXWLPLh6MH4Kb+bfHazX2EjrNjNhId4P819yfd7REBSplhtzoUsr1C005s6pwgYbQZWw3PBTHBQ3lvLu1u3SckXthxFHUalvASG0EWq03TwnxuBl+y2iAa1RPJN8R3yySkBSTkC2Tmzm2QgbxG8V0s2cUpzbTdRc7OwycAAKvqisJqn7tIL3d6YRLt6xwh9lrIYTeJYckWPRSp2/2elTBC+7yGeOKX3XBKE7HQVZafidkE5dTLEREklQ67vFFALLQ+LnrRRuVVAmYjgR+syhWUADZp5eqNL1TavbawYK2M+Xxe+GAV9hPt9/X9zrg6cIQE6CoOIGHyjb0dOZNxtJE1WHmEzNvirLSgLish1jHa5zX0rZbXCBJeODHqDL+76HTVAJMIE48dWBois5/u1MsauY5SkKqusZ+fXS2sqNfl9T4vvA67YgOZUtBJFAE4QiZHiJ3Re5GVYXzsr3q3NvyOF+1KW0+bxkTfeiiE6DE81YEjJFZP4UdrlXQqB5ahH5rFoYnHbGSYldlJRykNvHfrlgHt8KverTHhqjN9nfdHDxJebBLQxM27WOjYF9gt1uVEeOB1ry7Cql1HVRNidShk+8xaTYuso3nhNhsJZqlTjmNK2eXUFg35TxJHlPeq2ym5pvsb9RqWsK8nw4maO5grY47BWtlf95VV4OgJsRIUYXW8oNlI6btgqnlJbPGFK2mlg9dzOs+LqNlo/7GKaKJFI0dzu49URNg7q6gJJl7dA/k5WaR5SUWUq6Nk17ywtQLxMRvVhGRc8fL/UK0IJXHCbKQch0Ixmhfgq9XFOHCskutcdnxeJEnCtLv6Y/TgUzF6MNsB0ytyszOif/PU5DKucmuM/rsk9pxZSepEe0zfZ76LFjTlRbQfKOEqeGnx3CmLg07aAFAjWJHznKe/wwUT5wJwNkJNSTg/kPhxCaZ4oWgju0iSWmpPduGFZTYye2GcjcgAJny9Mfq5usYJh11ltFFs0rox7/FXOw9HG1nzeQGA3m2boXfbZtzHx5u8RkFMu6s/crMzucxcRruwJmi970QHfJZqPl6DtdV+72dfIqfgizZy7kdactJmYCW/z56jJwEY9027v9Yoz0v31mwN6ZTfnI073lmGZ68802YL4gMJLzYJSJLqBbRrVvE7rN9nGirtcFuUVLMSenCiKg+gTTIn2HjR2kaJZm8GICRcGQn1rD6jJ/CICgIxmheho50RcKyeg0f971b1ZT8RF7OR4DOK9E07yQmdTGyoxKgw4zu3snO3DOmUj/V/Ge54zTi3SIxW+hgJ6gHRyeqyfsSOM6mb87MTp47xeVGdX+wK2lBrc2dmodMnHKxCcUboCTyiAz7bYddgFS7q1OsSEkUbAXA6w66R5kXsSUf6Jk9/NE5saHCszR+sp3nJaxREboMM/QMUJIrgApDwYhtJSrwwVzuwHSzZv93NLLJOnDomw25ILcwItafuvwhe3hs/YKx5ETNDit6mP322zlY9Gyc0YpbNRhbydSQNqlB8B81GDmleIs2zE23knuYl9r0ykkkSuX+R8GKTgCSpJqmkNxsxeoyXgpsTUUzaPC+ywXe851JpXkz2TzbNixMVxA+Vx0b2WBnwV+8uVXyy7rBrBbeTfyVaWL1fEX1EEkPzUhuSsXZPqWlfZZW1sIUUq7FLxnmJhBcHUPbRJOwjKuxkAnYzr4HTvgna9P7CvhIQ08glos+LCEYpBFiaF72ipVYGfOUxPOUBnMRetJG51qFxtrkpINGJy5gq2BkiwoBW8yJJEp6ZvgGX/vMHPPnlesPjv1lX4mr/0/abRA+p14OEF5tIGrtRoq6EMtP1u0Lnwhzce0HH6Ge7DrtuvbBOm43CPi/Kz6I+L5r94xiJ5Qe0g6eh2YhxjmMVsVlrrQh5GQrJSXn05Lnb8Pp8/RIUevtbxapgGs6Uyt6nKYcfg5+xe3+dGm/FNS/h/9dqQqVlWcYbP2wHAExZ8LPh8Xe+s8xls5F6WzLmH0vCnxR/lF0wUX1e/mNQRXTGA4Nw0Rn1dUX867DrrG9CSKN5ER1ntEnuzMPIxc6fKDw8ojMAYIJB+CVrRVimo3mxcp9Yffapr2KLf6qFVvHrabH6bAMBc3NjdkaatZP7HG2GXSPcXisu/fmw7vYTVbXYuv84amzklzIUXmx2Oj1fKTIbEboozSEJqnhB77bN8I9rz9L9TpnVlO1gmegOu+rzqc8pqHnRHGGm/k82zUuEu4acis1PXWxYpZb1vujVC7Jyn5QFPMUdrx3Aaqg0R4W9ZDQHaLEz5vBi1C9+NXkh9pdV6H539eQFMf1R5HkYRz7ZQy9UOlEX1SxIeHGYRB5MjDp480aZ0b/LGenxvczzwnN9M7RJ6eyswLU+M2ZCbZLKLgCMTZJmlOkIL1buk9JPywkNnQiSjWuGfV5SE15fHsfMRoyOtbsuqZyWIyeqY3xe9M5jdGYjzYvdsUBP82KYqiCR5yuvG5AMJMu8oxVeLuvRCgAQTK9XTR/RiQCJYPYauOmU6rTmJVxVWnF+0fZALPNysmpezGDdl66tGjtyDbNbW3qyGp+t3KM4wJHLRk9lOUkdkj8AQI8+bZvikZGdo59Zt8ApcwjrEbECDWwlqXMo54wWCbH3LFF9MVlQhl0HSBZ/BWX/XvnYRar6NRHycsQK4ylJJJ8XQJ3e326otLnDrtj5kwXWmPrP63vh3Ge/t30Ns+d419RlWLDtkNHBtq9vK88L4/sP7uhnrUE+57939sdRHa2bHk4Jd6xHxBJQtD5TupoMg8ONhCInNC9arDjM+x3SvDhAsoS5Kvt3kwaZqhfxvdv74sELT8eIboWGx5uGA7uoo4r1URFH7bBrL8OutrCj9s5kZ6ShV5smeOPmPuHzJ0kfEsVIbX193zY4pUm2I9dQPUed22wouMAZJYzVcwQkianW79uhucUz+we95yFiLotHtBFLeOEt1KqHcYVr+2jPQT4vhC7JMu+kM+LpBpyah/svPI2tpuXQLrj1Ds3bfMB+fSOV2chZzYt2Emqf1xAf330uhnYpsHT+ZEHpT6Uk3UE1tyoEnmNq2LTvGG6dsgRr95R6WtsoLZAKlYtikSTzEPEIzjnsMkxDLr2cRkKRXROynsCrHdoHnpaHRsF0DOnUwta1vITMRg4QbydAtxh0egv0aJ2LrqfkGu5jQ3ZxRDtixN9nbbZ9Dla0kWizwwOTSKh0cvQhXv55XU9MX1OMy3q0wt3vxlbrdnKl+OYP2zF38wG8exu7MF2EdXvLsG5vGeZvOYDOhfb8bsIZdq0924DAJJ6q2EmayYtb+ViM3vl/zWXnHrKCdtH59q3noCYkIyOBahlpIeHFAZJl3slMD+Cze85j7sNaC5p5rv9v60HdxGN+gZXnRXQCqg3JTM2L9lalmvByWY9WuKxHK8zbfED3eyc1L/9duhsA8PLsbbj1vPbcx1XbyOGhxOpZ0tP0qwMnKy/f0AsNg+Epifd3O+awy3hIdt9No8XtwePWTU4stPX2wttiM+4qUwgkIiS8OEAqTTt2NC9r9pQ62hanUQ5SC7cdwo392kY/i45ftRqfF+1cHCu8iJ0/WTDqT25ERzRtqG+iYmFXq2on2igtBWKllRmCR57Zsv4LbrORM+1gPWe71mgjpq8pceW8eoIfRRsRuqTSopkpvCT4+6F8jjUhGe8t3ln/neAkFta8GLvsageYlHXYNZil3BhsWzcVdwB2xOfFogCUCj4vD1/cGXtLT+Las9uotivHEtbdc8psxI42ckl6cQlJiu1zyZhhl4QXB0gWnxceWMNponu0a9XDcxUmDdHxSxttpB1jGwXTNfuLnT9ZiKfmJSBJwsLIur1ltq9r9dkm42pZS16jIN69LTbk21/RRo5cIm7o3ZEEH5p1iYu3zssvv4z27dsjKysLvXv3xvz58w33nTNnTp23ufrfxo0b49FUS6TSojkZX4IIrElG1GlP6/MyrGuB6nttrZ+U1bzEUXipDcmeLDSsPtu0gJTwCwIniEeSOhY1NjUv8X61RfK8JDKuCy8ffPABHnjgATz66KNYsWIFBg4ciIsvvhg7d+5kHrdp0yYUFxdH/5122mluN9UyqTTxsM1GifOCLNh6EP9duku1jeWYJxouqTQbPfnLrrii5ynR76bfNxDt8hpqri10et/ym3PbAQB+N+x0rv0NzUaCfYknJ4zWCTsehKONrB2bFkjdaCPescRot77tmwldz02HXU/QNJmqSlvg+eefx29/+1vcdttt6NKlCyZNmoSioiK88sorzOPy8/NRWFgY/ZeW5t/qqcky8fDAjDaKYzvscv3rizHuo9V1uTxkLNtxGEdPGJc+0NYwMUNpNjq3Y55q5aM34CbkAKnDny/rio1PjkDPNk259jfUvJhEQvx+eCfV5+xM8/HBrZBXt6BQaXOMNAqv1yV/5MVNh9149zq9MZo0L4JUVVVh2bJlGDZsmGr7sGHDsGDBAuaxPXv2RMuWLTF06FDMnj3bcL/KykqUlZWp/hHukSgOu0fKqzBt2W6cqDIuJAkAe4+exKz1+3DVKwvx8LQ1hvvVCo5gNYoUvRHTZwQ/3Sc3yMrgX2gY3QqzUOnhXQtwsSLbczbHNcuravHRst3cbXMCGdYF0/RAaoVKW8HIvKisx8YDU/OSYEJvqpiNXHXYPXjwIGpra1FQoLb3FxQUoKREP0ysZcuWePXVV9G7d29UVlbinXfewdChQzFnzhwMGjQoZv8JEybgL3/5iyvt5yVJFs1csF4BP70ft0xZglW7jmLhT4cw8eoehvvtO1aJVbuOmp5PNOVHKFS/lpOgdtjVm5Dyc7LELpAkGEWLmA+2an8QHuFl6qIdOKxTWDQ9IAlr1kSwYzYi2Bj1EydvnUjf8MsT07Y4GftSXKKNtPZLWZYNbZqdOnVCp071KuH+/ftj165dmDhxoq7wMn78eIwdOzb6uaysDEVFRQ613BwJyaPy54Gd58U/L0hEIPl85V6m8PKnT9eq/FGMEA2X/PnQiejf4XQdbM3LIyM740RVDX59dvz6rh+wqnmRJLXgw2M20hNcgPDA7qrwYvE4kTT5qYrRpCyqaXCrPMCuwyfibq7ULQ+QhB3JVeElLy8PaWlpMVqW/fv3x2hjWPTr1w9Tp07V/S4YDCIYtF7p2C7qJPCpACvDbhyb4SAsX5cIduzeEtSTkN5427xREK/c2Nv6RRIUq9FGAUlS3UcezUtmegBVNbEPMj0gwYlcpwEp1v/NTnkAILGc4L3AqJuI3ja3zEYDnzN2eXALvZ+ehIoXd31eMjMz0bt3b8yaNUu1fdasWRgwYAD3eVasWIGWLVua7+gVpHkJfxe/ZnDDExrLsyqxo12TJO198+Od8gqjJHXsoSlsiqs/NivDfCirMZBAnUp0ZiRw2Vl4U09hY/TsRIU+dp6XxBrf9coDkObFAmPHjsWoUaPQp08f9O/fH6+++ip27tyJ0aNHAwibffbs2YO3334bADBp0iS0a9cOXbt2RVVVFaZOnYpp06Zh2rRpbjfVEhKAIZ3z8cL3W9GQQ3Wd6LDmcD+uErXtXbHzCGasU2sCeZpdY7POjVm0UapirHmp/16vz2kFQp4Cc0ZzkFMDe/g8ehexJ/gSxiRCbaN4o3dHyOfFAtdccw0OHTqEJ554AsXFxejWrRumT5+Otm3DdWOKi4tVOV+qqqrw0EMPYc+ePcjOzkbXrl3x1VdfYeTIkW431RIygF5tmmL6fQO5ck0kOqyETX4caLXDzhUv60W5mTfcjt1bO9H68DZ5hrHDZXi7kTggaRx2020UmXOq3+pNEHZqG6UySjNgM0ZNKqcET3aodOI9QO3v8ePYbJe4OOzefffduPvuu3W/mzJliurzuHHjMG7cuDi0ylnOaNXY6ybEBbsaCD/CsyixU99EkiSNw24SjiQWMboTkXsUkCTdla8kqZ+bnZWlUxOgkRaAzEbipAUkrPjTRQjJMjP03qnkaywB001nbjdIFYfdJMy7F1+Sr0uwqWZ4rvrxBakNyaa5XrjMRjYGsNhQaSKC2b1nfa/sb3bMB05p1I38L+yUJPDjOxUvmjbMRPNG7GCMeNQ2+ts3m/DavJ8cuY6WHkVNHD+nJAHdW6vPm4xmIxJeBJh2V/+YbYklk9uHNYkrX4+P7x6Ay89qhRY53kWCRfj7zM3M73lCvO1EHMSYjZJvHLGM0b2PjLVG32tDpZ1yurWDXhPC0UbWz6nsK4+M7IycLKqlq8Qx4c7kIT09fYMz11GQnZGGxi48TwlAbnYGVv25PjlsMgrBJLwI0LttMyx59EL849qzvG6KZ7RgrISU70evNk0x6dqeKGjsvfASyfny58/W6n7Po3q2p3lRZ9hNxoHEKka3Irrd8Ht1qLQdzYtTPilGq1unzn/HoFPxm3PbO3OyJCGR3yXXCm/WnTM3OyO6yQeyveOQ8CJIi5wgfnlWfVKzJOwTTNrlNcQL1/VUvRgR9FbJfnBWlCSgpLQC/164Q//7eGheLB+d3Fgdu7Wh0v7QvOg77NqLVvH+d/kZpx67F8OU1m/LsfPqbGvbvIHzF/IYEl5s4oO5Oe78okcr9O/QPGa7XxdBZsIJz+RiK9oIFCpthKFZCPXRRrrfS87dU6feYTf8CrS/ywcymq9wzOfFg4E8LSAxQ/ytChzKPjP1t31xy4B2uG1gB0vn8jMkvBCW0JssfDspS8DxSmOnXZYTcgRb4ZIxPi9+vVH+IXKLjG6VtuKyHfW7nQy4SozaYHT6r+47z/Sc2jPeMqAdippl487ByTcZWSEeodJukSZJyEg3noILGlurd6ZcEJx3Wh4e/0VXoWKpiQJ5f9kkVachXeHFp3dDAnDh83MNv6/mCP+2I7xoc5L48y55g2m0kaFmxrloI6fQ850K56nR7zsNM82HX+3PatIgE/N+fz4JwHUoNS9GCQ15cErzIvJYAgEJQYbmxay+V6pDmhfCVRJhjNWrd6PFlsOuxP6cypjdC8PxW1JPXHbGecfMRoJ5Xnj6gZ7wRoJLPU4tCpzqAyJCUJokIZOhebFqEkuV7kHCC2GbOwd3wH1DT7OV5dRLrJiN7hjEr7aXoM2wm5j3KZ5EJmijiVpb7NIPE7phnheb2ZkJY1Th8j6IOBMhLcAWXqxqXlKly5DwQthm/MVdMPai030xgehhNi5ZEV5Efmo4rNcZLUGqYBIpHeOw64eJS0/zEo420t+fR4j16SvlG9JUAqz18/D4vPBEHIqZjYBMhtnIrDipE21IZEh4ISyRSNoDs5VvJYfZSBuRJPL7tZqXBLp1nmGa5wXa8gDWr+VU7Rpjh13981s1GxH62Fo8cXQBpws0mpmN9DQvbZrFRiBd2fMU1WeOtVhSQMIL4RhORW04jdncVMXxtmtrOoloT2LCemlCimIWKm14nEabZWfi4tG88aBnNpJgr7YRdRU2ylvrtt+T06NbwCRUOk1jhm/TrAE+HB2b5f35a85SfXaqP/sdEl4Ia3AOFH6YqM1WTDwve4zmRUR40dyFVFHr2iEaKm30PdQCix2zkVOF94zmIaP+x6d5IVgoH52dsYZn4eUHzQtPG0h4IQhB9Fa/XuRPiGmDmebFQrSR0EApQTUL0YRkTuum2QAYDruSeqVtZ9XtlNnIONrISHjh8Xmh3sJCeW9taV44uoDTiuW0gIQgU3hRf8cbCs6T+iEZIOGFsEQiDalO+LzYyvMiqYUdmpDY/PWqM6NVcQ1rH0FSCQt+qJprHG2kvz9Pi73/Vf5G+W4n2nslWdC88IxCpHkhiCTBTO6otiC8iJmNCCP07uM1Z7ep/97wQLWw4IeJS7S2EZfZyPuf5WtCilfX7TwvTmteApJJtJFO6gmjiKcJV54Z/buGhBeCSA7M7MRcDru2NC80A1nFyJclIKkndh8oXsST1DGm22l3xTpmErHUqjQv1s/DI5jw+JsIRSFKENa8NGuYqbvvdefUC/zVDplB/Q4JLzZpl5d81TqTDVPNi8ZGfHP/tmicpU7dbquqtMlnwhhDs5Em2sgfZqPYbeFoI3HNS682TcPnJMGXiVNmIx7fPL6IJP5xQoJJtJGmT0sAGgbTVYLKnIeGxBzHo0lOBqi2kUX+e2d/rNtbivM75XvdFE/gHSj8EG0kGsItSVLMwKGtKq389r3b+6I2JGPUGz8anE/THqHWpDpGodRqbYsftFtGApRhnhfGuaIZhu02KslR3lr3HXadfXMDFjQvANCpoFH07xY5wZjvnYqe8zskvFjknPbNcE77Zl43wzMSaVAVHXMCkhST3ZLlBNejdRM0DBq/StpU9kQ9Vm9LbIZdZ9pjByMtScio63C0+fzO+fh4xR40yEy+qsBOoI42crcT8MgEYlGIZrWN9L9TCup6PzlVHHZJeCGSHtH8DAEpNmfHsYoaw/1NKyNrNS8+TebnR1jRRk7leXEKI82LodmIY6K7tHtLNGmQgc6FjW21LVlRWnzt+bzwqF6sn1+PgARmVWmerNF6fShVhBfyeSEs4YO5ghvRMScQkGKcL2OEF8bE2bVV7ESTQLfLVxgmqZPUdW2MnGXjiajDLg+SJGHgaS10zQOpxp2Dw8VQr+lTFN2mFjrs+Lzw7OOs9CIBSBeobVRfrFS5LfY4yvNCEAx61OXhSARENS+SFBumeLi8yvL1JQlonJ0R/az8m2DDkknUodJxaIwJRnlejKZGP7Q5kRg3vDM+v+dcPH1Ft+g2Hp+Xds3Ngyq8SFIXkCSk64RDR8jQcdjlIVU0L2Q2Iiwxqn9bSBJwbsc8r5tiiiWfF4GZRbtrzOe6qIIVf7oIAJgRBqmGqcnNyGFXcq48gFOIh0oDwfQAV5JEImyW665ZNCnzLxl1gS4tG+PnQyeY5+bRqiz5+bDpPiJIEpDBqBytl+cFMBZiWuZmobi0AsO6FjrQOv9DwgthiYy0AH5zbnvVNu+nD31EfUwCEmsVHUb5rdnEGfm6qUGOBsIYo8cgQVKXB/CBPKjXhhNVtZi5rkR3f0mSsOSPF6L74zNdblny4pTDLs8Qccc7yyyfXw8JbM2LUbSRUkpT/uSv7x+ITSXHUiaQhIQXwjH8amkV9TkISJLxwKFDbB4Xa+reVCS/cRbze3ZtI39pXvTaMGXBz4b7SwAaZ5EJ0Q7dTsnF4u1hjYidHuDJ2CWxtbAx0UaS6n91f9d/atIgE307NHewgf7GB+sVgnAXYZ8XiE2GkQm2Q4uGAIBfntVK93silsZZGZj54CDh47R5XrRF7LxANFEedQvrzHhgIJ654kxcftYp0W2J9p5JADKsaF6U50isn+wo3r/xBOEyoj4veknqYvep/zuy6yd3n4v37+inyoAJkObFjNMLcnBz/7ZCxwQkdag0S/0eL0QjnvyQwDFR6VzYGNf3baN+Dw1mM57H4kX2gvQ0ySTayFjrGP3b6UYlECS8EI6h9yL5YWUg7vMillQuMonmZmegX4fmMROpH+6B32GZh4y2K4UF1go2Xpj5SRHuYk8YjL/0EpCkmIgiJVrNi/74mrp9joQXwjGcfP0jOR2cQNznxd71tCvwVB5g7MKsbaQYvYyykcYTkl28xcprVljnc+WU5kUkF0xagK15yTbIqkwauzDev/EEoYOTScdEa33wrKBZA4gfigQmC6z7rPRLYq1g44Xwc/e+yUmFqNN258IcjKozVzolvIgMNWkmeV4A4LWb+sRsI7NRGBJeCMdw8kVyUgCoMSwuo48k2VvdkKZFHJZ5yGib2ufF+6HMDxFPqYzR3Td6l5WRXk5lzxUxUacFJGael4Ak4aIzCmK2q6KNUrjLxeWNf/nll9G+fXtkZWWhd+/emD9/PnP/uXPnonfv3sjKykKHDh0wefLkeDSTiDM392+rSvWtxMmJoEYwXbYT1ybtizPo2vnr/q+8xX6438JtqOuW57RLjbwcrmMsvRgfUvedc2YjfsJmI34Nrt6iKJUXSq4LLx988AEeeOABPProo1ixYgUGDhyIiy++GDt37tTdf/v27Rg5ciQGDhyIFStW4JFHHsF9992HadOmud1UIs6cVpBjGCHg5GRUJZgum+fSZmMGVQEWwziTrvGAHfCbw67FieSVG3s53JLUxMr9j/Q7p/z1RISgQICdT8ro96SwvKLCdeHl+eefx29/+1vcdttt6NKlCyZNmoSioiK88soruvtPnjwZbdq0waRJk9ClSxfcdtttuPXWWzFx4kS3m0r4CEfNRsLCi1i0kR4kvDhDImlerAovzRtR0UUnsNIFIo/so2W7HWmDkNlIE+4f870P+rSfcVV4qaqqwrJlyzBs2DDV9mHDhmHBggW6xyxcuDBm/+HDh2Pp0qWorq6O2b+yshJlZWWqf0Qiof+COmk2slPV1yqpUtnVKYIZRkk6dDbVbVNrXrz3efFBE1IaP/gciZqN2N/rb6doozCuvm4HDx5EbW0tCgrUTkcFBQUoKdGv91FSUqK7f01NDQ4ePBiz/4QJE5Cbmxv9V1Sk70NBJBZeTgQByXx4MPv+eGWNU81JCe4c1AGdCnLw++GdVNv1NS+xZiORcg5uIZrnJSuTpJ14wHoqTvcaEbORmfCiFcakmD9Sm7jUNtKqxmRZZqrL9PbX2w4A48ePx9ixY6Ofy8rKSIDxiEZZ4t3JqBt4uYpyYh6sokrBQjRpkIlvdMoE6Pu8hP+v9JfyQ3kA3j575+AOuLp3awTTybToJFacV50eZkRKkZilgzDMsCvUouTFVeElLy8PaWlpMVqW/fv3x2hXIhQWFurun56ejubNY4tOBYNBBINkM/YDf7rkDOw5chI3CaZ6jzD1t31x4xuLAXi7kg4EJPKK8wn6WUUj/1eGSnv/vKo5BdaOLRqhY36Oy61JPYSHDJspEfQQddhlYSTcpHKEkRJXlyuZmZno3bs3Zs2apdo+a9YsDBgwQPeY/v37x+w/c+ZM9OnTBxkZVIHVzxTmZuHTMefiyl6to9tEXjPlO+mGs1r31rmc7eAwG5nscN8FHcP/H3oa1zUJfXTzvOiZjXwgvJCp0FuMo3NYWn5n2yDm88L+XivckMyixnVd69ixY/H666/jzTffxIYNG/Dggw9i586dGD16NICw2eemm26K7j969Gjs2LEDY8eOxYYNG/Dmm2/ijTfewEMPPeR2UwkPkIz+duFNzclKx11DTjXdzwm56YELT8c3DwzCAyS82EJPjNTrGn4wG/EKL2YT3J8uPcN+YwhPEIk2MuuzZDZi47rPyzXXXINDhw7hiSeeQHFxMbp164bp06ejbduwaaG4uFiV86V9+/aYPn06HnzwQbz00kto1aoVXnjhBVx11VVuN5XwEW5oXgKSxFV2wAl/m0BAQqdCMg3YRV/zEkbpX5Asmpe8RkH89rz2DrQm9TAaMpgOuw4tku4behpe+G6LmNnI5Np+iJ7yM3FZrtx99934+eefUVlZiWXLlmHQoHrHvClTpmDOnDmq/QcPHozly5ejsrIS27dvj2ppiOQi5j1Xmo1ceHEDksSlVeFxeSG7c3x4+opuMdsi9z6kiIFnpVmPF22bNeDbkaLoXeHOwWGtql5KfSP03uJ+HcQzHkeOESvMaPY9Jalj4f0bTxB1KE0EomGnPKRxOuKSYOIfereNnUjqNS/127xM6PXpmHPx++Gd8OuzKcrRS0ae2RLzx52PyTf25j5G71X/4yXiZrvI2CWSU0rUYTdyDRqewpDwQniGBE2FVJXDrvPX48nfApBN2W/ceq7GjFL3gGoVM4WX0WlnFTXBmPM7IpOz07JW5zQx2aOoWQMhQZYVzSZC5JJWQqUXPzIUE6/uEXtOSlLHhIQXwlOUL6LylXTD3psW4Dsvr5BDxIfxIzujqFl29HPk2SidI93Q1InC22X15rcLu4RNHeTv4jys5+KUljVS1VykCGxE4C5onIWBp+XFfE/lAdjEJUkdQYjilsMuz1jlA/cJQkFGWgB92zfHrsPh+jORCafWqVLADmFnInz5hl7YWFKGbq34wvmJMB1aNARgXfPGCsUXIVIYVKSOmlLg1muHcZ4XsbYlKyS8EL5BOfi74rAb4BuWAiYF04j4o5ybIo+m1ouiVQx4e4xeqzPTA+jeuomDrUkNGmSmY/Xjwyw7bDtlNorU1qoS0Lwoxzi9kYnyvLAh4YXwDcqX0xWHXUniOq8TSeoIZ1E5c9fdfDPFS0CKb1FOEni9oXEWO3kp86kwyk+IENW8hNzXvBBhSEFOeIYMjcOu4js3XtzMdL7ubiTfkAnaO5QLa708L7rHxHnw59a8+EthlLI42TsimhfeEhGA2tSl1xbjUGkaiAASXgi3MXnRjL51w+elRQ5fDSwj3xhKGuUdGYpInshjOKNVY+Yx8Xbipe7hT0Qneys+LxGH3WoBVV+zhpn119RpY6T/PjqyCzLTAphw5Zl17SMAMhsRPsJts1F+ThAnqmpN9zPUvCjsEBSPFF8yVGHI4XvfubAx3ru9L1rlZuseE2+1O2+fEElkRriMjhrMjtmomsNhNz0g4ZdnnYIrep5Sf02d/SL99/ZBHfCbc9tFBSQiDN0NwjN0rM3Rv9xYNLfICXLVHjFaqUXCWAef3sLRdhHmKE1+yscz4NQ8tMtrqHtMvENN7YRKE96gpyix0msiDsM8z/bqPq3x91/3UAkjuppexexstm8qQsILETf0Ur0rUSWpc+ENzc/J4hpcJOivon95Vit8/7vBeP3mPo63jWCjTADH2zP86qNEsos/kKHvN6UcenhDsDM4/enqrqCzJXYbOeyyIeGFiBs39G0bs02p5VAlqXNh5mkYTOPaz8i3JSBJ6NCiETLSArT6iTNGmhcWftW8EP4gmB5wTAsmkmdGt5/obDOqOk2+d2FIeCFc5eHhnQAAtwxoJ3ScGxNPeiDAteo1ShlBg4Z3qDUvfM8h/sIL2Y38iNFTCaYHDCLWrCSp459KeXPLGJcHIABy2CVcZkDHPKz9y3A0CsZ2Ne2woRz83SkPIPGZjSRJd4RQzoV5jfgilwhnsKJ58WuoNBFnDB5MMD1Ndzyw0m3SAhJ3XiH9rL7659Sj2ymUhRkg4YWIA3qCix6qPC8uVZXmifRgmY0iXH5WKyzbcQT9OsRWPSacJ8OSz4s/zUakd/EHwfSA7nhgtddkpAVQyZHnRTebrl6otEGHKmrWADMeGIimDTJ1v08VSHghfIPbDru855SgP4ApB5N0Rd4Fwn3Umhefmo14Q6VJevEFwYyAoxmYeYUXPXQz7DL6b+dCdo6jVIB8XghPMZqH3CiOmJbGazYSs0ET7qMUXvYcPcl1TLyfF7lE+RMjodLYbGTtQUZyvZi2R9dsRNFGotBwTPgSV8xGkmRLZU8Ou96RyTkxKIl/kjo+eHINEe4T1rw4ZzbiTSLH77BL4w0LEl4Iz2C9mu5Ulebbz2ilRsKLd/DWpVIS98GfukdCEUwL6AqSVkPxM3mFFxpHHIGEF8IzWOtPNyae9ECA2+FA36nO6RYRvGSm8eXoURL/aCPe8gCEHwhmpBlk2OX3jVOSbkE7GD0XjS3CkPBCeEo8tRy2zUYkvXgGrz+Bkribjah7+JasjNipzjjPCx/aMUok14sWqpUmDgkvRFzJa6QO71O+/8pxxFOzkZHDLs1OnmGlKF28Hxe/z4urzSA0SBLw5b0DY7aHhRf9/SOwHpW2f/Fm2dV12KWhRRgSXoi48t3YIVz7uREpkh6wlw6cFC/eUVljXg1cix01vhUaZaUjNzvDdD+SXeKLBKBjfqOY7cH0NFNJkqWZsap50dOy0NAiDgkvRFzJbWA8uKvyvLggKQQC4EpSZ3RlcrTzDitCZ7zNRsH0NMx4YCAu7d6SuR9FG/mD1s2yzTUvjEel7V68ZmV9zQuNLaKQ8EL4Ei+T1AHiSaMId+nbXjyTsRcTQsvcbOTnZMX9uoQY4y/ujP4dmnMtZoy47bz2AIDzO7UAAPAq+nRDpS23InWhDLuEpyhfWuUqxw3nWN7aRkYjCcku3pGeFsCsBwfhov+bhx6t+Wq7eCVs0iLaX2ifR78OzXDn4FMB6Nci4hV6r+rdGr84qxXaNm8IgL+/6ZaCpD4jDAkvhC9xQvPSKjcLe0srop8lgWgj3vojRPw4rSAHSx69kMuvBKAMpalOWkBCbUjGwNNaqLYr3207SeokSOiYn1P/mbO/6V5Tc2xThnmdCEPCC+FLnNC8vHd7PwyZOEf4OAkSRQT4lBY5/NW8/fq8yOUlPiz4wwVYv7cMQzqphZeurRR1gWxUldbuxyssGz3/X/dpjUPHq/DMlWeicRYJL2aQ8EJ4itH7blfln5EmoTA31vfAzsRBK3l/k5udgdKT1dHP9LhSm4LGWShoXD8GfHnvefhmXQnuGnJqdJu+5sVax+E2GxkMQs/9qoel66Yq5LBL+BK7goIESXcwseOgR2YjfzPtrgGqz359Xnb6IGGdbqfk4nfDOqFBZv2a3UktGK+22MlK1qkMCS+EpyhtvbJqu/1zOz15+XQuJOrQ5vLwk/ByVa/W0b/JbOQfzEKlWcSESnMeR8KrM7gqvBw5cgSjRo1Cbm4ucnNzMWrUKBw9epR5zC233AJJklT/+vXr52YzCY/ICRpbLW3PO5LBYMIxbhhdm3IxJBZePS69yypNFX07NI9fYwgmthx2NR2MV1tcG+K8AMHEVZ+X66+/Hrt378aMGTMAAHfccQdGjRqFL774gnnciBEj8NZbb0U/Z2ZmMvYmEo0nL++GH7cfxqXdW2J9cZlr19ETNrijjRTHdm+dixaN+B1FCX/gJ2EzIAE/PjIUu46cxFlFTbxuDlGHrv+JxXwtvGYjSlLoDK4JLxs2bMCMGTOwaNEi9O3bFwDw2muvoX///ti0aRM6depkeGwwGERhYaFbTSM8ZlS/thjVr23Mdkn1t3cTj/bKn40511cTIeEM+TlB7D9WGZdrBSQJ+Y2zkN+YEtj5CTv+J05HGxFiuGY2WrhwIXJzc6OCCwD069cPubm5WLBgAfPYOXPmID8/H6effjpuv/127N+/33DfyspKlJWVqf4RiYNxKn53rmdl1UOCS2Ji9tRGnslO4+8k1IX8iZPRRrz12OxUsibqcU14KSkpQX5+fsz2/Px8lJSUGB538cUX491338X333+Pv//971iyZAkuuOACVFbqr5AmTJgQ9anJzc1FUVGRY7+BiAMGFVydHusjkwfPuJGeRgXqkwE/ZUT2k/MwUY+dbLfaUYL3GZPo4gzCwsvjjz8e41Cr/bd06VIABj4HssxcyV5zzTW45JJL0K1bN1x22WX4+uuvsXnzZnz11Ve6+48fPx6lpaXRf7t27RL9SYSHtG7aQHc7r7bDKLeCdivP2a7sdQoGnNocZxU15bo24W/MJhM78gTrWEpwmDhcoqN943fYVX/mzfNCmhdnEPZ5ueeee3Dttdcy92nXrh1Wr16Nffv2xXx34MABFBQUcF+vZcuWaNu2LbZs2aL7fTAYRDBIzpSJyrVnF2HnoXKcF5PCG3j+1z0w9r+rmMe3z2uIrfuPx2zXDiwRYYg1bPz1qu7RsvY02SQ+Zs/Qjn5NgtgKmjQv/uTcjnn4+v6BqK4N4Rcv/g8A/8IpxmGXfF7iirDwkpeXh7y8PNP9+vfvj9LSUvz4448455xzAACLFy9GaWkpBgwYYHJ0PYcOHcKuXbvQsmX87NNE/MhIC+DRS84AAKzcdTS6XZKAfhwhpSFOj7vIsMIaOJSDD001iU9jkxpI9jQvktAsRMKLf+nSsjF+OhC7ADIlJs8LaV7iiWs+L126dMGIESNw++23Y9GiRVi0aBFuv/12XHrppapIo86dO+OTTz4BABw/fhwPPfQQFi5ciJ9//hlz5szBZZddhry8PFxxxRVuNZXwIWETpPl+tZwDAc+5lFrfiDPnKU2yuc5P+I+cYDreu70v/ntnf93vebT8pzTJxhO/7Gq7LX7yvyFisbJw0Wru0rgddjkvQDBxNc/Lu+++i/vuuw/Dhg0DAPziF7/Aiy++qNpn06ZNKC0tBQCkpaVhzZo1ePvtt3H06FG0bNkS559/Pj744APk5OTEnJ9ILqxEAtVya14iZiPj/ZXq4qt6tcYpTbNxRsvGhvsT/kYGMOBUYy0xj3ngvdv7om3zhnjss3Xqcwv2VYpY8zfKx2P1UdmtbUSI4arw0qxZM0ydOpW5j/JBZmdn45tvvnGzSUQCwaOG5TUbRRZJvONGICAxJz7C/5ip53mmGqM+eH3fNpi6aCcGnd5C9/vY83DtRniEUovC6wtl5FdnBskuzkBVpQnfwjMUcJuN6v6fmU7lvFIF065hQ6D44yVnYMjp+eh3aqxf1jVnF+G1+dvVlyLNi69RPR4p/Nms/2ifKHeSOgqWdgQayYmExqhOiHb1FBlX7hzUAR3yGrrcKiIR4FlhG6V8z8pIw4VnFKCRTn2ujvk5+HTMuWjSoN5hODsjzXpDCdfRPmceQSSmthFvqDTVNnIE0rwQ/oVjLDCyHxuNPc0bBfH9Q0NwxmMzcKKq1kbjCL9jtr7lmWus6kvOKmqClY8Nw5rdpaisqUV2JgkvfkaleJEi5kIxsyNFG8UXEl4I35LOkW+b32xEavtUw1TtzxWBZq/fnNk619bxRHyI9V8RP4bXr4mijZyBzEaEb2jbXG3OadYwE9f3bYMb+rYxPIY32qhhkFa+qYZZVAePQEuuKqmBUkiVZX4TkBL+Y0h6cQLSvBC+oVnDTHz3u8Eq/4BnrjgTAPDu4p26xwTTAzjGcW493wQiuXFC80LCS2qgfM6yLHNp3GJqG3GXBxBqGmEAaV4IX3Fqi0ZoJZAY7tWb+uhu105cDUl4STnMojp4phoyN6YGyucckjmFVm1tI/J5iSskvBAJzVmtm2DaXbHlJrS+MCS8pB7KLnBz/7bI0fQBnvBlys+SGgQz6qfC9DSJywREPi/eQsILkdAYzT9VNep4RDIbpR7KOeLxX3TF8scuUn2v7TtNG8TWQqL8LKlB46wM/OHizvjDxZ3ROCuD02ykxshspA2Tpwy7zkDCC5FQ5OcEcefgDtHPEkdII0DCSyqinCMkSYpWDI9u00w/S/+oFm4A0rykEqMHn4rRg08FYC3KzMhspA2TJ9nFGUh4IRKKoV0K0K65eJI5vVU1kdy0adaA+b12rkkLSJj14CA8dukZ9fuQz0tKwpUDSOJz2NVqXsjnxRlIeCESCtEF0bgRndChRUPcO/Q0dxpE+Balhk4PvTnktIIcXHRGQfSzRCNkSmLJbGRwTAPSvLgC6dKJhENEfrl7SEfcPaSja20h/MmpLRoiyyQlfy1HnnbSu6QmVhx20wwE3QYakzVpXpyB1hVEwkE+lIQZPI62PNmZ7WbYJRITrhxA2jwvBged2kJt5ibZxRlIeCFSEhpAkhsekaOGI2aVZJfU5O9X90BaQMKfFP5PWmJDpfU7yx9GdFZ9Js2LM5DZiEgoJDjvRPnubX1NnTuJxIJH6Kit1Z9ElHMLaV5Sk74dmmPjkyOQkRbAk1+u5zpGz9Q0bkQn5GSpgwVIeHEG0rwQiYfD88m5HfNQRMJLUnBVr9YAgPuHnm66L4/mhUhdtKH1ZuhFGwUkCVkZAfQoahLdRr3OGUh4IRIeWsgQESZe3R2LHxmKS7q3NN3XqKinWVkBggB0HHZ1tHQSwv5XnyiygJPM7AwkvBAJRUBiG40evDC84u7bvll8GkT4CkmSUNA4i2tfpcPu6QWNDM4Xu23Fn2KT2RGpR6zDrs4+dduUWhnKsOsMJLwQCcGDF56OU5pk494LOjIjSe69oCM+Gt0fb/3m7Di2jkhElD4vb9/aN/p380bB6N/pgdghsmnDTHcbRviW/7umR/TvGIddHeklOzPWrZR8XpyBHHaJhOD+C0/DfUP1BZfc7HqHuEBAQp92pHUhzFH6vBTm1mtrGgXT8c0Dg7gL9BGpwcDT8tBWkd1b2zP0zEbaYqAAmbmdgoQXImGICC7aIeK0ghz8fngn5OcEYw8iCANYK+BOhTlxbAmRDOgJuno11cjnxRlIeCESDj2r0ZjzKYsuIQZFGxEiaNMpaLXAeuNSoyw9zQv1OycgnxeCIFKSEAkvBAfv39EP155dhHGaZHNaWUWvHAVVs3cPEl6IhIPyhhFOUMNR24gg+nVojmev6o7c7AyVv4p2HNJWjwaAxooEdc9ccSaaNczExKt7xOxHiENiIZFwOJ1hl0hNjPK8EAQPWrORtno0oDYbXd+3Da47p4ir7hZhDmleiISjTXP72XAvPKMAANAhr6HJnkSyQj4vhJPomY0aBtXbSHBxDtK8EAlHrzZN8bdfdVeFLYryzBXd0KdtU1zcrdDBlhGJBGleCFFYsodS85ITTMewroUIpscKNIQzkPBCJCRX9ymydXxOVgZuHtDOmcYQCQkJL4QorEChBoqEdG/95mzKN+UyZDYiCCJlOPOUXADAWUVNhMxG/xrVGzlZ6XjrFsrcTOijdNglk6T7kOaFIIiU4Y1b+uDDpbvx6z5FGD11Gfdxw7sW4qIuBbop4AkCALIVZqOK6loPW5IakOaFIIiUIT8nC2PO74gWOUHcem57AMCQTi24jiXBhWCRkVbfPyqqKQzfbVwVXp5++mkMGDAADRo0QJMmTbiOkWUZjz/+OFq1aoXs7GwMGTIE69atc7OZBEGkIJd0b4k5Dw3B6zf18bopRBKgjCSqrCHNi9u4KrxUVVXh6quvxl133cV9zHPPPYfnn38eL774IpYsWYLCwkJcdNFFOHbsmIstJQgiFWmX1xDpaaSAJpyFMuu6j6t3+C9/+QsAYMqUKVz7y7KMSZMm4dFHH8WVV14JAPj3v/+NgoICvPfee7jzzjvdaipBEARBMClozC7++vyve2DlrqM4v1N+nFqUuvhKPNy+fTtKSkowbNiw6LZgMIjBgwdjwYIFusJLZWUlKisro5/Lysri0laCIAgitWjdtAEm39gbTRpk6H5/Za/WuLJX6zi3KjXxlb60pKQEAFBQUKDaXlBQEP1Oy4QJE5Cbmxv9V1RkL/8HQRAEQRgxolsh+nVo7nUzUh5h4eXxxx+HJEnMf0uXLrXVKG0KZVmWDdMqjx8/HqWlpdF/u3btsnVtgiAIgiD8jbDZ6J577sG1117L3Kddu3aWGlNYGE7VXlJSgpYtW0a379+/P0YbEyEYDCIYZNshCYIgCIJIHoSFl7y8POTl5bnRFrRv3x6FhYWYNWsWevbsCSAcsTR37lz89a9/deWaBEEQBEEkFq76vOzcuRMrV67Ezp07UVtbi5UrV2LlypU4fvx4dJ/OnTvjk08+ARA2Fz3wwAN45pln8Mknn2Dt2rW45ZZb0KBBA1x//fVuNpUgCIIgiATB1Wijxx57DP/+97+jnyPalNmzZ2PIkCEAgE2bNqG0tDS6z7hx43Dy5EncfffdOHLkCPr27YuZM2ciJyfHzaYSBEEQBJEgSLLMqpOZeJSVlSE3NxelpaVo3Lix180hCIIgCIIDkfnbV6HSBEEQBEEQZpDwQhAEQRBEQkHCC0EQBEEQCQUJLwRBEARBJBQkvBAEQRAEkVCQ8EIQBEEQREJBwgtBEARBEAmFq0nqvCCStqasrMzjlhAEQRAEwUtk3uZJP5d0wsuxY8cAAEVFRR63hCAIgiAIUY4dO4bc3FzmPkmXYTcUCmHv3r3IycmBJEmOnrusrAxFRUXYtWsXZe91EbrP8YPudXyg+xwf6D7HB7fusyzLOHbsGFq1aoVAgO3VknSal0AggNatW7t6jcaNG9OLEQfoPscPutfxge5zfKD7HB/cuM9mGpcI5LBLEARBEERCQcILQRAEQRAJBQkvAgSDQfz5z39GMBj0uilJDd3n+EH3Oj7QfY4PdJ/jgx/uc9I57BIEQRAEkdyQ5oUgCIIgiISChBeCIAiCIBIKEl4IgiAIgkgoSHghCIIgCCKhIOGFk5dffhnt27dHVlYWevfujfnz53vdpIRiwoQJOPvss5GTk4P8/Hxcfvnl2LRpk2ofWZbx+OOPo1WrVsjOzsaQIUOwbt061T6VlZW49957kZeXh4YNG+IXv/gFdu/eHc+fklBMmDABkiThgQceiG6j++wce/bswY033ojmzZujQYMGOOuss7Bs2bLo93Sv7VNTU4M//vGPaN++PbKzs9GhQwc88cQTCIVC0X3oPoszb948XHbZZWjVqhUkScKnn36q+t6pe3rkyBGMGjUKubm5yM3NxahRo3D06FH7P0AmTHn//ffljIwM+bXXXpPXr18v33///XLDhg3lHTt2eN20hGH48OHyW2+9Ja9du1ZeuXKlfMkll8ht2rSRjx8/Ht3n2WeflXNycuRp06bJa9aska+55hq5ZcuWcllZWXSf0aNHy6eccoo8a9Ysefny5fL5558v9+jRQ66pqfHiZ/maH3/8UW7Xrp3cvXt3+f77749up/vsDIcPH5bbtm0r33LLLfLixYvl7du3y99++628devW6D50r+3z1FNPyc2bN5e//PJLefv27fKHH34oN2rUSJ40aVJ0H7rP4kyfPl1+9NFH5WnTpskA5E8++UT1vVP3dMSIEXK3bt3kBQsWyAsWLJC7desmX3rppbbbT8ILB+ecc448evRo1bbOnTvLf/jDHzxqUeKzf/9+GYA8d+5cWZZlORQKyYWFhfKzzz4b3aeiokLOzc2VJ0+eLMuyLB89elTOyMiQ33///eg+e/bskQOBgDxjxoz4/gCfc+zYMfm0006TZ82aJQ8ePDgqvNB9do6HH35YPu+88wy/p3vtDJdccol86623qrZdeeWV8o033ijLMt1nJ9AKL07d0/Xr18sA5EWLFkX3WbhwoQxA3rhxo602k9nIhKqqKixbtgzDhg1TbR82bBgWLFjgUasSn9LSUgBAs2bNAADbt29HSUmJ6j4Hg0EMHjw4ep+XLVuG6upq1T6tWrVCt27d6FloGDNmDC655BJceOGFqu10n53j888/R58+fXD11VcjPz8fPXv2xGuvvRb9nu61M5x33nn47rvvsHnzZgDAqlWr8MMPP2DkyJEA6D67gVP3dOHChcjNzUXfvn2j+/Tr1w+5ubm273vSFWZ0moMHD6K2thYFBQWq7QUFBSgpKfGoVYmNLMsYO3YszjvvPHTr1g0AovdS7z7v2LEjuk9mZiaaNm0asw89i3ref/99LF++HEuWLIn5ju6zc/z000945ZVXMHbsWDzyyCP48ccfcd999yEYDOKmm26ie+0QDz/8MEpLS9G5c2ekpaWhtrYWTz/9NK677joA1KfdwKl7WlJSgvz8/Jjz5+fn277vJLxwIkmS6rMsyzHbCD7uuecerF69Gj/88EPMd1buMz2Lenbt2oX7778fM2fORFZWluF+dJ/tEwqF0KdPHzzzzDMAgJ49e2LdunV45ZVXcNNNN0X3o3ttjw8++ABTp07Fe++9h65du2LlypV44IEH0KpVK9x8883R/eg+O48T91RvfyfuO5mNTMjLy0NaWlqMlLh///4YqZQw595778Xnn3+O2bNno3Xr1tHthYWFAMC8z4WFhaiqqsKRI0cM90l1li1bhv3796N3795IT09Heno65s6dixdeeAHp6enR+0T32T4tW7bEGWecodrWpUsX7Ny5EwD1aaf4/e9/jz/84Q+49tprceaZZ2LUqFF48MEHMWHCBAB0n93AqXtaWFiIffv2xZz/wIEDtu87CS8mZGZmonfv3pg1a5Zq+6xZszBgwACPWpV4yLKMe+65Bx9//DG+//57tG/fXvV9+/btUVhYqLrPVVVVmDt3bvQ+9+7dGxkZGap9iouLsXbtWnoWdQwdOhRr1qzBypUro//69OmDG264AStXrkSHDh3oPjvEueeeGxPuv3nzZrRt2xYA9WmnOHHiBAIB9VSVlpYWDZWm++w8Tt3T/v37o7S0FD/++GN0n8WLF6O0tNT+fbfl7psiREKl33jjDXn9+vXyAw88IDds2FD++eefvW5awnDXXXfJubm58pw5c+Ti4uLovxMnTkT3efbZZ+Xc3Fz5448/ltesWSNfd911uqF5rVu3lr/99lt5+fLl8gUXXJDS4Y48KKONZJnus1P8+OOPcnp6uvz000/LW7Zskd999125QYMG8tSpU6P70L22z8033yyfcsop0VDpjz/+WM7Ly5PHjRsX3YfuszjHjh2TV6xYIa9YsUIGID///PPyihUroilAnLqnI0aMkLt37y4vXLhQXrhwoXzmmWdSqHQ8eemll+S2bdvKmZmZcq9evaIhvgQfAHT/vfXWW9F9QqGQ/Oc//1kuLCyUg8GgPGjQIHnNmjWq85w8eVK+55575GbNmsnZ2dnypZdeKu/cuTPOvyax0AovdJ+d44svvpC7desmB4NBuXPnzvKrr76q+p7utX3Kysrk+++/X27Tpo2clZUld+jQQX700UflysrK6D50n8WZPXu27ph88803y7Ls3D09dOiQfMMNN8g5OTlyTk6OfMMNN8hHjhyx3X5JlmXZnu6GIAiCIAgifpDPC0EQBEEQCQUJLwRBEARBJBQkvBAEQRAEkVCQ8EIQBEEQREJBwgtBEARBEAkFCS8EQRAEQSQUJLwQBEEQBJFQkPBCEARBEERCQcILQRAEQRAJBQkvBEEQBEEkFCS8EARBEASRUJDwQhAEQRBEQvH/gBstwwtblkQAAAAASUVORK5CYII=\n",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {
|
|
"filenames": {
|
|
"image/png": "/Users/mhjensen/Teaching/MachineLearning/doc/LectureNotes/_build/jupyter_execute/chapter13_3_0.png"
|
|
}
|
|
},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Metal device set to: Apple M1\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Model: \"sequential\"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Layer (type) Output Shape Param # \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" simple_rnn (SimpleRNN) (None, 32) 1184 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense (Dense) (None, 8) 264 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense_1 (Dense) (None, 1) 9 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Total params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Trainable params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Non-trainable params: 0\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 1/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2023-11-08 15:25:18.982829: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 3s - loss: 1.7073 - 3s/epoch - 65ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 2/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.5091 - 450ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 3/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4349 - 447ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 4/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4192 - 447ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 5/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4095 - 448ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 6/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4056 - 450ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 7/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4055 - 451ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 8/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4025 - 449ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 9/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3975 - 452ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 10/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3994 - 451ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 11/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3976 - 449ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 12/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3973 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 13/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3951 - 452ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 14/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3957 - 454ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 15/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3953 - 472ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 16/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3940 - 479ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 17/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3939 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 18/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3897 - 483ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 19/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3936 - 467ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 20/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3929 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 21/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3913 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 22/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3900 - 463ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 23/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - loss: 0.3921 - 516ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 24/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3903 - 485ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 25/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3902 - 471ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 26/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3887 - 472ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 27/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - loss: 0.3895 - 519ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 28/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3873 - 477ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 29/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3879 - 500ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 30/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3876 - 485ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 31/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3865 - 472ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 32/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3856 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 33/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - loss: 0.3856 - 502ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 34/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3839 - 479ms/epoch - 10ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 35/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3853 - 459ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 36/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3827 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 37/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3824 - 453ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 38/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3824 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 39/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3812 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 40/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3823 - 469ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 41/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - loss: 0.3814 - 527ms/epoch - 11ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 42/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3814 - 464ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 43/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3812 - 453ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 44/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3779 - 456ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 45/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3797 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 46/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3808 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 47/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3788 - 460ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 48/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3792 - 455ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 49/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3765 - 457ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 50/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3775 - 454ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 51/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3777 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 52/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3770 - 458ms/epoch - 9ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 53/100\n"
|
|
]
|
|
},
|
|
{
|
|
"ename": "KeyboardInterrupt",
|
|
"evalue": "",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
|
|
"Input \u001b[0;32mIn [1]\u001b[0m, in \u001b[0;36m<cell line: 58>\u001b[0;34m()\u001b[0m\n\u001b[1;32m 55\u001b[0m model\u001b[38;5;241m.\u001b[39mcompile(loss\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmean_squared_error\u001b[39m\u001b[38;5;124m'\u001b[39m, optimizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrmsprop\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 56\u001b[0m model\u001b[38;5;241m.\u001b[39msummary()\n\u001b[0;32m---> 58\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrainX\u001b[49m\u001b[43m,\u001b[49m\u001b[43mtrainY\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m16\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 59\u001b[0m trainPredict \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(trainX)\n\u001b[1;32m 60\u001b[0m testPredict\u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(testX)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/utils/traceback_utils.py:64\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 62\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 63\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 64\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 65\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e: \u001b[38;5;66;03m# pylint: disable=broad-except\u001b[39;00m\n\u001b[1;32m 66\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/engine/training.py:1384\u001b[0m, in \u001b[0;36mModel.fit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 1377\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m tf\u001b[38;5;241m.\u001b[39mprofiler\u001b[38;5;241m.\u001b[39mexperimental\u001b[38;5;241m.\u001b[39mTrace(\n\u001b[1;32m 1378\u001b[0m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mtrain\u001b[39m\u001b[38;5;124m'\u001b[39m,\n\u001b[1;32m 1379\u001b[0m epoch_num\u001b[38;5;241m=\u001b[39mepoch,\n\u001b[1;32m 1380\u001b[0m step_num\u001b[38;5;241m=\u001b[39mstep,\n\u001b[1;32m 1381\u001b[0m batch_size\u001b[38;5;241m=\u001b[39mbatch_size,\n\u001b[1;32m 1382\u001b[0m _r\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m):\n\u001b[1;32m 1383\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_begin(step)\n\u001b[0;32m-> 1384\u001b[0m tmp_logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1385\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m data_handler\u001b[38;5;241m.\u001b[39mshould_sync:\n\u001b[1;32m 1386\u001b[0m context\u001b[38;5;241m.\u001b[39masync_wait()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 150\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 152\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:915\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 912\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 914\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[0;32m--> 915\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 917\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[1;32m 918\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/def_function.py:947\u001b[0m, in \u001b[0;36mFunction._call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 944\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m 945\u001b[0m \u001b[38;5;66;03m# In this case we have created variables on the first call, so we run the\u001b[39;00m\n\u001b[1;32m 946\u001b[0m \u001b[38;5;66;03m# defunned version which is guaranteed to never create variables.\u001b[39;00m\n\u001b[0;32m--> 947\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_stateless_fn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# pylint: disable=not-callable\u001b[39;00m\n\u001b[1;32m 948\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_stateful_fn \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 949\u001b[0m \u001b[38;5;66;03m# Release the lock early so that multiple threads can perform the call\u001b[39;00m\n\u001b[1;32m 950\u001b[0m \u001b[38;5;66;03m# in parallel.\u001b[39;00m\n\u001b[1;32m 951\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:2956\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 2953\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock:\n\u001b[1;32m 2954\u001b[0m (graph_function,\n\u001b[1;32m 2955\u001b[0m filtered_flat_args) \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_maybe_define_function(args, kwargs)\n\u001b[0;32m-> 2956\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 2957\u001b[0m \u001b[43m \u001b[49m\u001b[43mfiltered_flat_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mgraph_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:1853\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1849\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[1;32m 1850\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[1;32m 1851\u001b[0m \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[1;32m 1852\u001b[0m \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[0;32m-> 1853\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_build_call_outputs(\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1854\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcancellation_manager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcancellation_manager\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m 1855\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[1;32m 1856\u001b[0m args,\n\u001b[1;32m 1857\u001b[0m possible_gradient_type,\n\u001b[1;32m 1858\u001b[0m executing_eagerly)\n\u001b[1;32m 1859\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/function.py:499\u001b[0m, in \u001b[0;36m_EagerDefinedFunction.call\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 497\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m _InterpolateFunctionError(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m 498\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cancellation_manager \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 499\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 500\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mstr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msignature\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 501\u001b[0m \u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_num_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 502\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 503\u001b[0m \u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 504\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mctx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 505\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 506\u001b[0m outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[1;32m 507\u001b[0m \u001b[38;5;28mstr\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msignature\u001b[38;5;241m.\u001b[39mname),\n\u001b[1;32m 508\u001b[0m num_outputs\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_num_outputs,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 511\u001b[0m ctx\u001b[38;5;241m=\u001b[39mctx,\n\u001b[1;32m 512\u001b[0m cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_manager)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:54\u001b[0m, in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 52\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m 53\u001b[0m ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[0;32m---> 54\u001b[0m tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 55\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 56\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 57\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Start importing packages\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras.models import Model, Sequential \n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"from tensorflow.keras import optimizers \n",
|
|
"from tensorflow.keras import regularizers \n",
|
|
"from tensorflow.keras.utils import to_categorical \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"# convert into dataset matrix\n",
|
|
"def convertToMatrix(data, step):\n",
|
|
" X, Y =[], []\n",
|
|
" for i in range(len(data)-step):\n",
|
|
" d=i+step \n",
|
|
" X.append(data[i:d,])\n",
|
|
" Y.append(data[d,])\n",
|
|
" return np.array(X), np.array(Y)\n",
|
|
"\n",
|
|
"step = 4\n",
|
|
"N = 1000 \n",
|
|
"Tp = 800 \n",
|
|
"\n",
|
|
"t=np.arange(0,N)\n",
|
|
"x=np.sin(0.02*t)+2*np.random.rand(N)\n",
|
|
"df = pd.DataFrame(x)\n",
|
|
"df.head()\n",
|
|
"\n",
|
|
"plt.plot(df)\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"values=df.values\n",
|
|
"train,test = values[0:Tp,:], values[Tp:N,:]\n",
|
|
"\n",
|
|
"# add step elements into train and test\n",
|
|
"test = np.append(test,np.repeat(test[-1,],step))\n",
|
|
"train = np.append(train,np.repeat(train[-1,],step))\n",
|
|
" \n",
|
|
"trainX,trainY =convertToMatrix(train,step)\n",
|
|
"testX,testY =convertToMatrix(test,step)\n",
|
|
"trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
|
|
"testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(SimpleRNN(units=32, input_shape=(1,step), activation=\"relu\"))\n",
|
|
"model.add(Dense(8, activation=\"relu\")) \n",
|
|
"model.add(Dense(1))\n",
|
|
"model.compile(loss='mean_squared_error', optimizer='rmsprop')\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)\n",
|
|
"trainPredict = model.predict(trainX)\n",
|
|
"testPredict= model.predict(testX)\n",
|
|
"predicted=np.concatenate((trainPredict,testPredict),axis=0)\n",
|
|
"\n",
|
|
"trainScore = model.evaluate(trainX, trainY, verbose=0)\n",
|
|
"print(trainScore)\n",
|
|
"\n",
|
|
"index = df.index.values\n",
|
|
"plt.plot(index,df)\n",
|
|
"plt.plot(index,predicted)\n",
|
|
"plt.axvline(df.index[Tp], c=\"r\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "897a47c7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## An extrapolation example\n",
|
|
"\n",
|
|
"The following code provides an example of how recurrent neural\n",
|
|
"networks can be used to extrapolate to unknown values of physics data\n",
|
|
"sets. Specifically, the data sets used in this program come from\n",
|
|
"a quantum mechanical many-body calculation of energies as functions of the number of particles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6776ae2a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"# For matrices and calculations\n",
|
|
"import numpy as np\n",
|
|
"# For machine learning (backend for keras)\n",
|
|
"import tensorflow as tf\n",
|
|
"# User-friendly machine learning library\n",
|
|
"# Front end for TensorFlow\n",
|
|
"import tensorflow.keras\n",
|
|
"# Different methods from Keras needed to create an RNN\n",
|
|
"# This is not necessary but it shortened function calls \n",
|
|
"# that need to be used in the code.\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras import regularizers\n",
|
|
"from tensorflow.keras.models import Model, Sequential\n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"# For timing the code\n",
|
|
"from timeit import default_timer as timer\n",
|
|
"# For plotting\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"\n",
|
|
"# The data set\n",
|
|
"datatype='VaryDimension'\n",
|
|
"X_tot = np.arange(2, 42, 2)\n",
|
|
"y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,\n",
|
|
"\t-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, \n",
|
|
"\t-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35227d36",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The way the recurrent neural networks are trained in this program\n",
|
|
"differs from how machine learning algorithms are usually trained.\n",
|
|
"Typically a machine learning algorithm is trained by learning the\n",
|
|
"relationship between the x data and the y data. In this program, the\n",
|
|
"recurrent neural network will be trained to recognize the relationship\n",
|
|
"in a sequence of y values. This is type of data formatting is\n",
|
|
"typically used time series forcasting, but it can also be used in any\n",
|
|
"extrapolation (time series forecasting is just a specific type of\n",
|
|
"extrapolation along the time axis). This method of data formatting\n",
|
|
"does not use the x data and assumes that the y data are evenly spaced.\n",
|
|
"\n",
|
|
"For a standard machine learning algorithm, the training data has the\n",
|
|
"form of (x,y) so the machine learning algorithm learns to assiciate a\n",
|
|
"y value with a given x value. This is useful when the test data has x\n",
|
|
"values within the same range as the training data. However, for this\n",
|
|
"application, the x values of the test data are outside of the x values\n",
|
|
"of the training data and the traditional method of training a machine\n",
|
|
"learning algorithm does not work as well. For this reason, the\n",
|
|
"recurrent neural network is trained on sequences of y values of the\n",
|
|
"form ((y1, y2), y3), so that the network is concerned with learning\n",
|
|
"the pattern of the y data and not the relation between the x and y\n",
|
|
"data. As long as the pattern of y data outside of the training region\n",
|
|
"stays relatively stable compared to what was inside the training\n",
|
|
"region, this method of training can produce accurate extrapolations to\n",
|
|
"y values far removed from the training data set."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "7dc577d1",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# FORMAT_DATA\n",
|
|
"def format_data(data, length_of_sequence = 2): \n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" data(a numpy array): the data that will be the inputs to the recurrent neural\n",
|
|
" network\n",
|
|
" length_of_sequence (an int): the number of elements in one iteration of the\n",
|
|
" sequence patter. For a function approximator use length_of_sequence = 2.\n",
|
|
" Returns:\n",
|
|
" rnn_input (a 3D numpy array): the input data for the recurrent neural network. Its\n",
|
|
" dimensions are length of data - length of sequence, length of sequence, \n",
|
|
" dimnsion of data\n",
|
|
" rnn_output (a numpy array): the training data for the neural network\n",
|
|
" Formats data to be used in a recurrent neural network.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" X, Y = [], []\n",
|
|
" for i in range(len(data)-length_of_sequence):\n",
|
|
" # Get the next length_of_sequence elements\n",
|
|
" a = data[i:i+length_of_sequence]\n",
|
|
" # Get the element that immediately follows that\n",
|
|
" b = data[i+length_of_sequence]\n",
|
|
" # Reshape so that each data point is contained in its own array\n",
|
|
" a = np.reshape (a, (len(a), 1))\n",
|
|
" X.append(a)\n",
|
|
" Y.append(b)\n",
|
|
" rnn_input = np.array(X)\n",
|
|
" rnn_output = np.array(Y)\n",
|
|
"\n",
|
|
" return rnn_input, rnn_output\n",
|
|
"\n",
|
|
"\n",
|
|
"# ## Defining the Recurrent Neural Network Using Keras\n",
|
|
"# \n",
|
|
"# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.\n",
|
|
"\n",
|
|
"def rnn(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with one hidden layer and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer\n",
|
|
" hidden_neurons = 200\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to \n",
|
|
" # the network immediately after the input layer\n",
|
|
" rnn = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\")(inp)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1978b6a1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Predicting New Points With A Trained Recurrent Neural Network"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "d4ad417a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_rnn (x1, y_test, plot_min, plot_max):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" x1 (a list or numpy array): The complete x component of the data set\n",
|
|
" y_test (a list or numpy array): The complete y component of the data set\n",
|
|
" plot_min (an int or float): the smallest x value used in the training data\n",
|
|
" plot_max (an int or float): the largest x valye used in the training data\n",
|
|
" Returns:\n",
|
|
" None.\n",
|
|
" Uses a trained recurrent neural network model to predict future points in the \n",
|
|
" series. Computes the MSE of the predicted data set from the true data set, saves\n",
|
|
" the predicted data set to a csv file, and plots the predicted and true data sets w\n",
|
|
" while also displaying the data range used for training.\n",
|
|
" \"\"\"\n",
|
|
" # Add the training data as the first dim points in the predicted data array as these\n",
|
|
" # are known values.\n",
|
|
" y_pred = y_test[:dim].tolist()\n",
|
|
" # Generate the first input to the trained recurrent neural network using the last two \n",
|
|
" # points of the training data. Based on how the network was trained this means that it\n",
|
|
" # will predict the first point in the data set after the training data. All of the \n",
|
|
" # brackets are necessary for Tensorflow.\n",
|
|
" next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])\n",
|
|
" # Save the very last point in the training data set. This will be used later.\n",
|
|
" last = [y_test[dim-1]]\n",
|
|
"\n",
|
|
" # Iterate until the complete data set is created.\n",
|
|
" for i in range (dim, len(y_test)):\n",
|
|
" # Predict the next point in the data set using the previous two points.\n",
|
|
" next = model.predict(next_input)\n",
|
|
" # Append just the number of the predicted data set\n",
|
|
" y_pred.append(next[0][0])\n",
|
|
" # Create the input that will be used to predict the next data point in the data set.\n",
|
|
" next_input = np.array([[last, next[0]]], dtype=np.float64)\n",
|
|
" last = next\n",
|
|
"\n",
|
|
" # Print the mean squared error between the known data set and the predicted data set.\n",
|
|
" print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())\n",
|
|
" # Save the predicted data set as a csv file for later use\n",
|
|
" name = datatype + 'Predicted'+str(dim)+'.csv'\n",
|
|
" np.savetxt(name, y_pred, delimiter=',')\n",
|
|
" # Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
" # for the training data.\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.plot(x1, y_test, label=\"true\", linewidth=3)\n",
|
|
" ax.plot(x1, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
" ax.legend()\n",
|
|
" # Created a red region to represent the points used in the training data.\n",
|
|
" ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn(length_of_sequences = rnn_input.shape[1])\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2cad4fc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Changing the size of the recurrent neural network and its parameters\n",
|
|
"can drastically change the results you get from the model. The below\n",
|
|
"code takes the simple recurrent neural network from above and adds a\n",
|
|
"second hidden layer, changes the number of neurons in the hidden\n",
|
|
"layer, and explicitly declares the activation function of the hidden\n",
|
|
"layers to be a sigmoid function. The loss function and optimizer can\n",
|
|
"also be changed but are kept the same as the above network. These\n",
|
|
"parameters can be tuned to provide the optimal result from the\n",
|
|
"network. For some ideas on how to improve the performance of a\n",
|
|
"[recurrent neural network](https://danijar.com/tips-for-training-recurrent-neural-networks)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c39f1516",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer, increased from the first network\n",
|
|
" hidden_neurons = 500\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Create two hidden layers instead of one hidden layer. Explicitly set the activation\n",
|
|
" # function to be the sigmoid function (the default value is hyperbolic tangent)\n",
|
|
" rnn1 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=True, # This needs to be True if another hidden layer is to follow\n",
|
|
" stateful = stateful, activation = 'sigmoid',\n",
|
|
" name=\"RNN1\")(inp)\n",
|
|
" rnn2 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False, activation = 'sigmoid',\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN2\")(rnn1)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn2)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn_2layers(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "842c7602",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Other Types of Recurrent Neural Networks\n",
|
|
"\n",
|
|
"Besides a simple recurrent neural network layer, there are two other\n",
|
|
"commonly used types of recurrent neural network layers: Long Short\n",
|
|
"Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short\n",
|
|
"introduction to these layers see <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>\n",
|
|
"and <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>.\n",
|
|
"\n",
|
|
"The first network created below is similar to the previous network,\n",
|
|
"but it replaces the SimpleRNN layers with LSTM layers. The second\n",
|
|
"network below has two hidden layers made up of GRUs, which are\n",
|
|
"preceeded by two dense (feeddorward) neural network layers. These\n",
|
|
"dense layers \"preprocess\" the data before it reaches the recurrent\n",
|
|
"layers. This architecture has been shown to improve the performance\n",
|
|
"of recurrent neural networks (see the link above and also\n",
|
|
"<https://arxiv.org/pdf/1807.02857.pdf>."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "6f0e9b62",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons on the input/output layer and the number of neurons in the hidden layer\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input Layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)\n",
|
|
" rnn= LSTM(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True, activation='tanh')(inp)\n",
|
|
" rnn1 = LSTM(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True, activation='tanh')(rnn)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn1)\n",
|
|
" # Define the midel\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the model\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with four hidden layers (two dense followed by\n",
|
|
" two GRU layers) and returns the model.\n",
|
|
" \"\"\" \n",
|
|
" # Number of neurons on the input/output layers and hidden layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden Dense (feedforward) layers\n",
|
|
" dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)\n",
|
|
" dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)\n",
|
|
" # Hidden GRU layers\n",
|
|
" rnn1 = GRU(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True)(dnn1)\n",
|
|
" rnn = GRU(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True)(rnn1)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Define the model\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the mdoel\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Change the method name to reflect which network you want to use\n",
|
|
"model = dnn2_gru2(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)\n",
|
|
"\n",
|
|
"\n",
|
|
"# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)\n",
|
|
"# \n",
|
|
"# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"# Reshape the data for Keras specifications\n",
|
|
"X_train = X_train.reshape((dim, 1))\n",
|
|
"y_train = y_train.reshape((dim, 1))\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Set the sequence length to 1 for regular data formatting \n",
|
|
"model = rnn(length_of_sequences = 1)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(X_train, y_train, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict the remaining data points\n",
|
|
"X_pred = X_tot[dim:]\n",
|
|
"X_pred = X_pred.reshape((len(X_pred), 1))\n",
|
|
"y_model = model.predict(X_pred)\n",
|
|
"y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))\n",
|
|
"\n",
|
|
"# Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
"# for the training data.\n",
|
|
"fig, ax = plt.subplots()\n",
|
|
"ax.plot(X_tot, y_tot, label=\"true\", linewidth=3)\n",
|
|
"ax.plot(X_tot, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
"ax.legend()\n",
|
|
"# Created a red region to represent the points used in the training data.\n",
|
|
"ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0752ba7f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Generative Models\n",
|
|
"\n",
|
|
"**Generative models** describe a class of statistical models that are a contrast\n",
|
|
"to **discriminative models**. Informally we say that generative models can\n",
|
|
"generate new data instances while discriminative models discriminate between\n",
|
|
"different kinds of data instances. A generative model could generate new photos\n",
|
|
"of animals that look like 'real' animals while a discriminative model could tell\n",
|
|
"a dog from a cat. More formally, given a data set $x$ and a set of labels /\n",
|
|
"targets $y$. Generative models capture the joint probability $p(x, y)$, or\n",
|
|
"just $p(x)$ if there are no labels, while discriminative models capture the\n",
|
|
"conditional probability $p(y | x)$. Discriminative models generally try to draw\n",
|
|
"boundaries in the data space (often high dimensional), while generative models\n",
|
|
"try to model how data is placed throughout the space.\n",
|
|
"\n",
|
|
"**Note**: this material is thanks to Linus Ekstrøm."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "784138f8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Generative Adversarial Networks\n",
|
|
"\n",
|
|
"**Generative Adversarial Networks** are a type of unsupervised machine learning\n",
|
|
"algorithm proposed by [Goodfellow et. al](https://arxiv.org/pdf/1406.2661.pdf)\n",
|
|
"in 2014 (short and good article).\n",
|
|
"\n",
|
|
"The simplest formulation of\n",
|
|
"the model is based on a game theoretic approach, *zero sum game*, where we pit\n",
|
|
"two neural networks against one another. We define two rival networks, one\n",
|
|
"generator $g$, and one discriminator $d$. The generator directly produces\n",
|
|
"samples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a42f89ee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" x = g(z; \\theta^{(g)})\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abe7212f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The discriminator attempts to distinguish between samples drawn from the\n",
|
|
"training data and samples drawn from the generator. In other words, it tries to\n",
|
|
"tell the difference between the fake data produced by $g$ and the actual data\n",
|
|
"samples we want to do prediction on. The discriminator outputs a probability\n",
|
|
"value given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0821eaee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" d(x; \\theta^{(d)})\n",
|
|
"\\label{_auto2} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55e0ccf6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"indicating the probability that $x$ is a real training example rather than a\n",
|
|
"fake sample the generator has generated. The simplest way to formulate the\n",
|
|
"learning process in a generative adversarial network is a zero-sum game, in\n",
|
|
"which a function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f37ece14",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto3} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d6d6d5fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"determines the reward for the discriminator, while the generator gets the\n",
|
|
"conjugate reward"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c74b45c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" -v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto4} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "605ac8c9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"During learning both of the networks maximize their own reward function, so that\n",
|
|
"the generator gets better and better at tricking the discriminator, while the\n",
|
|
"discriminator gets better and better at telling the difference between the fake\n",
|
|
"and real data. The generator and discriminator alternate on which one trains at\n",
|
|
"one time (i.e. for one epoch). In other words, we keep the generator constant\n",
|
|
"and train the discriminator, then we keep the discriminator constant to train\n",
|
|
"the generator and repeat. It is this back and forth dynamic which lets GANs\n",
|
|
"tackle otherwise intractable generative problems. As the generator improves with\n",
|
|
" training, the discriminator's performance gets worse because it cannot easily\n",
|
|
" tell the difference between real and fake. If the generator ends up succeeding\n",
|
|
" perfectly, the the discriminator will do no better than random guessing i.e.\n",
|
|
" 50\\%. This progression in the training poses a problem for the convergence\n",
|
|
" criteria for GANs. The discriminator feedback gets less meaningful over time,\n",
|
|
" if we continue training after this point then the generator is effectively\n",
|
|
" training on junk data which can undo the learning up to that point. Therefore,\n",
|
|
" we stop training when the discriminator starts outputting $1/2$ everywhere.\n",
|
|
"\n",
|
|
"At convergence we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfec7462",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto5\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" g^* = \\underset{g}{\\mathrm{argmin}}\\hspace{2pt}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto5} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37c65d4c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The default choice for $v$ is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c868a092",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto6\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)}) = \\mathbb{E}_{x\\sim p_\\mathrm{data}}\\log d(x)\n",
|
|
" + \\mathbb{E}_{x\\sim p_\\mathrm{model}}\n",
|
|
" \\log (1 - d(x))\n",
|
|
"\\label{_auto6} \\tag{6}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ad465af3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The main motivation for the design of GANs is that the learning process requires\n",
|
|
"neither approximate inference (variational autoencoders for example) nor\n",
|
|
"approximation of a partition function. In the case where"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "27858a4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto7\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto7} \\tag{7}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86006023",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"is convex in $\\theta^{(g)} then the procedure is guaranteed to converge and is\n",
|
|
"asymptotically consistent\n",
|
|
"( [Seth Lloyd on QuGANs](https://arxiv.org/pdf/1804.09139.pdf) ).\n",
|
|
"\n",
|
|
"This is in\n",
|
|
"general not the case and it is possible to get situations where the training\n",
|
|
"process never converges because the generator and discriminator chase one\n",
|
|
"another around in the parameter space indefinitely. A much deeper discussion on\n",
|
|
"the currently open research problem of GAN convergence is available\n",
|
|
"[here](https://www.deeplearningbook.org/contents/generative_models.html). To\n",
|
|
"anyone interested in learning more about GANs it is a highly recommended read.\n",
|
|
"Direct quote: \"In this best-performing formulation, the generator aims to\n",
|
|
"increase the log probability that the discriminator makes a mistake, rather than\n",
|
|
"aiming to decrease the log probability that the discriminator makes the correct\n",
|
|
"prediction.\" [Another interesting read](https://arxiv.org/abs/1701.00160)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2fee38bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Writing Our First Generative Adversarial Network\n",
|
|
"Let us now move on to actually implementing a GAN in tensorflow. We will study\n",
|
|
"the performance of our GAN on the MNIST dataset. This code is based on and\n",
|
|
"adapted from the\n",
|
|
"[google tutorial](https://www.tensorflow.org/tutorials/generative/dcgan)\n",
|
|
"\n",
|
|
"First we import our libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "004a0b53",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import time\n",
|
|
"import numpy as np\n",
|
|
"import tensorflow as tf\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from tensorflow.keras import layers\n",
|
|
"from tensorflow.keras.utils import plot_model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af161",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define our hyperparameters and import our data the usual way"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "8cbaf16a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"BUFFER_SIZE = 60000\n",
|
|
"BATCH_SIZE = 256\n",
|
|
"EPOCHS = 30\n",
|
|
"\n",
|
|
"data = tf.keras.datasets.mnist.load_data()\n",
|
|
"(train_images, train_labels), (test_images, test_labels) = data\n",
|
|
"train_images = np.reshape(train_images, (train_images.shape[0],\n",
|
|
" 28,\n",
|
|
" 28,\n",
|
|
" 1)).astype('float32')\n",
|
|
"\n",
|
|
"# we normalize between -1 and 1\n",
|
|
"train_images = (train_images - 127.5) / 127.5\n",
|
|
"training_dataset = tf.data.Dataset.from_tensor_slices(\n",
|
|
" train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "822b8cc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### MNIST and GANs\n",
|
|
"\n",
|
|
"Let's have a quick look"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "52b5965c",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.imshow(train_images[0], cmap='Greys')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "21c5199c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our two models. This is where the 'magic' happens. There are a\n",
|
|
"huge amount of possible formulations for both models. A lot of engineering and\n",
|
|
"trial and error can be done here to try to produce better performing models. For\n",
|
|
"more advanced GANs this is by far the step where you can 'make or break' a\n",
|
|
"model.\n",
|
|
"\n",
|
|
"We start with the generator. As stated in the introductory text the generator\n",
|
|
"$g$ upsamples from a random sample to the shape of what we want to predict. In\n",
|
|
"our case we are trying to predict MNIST images ($28\\times 28$ pixels)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "356759c7",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_model():\n",
|
|
" \"\"\"\n",
|
|
" The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to\n",
|
|
" produce an image from a random seed. We start with a Dense layer taking this\n",
|
|
" random sample as an input and subsequently upsample through multiple\n",
|
|
" convolutional layers.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
"\n",
|
|
"\n",
|
|
" # adding our input layer. Dense means that every neuron is connected and\n",
|
|
" # the input shape is the shape of our random noise. The units need to match\n",
|
|
" # in some sense the upsampling strides to reach our desired output shape.\n",
|
|
" # we are using 100 random numbers as our seed\n",
|
|
" model.add(layers.Dense(units=7*7*BATCH_SIZE,\n",
|
|
" use_bias=False,\n",
|
|
" input_shape=(100, )))\n",
|
|
" # we normalize the output form the Dense layer\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" # and add an activation function to our 'layer'. LeakyReLU avoids vanishing\n",
|
|
" # gradient problem\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" model.add(layers.Reshape((7, 7, BATCH_SIZE)))\n",
|
|
" assert model.output_shape == (None, 7, 7, BATCH_SIZE)\n",
|
|
" # even though we just added four keras layers we think of everything above\n",
|
|
" # as 'one' layer\n",
|
|
"\n",
|
|
" # next we add our upscaling convolutional layers\n",
|
|
" model.add(layers.Conv2DTranspose(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(1, 1),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 7, 7, 128)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 14, 14, 64)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=1,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False,\n",
|
|
" activation='tanh'))\n",
|
|
" assert model.output_shape == (None, 28, 28, 1)\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "854bcd6b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"And there we have our 'simple' generator model. Now we move on to defining our\n",
|
|
"discriminator model $d$, which is a convolutional neural network based image\n",
|
|
"classifier."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "41473304",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_model():\n",
|
|
" \"\"\"\n",
|
|
" The discriminator is a convolutional neural network based image classifier\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
" model.add(layers.Conv2D(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" input_shape=[28, 28, 1]))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
"\n",
|
|
" model.add(layers.Conv2D(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same'))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
" model.add(layers.Flatten())\n",
|
|
" model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af567",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Let us take a look at our models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "f899d4e3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generator = generator_model()\n",
|
|
"plot_model(generator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "87ef384b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"discriminator = discriminator_model()\n",
|
|
"plot_model(discriminator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2bef82d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we need a few helper objects we will use in training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "e397847a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
|
|
"generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n",
|
|
"discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "db3396cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The first object, *cross_entropy* is our loss function and the two others are\n",
|
|
"our optimizers. Notice we use the same learning rate for both $g$ and $d$. This\n",
|
|
"is because they need to improve their accuracy at approximately equal speeds to\n",
|
|
"get convergence (not necessarily exactly equal). Now we define our loss\n",
|
|
"functions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "931eaced",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_loss(fake_output):\n",
|
|
" loss = cross_entropy(tf.ones_like(fake_output), fake_output)\n",
|
|
"\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "0c4a44bb",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_loss(real_output, fake_output):\n",
|
|
" real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n",
|
|
" fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)\n",
|
|
" total_loss = real_loss + fake_loss\n",
|
|
"\n",
|
|
" return total_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fcf8f066",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a kind of seed to help us compare the learning process over\n",
|
|
"multiple training epochs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "eea2bbee",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"noise_dimension = 100\n",
|
|
"n_examples_to_generate = 16\n",
|
|
"seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94a6e341",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have everything we need to define our training step, which we will apply\n",
|
|
"for every step in our training loop. Notice the @tf.function flag signifying\n",
|
|
"that the function is tensorflow 'compiled'. Removing this flag doubles the\n",
|
|
"computation time."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "8d48470b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tf.function\n",
|
|
"def train_step(images):\n",
|
|
" noise = tf.random.normal([BATCH_SIZE, noise_dimension])\n",
|
|
"\n",
|
|
" with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n",
|
|
" generated_images = generator(noise, training=True)\n",
|
|
"\n",
|
|
" real_output = discriminator(images, training=True)\n",
|
|
" fake_output = discriminator(generated_images, training=True)\n",
|
|
"\n",
|
|
" gen_loss = generator_loss(fake_output)\n",
|
|
" disc_loss = discriminator_loss(real_output, fake_output)\n",
|
|
"\n",
|
|
" gradients_of_generator = gen_tape.gradient(gen_loss,\n",
|
|
" generator.trainable_variables)\n",
|
|
" gradients_of_discriminator = disc_tape.gradient(disc_loss,\n",
|
|
" discriminator.trainable_variables)\n",
|
|
" generator_optimizer.apply_gradients(zip(gradients_of_generator,\n",
|
|
" generator.trainable_variables))\n",
|
|
" discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,\n",
|
|
" discriminator.trainable_variables))\n",
|
|
"\n",
|
|
" return gen_loss, disc_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "62015b88",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a helper function to produce an output over our training epochs\n",
|
|
"to see the predictive progression of our generator model. **Note**: I am including\n",
|
|
"this code here, but comment it out in the training loop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "b189ed96",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_and_save_images(model, epoch, test_input):\n",
|
|
" # we're making inferences here\n",
|
|
" predictions = model(test_input, training=False)\n",
|
|
"\n",
|
|
" fig = plt.figure(figsize=(4, 4))\n",
|
|
"\n",
|
|
" for i in range(predictions.shape[0]):\n",
|
|
" plt.subplot(4, 4, i+1)\n",
|
|
" plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n",
|
|
" plt.axis('off')\n",
|
|
"\n",
|
|
" plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')\n",
|
|
" plt.close()\n",
|
|
" #plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ba2c82d5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Setting up checkpoints to periodically save our model during training so that\n",
|
|
"everything is not lost even if the program were to somehow terminate while\n",
|
|
"training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "a0e2fc8a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setting up checkpoints to save model during training\n",
|
|
"checkpoint_dir = './training_checkpoints'\n",
|
|
"checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')\n",
|
|
"checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n",
|
|
" discriminator_optimizer=discriminator_optimizer,\n",
|
|
" generator=generator,\n",
|
|
" discriminator=discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d93a0f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our training loop"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "a1275556",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def train(dataset, epochs):\n",
|
|
" generator_loss_list = []\n",
|
|
" discriminator_loss_list = []\n",
|
|
"\n",
|
|
" for epoch in range(epochs):\n",
|
|
" start = time.time()\n",
|
|
"\n",
|
|
" for image_batch in dataset:\n",
|
|
" gen_loss, disc_loss = train_step(image_batch)\n",
|
|
" generator_loss_list.append(gen_loss.numpy())\n",
|
|
" discriminator_loss_list.append(disc_loss.numpy())\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epoch + 1, seed_images)\n",
|
|
"\n",
|
|
" if (epoch + 1) % 15 == 0:\n",
|
|
" checkpoint.save(file_prefix=checkpoint_prefix)\n",
|
|
"\n",
|
|
" print(f'Time for epoch {epoch} is {time.time() - start}')\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epochs, seed_images)\n",
|
|
"\n",
|
|
" loss_file = './data/lossfile.txt'\n",
|
|
" with open(loss_file, 'w') as outfile:\n",
|
|
" outfile.write(str(generator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write(str(discriminator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ff3a75a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"To train simply call this function. **Warning**: this might take a long time so\n",
|
|
"there is a folder of a pretrained network already included in the repository."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"id": "371ed41a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"train(train_dataset, EPOCHS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "654399f1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now to avoid having to train and everything, which will take a while depending\n",
|
|
"on your computer setup we now load in the model which produced the above gif."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"id": "dec4b560",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))\n",
|
|
"restored_generator = checkpoint.generator\n",
|
|
"restored_discriminator = checkpoint.discriminator\n",
|
|
"\n",
|
|
"print(restored_generator)\n",
|
|
"print(restored_discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "296bfa5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We have successfully loaded in our latest model. Let us now play around a bit\n",
|
|
"and see what kind of things we can learn about this model. Our generator takes\n",
|
|
"an array of 100 numbers. One idea can be to try to systematically change our\n",
|
|
"input. Let us try and see what we get"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"id": "eecfbb1f",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_latent_points(number=100, scale_means=1, scale_stds=1):\n",
|
|
" latent_dim = 100\n",
|
|
" means = scale_means * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" latent_space_value_range = tf.random.normal([number, latent_dim],\n",
|
|
" means,\n",
|
|
" stds,\n",
|
|
" dtype=tf.float64)\n",
|
|
"\n",
|
|
" return latent_space_value_range\n",
|
|
"\n",
|
|
"def generate_images(latent_points):\n",
|
|
" # notice we set training to false because we are making inferences\n",
|
|
" generated_images = restored_generator.predict(latent_points)\n",
|
|
"\n",
|
|
" return generated_images"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"id": "333a593d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def plot_result(generated_images, number=100):\n",
|
|
" # obviously this assumes sqrt number is an int\n",
|
|
" fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),\n",
|
|
" figsize=(10, 10))\n",
|
|
"\n",
|
|
" for i in range(int(np.sqrt(number))):\n",
|
|
" for j in range(int(np.sqrt(number))):\n",
|
|
" axs[i, j].imshow(generated_images[i*j], cmap='Greys')\n",
|
|
" axs[i, j].axis('off')\n",
|
|
"\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"id": "2f5f0154",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generated_images = generate_images(generate_latent_points())\n",
|
|
"plot_result(generated_images)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff581bf2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We see that the generator generates images that look like MNIST\n",
|
|
"numbers: $1, 4, 7, 9$. Let's try to tweak it a bit more to see if we are able\n",
|
|
"to generate a similar plot where we generate every MNIST number. Let us now try\n",
|
|
"to 'move' a bit around in the latent space. **Note**: decrease the plot number if\n",
|
|
"these following cells take too long to run on your computer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"id": "d3617ad8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 225\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=-5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=5))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a074f93",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Again, we have found something interesting. *Moving* around using our means\n",
|
|
"takes us from digit to digit, while *moving* around using our standard\n",
|
|
"deviations seem to increase the number of different digits! In the last image\n",
|
|
"above, we can barely make out every MNIST digit. Let us make on last plot using\n",
|
|
"this information by upping the standard deviation of our Gaussian noises."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"id": "4ef8937d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 400\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=10))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "385a2d0a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"A pretty cool result! We see that our generator indeed has learned a\n",
|
|
"distribution which qualitatively looks a whole lot like the MNIST dataset.\n",
|
|
"\n",
|
|
"Another interesting way to explore the latent space of our generator model is by\n",
|
|
"interpolating between the MNIST digits. This section is largely based on\n",
|
|
"[this excellent blogpost](https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/)\n",
|
|
"by Jason Brownlee.\n",
|
|
"\n",
|
|
"So let us start by defining a function to interpolate between two points in the\n",
|
|
"latent space."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"id": "57de87b8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def interpolation(point_1, point_2, n_steps=10):\n",
|
|
" ratios = np.linspace(0, 1, num=n_steps)\n",
|
|
" vectors = []\n",
|
|
" for i, ratio in enumerate(ratios):\n",
|
|
" vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))\n",
|
|
"\n",
|
|
" return tf.stack(vectors)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfb76bb6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have all we need to do our interpolation analysis."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"id": "e25decef",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 100\n",
|
|
"latent_points = generate_latent_points(number=plot_number)\n",
|
|
"results = None\n",
|
|
"for i in range(0, 2*np.sqrt(plot_number), 2):\n",
|
|
" interpolated = interpolation(latent_points[i], latent_points[i+1])\n",
|
|
" generated_images = generate_images(interpolated)\n",
|
|
"\n",
|
|
" if results is None:\n",
|
|
" results = generated_images\n",
|
|
" else:\n",
|
|
" results = tf.stack((results, generated_images))\n",
|
|
"\n",
|
|
"plot_results(results, plot_number)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.10"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |