Morten Hjorth-Jensen
3240 lines
158 KiB
Plaintext
3240 lines
158 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d2db899",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- HTML file automatically generated from DocOnce source (https://github.com/doconce/doconce/)\n",
|
|
"doconce format html chapter13.do.txt -->"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f4908a97",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Recurrent neural networks: Overarching view\n",
|
|
"\n",
|
|
"Till now our focus has been, including convolutional neural networks\n",
|
|
"as well, on feedforward neural networks. The output or the activations\n",
|
|
"flow only in one direction, from the input layer to the output layer.\n",
|
|
"\n",
|
|
"A recurrent neural network (RNN) looks very much like a feedforward\n",
|
|
"neural network, except that it also has connections pointing\n",
|
|
"backward. \n",
|
|
"\n",
|
|
"RNNs are used to analyze time series data such as stock prices, and\n",
|
|
"tell you when to buy or sell. In autonomous driving systems, they can\n",
|
|
"anticipate car trajectories and help avoid accidents. More generally,\n",
|
|
"they can work on sequences of arbitrary lengths, rather than on\n",
|
|
"fixed-sized inputs like all the nets we have discussed so far. For\n",
|
|
"example, they can take sentences, documents, or audio samples as\n",
|
|
"input, making them extremely useful for natural language processing\n",
|
|
"systems such as automatic translation and speech-to-text.\n",
|
|
"\n",
|
|
"More to text to be added"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "43cb4913",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## A simple example"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "cf6b9dab",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAACYwUlEQVR4nO2dd5wU5f3HP7NX9g44juY1OkqTJoIKSFMUFXtMYkVN1IiKjZ/BoCm2iEmMIcZCNCpRbDGgsaKgNKV3pCoCBwfHAcLdUa7tzu+PY3efmX1m5pm6s7vf9+uF3s4+M/PsPM88z/f5tkeSZVkGQRAEQRBEkhBIdAUIgiAIgiDMQMILQRAEQRBJBQkvBEEQBEEkFSS8EARBEASRVJDwQhAEQRBEUkHCC0EQBEEQSQUJLwRBEARBJBUkvBAEQRAEkVRkJroCThMOh7Fnzx7k5eVBkqREV4cgCIIgCAFkWUZ1dTVKSkoQCOjrVlJOeNmzZw/at2+f6GoQBEEQBGGBXbt2oV27drplUk54ycvLA9D445s3b57g2hAEQRAEIUJVVRXat28fncf1SDnhJWIqat68OQkvBEEQBJFkiLh8kMMuQRAEQRBJBQkvBEEQBEEkFSS8EARBEASRVJDwQhAEQRBEUkHCC0EQBEEQSQUJLwRBEARBJBUkvBAEQRAEkVSQ8EIQBEEQRFLhqvDy4osvom/fvtGEcYMHD8Znn32me878+fMxYMAA5OTkoEuXLpg6daqbVSQIgiAIIslwVXhp164dnnrqKaxYsQIrVqzAueeei8svvxwbNmzglt++fTvGjBmDYcOGYfXq1XjooYdwzz33YMaMGW5WkyAIgiCIJEKSZVn28oatWrXCX/7yF9xyyy1x3z344IP48MMPsWnTpuixcePGYe3atVi8eLHQ9auqqpCfn4/KykraHoAgCIIgkgQz87dnPi+hUAjvvPMOjh49isGDB3PLLF68GKNHj1Ycu+CCC7BixQrU19dzz6mtrUVVVZXiH0EQBEEQqYvrwsv69evRrFkzBINBjBs3Du+//z5OPfVUbtny8nIUFhYqjhUWFqKhoQEHDhzgnjN58mTk5+dH/7Vv397x3+BH1u0+jNe+2Y5w2FPFGUEQBEEkHNeFl+7du2PNmjVYsmQJ7rjjDtx0003YuHGjZnn1bpIRq5bWLpOTJk1CZWVl9N+uXbucq7yPuey5b/DoRxvx0bo9ia4KQRAEQXhKpts3yM7OximnnAIAGDhwIJYvX46///3v+Oc//xlXtqioCOXl5YpjFRUVyMzMROvWrbnXDwaDCAaDzlc8SdhWcSTRVSAIgiAIT/E8z4ssy6itreV+N3jwYMyePVtx7IsvvsDAgQORlZXlRfWSjswMStVD2MNjn32CIAjbuDrzPfTQQ1i4cCF27NiB9evX4+GHH8a8efNw/fXXA2g0+dx4443R8uPGjcPOnTsxYcIEbNq0Ca+++ipeeeUVPPDAA25WM6nJzOCb0whChE/W7UW/R7/Awu/2J7oqBEEQwrgqvOzbtw9jx45F9+7dMWrUKCxduhSzZs3C+eefDwDYu3cvSktLo+U7d+6MTz/9FPPmzcNpp52Gxx9/HM8++yyuuuoqN6uZ1GQFSPNCWOeut1ahqqYBY19ZluiqEEnG8bpQoqtApDGu+ry88sorut9PmzYt7tiIESOwatUql2qUepDmhSAIr1m87SCufXkJxo04Gb+5qEeiq0OkIbRsT3LI54UgCK/546eNEaNT529LcE2IdIVmviSEdbDMCmhrXrbtP4IjtQ1eVIkgiDRhdekhfFtGyUCJxELCSxISYhLTZWloXr4tq8Sov87HiD/P9apaBEGkAVe+sCjRVSAI9/O8EM5TH4oJL+VVNQiHZQRUGpjZG/cBAA4erfO0bgRBpCazN+5DbQM56RL+gISXJKQuFI7+/ZfPt6Ds8HE8eWUfRZkQbRuQVtQ1hLF1XzVaNs1G2xa5ia4OkWKEwjJue31FoqtBEFFIeElC6hrCis9vLS2NE14aSHhJG8JhGd1++1n0846nLk5gbYhUJEyJDAmfQT4vSUh9KGxYZta3ez2oCeEHahuM+wNBEEQqQcJLEmIkvGzbfwQ7Dh7zqDZEotHYs5QgHENP8fKfFemxGS7hL0h48Qlzt1Rg0JNf4uvvDhiWNRJedh867lS1CIJIQVaXHsLWfdXC5fXMRhP/u45SMhCeQ8KLT/jFa8tRXlWDG15ZaljWyEyQqZP7hUg91PPKnsPH0SBgWiTSkwNHanHlC4sw+m8LHLtmbT1FIRHeQsJLEsKGSvPIIOElrZCh7A9DnvoK171sLAQT6cmew+Y1s+SvS/gNijZKIiqP1+PT9XvRpllQtxxpXtIL3sSybMeP3leESAqsBCKqBWSCSDQkvCQBlcfq8eCMdZi1oVyovDphHZHa0LRCmIHnvyLLMp76bDPat2qCGwZ15Hyvf03qg4TXkNkoCfjbnK3CggtAKt50QxZscNFyRGrD6wfrdlfinwt+wG8/+FZxfOF3+3HP26tx6Jh+pm7qWoTXkPDiUxZ+tx8zV+0GAOw/UmvqXEoolV6ItPaq0kPo//hs3bDWhd/td65ShG/hDQ/VNfxoobGvLMOHa/fgT7O26F+TdC+Ex5Dw4lPGvrIME/6zFqtKDyFgMpEHbQ2QXojIqndOX4XDx+ox8b/rNMuMfWUZ9/j8rfsxZPKXJNykCGx3EdXGlVdS+gXCX5Dw4nMWbj0Asy4sYRJe0guB5razMr5j+krsqazRFG6I5II3PhitjzIDBlOFQPd6c+lOXPbc19hfbU6TTBA8SHjxOZvLq8xrXshslFa4rbLPzqRhIpVgZRfRoSIzQ38MElkvPfz+t1i3uxLPzNY3QRGECDQqJZAdB47i34t26G4zf6wuBLOxQyJmo10/HiMHzhTB7WZs37KJuzcgPIUVdiN/GY0xWRn6U4UZP7tjdZTQjrAPhUonkJFPzwMAHDyq7ckflmVIDvu8vLF4B373vw0YO6gjHr+it6lrE/5DZNqwI+CUtMjB+rJK6xcgfIWs0LzIACRD6cUod5QZ4YXWTIQTkObFB6wwSChm1ufFSHh56rPNAIA3luw0d2HCl7itQcvOzPDsXoT7KIQXzvcRnxjWQdtI80LdgvAa0rz4HFmGaZ8Xo1WQ2esR/sbteYN1dwjLys9E8sGOD2WHjmP3oeOKLUXCsoyqY/UKB21jnxeSXghvIeHFY77ctA9rd1fi/vO6apZhowFkyDBy9FdjuCcfTT4phZPzRigsx+2NxWZsbgiHkRHIUJ9GJBGsoBExXd/HjEdhGTh0rF5xjlG0kZkAR1o7EU5AwovH3PLvFQCAXiXNo8fUkw87uDT+6Wy0EY0dqYVItJHohFEfihdOWE0d5RBKfngtuHLnoejfPC2KXZ8XK5tBEoQeJLwkiH1VNdG/1ZMPOz+EZdnQ50U9sLCaG965tPdRiiGS50VWf5a5TrgNHOEkg4SXlMLIb4n3tdGYoXfNw8fqMOSpr3SvTxBmIYfdBKE3FLCrmLCAz4t6YAkpzE7m7k34C1mWcdebq3D326u1y1i47nsrd+Oy576JO17fEG9zZPsXCS/JD094YCMaeVoUo8SXel9/X3FEuG6E/wmFZeyrqkl4skESXnxIWBUOYKQoYVfGx+tC+HjdntjpBgMV4W8OHKnDJ+v34qO1e1B5rB7/W1OGxz/eqPSLsiBPvLucv8dRfTheeGEDTXiaGSK54DWhpPg+voBRu5PDbvqwr6oGZz35Jc5mtGmJgISXRKEjQCgyYMI4zwvrYPn7/32LuVuUe9CoVbokuiQPyoRiMu59Zw1e+Xo7vtxcwS0jfF2NyaY+FH+cLcoKTYdP7DRcHwpj7uYKVB6vV59K+BCeoMEOMTw5pYEj1LJUVFHK/3Qhon01G0jiNCS8+BC1w66RooQVXt5buZtzPeVnVhh6ffEOHKeMl0nHoWOxxIZWFr1apzRwQtXYspEV+LNffofTHpuNd5aV4oW52/CLactx/b+WmK8I4TlG/YUn2DZwhFqWG1+lfa/Shcj8ZLjflcuQ8JIgdH1eGGmj0WFXXPPCvZ5a88IU//3/NuBvc7bqnk/4A7YZ2RZ3UmFfzxNemBtEVl3PzG7sM7+ZuR7vr24UmL8tq3KwJoRbGDnsWtG8EKlNOCzjwJFG7VpkAZPouA8SXnyI0mxk3EmM+lBYllF2+Dj+tfAHVNfUx5Vf8sNBC7UkvEASMPJZyXqrdQrPbMSidtiVJPKhSjZ4LawwDXI6x+cb9pm7hyxjVekhHDxC5qRU4K63VmHgE3Ow6PsD0cW10aLZbVwVXiZPnowzzjgDeXl5KCgowBVXXIEtW/R3FJ03bx4kSYr7t3nzZjer6jnseK+X50Uk2sho7pBl4Irnv8ETn2zCHz7cEFc+J4uSjiUbbJ9w0leSp3lhpzu14yaJLckHTziRDb43y7dlVfjJC4sw/M9zbV+LSDyffVsOAPjngh+iecRSWniZP38+7rrrLixZsgSzZ89GQ0MDRo8ejaNHjxqeu2XLFuzduzf6r2tX7Yy0qQY7eITCYVMrW15RWUY0rG3hdwfihCESXpIPu8oOrenJyGE3XvNC4kuywTMLySo/O7ts3NuYQ+hoXQjVNQ32L0j4hoj/U6KFF1eT1M2aNUvx+bXXXkNBQQFWrlyJ4cOH655bUFCAFi1auFg7/6AeK1jzcihszrYo8a6ndgBWfZ+bRdbDZIBtV7OaF9H5iOuwqye8CF6X8A9GZkYncvkUNM+J/r1xL/lCpQqSFJtPMhK8cPF01qqsbJTGW7VqZVi2f//+KC4uxqhRozB3rrbqsba2FlVVVYp/yYCeL4Na82K8kWLse95KWD0UqcuQ5sW/KM2LMv+4SZfd8W+t0pR4tu1v1IrOXLUbC7bux/Nzv8fm8tg7xZvYSIDxF+WVNfjPil2oqedHEfKaXmmqdkD1wlyiTpX48FhdA2au2o3KYxRan4zEQqVTWPPCIssyJkyYgKFDh6J3796a5YqLi/HSSy9hwIABqK2txRtvvIFRo0Zh3rx5XG3N5MmT8eijj7pZdddZtv1Hxed1uw9H/24IG28PwMIrqhdtBAA5mSS8+BWRpjc713y8bi/6tsvnfvfQ++vRt10+JvxnLfd7ddQJ7VDuPy5+diEOHq3Drh+P4f9Gd4/7nifsss0qy9acwBXXY7W9qu/mbKrAnE0VOKtzK7x7+2Bb9yG8JxYqnSbCy/jx47Fu3Tp8/fXXuuW6d++O7t1jL9zgwYOxa9cuPP3001zhZdKkSZgwYUL0c1VVFdq3b+9cxV1Cb8wf/1YsFXw4bJykzghZMd/IcffOzSbhxa/IGn+zfcLKNKNnGth58Jj4eSS7+I6DRxtzAC3Yup8rvPCinlmBJixbSXuouh4bMakhCC1VLdoI71lVeggvzf8BVTX1uHFwJ1zYu0i3vISYz0taaF7uvvtufPjhh1iwYAHatWtn+vxBgwZh+vTp3O+CwSCCwaDdKvoKNqKjIRwvbKhhv+eVjdO8gMxGyYJWCKsiz4uFVbKW8NKjKE/XbMA9jwQYXxLU0Kjy2nfJDz8y39t32nU6eolwh5+8sCj696JtB7HjqYt1y0uSFIs2SmWfF1mWMX78eMycORNfffUVOnfubOk6q1evRnFxscO1SyyizR4OGyepMyJOdFFdLjsz1g0+WF2GpZT3xTco1sDMnwGbmhetvWqaZGeYEl5IbvEvQY4j/pHaBuytrOGUjhGWZdtmI1nhQyN2Tl1DmOswTiSWv8/5Lvq3hJjmLqWjje666y689dZb+N///oe8vDyUlzfGiufn5yM3NxdAo9mnrKwMr7/+OgBgypQp6NSpE3r16oW6ujpMnz4dM2bMwIwZM9ysqm/ZU1kTzWYqgsSJN1JORhxh6MT3G/dU4b531wCAoQROeIRC8xL7Wy9PkAhak4RscL24PC8kvfiW7Ix44eWMJ+bguIYjbwTZAbNRWENjqEVdQxgDnpiN1k2zMe/X59i8O+EUuw8di8vA7pc8L64KLy+++CIAYOTIkYrjr732Gm6++WYAwN69e1FaWhr9rq6uDg888ADKysqQm5uLXr164ZNPPsGYMWPcrKrnODnoS5ofGok3G6m/b/z/zoPG+XcIb2FbjnWWlTRLiaGleZFl/ckmZGCCJPwDq1GNYCS4AM6Yjdg+KXKtbfuPoLqmgXLC+IxjnH3vQifGoZQWXkRUj9OmTVN8njhxIiZOnOhSjdKTuGZQ9bnIhGS07T3hPcodnWN/G+0CbISWz4us8x0AhDhJ7Eh88SdBjvAiQqPZyN699XIDad2T8BeyLMeFuUtSY+4xIPGRhp5FGxHewOtO6nEhXvPSWMCJ5FSEeyjDlO1tD6ApqBpMXOrzArS3kW/Rctg1wok9GM067Kqjk6hPJZ5b/70CX26uiDsemSfSJlSaUOKkut1MtJEsx0vM4bCM3//vW7y+eKdjdSKcgfU+YIVLdtwQ8VBQzx96mhe9ySY+ZxBNMn6FZzYSISwbRziKXCOCUAZoWfk3davEwxNcACn9ktQR3sATioyS1IVlkODiU9imY7UeijwvGpND5fF65OdmAUB0O/votTQcdtftrsRJzbR3ECbTYvJg1Wxk1YLDakyMdqlWo5fUjvAXaREqTejgYLvvq6rFTa8uw+5DxzQ3ZlTeWqV5IXuzb2FbhtWWKPO88M+9+p+LAQBfbooXRvSEEP6KK1IHlQ1csyThBTX1IawuPYTwifZk/Qytal5CFn1eFNoT5riQ2UhxHRqP/IokIdrXEu2wS8KLh7j5Us7fuh8PvLfW0OdFBkfzQqtpX7C38jgeeG8tNuypjB5j+4zCbMS8uVpmo83l1aipD+GtpaVx31nVoMQpbCQSYBLJ7W+sxJUvLMJLC38AANQyDpaWfV5k2fR+WYC2ACLS1UjzkjyESHhJPxR5Oly4/pbyahzlhLYZ+SnwBhcSaLznnrdX478rd+PiZ2NbaGiajQQddvdX13InA6vO2aR58Rfzt+4HAPx70Q4ASuHFquZFlmVLEWyyhp+LyKJN7fNCuMOuH4/hPyt2xUURqdEb//0ivJDPi4e4rQ49pLFLq3Gel/h6hWQZAZqaPGVLebXu9yLq9/N6FmDOppjZR2uTPavCS3ySOuojfiDihF/L5HGxGg0SloGAhbFKoXlhPlUJ5G5RCD6ke3GNkU/PQygs48CRWtw58hTNcnUaPnESYj4viQ6VJs2LhygzpHrX8HFpXuIcduMHC/KD8Qdskqh6ZkDhRXMU5+fg95f0UpxvzQCgjXpFJkkUGeInag1W1CKEw1Y1L7G/v/k+tsXIJ+v2Gp+rcR3CWSKLlsXb9LeAqdfZpiGmeXGuXlYg4cVDErWiUK+840Ol489xItcDYQ61QLvo+wO4YMqC6Oc731wV/Vvpx9T4QUK8IOH0RFCvSlJHcos/iLQ7q1GzOt40XsKK5qXxnM3lVfjvyt3m7klmak8xMvmo3/MIjUnqInleEis+kPDiIexE4uWgrx4XSPPiHyqPx0x96nZ54pNNis+HGbMgb6UqSfGbbspwVoChfuFPIu3OmvXshDzbiTbavFff/MmDHaOoi7mPlknx3KfnYe6WCl2fGL/keSHhxUMS9VIa+byo96vROkY4y1tLS9Hv0S/wytfbud/rmWOUPgLsOd6GwZPPiz+IOHA7kSU7LFvbciKCtUgl8nnxEi1/lR8OHMUvXluuazaKjCkZCX71SXjxkMSZjWJ//3i0Dmt3V2p+Hz1GZiPXeej99QCAxz/eaPpcXliqxAlblmVnQ095W01oZYtetv1HXP7c11iz67CDNSB4xDQvsRfXars37m1kQQCxI/CQ5sVTjMxGWr5TEpNhN4PMRqlLTX0Ir369HTsONO7WrHTY9a4eRqtvrWgjwlvMdAleQjBJ4q2orE1EWhhla2b5+T8XY+3uSox9Zalj9yf4RNo95IDZqDHPi3kiZ1m5L+V5cR82q7ZRpJCe5qWBHHZTnylzvsNjH2/EyKfnAUhc5kij2/JUzeTb4G94OTUkxPu8OO0HGX85Y5GrWiBUlrBHpBUaHHDYlWVr77+dIUPp80Jjj9M0hMIY8tRX0c9G/irHOPnCAMqwmzYs3a4MR0uU5sVoLOB9T97//kbmfNIyGzl6X47zt1Ffzk70Ei0dONEGTry3YYu2Rjt3JoHFXUp/PIaK6tj+Zkb+Kn/8RNuUTXle0hGfOOyq4Wte3KoNEUG9cDHj/MrzEZCi/2G+c7jTGTl/87C6QSAhDlfzcuLP43Uh1NTzV9I8whb9pOwIIFr7IhHuYKR5WVV6mHtcGSpNwkvawA78Wk6Obt+XB0UbJQajPAm60UaK6IxIeSmuXzVm2LVaQ2cIZlnbY4cQR+L4vABAXUMYPX8/C6c/PltYKxMKy9bMRqbPYM+176tD8AmHZazYcUhxzM6O0BQqnQbEu04mBqMxizeokdnIfezYjLU0L+pLXvnCN/j6+wOG1/v1Bd0F7yvusBuBNC/uw9e8yCg7fBxAow+DVsp3NXbzvFhy2GWrRkOPo7zy9XZMnLFOcczO2BNZ2NoRgJyARhUXmLNxHy79x9fYtv+o4njinGDNa17IYdd91AOImaHg7rdXY+kPjT5VMiO9qE1PNfViE5boYKaWaUXs3jlZNMy4TSzaSNneRhvw8bBqNrIjdChC/0l6cZQ3l+6MO2ZVayJBijnsJjjRC40qLnDr6yuwvqxSkT0V4Kd09wIjJQr5vCSGOOHF5Fhw9UtLsOvHYzGzEaxnbhYdy7h5XgwqHswks5HbxLYHiB2TZaXwIrogCctWzUYnQqVNn8nfq4twBt7jtKo1kSFHI5ESrXmhXaU9JFErCiMTEE94cSJTJ6GPEw5v97yzGhMv6AGAvz2AKKKRAyJ9+L8rd2P2xvLoZ9K8eEdIlaSuLhRz1BV9pS1HG9nRvJDDrmvw2sWq2ejT9bH3OtHRRiS8eEiiskha0bzIsozJn23CoaN1+NNVfSkNvAsYDSAiTt1lh46rNma0qA4WPC9+n6z48x54b63iM2le3CcykaijjeoaYp9FtSmNsot1h11r2Xll7t+EOzgheJDDbhqRKOHFaCDiDWoNYRn/nP8D/rNiN7YfOMo5i7BDTX1IkXeBh8j4EpCk6KwhknNF+zqCBVV95Xh9CJv2VumeEiTNi+vwdpUGoHDSFd3yIyzLlnaVtyN0hEnz4hq88T8jYF9ITLTZiEYVD0lUCmwrGXbZvBBkQXKev3y+hXPU/GDAyC6ID5QWR9xspOTHo3WOXZuwTnRvoxA7xsiot+TzYjHPi4VzYvcknxcvCQQk264BCVa8kPDiJQqPeg/fUOM8L/HHjjPCS6LTQKciszfuc+xaskLzYq2tRJuYotD8STTaSCUEsJoX0dxNidiYUbmwoz7mJFyfF0myncuLzEZpBOs46yfNC8+hl9W8kOziXyQoB3urTSUq9JDs4k8irRdSjTFWoo1kWbakbbUjdITIbuQpGQHJkmmQJdEaVRJeEoWHL+iNry7T/b6BK7ywg57jVSI4WBkLJEliNC+S5QHFqtmI8Akch11A5fMiHG0E2NncyFqodNxlCBcJSBIabEovid6yjIQXD/GrapSneTnO7CoaCss4fKwOk2auw4odP3pZNcIApc+L+w67ZDbyJ1HNiyrRixXNS+P2AObrYMvnhbMnE+EMWqHSpHkhhElUtJERPNsn6/MSCst44pNNeHvZLvx06mIvq5ZWqIcCkaFBkmL+U3bGEuGByEf9logRddhVmY3qWZ8XQYlEhrXxKXqOhXP9urBLVTICDvi8UJ6X5OXvc77DrkPH8JefiuVBSVS0kRFGPi+hsIwf9h/xskqEIBIkZmNG6wIMyS7JDc/n5R9ffa8oIzxXybLFPC82fF78tJpLA9jdoa2SaF9I0rxYpD4Uxt/mbMV/V+7G9xViE7sy2sidelmBN3AoQ6VpLeQF1nxeEMvzAvd9Xqxs2KkVuUKbfzoHL9pIjbDDLqz5udmLNnLmOkQ8vPdPlu0LL4mOQnVVeJk8eTLOOOMM5OXloaCgAFdccQW2bOHlt1Ayf/58DBgwADk5OejSpQumTp3qZjUtsfdwTfTvLEHPpUTtbWREAydWmnXYDYWt7TJLuA8bbSRJ1qONRAciu86YEXYePIoBT8zGc199Z+GKhJpokjpe3oMTiM5VsmwxS270/+bPZX11aKjxBrtzUKKzrrsqvMyfPx933XUXlixZgtmzZ6OhoQGjR4/G0aPaGVu3b9+OMWPGYNiwYVi9ejUeeugh3HPPPZgxY4abVTXN7kPHon+bCUGM/e14lSzDqz/r88KLRmKpD4WxbvdhWkk7yHsrdmHt7krDcjsOHkPtCUHTzvYAojkbrPRbXv+a/OlmHDpWj6e/2Gr+gkQckfSEeu8qu9Ju0yyoWc6qU7bV3FUrdx7CIx9tjH7eqcroXVMfil5796FjeH7u96g8ptz0ltCG1yqybH9BmmifF1eFl1mzZuHmm29Gr1690K9fP7z22msoLS3FypUrNc+ZOnUqOnTogClTpqBnz5649dZb8ctf/hJPP/20m1U1ze7Dx6N/m1HH8v52C9GN/3jqw+MmzEb3v7sGlz33DZ6f+71OqfTl34t24BevLVOY4niw+XGnzBHXSDwXee6SnQy7YuWsTGyRc74tq8T8rfsB2HMuJjhobA/AwgoXeu0ty9baOXKK2VNvf0M5H1z3r6X4clNjEseK6hr0+N0s3PTacgDA9f9air98vgUP/Hdt3HUIcSzuvakgrUKlKysbV5KtWrXSLLN48WKMHj1aceyCCy7AihUrUF8fL23X1taiqqpK8c8L2FBiI81EBMWA4IHqRdQUwBuoalSh0nr1/XjdXgDAi/O3maxhevCHDzdg7pb9eHf5LuFzyhjh2IgNexr7vL1QafekiUhI5iX/+Bo3vboMpQeP6Z9AKKipD2FfVY1umUjr6Y1F7Fd6zW0n2uirzfvwm5nrTZ13vK4h7tibS0sBAB+u2QMAWHBC6N15ou/M2eRchup0pLGNyWwkhCzLmDBhAoYOHYrevXtrlisvL0dhYaHiWGFhIRoaGnDgwIG48pMnT0Z+fn70X/v27R2vOw+24dnVzitfb9c5h/nb4n2vPVP894kKL9y9jRoYzYvqe1mW8c/527Bom7I97DqApTpHauMH6QhObBfh1+0BQrKMY8wEtfswCS9mGPXX+TjryS91N0iNNLte+0S+a/SN0m5wqyYFGTJ+OW2F6fOOczSSRv3MT2Z3v8N7VrL+elSItNmYcfz48Vi3bh3efvttw7LqATiWxyL+YU2aNAmVlZXRf7t2ia9u7cC2O5vs5/GPN8aVjZaz6fNyz6iu6FHUXLi8HeFFrVliS3y+YR8mf7YZ17281PA6hBhOPDo7Q4mZ7QH2G+yGrWZLeTVO/f3n0c9ZGQEyG5kgooX7UkfbsOSHHzH5s02o1TFNRt5PIw2dXbORWXh9PywDn63fiyc+2WTtooQuTgSMBBJsNvIkz8vdd9+NDz/8EAsWLEC7du10yxYVFaG8vFxxrKKiApmZmWjdunVc+WAwiGBQ2/nMLdgXNRKeaLR6VlqNzHceCeZi6+34vCiijVQrsR0H+StAytWgj1Gb253Q7ahxRc1GX27ahzeW7NT8nvcbK48rzb2i0XmEOf45/wfk5WgP6ew2EnqtbS3LizVt8rvLS/nXkmXc8eYqC1ck1PBa0wnNS0o77MqyjPHjx2PmzJn46quv0LlzZ8NzBg8ejNmzZyuOffHFFxg4cCCysrLcqqpp2HYPhcPY9eMxzPq2XLM8YN9sZNYskCk4SfBWPmqzEfsCaK3KSHaxjizLtjQngL2kUaLn7qnU972QZehOoIC4UE0oEdGGVNfomCZPvMNGj1+WYThADevahnOe+QHgwRl8/xitS5HGzjx8s5FVETVGSgsvd911F6ZPn4633noLeXl5KC8vR3l5OY4fjzkjTpo0CTfeeGP087hx47Bz505MmDABmzZtwquvvopXXnkFDzzwgJtVtUUoDAz781zDlQLbWaxM9GaTkInaJHkaEzateIMqzwuFRFtDT/B04olajzVybiAKC4QxkOYlMcTMRpJhXzQSlN645SzueU6hNbE2ycpw8C7pi1WnbJaUTlL34osvorKyEiNHjkRxcXH037vvvhsts3fvXpSWxlSHnTt3xqeffop58+bhtNNOw+OPP45nn30WV111lZtVNY2Ww67+OczfFu9rZo6x4/PCJq4Lq4UXkl0sobcylWUHvPdtnO7UIios4C+REZBsCVrpishko9eO0ffW4NFbzajtpOZVa9PAJkHa0cYs/Dwv9q+baAWqqz1BRI04bdq0uGMjRozAqlXJY+8UdW5TOuxa8HmRTPq8ZFgXXhQbuqkGM9pZ2D7qScYJBzo7Y4lTmhcZIjsSy/Yqm6aI9JDsjABqG/gzfzTwAcYOu9ZecefGBa0xpml2BvY7dpfUo/TgMWw7cATndC/QLWffaJTiZqNURuGwa2K3VjuYzaBqJ5StntG8qH8fmY2soZfmp/JYvancLjw82VXagEarEYW5JorsTO0hPfLaBiTJsK94GW3EvZbG8dxs0rzoMfwvc/GL15Zj0ffxaUVYrG4BwULCS5LCDtAfrCkTO8dkqHSrptmKz42aF/EOI5rynUdDSL23Eeuwa/myhAZD/zTX9jVs+bw4NBKEZRHNC2EFkTEjW8efKOrzYuiw6120kV4deJC7lBirdx2O/s112IX99kp0qDR1BYuwHWLmKlHhhflboOu0jhNeJFdCpXnU6WleaOlsCb2nVhfSMPKbwA+al7DAik4GWY2sYFfRrzAb6bRA2KI3p6M+L1rRRtRzhDAWUClUOm2x0u5mt30/t2e83dKU5sVG52pgPObUwgrlc/EnfhBeZAHNC3Uf99B7N0XNRo1J6szfe+n2g+ZP0oAWSPZQvs/xz1J2QPdCwksSsePAUTz20UbsrTxuce8PxmwkUL5pdiYevaxX9HNjnhfx+9npW2y0kTpUmsYVcdg2Z5vDjffeXqi0M3UQ07xQB9JjzsZ9+NfCH+KOqx/rwSPxmY71/O+iAoGkr/mSIVvyh/j9/zaYPkezDlqaF1K8CGH4PjugeUl0qDR5P5ng5/9cjIrqWizf8SMu6lNk+nwjzcsTV/TGbz/4Nvo5ICk7iNk8L3YkYzbaKC5JHTk1CKNoc5fvZWdgd2qTNVHNS6I3dfMzt77euD/Q6R1b4vQOLaPHWYHiw7V7cM/bq+PO1Xs3Q6zZSC/Pi0XNi5PQCGMPduznzTVWw+FZEv0Kk+bFBBUn9nRZX1ZpWmqtD4XxfUV19DOv62SpQpsbfVwY4cVDh916lcMuC5mNxNFaCfvtETq1iBKJvPPbb/crFTo7Sf/xE/4eanrvpt4ecepyiW4iJzYqJRrhPUknHm+iN2YkzYtHjH9rFT7fENtYTaTzZASUtmmzexvZmZDYOSikUjHSuCKO3c04zeDF3kZGiAq2pHcxD/totUyEWsnd2O8ChmajxAsPWj4v1G/EMHqfLfpkK++Ryhl2UxmzLzcruGhfU/lZPciY3dvIqa4VCocpSZ1FvNxp2w9J6vQmzwiJX9cnB+rXjP2o1Vx6wuOKnYdOnKsvvTgRiWKXb8uqEluBJIeVK3hzlUg+JuN7kOYlKbH7cosIPwG12QjmQqWd6lxPfrpZ8dnLCTnZYScTdrBwxWHXB9sDNAhIL40+L87cL5XRe8u0Hp/euzl1/rbouYYOu34VMKnjCGFoGoRsX/OSytsDpDJ2X22xfUqUKyTTPi8uvegku4ijdqCcu6UC/160A+UGuzNbwQ+aFxJs3UNhNrLRXkYa3LAPNC9akOgihsjO4RRtlKbY1rwIlMngCCuJzmoIxGuNMgISTVoq7n93DSqP1+PPP+2rOP6L15a7dk9bPi8O9atjdSHDMn6dGP2OU9oQY4ddWqAkO2wbazVlspuNfDAVJif2s10alwkEJJXPi/5W9l6h9nlJtATuN0JhGe+vLsNXmyuwreJI9LjbIeZ+0LwcPlZvWKamwVjAIeLHiA/X7kF5ZQ1+PnWxrX2wktVsdM1Li7GGSXtPaGPosCvbNxsleioi4cUi9jUvxheQJEmxIm6MNkq8oKDOZJ/okDm/wWqmGsL8v93ADxl2Dx2rMyzzs6mLaWUvgHqM+GH/UQya/CWW7fjR1nUNk1361Gy05Ad7vzsV2VJejTF/X4gvN+1TjDuGZiMH7p3oRSsJLxbxwuclQ5IUIZGNPi/i99DbYdYOPLMREYN9Oqw5zf38ONbbwak2FNG8AEDVcbFy6cD0JTvxjy+/izvuVneRIHFDrSN9ICxby7BLeM8db67Exr1VuOXfKxQLAqEkdbYddsnnJTmx2fIiZwdUKyQzmpehp7RBbnaGpboZoTYbkeyihH08rMASCvlZ8+JMHUQ0L4SSSFbtS/uVoFObpq7fTz2uRDipWRDlVTW+CJUWQZZlX5jRE0kls1hgI/3Yx6IVKm2XRAsvpHmxiO2253Uo1eeAysdFUm2o9vjlvVDYPMi9/PVndbBbQ03UczBpXpSwwl3YQ82LH3xeDh0VE17mb93vyP1SiaN1DZ7cR2vCj7zHMpIjPX8yCFhuo8i/xZjzjRx2G9vYrsOurdNtQ8KLRbyINlKbidSh0pIkIVMjTMQgD5Ut4h12tbvR5vIqHKn1ZlD2IwqzkY99XpxaRC3+wbmdhdMBPfOM13Nz5DVujDaK3b1Fkyw0z/Gfkp5kF2X/CZnxeUmBUGkSXiziRbRRRkDl8wK18KLXgdzrWPE+L/xyW8qrceGUhTj36XlpZUNnB35PhRdbu0o701827KHMqGbQS/nv1jsTCPC1L5E+wPOH8KN55tqXl+CjtXsSXY2EouVfZ/w+248nI7NRkmJ3XBFJsd+YYZc5oPosQdIUXiTJvZWJehLWijbatLdxIquorsXWfUe4ZVIRLZ8Xt7dV8IPmhTCHSMp/p+G766qcPL2pii2Wbf8Rd3N21k5XWBO1oi9xGrNR82LTbESal+TE7sstsghXhzRKgMoHRlvz4ma3Utc9k1G9sC9QTlbs+OE0cuTUWg01+Nhhl5IMJgZWoPVMeNFw2I0MJTIn2ohX/qcD2rlQO8IqIUVf0u9M7yzfhblb7Pmdkc9LkuLV3kbQCZWWAGRqal7sGBH0YeteUx9CiyZZ0c+by6ujf7P5YNJpagxraFvcd9i13uJNs/3n05AOKIQXj5Lfa90lonmRoRzfepU0x8s3Dowr//jlvfGPa/vjpDx+0ADhPmw7sQtHdozWGnWe5YTnm8Gr/qoFCS8WsWsxFDEhZATiN2JU+7xo2R0leGM2OuOJOVi3uzL6OWIqAlSbEqaR9KIwGzECXE29y5llT3SFqTecbuq0i/sWuxZWDwBZGWST0iIR70VAFbXIHgdO+Lwwo8fffn4azujUKq58bnYGLu1X4ktn3nRBy2HXC9w2gxtBwotVbPu8GJdpzMfAOuxK8dFGCZgY2LpXqyKJFD4e7EognXQvGquhT9eXu3rbSE+4sHcxLu5TLHxe98I8dyp0gk2PXejq9ZMZ5WaL2t85isRfNUd8GNi9jX55dmcUNM/Rvxw5TCUMLRM123fccvxO9IhOwotF7DYcrz/xPPzVodJqHxg9h12vQqUVMF8p0uEnuqd7iJemIhZJ3TlEz4O7/haZWuFohMpspKS2IYRfvLbM8XtqtXekmWTEhG6RpiPRxR+weV7CAmYj+/dL7KBO+j6L2JVmhcxGKvVufKi0pBnp4+ZkpFd3VsOS6M6dKNhf7fZ+RiySxt9GBAJa8SeE2+hFG81YWWZ7HyMeWv5wUZ8XObaKF4koIcVLAtGIbHR7zXRqcXO0bZHr7k0MoCWRRTxx2FXlY5AkIDsz9rmmPqQTbSS5JnHX60TNaIUJp5MYo9C8qHexdBGFoGtyRqEJKDEoFwLKRthXXePKPbXkESkqvMhRs5HIpqsk+CYOTbORy/f9+O6hFCqdrHgTKi2pVtMSOrdpFv3crmWuts+Li/2qXmdCZn+Xlg021VEKcN7dl21yM+OKEybG+87rqvv9o5f1snTdmvoQ3lpaij2Hj1s63+/IipTuyu+OupSZWgI/VjoWKh0TqkQSkZHgmzjYRbDCVOSyFibRggtAwotlvEtSp7QbZQQkrPrd+Zh6w+kY3vUk3Wgjt7qXnvCiMBspNC/pI72wv7XBYc3LPaO0hQSlc7c46j20rGA0yd00pJOlqJS/zd6Kh95fjzHPLrRaNV+yfMePmPjftfhRJ/9RdY1LwouGsBrRsoRlGSt2Npqr/DBJEWLYXSw+/bN+DtbGfcjnxSL2Q6WNr5khSYpjkWGkVdNsXNi7MZpEL8+LW9Q3iJmN2KRs6ap5cdrnRa9VFVo6E+3vhKAb2alY3c4DO7a0dd3IBo6Hmd1zU4GfTV0MoDH7dIR4h133TI56odLvrdwdPSZkNiLViy8IaUR3is5V7Vom1ofFLCS8WMSbJHWArEhSFz9IaG2K6OZwoq95iZHoPACJws0wRd15wlqwkWbGVTNIJ7SEET+nZsFM/O3q03BWl/j8IESM7yuU22Z4sQdYQMNhVy8CSQ8SXRKH1nhrpRsleq8is5DZKEGITOzxPi/xaA0ubvZD3RWhxqaE6STGsCsdJ61Gj13eS9c5UjIQdLVwwmykzv6ckxXA+acWonlOLPuyUR+oD4Xx3opdKGP8W1Jd/lVvy+DF79Vqap7zP0Ub+Ytvyyrx0xcXYeUJs54yISZrsjd/7WSzELoqvCxYsACXXnopSkpKIEkSPvjgA93y8+bNa5ywVf82b97sZjUTgkjnyggoJxXeIJGpqXlx0WwkqHkJaTiQpTK1DSGl07JDv7trQTPcOLiT7kShjDZy5LbCSFALQOYr8NKCH/Dr/67DqL/Oc6xefqdBJeB78ZY0atp4WlzOMXLY9RXXvrwEK3YewlUvNpodlQslDbORYKdKNv8mV4WXo0ePol+/fnjuuedMnbdlyxbs3bs3+q9rV/1IhkQwbdEOW+eLdKiIH0EEvlpXO8+LEwNhXjDesqgrvLATdyi9NC9/mrUZPX43C9+WVUaP2d0/JEJRfmOWU73xxXKeFwdmH7Xmxco4uOCEf0tNvXfh5YlG7WSZWLORmEATdx4ZjjxD7cStpXlRmK4Fry0iqPoJV31eLrroIlx00UWmzysoKECLFi2cr5CPEBmkGs1GrMOu+ODiVDd8+uf9cPsbKxXH9MxGidxrI9G8OG8bAODON1c5fu3IxKJn3rGqeXFizApISiHIikDE6y2pHqWmXAh481sl6O8qrShLmpekwW5eLRFB1U/40uelf//+KC4uxqhRozB37lzdsrW1taiqqlL8SwZ4Pi9ZKgeWgKRMrW5G8wKd3B13jDxZtJrcTfWO1WlvMKhwIEvT7QHUfgxOIDKssM7bZoQHJ4Ys9b5blsZBXgReivebOJ8XL24q8TMq8/qMyNZpyTXlpRbK8ZY5bkH1kmxCqK+El+LiYrz00kuYMWMGZs6cie7du2PUqFFYsGCB5jmTJ09Gfn5+9F/79u09rLF1ePPbZf1KFJ8zAhJysvSbSCtUWo8HL+whXFYrmkkL7Qy7KT4LuUxkYNEbYIKZ+oKuFk7YutW5Q3gr9iQbGz0h3mzk/j212oErvIj0jWSb9VIIdoFod3uAZNO8+CpUunv37ujevXv08+DBg7Fr1y48/fTTGD58OPecSZMmYcKECdHPVVVVSSHAfLh2T9yxnKwMTLywO/48awuAxsEkmJkR/Z43IWhNPFrbA7x/5xBT9TRrB2Xv2ZCmGXbdQIr+X7s9WOHFjKjgiOZFvQ8X56JGXYAn4KZ6t1FH5Hkh5Ac0EvvwhhKhaCP7VbJFOCwnnbOpE3xfUa0YY8OK8VZGKCzjiU82ok4w5JFCpR1m0KBB+O47bafHYDCI5s2bK/4lM8pwV9VqmlNeO0kd//qd2zQ1VR+e2UgPRbpqEl4cI+bzol0m26LmxYmVc0BSTnRODYRsf/rJC9/g4ffXO3Jdv6AW8L0JleaLwMkabTR7077EViBBPPrRRsVndaj0zFW78do3O4SvR8KLw6xevRrFxcWJroZnsP0nIyAhmGXN50VjcWU6n0dWpvUu4uGehCmPpPo/j+wM1udF/NpOLFolqB12zV/DaOJeVXoYby4tNX/hJMEr06q2w66+ZnfIya0BAKNPLYy7XiJRBxSkC3WqwAm1w+6uQ+b2AyOzEcORI0fw/fffRz9v374da9asQatWrdChQwdMmjQJZWVleP311wEAU6ZMQadOndCrVy/U1dVh+vTpmDFjBmbMmOFmNX0F233UZqMGzi5/Zs06ZoXrbJEUmwysI3LYpvd7MuF2iGuk3fRWRwrNiymzkQOal4CkEFisJL1L9T5ihFfaSd42DgDfRMSOLy9ePwBfbCzHhb2LVNeLP29MnyLsOHAMG/cmRwBFMqJOWaE2G9XWawdW8Egy2cVd4WXFihU455xzop8jvik33XQTpk2bhr1796K0NLaSqqurwwMPPICysjLk5uaiV69e+OSTTzBmzBg3q+kr2HEgIEHhsKuWtAEgw6RZx2z/zDapedHOO5DaU5Ob+9CwuGE2ckJb3LiaV5o8RZFlGZIkcftIaveaeLwyG/Hg+7zE/s5vkoWfDYz3J+Rd7WcD2+Oc7gXo9JtPLNaSMKJetZhVp6YwOyYlm9nIVeFl5MiRupPWtGnTFJ8nTpyIiRMnulkl36NO8c5qPmo5dhiz0UZmV8RmNS+aDrumrpJ86IWPO4Nxu2Ub+Edp4ciKS1JqXswMhLIcL+xUVNWgoHmOAxVLHmTZG9ORmWgjkXYUNUERzqLWvCh9XmTUmNS8JFuT+d7nJd1Q+Lyo9pzhqQFNm41M1sesz8v63ZU4+6mv8NHaPWnlsHusrsG4kAn+cOmpis9CZiNG0FRn4tTDEbORJODzotEHZM7XZz75JY7XhbjZnFNViydD9uw94d3GVB4p9jxO//HDPNgQCmNfVU2iq+EacWYjpvO8vHB7ymteSHjxMerOxAt50wsR1BugOrVuIlQHs5qXT9bvRdnh47j77dVplWH3uMOalxZNshSfI62sN5mwmpfjZlZdjpiN1JuImtG88PvJve+sxq4f450O3UgC6AdkGZ5MtnL0P0p4CyGhxRGniB/mwWteWoKznvwSK3ceSnRVXCHObMRMD/ura7HixOaNqQoJLz5Dz2+A6/OiMUpoDe+RSeWt2wbhnlFd0eUk/dBpsz4vLIoMuyluOKqqqXf0elqrIL2FMBtWb2bV5dTeRpZ9XiL/V3WRLzbyQ2AbUlR4AYBz/zo/Yfe2vKs051iiV/H/XbkbK04ILe8uj/lVHq8L4e9zvsPGPcnvSKznsAsAew6bE4T9IHCagYQXn6GINlINHLwJSW9w4YdKN/6/pEUuJpzfDYV5+n4FZjUvLOmUpK7yuLPCi5aGRa+9sxXCi7jmxYkxKxSWFc6d6q0u9AjLMg4fq+MK51r3Iqyj1d7cjRkt+rwkeh584L213OPPzf0Of5uzFWOeXehxjZwnzudFVmtizL0nogJn77b+yKXmqwy7RLzPC0stZ7ddrQ4nvg269neZAcmWNG53o7Bkouq4sz4vWpOGvs9LLKyetzNzblYG15xkcgcILqGwrKhbpokouP3VtRj6J/09zFhSVfPipb8LzzGYJ29a9nnx0TJegoSa+hBWlx7Gut2Via6OY8TlebH5Xoi22H9uH2zrPk5Bmhcfox436kIch12TLageU/Qmw8wMe8KLWo2ZyjiteVFrWCJPUm8lzGpeeJEGM+4YglE9CnBO95MUx51w2FULL2Y0L998f8DUvZKpX63ZdRg3/GupkJnCqyR1WvIIbywQef95wq+PZBdIEjBu+kpc+/ISLPzOXF/zM2qfF95mv6YQbLMm2f7QeZDw4jMUTo8nRoBL+jZmGL55SKe48mZty+qJSu/8rIyArYlNvelcKlPltNlI1S6R5ye6MSNPeDm1pDleufkM9ChWqn2dmGhCYVlxHd62EkZ+WKIkk+blyhe+wdffH8B1/1piWNazJHWQ+EnqLG7MmOhoIyO/PQCYt2W/BzXxFrXZiJfE1AwtcrNtne81/hChCC6RgeMf1/bHn67qi6bB+OYyLbzEaV60ywYkm5qXNNpV2iufF9FoIz2HXfXE5YSKPySrzEZO2KK07pVEwkvkWR8+Ztw/vPpVkmQiVNpqnhcP07V2adNM93s/aYGcRC3E29W8ZGcGsPzh8yBJwMAn5ti6lheQ8OIz2O4Xef8lSeIKLoD5/SjUwo7e+XUNYVsrqHRy2DWTV0UErcFfT1jNydL3eYmgDk12ymGXvY4Zs5HZyaUhnJqbZnmVv0brefMEFatCiLfyQooPLoI4IdSflBd0oCbeQGYjn8GOXyIrYr3BpU/b/Lhj6tJ696gLhW2tykNplGG33uEJVdNhV6e983NjuWH09jVRt4UTK9OGkNrnRfyiZvtYMmle/IiWmY67t5GI2YjrK+Od+ELdoZF0yqsFkPDiO3iaFz30yvxqRBcM69pGccyM2Ui9mjaLbQeyJMLpn6plddFrr7ycmHauRidUWr3CFzU9XntmB83vQrLa58WE5kW4ZCPJ5PNiBk/NRpwOa3l7AI17eIWxxio17UbqZ5xMjuxOQMKLz2BfRJFVj55NOpiZgbvP7ao4pl4RGd3DVqh0Gm3M6LQ2QGvS0GtvVmDQa9c4nxfBOvEcxiOEbYRKmyVVNS/vLtvl2b14T5DXZcQ0L5xj5qtkmdTsDcaoxwhOAvaUhoQXn8FOLCKrHqMy7ODTszg+uZCRetcps5EoFVU1eG/FLtObiiUap7VMWu0q2h6v3XwmivNz0Kskvs1FzEbnn1rIqZP2/UKyMkkdL7mhlgBr2ufFZlSFX3l3hXnh5cJeRabP0epD3Ay7Fl9/LzPspqgsa4i6bchsRCQUNipHaFsRjTKR67C7Tj8wultcOTcHGSuh0le+sAi//u86PP35Fpdq5Q7OCy/mjqsZfHJrLJ40CqN6xgshRtFGj1x6Kl6+cWD8vXVubidJnVlSVfNiBStBXQGJ/z5aXajwrxV/7NcXdMfJAmHN5u+v3x9SNdpI3V5kNiISCtv/RMIUjdS67Pe8srw5prC5Mx7n7AJZNFS67HDjRnxzNvH3tfErTgfAxE8kjc/PbHTZyBMJ6ZpkxyKR1IKW6BX1BN3GPC/WQqUp2sg6VhYfWmdY1bLwBHdeve465xQ004iaJMzjpuaFHS/8CgkvPsOs2choMmNXwHy1cPyxk0/Sz5sgSqquBL7YUI5Z35YrjnmneeF/MbBjS+7x0zu0xCf3DMWi35yrea/C5vr7W0XQ62mNmpfYZzMbeppNUkealxiWhBcLW0/4mW++P6C7l1dy/ipj1O3l5Hi7cOI5jl3LLUh48RnmzUaadiMASrOR6K6xk3/SB8X5OfjtxT2NK6BDKuZ5qakP4VdvrMS46StRzewk7ZXPC6+93r9zCN645SzNa/UqyUeLJrHsmWxI9e8uORX92rcQqpNef2wIK3MCZZpYxpvXvKRIZ3IAs5o4oHEy5z1B3rWsdmsv5aCwDDzy4QZf1MVL4h12nXsvWjcLolPrJo5dzw1Ih+czzOZ5MTItZTDqe24SKs7pHVs3xeJJowzvbUQ4BYUXdnPM43Uh5OU0CgJOz6eawgvncK+SfFOajtuGd8H6skpc3KcYVw1oJ3yenoYkFFbW2UyotFlSVaNnBSsTsySB+0Ja1byIanTd5O1luzD5J309vWeiUT/ifdW1jl7f728ZaV58htmB2WiOYFfAPCdKVx1202hXacdDpTXalSeAmm3CZsFMvHrzGaYEFyNuGdrJMEmdU0+INC8xrL2/4oKxyJM20ycT0XJObDzqR9Rt/9HaPY5e3+8LThJefIbZ/mKknWFXRdwkVC7uQfJ9xRHXru0H2LZy22wU25jRfQHUbNTJN785F6cU5FlOUmcW8nmJIeLUr0ZrbyO+2cj4Wev1SbUQ6+aEOGfjPjzx8UY0pEnCE7e3j/L7fnQkvPgMsy+3sdlIPwLEq/3TUj1JndM/T3P/GU6DOdWENw/phM5tmgppZNjMzU1PRCYoQ6XFhxazz440LzGshEpr9Zcm2fFeBLxjanhNHbmH1Q0627bINX3Ora+vwL++3o6Zq8os3dMP1IfCuGXacrww73vDssnqYO0UJLz4DLMreHYy43mIKzQvnNb26gVI9enGqwy7PGHTqSZ85LJe+Or/RmiGs7IrMbZMZOXN9i8zexuZfXahJAqVZttmb+Vxx69v5f0NSBJXYMzNVg4Qtw3rjO5FeYbX4wrUJw49dnkvAMCdI082Vcf+HVqYKs+yR/Wck2mOn/VtOb7cXIE/zzLOc+X2/lF+X2+S8OIzzJuNYn+3bxXvHW4U9eGZ9O7zF8EubodKR67OM/M5OYiJXov9vZFTrDrsmn12yaR5YZ/JpJnrXb2+KI1mo/hnmJ2hzO3x8MWnWq5DpB/9bGB7rPjtefj1Bd1N1dFKFFWEJOoecRyvE88sfuCIsw66aq4/q2Pcsc/uHebqPc1AwovHjOh2Ei7pW6xdwIbmxcr36oHnkUvFBiyz+N1+KorWROu08OKHXBy3j+iCnKzYEKH1E3l1MhMqbfbRJZPPC2vW3XPYDc2L+XOc7kJGY0ybZkHTArYVX54Ioibqo7UNlu+RDvxqeBf85/bB0c8Bib/FTKIg4cVjpv3iDLRkcm6oMTssa73kkesY2ZzZcad7YR5uPruzyRqkF1rt41WotJ1B3SyTLuqJ5Q+fF/3M/kRlMsXI/2N144Vua80pZjODJtPeRmxzWfX/0L++lTwv/HOsZmjl9Um7srydQALRjUd7/eFz3eR2iUC9yJNlGfe9sxp/m73V87pkBCSc2blV9LPbZiqzkPAiiFMOp5Ik6a5UzK7gzUQbGX2f42JK6MjPOl4XSuooJLZ92KZyWhvQPIfvd+L1+KElRIXl+DIBixO12T6fVJoX5qGY8QMSxZImTuILF1Z9ifiChr02MqO5U2OmP+0+5Lw2zA7qqq/edRgfrNmDv3/5naP3OfuU1qbP8ZfoQsKLME5aBfQECtPRRhrXCp5Y+bKDgNFmbG52zsitL/nHQpz3zHws3nbQxbu5B/sMdxw8yhx3roNc3KcYrZvx95fyOsLAzO2sbsxoVhaJTE6yLOPZL7/DFxvKDc5wh0XbDuDy57/Bt2WVmmXYZ2LHj0MLi7KLhvAS+/uJK3oLX4+nebErX9rRvKjvrbfAS4RTqizLqKkX0/iIljNDs2Ambh3WxfR5PlO8kPAiipM+DbrCi+lrKT/ff143XHFaCQac2OuGHQSMtrx3s3NGHt+2/Y0T/sfrjBMqffP9AcxYudtXYdasWveal5bgg9WNYZlOKgMu7Vei+Z0bE6AeWiYGtk0iEzQ7SWSbCpU2qXk5Uf6b7w/imdlb8as3VmLCu2uiAnFdgzfRSNe9vBRrdx3GTa8ui69jWMacjftwhPGrMBM+LoqV3iBJElcLxCbIvGFQvLOmFjxBw+54aaebm+tP3o8tt72+Aj1/PwsV1TW6tbn5tWWYt2W/4/eXZdmao7fPdC+0PYAgTk5Oeh3H7DvfqyQfzXMy0bZlY6TRved1jStz7ZkdsL+6Bj04YY/sZOiu5kX5w0TS2V//r6XRsnoTupeo2+efC37AFf3bOmrK0BtXPJZdFHVhJ4WwQnhR/h8wp3kx++wixdnBf+bqMsxcXYZfX9Adf5/zHd65fRBO79DS1HWtcvBoXdyxNxbvwCMfbVQcc8NsZFXz0pQTDm/Z54XzKmtdStRx346G0YzgFCm68+BRVB6vR992LSzfV5Q5myoAAB+u2aOrAZm3Zb8rwgtgVeh1vBq2IM2LIE5qXrJ1BjGzq9CcrAys+O35+PjuoZplJv+kD/510xlc9anCbGSid9qdRFfuPKT7PVuX91f7J+mUunki7eV0tFH8fRuv72ZGZB4K4QX8v6N5XiyGSq/dddhUnaLPgtNf//L5FtSFwnhUJTh4zecb9sUd843DroS4XD43De5oOQSdbzYyd635vx6JImZ3czu9PM5hV+dikaIj/jIPlz33DXYfOmbjzubgPSO9x3bEoegoSZIsh9j7CRJeBAlmBjBdZ+deM7BaB7VjmpXhIzszYNmcYHUuNKsCl2Wg8nhsF+Z1uyt1Jy1WiKtPQLrviqoaXPXioqhZKIJ6wIl8dFJ44TVJ4YmB3XOfF02zUezvSB9Sbg/AiUDR6N0frDG3J0vEvKH3KLQcnr3gwJFaLP4h3qfLFc2LxXOaqZ7Po5f3trzhJU+g1tS8aBwvbJ6Ddi1jWXXtRLaY+Rnq+jgdTPDsl9/hwikLFGNfBLOPu/cfPneoVhZD7H1mNnJVeFmwYAEuvfRSlJSUQJIkfPDBB4bnzJ8/HwMGDEBOTg66dOmCqVOnullFYSRJQgdOEjgrBDNjUT1q9brXO+ayqyYzXdNsNMALc79Hv0e/UBxbp+PoyJIIl5enPtuMlTsP4b531yjroioXjmpenLs3b+Du3TYfgLeh0oBykGPbwUnNi1kiz1pPYM/PzXL8vsfqGjDqr/Pwh/99q1vuua/4qd299lfSIiBJ3CzKVk2fTgjUTnZrU2ajuNBk5+oBAM/M3orN5dWY9s2OuO8SGjVnRXjxR/eN4qrwcvToUfTr1w/PPfecUPnt27djzJgxGDZsGFavXo2HHnoI99xzD2bMmOFmNYVxqvGCTOKvLJUq2ev+zK6azPw+swPxnsp45zRREpHgrlpDRas260U+ua15ubhPY2JDrwcQrRUwz7zpdk6TCGEds1GEFk2cF14+WrsH2/Yfxb8X79Qtx1tlA+447LKdRTiBGMdsBFjvw+xCpkubpsgLZuKUgmamruGkRlHdN7884WPCL6v87Jb5ty4UHzXEe4e8GuvUz7t3W+O+47e9lFzVrV500UW46KKLhMtPnToVHTp0wJQpUwAAPXv2xIoVK/D000/jqquucqmW4jjVdmwkhlrz4vVEHVBoXvR/YFHzHJRXNQohbq6s1SRC86Kl4tca7NzUmD0wuhtaNm1MbOh9tBGL/m9Ual7cq2dMeNEu44bmRbSJqzSElywX2s5q1IjabARY33aBXQDNnjACDeGwQrssdA0HJ0b1zyj9UdyPxa2xhvdo/aJ4Gda1Dab94kxT5/gBX/m8LF68GKNHj1Ycu+CCC7BixQrU1/MHhNraWlRVVSn+uYVTLxirebn73MbooEg0jdcTtWI8Nfh5WZmxAnke+hQkQnjR0hzE1SXq8+LcvdXdLJfZ2TcReV4iAlO7ljGzKa9N2L7kpnD79fcHMe2b7bp+EW4IL6JPvrqGr7VzO1Ra1NlfkoCCvJy44xHtXr92+abqwLZ7RkAyLbiorwGY2+NHjZkFYJwDvuW76sPT6Jh12HUSVuBsFswUWxT5THrxVah0eXk5CgsLFccKCwvR0NCAAwcOoLg4fk+gyZMn49FHH/WkfnYnjs5tmgJQ+rwM69oGq353Plq6oOYWwcxvysnMwIvXn46QLOOlBT/YvrfonRNhNtIK9Y1z2D3xfzdDpRU5VTxebkiShA2PXoCGsIycrFi/5Q28VpPUmWXB1v1YsHU/bhvWWbOMG0Ke6DWrajQ0Lz4Klb78tBI88N5axfH2rZpg7e9Hc7UyejjhhyVJkuIttxNZY8phVzW2uGU2kmXgu33V+B/jnJ4ozYsEa/m9fCa7+EvzAsTb2CMDt9Yqa9KkSaisrIz+27Vrl4t1s3be/ed1w88GtMPrv2xUzQWZaKOMgIRWTbOjv8/tkFs1Cp8Xg7KSBFzUpxiX9C1xZJ8LvUuw1zfzktfUh1Buw78mgtoXKUKc4oXJ9uoFibA752RlxPlIGGVr5iWpc/oRlVe5u6uuGtFHr6V58YvDriQ1asauPbND3Hf5TbJM19ON8H0tnzMRzLyL8akPtMvO+rYcz8/93tK7Hg7LOP9vC/Dc3JgzN9/nxSvE3QWi5dLJ58UsRUVFKC9XpvquqKhAZmYmWrfm78UQDAYRDPJTqTuN1bbrVthMkTxOGSqtHOQTaTYy+n0SxAUdEWrrw7hwygL079ACk3/SV7ugwTNZvuNHPPnpJjx6WS/c/fZq7Dx4DF/+3wicfJI5p0GWDE2fF/VKTfl/J9AbTLyONtKCpw1T9GsPfKKCOokO3XiPRAdvLZ+XyLue6IzRkf7lVFdyKgqT7dtHNLRXIiyysfWIXtuMm74SAHBGp1aKDQuFrss5xl2oerYIYj6Ial78MfRE8ZXmZfDgwZg9e7bi2BdffIGBAwciKysxZhUWx3xedEKl/eywK5kQdET4YmM5NpdX4+1l8doydhAxeiY/m7oYq0sP47qXl2LnwUbnvM9t7nej5VypFlIidbOanZSLzrP1i8c/L1qFFSbcdNjl3U+NGxpMUQVDrcb2BA3hMD5etwf7q61pjIZ1bRN3jJ3wRX+y06bHy09riztHnozXfnGGreuwvoBHa637vETGACuIPEJeWn8j+D4vpi/jGMpxXwx/jDwxXNW8HDlyBN9/H1OTbd++HWvWrEGrVq3QoUMHTJo0CWVlZXj99dcBAOPGjcNzzz2HCRMm4LbbbsPixYvxyiuv4O2333azmsJYbTz1fBPUSVKXNKHSDkyix+vFks+JDsqsnby+wd6D1NYcqDQvJ36Ck6tpvSfrtc+LFpf0LUG7lk0Uq25FCgBPNC/ajqFuvEbsgP/8XH4uFz2mLynF9CWlKGxuTVP83LWno99jylxJ7PsrvvBpPKfNiQg2u2QEJEy8sIdQWb3XhG1Pp7LJGmHGbOTkfTSPuXP7OJQLUTIbxbFixQqcc8450c8TJkwAANx0002YNm0a9u7di9LS0uj3nTt3xqeffor7778fzz//PEpKSvDss8/6IkwasLPqVZ7HalvUk2Qbjd2E3cKMuZrtvE7Y70U30LOyiq4LhXCktgHBzICliVTbYZdf3quEU4nWvEy/5Sws3/EjLutXgiv6t1V8l8NqFD3w79DbH8sds1Hs7798vsXydfZZ9dXhPNIMC75hkVNuH3Eytu47gjF94wMhEgEr/HolpFtx2LXSt8oOH+dcJ3GqFyvjiE9ctqK4KryMHDlSt4GmTZsWd2zEiBFYtWqVi7WyjtWJQ30a6+eiFgJuH94F2yqO4GKPBhSF+tDQ5yWGEyvrugYx1bCVV3x/dS16/+FztG2Ri29+c67p87UmX808Ly5n2I2QaOFlaNc2GMoxXwDKyccL51Q94eXvX27FkJNbo1/7Fo7dL9ErT94jZTUvokJ+5IymwUxMHTvAgZqJo1dDVvh99pr+uPedNaZytFiuk8aGo04ye2P8XleJDJVmEe/V/pJefKKEThIstp36NHbeV/sGNA1m4vnrT8eYPt4ILxkWfV6cmJy0fAPUWHmh/7NiNwD+ikcEVsDU87+J7m3kZKi0znd+MRvxYCcf3kTv9List8FpTX0Ylz//jaP3S3RoP09wZd9DXh/k5WzxmfY/Civ89u/QEgsmnqNT2hlk2fz44lTrJmDLtkYkc4vW20c07nz9+0tPdbNWpvHxUOg/rM7X6oGc7TiJXkmbccJVpn/3zmyUCOUqK1SymUfDqirHNC/u5XlhYfvLmD5FmHrD6Y7d1y5sHhgv8D4yT7th3lpaign/WeOq+ZB3e/Y15N36f+Pjd5v32wZ7EXIsJLeziwzg7eUx1wUvU1XwNS8e3F9WLoKMesOki3pi9e/Ox2UnEqn6BRJeTGDZbKT63LZlLiQJaJqdwc2H4SVWf5MTScj0dotWCHwevND7qmpw55srsfTEbsCsLxJbzzjNy4n/e+VozWrK7j63Ky7s7Q9/BUA/+scN/jp7q6f305PXH3p/PWauKsOn6/e6eP/4CrDHRDU+fvNdiMBqXrzk4fdjG22KDDVOCRiR6/xz/jZcOGUBDh2tc+S6IijSXgjMAS0dcu52El/lefE7VpUk8dFGGdjw6AUISJIrCZ7MwKqdDx3Tf3nYDu/ExnuJ0ry8t2IX5m6pwDM/Py2qLZg0cz2+2lyBT9eXY8dTFys0S/UNMnDi3dWKTnAyVFpvZeyxTGeKs7o05r7wwlk3EYi8/5XH60+0n/ONw7u/0mwkeh1/to/Xwi/A2WjVw3cqsuCZ/NlmAMDU+dtQnB+/bYMbKPJ7eXJH5yHhxQROmniaZPvj0bM/6dsy/X2hnPZ5qdPRvCj8TBweUH7933UAgNM77MStwxrtubtUjoHs72PrGS+8OJ9hVzTzcG6292p2Pdq1bIJ5D4x0ZUdnP5DoSZ8n1GZYcNj1Cx1aNVE45FrZE8kuao1pIs1GtQ1hb0zkkmqMSVLphcxGJnBK8+InaurFk0GxP8MZs5H2qyor/nbnlT5wJKZpit9PKPY3azbi7W1U1xB21ddBPZ4+cumpuOfcU6J7ZfmJTm2aokUT/6mYnSDR/mncaCPJvPDil/HomjPbKz7nJMBs5AfNC1sXr+7PCuJ+9YEygoQXE1htZD93jgozOSfYjfcENC8PjenBzQoqAvsSu/VCNzBCibqN2FsqfV6U/Hi0Dj1+95nD2wPoc/PZnTFhdHfnbugVCVYMvLW0FL+Zsc5yZFiio414mh+l5kXwOj4dj/ygeRFpO6fGIzf2NupemIdfnN3JsJyZaCO/QsKLCUQtJc3Vu7L6uHOY2QCN/RkZAj4v/dq1QGFzazZcdhBxS6nBCiV6OzkrhBcv0nz7uL8kMw+9vx7vLN+FrzZX6JarD4WxfMePcT5ZIm5ebspnvPFHuT1Acmle1NUdcnLj/nVeukxZ0bw4JZxq7VBvB0kSy37uky5gCxJeTCBq8+5elIefMNlH/dxRbhrSSbgs+/NF9q4JBKyv8ZSaF3emhHodqYO9ZV2D+4IUi19XxqlCda3+pn9//GQTfjZ1MR56f73ieKLbhTf+sIdE+yZvN2mv0HuXO7Vpii//bwRW/vZ8z+oT7/MSX6byWD0u+cdCw2uFwjIOHBHXZKvvtaW8Go9/vFH4fC1EgkCs7G3kN0h4MYGZFcEZzK6jiXb006Nti1zhskrNi9gL4uOfrjQbqSrKrq5klQcOkdxoCSHhsIxP1u3FtEU7AAD/Xbk7+p0sy2J9WZY9FXLM+LzkZmVg3SOjcUqB9d3W7dK/Q0vd708+qZmnYblxmhfO+z11wTZFMIPWY7725SUY+MQcfFtWKXRvdXst3f6j0HlGiPRTpzfZTQQkvJhAVAiR5dQIRVPD/n6R7QHsqH+98XkR07ywf3uieUmVDuNTtJ7vf1fuxl1vxW9NMuE/azDy6XnCGaEThZEvT0ACmuckNhLsoTE9cN95XTFnwvCE7u0TQUTzIpp/ZdkJ4ePd5buEyrv1843MRhJUwkuSzlD+iNdNQVLBIUqNWc1LRkBy5MVwywGyTuGwq74n87cHghRLfF0SP8inA99sO8A9PnNVGQBgDmd/GjVet5RWP+XhBw1wXk4W7juvGwBAlssTXBvOu8V5iMdVEZlOjQFuhWWLRMWlwvxEmheXSHRYpRms5GwRiTYyYzbSW4WZ0Xa05OQY0bo2q3lR15MdWBQmJA+kFz9MMm7gFyFM6/karVhFa5+o36k1GXY5qTGk/vxTC72sjiF+6A3x0UbxHKsTTydh595OITKcp4LZiDQvLiBDvXeEv3tHMDMg9IIq9zYSk3tFX4ywDLA+wOyGimYEBl5JWebXg92zSC/Pi1Wz0YhuJ2H+1v3iJxCeoNUljYRGPyaBU+6IzC/zzm2D8PnGfbiSCSIgGlG3Kc/0dlw1NjrVC9zoT41jnTnNS7I6NpDmxSWSSS0nmpZbsT2AQLRR4+8W9RMSS1hnfB3x8xsE86lbTZiXikkNUwGt52u0YvWh7KLojVo70Rc0z8HYQR3RLOivtaofnqd63InILrIsY8OeStQ2hHCsTjydhJ17O4WI1j8VNC8kvDjEbcM6Kz4nUyhatqjwwvwQJ7YHYNF9jU2849zETxqDRL1ekjrFipY1G4nXxamNPAln0dKEGvVpkSzKsuytpvXsUxqTQLZokoXHLu+FP1/VF+PPOcWz+9vBD2ZE9fscedffW7kbFz/7Ne56cxWO14dV5ziU58Ul/28hs1EKjDL+EsWTGFZVJ8uycuLyeT+xIryI+LxIEPd50VOhmtK8cK/NL1uv4/OiZTbSG7eyMwOKxGaWc9xYPI8QQ6tPJqPZqG2LXCx7aBTycrKQm52Bn5/RHq98vT3R1RJCJJma2/zhww3c428u2QkAmLOpAp1aN3Hl3q457Ar5I8b+TnwrWIM0Lw7BdoDMQEDVOfzdPU5qFhQqp9xVWkw1KfrL9d5jUysdrtlIy2FXe+mjFHi0cr4oyVaFj1t1vNXauZpwBq1WMYr+92s7FDTP8d0mnSLcOLgTOrVugjtGnpywOuw+dFzxOSJQFDG7O8dFGzl0b7ccdo2GHUmSFItrn3ZrQ0jz4hRMh2kazFBIvz5YYOjy15+fhnveXm04iCjMRgJ5XiRJ/LfLsraQYlfzopWtl03zoheezA4yegNOdmYAYBJsWm13P+S/cAO//Cxtnxf7mhdZlhNuDkmW/pPfJAvzfn1OoquhIPJ+F+fHkncerTUXbeREhKUdjPqxcMJFn0PCi0OwWokmwUyFffykPDHNRqLo3KYpPrp7qGE5xfYAgj4volonGbKmYGBK8cL1eYn9zbaL4ieoM+xqmo20K6PWRll1C1LfYVCX1tYuRERRthu/YYwGfTd3Dif8QaSN2zSLZfk9ot7/zcd5XmTIgqHSrJuD49XwBBJeHIId95pmZ2BfVU30c5c2TRNQI+dhBRERh10z5jJZ1n6ZzaxkuZoXsNoWVnjRdqpWJv8Sc9hVZx22ai6MPIflD5+HssPH0a99C0vXIWKwcodW1zXWvBjfJ0nnAd/T5aSm+GH/UdfvEwmV9iLXklt9xWy0UbL2WvJ5cQi2LzTJzsTFfYrRsXUT3HPuKSmTdEzhsCsYKm3GYVdLeDHjlc+7hMLsw1xLT/5SCCzscZ0XXR1yLpgKh3Pzxv+dlBfEaSS4OALbtzST1Bn6vCTHIN+iiXd7A3mGR48+MlbobbXglFnQje7UuDVNfP8e1rVN9G+1z0uyQpoXh1BoXoIZKGieg/k+s+c6iUiSOgkmHHbhzMvMG1iUfi78SUwv2oid+PQEKXXUllXNS3JMkcmFQnjRKOOEz8ufZ21RRLElgstPK8GibQcw5OQ2xoWTBK+eaGR88MJC6J7mJf5YblaGZpkkkcnjIOHFIdiJqqnPkkG5gUi0UWZGQHwzyzAQDmiYjcxk2OVGG8XQ8nnRc9jV+DMOtdlIJGSRR7IOJn6GfabWQ6WN76OOTEkEWRkBPPPz0xJdDUfxSusV0biEdO7nVFVcc9jljDtqwZydr5J1vCGzkUMofV5SU3hhB3e9ibkkPwc/Ob2tqfwIMmTNlygky3h+7vdYvO2gwHU4xxhtiVJ4YTUvOg67iuM6DrsqU1p+rng/YH2I/JhPxAkS+auUZiN+GSOzUaq2SzLg1ZOPtLEXwpJbt+AJ4epD7OdER8dZhYQXh2D7RvtWuZrlkhn2N+qp2B+/ojee+flppnx9wjoOu/uqavGXz7fg2peXGF/IIM9LSGc/I63LsNV640TyKh7qpFutm4pHmTmcsDgpadXUPV8NEa2JUdI0El68p+OJBdBFvflbHzhNzGyk5/Oij+ir7FR/eu0XZyg+88aSOM1LCpiNSHixwZSrT4t9kCQ8d11/3HXOyTine0HC6uQmyu0BtMuxvh9mch44YWfm+7zE/tZK9a+upsLPhfl74XcHNO+t/q1suKURitBF4bNSi5IWOcaFLKL0eeF3SkOzkUvp3Alt3r/zbLx4/em46xxvEtlFzEZ2xqLIqUbaG6dC7wd2bKn4zF1Yqg6Rw26aM+TkWP4NCcAlfUtwSd+SxFXIZUQ1L6zvh3ieF2dUtfxoI77mRdcPQtDPhUU9+bUWzFwMKJ9tuq7w3fzZrOlQq0sahf+na7skklZNs3FRn2LUeORLJBRtJNgNjMo51Z3UCx8RzQv7OVnTF5HmxSKSKpQmBQRZQ9iXRG+gt6J5CTumedE/xgovkcmovLIGtQ1hzXNEhSr1T21jRnhhT07SwcQubsoGWvl9WIxMdyS8JA6vxtdQVPNiva1fX7wT76/ebfga6zkFm0H9aHgaxIBqrmLH75qGxDuZW4GEF4sEVLHyft+/yAlENS/qPX6E0PF5MXUZzjW0TEAygB0HjmLQ5C+xbnel5nXENS/Kzz2L89C2hZj/k8L7P02lFzd/tUjfciLayEu+uH94oqvgGV6Nr5F+otfWIu/n/e+uNexzTpmN1N2WNzYr1kaq2x6pUWUQThJIeLGIOodJemheYn/rCi+s5kXw2noOu2bgXoE5qBgwZODLzRWa9YkWE6xXS1VysLycLMx9YCQWTjTO95MKDnRGJDLJm9LXiV+PLzftE75GojmvZwG6FeYluhqe4bXmxQnBwthsJGP+1v2276MMe+ZvD6A3Xsdtf5AkeCK8vPDCC+jcuTNycnIwYMAALFy4ULPsvHnzIElS3L/Nmzd7UVVhGrPHpoHEAuDcHo0OyL84u3P0mJ7ZSOHzIuqwqxMqbQajPC9KzYv2PiBaextpcc0Z7XFKQbO449mZARTnm3NE9dsK3yvcFG6+ZhyteXdZtO0AVpUe1r1GuraLH/BqpI20sV5fFPZ5MdDQhGQZN726TLRqmqjHWO7YrOPiUF1Tb7sOicB1h913330X9913H1544QWcffbZ+Oc//4mLLroIGzduRIcOHTTP27JlC5o3bx79fNJJJ7ldVVNIUjoYihp5+caBOHCkFoXNY5Ownn9AFpPvRDhJnQyEXTIcsAJLTX3Mt0WWtQdFduARGazuGHky3luxm/udWSE3WdLQJwvffH8AE/6zNnaA83jXq8yGPPzULj6qiid4tVAMnQgpc8T/zuAaIZei17h5XnRmq2oyG/F55plncMstt+DWW29Fz549MWXKFLRv3x4vvvii7nkFBQUoKiqK/svIyNAt7zUSlKq4VB5MMgKSQnAB9JPUWTMbOaN54RFRAVcer8cl//g6elyG9u/QSlKnhZ5aVuQZkL+uW3u9yHHh7bwVscjc6CezkX9q4g1eLRTrQzJkWdbPsOvQvZwShuN9XuLL6C02yeeFQ11dHVauXInRo0crjo8ePRqLFi3SPbd///4oLi7GqFGjMHfuXM1ytbW1qKqqUvzzgoAkKTbe69+hhSf39Qt6Cb2sOOzq7Sptl6F/mou9lccxZ6PSp0GWZaFBUczZU/y7N289S/daPpojPcUNR2VZBhpUS1wzz5edYCjPS+LwykL//uoy3DF9lWtpG1gcc9hVjWJch13WbKT6rpp8XuI5cOAAQqEQCgsLFccLCwtRXl7OPae4uBgvvfQSZsyYgZkzZ6J79+4YNWoUFixYwC0/efJk5OfnR/+1b9/e8d/BQ5IaHTJvH9EFtw/voti1Mx0QddgVXTLJsruT9ovztsXZgmVor6K0ktlpIUmS5uSrVuPyIpCUZdJUeuFg972SATSoJgkz/UyxI7mPpEo/mbC8wEv/wlkbyvUFVcFnbxht5NbeRpxHdVbnWE6yyF0jecou7Zecuck8SVIXv2+MrNkZu3fvju7du0c/Dx48GLt27cLTTz+N4cPjQwMnTZqECRMmRD9XVVV5IsBEaj/pop6u38uP6G0qbS1JnXtmI+CEf4uqKjX1Yfz+fxs0yzOfDK9vJr2/UQLMVJ2XrPwso8RxRoRlGQ1h7Rw+RsgmhVivIOdhd3FCsDC6gnNJ6pT3ZOfWZ6/tD1mWcWnfEvzfe2sV5714wwDM3rgPF/RSKheSBVeFlzZt2iAjIyNOy1JRURGnjdFj0KBBmD59Ove7YDCIYFA8GZhTpEukkRZ6kwq747T49gDWJjfRFWhYluO0RZv2apsYzTrsBiRJWFDjllMNQEQjmXpSsgBhWUZDSK154fi8aLSdVqQakdqY2dto3e7DXE20V9sDqO/M1qVjqybo174F97z83Cz8dEA7R+qQCFw1G2VnZ2PAgAGYPXu24vjs2bMxZMgQ4eusXr0axcXebMwlSnqLLvo+L6xgZ8Zhd9l2412j488TL2dmFa/cDylyTPtmkiTus6H16CKb0A3q0ppfIMXhPd5Mm5oXWeaYjSzWyU/Ci39qkproNTX73fG6EC577htFIEC0nME9nEtSp1z5qLPpxpV35K6Jx3Wz0YQJEzB27FgMHDgQgwcPxksvvYTS0lKMGzcOQKPZp6ysDK+//joAYMqUKejUqRN69eqFuro6TJ8+HTNmzMCMGTPcrqo5UqUHWMRpzVMoLOPBGetNnXOsrgFlh44LlZU5mhc9lLns5BPX0C5vN3BeAjBnwgjU1IeQl5Nl61rJCk84yMiw91wbNS9qh13xSYMVSBNpqrErxBHm0BMs2P6jt+eSSJI6J9DTvKRyQg/XhZerr74aBw8exGOPPYa9e/eid+/e+PTTT9GxY0cAwN69e1FaWhotX1dXhwceeABlZWXIzc1Fr1698Mknn2DMmDFuV9UUqbArpx1EtRiij6k+ZP5FHvXX+dhbWSNUtnFiFLuHrCobGWP0znZibsnKCCj8hQhnNC/1Ag67Wv3UD5qXV24aiF4l+Ypj6eaw6zV6wgv7la5wbRRt5FKotGSgeUkVPHHYvfPOO3HnnXdyv5s2bZri88SJEzFx4kQPamWPVO4UIuiZjVhEJX8rE4Oo4AI0TkJ1ggKSOvIpUje9CSMgScIOeJIETL/lLGzcW4knP9184liadyjwx3q7Pi8NIY7mxeK1EiG8TDi/G0b1TE6HSi9o2yIXZYfFtK9mqNPJIMf2Ar0u4d3eRtqh0qk8rNAyzyIp3CeEEH0pRMuJvshWV5xhTr4P7bL8yCe9KprRxEmShKFd2+BXw09mjgmfnrSon+nI7ifho/FDdc+xq3kZ/pe5OFanVO2by/MS+9vrPC9f3D8cd597irc3TQIi25UAwK+Gd3HlHrM3au9zpRiD9HxjDO5x4EiduUoJwmrFyWxExEFmI2d/v9qpUgteyLPYefFRJ1qEZJmb50XXIddMqLS901OC4d1OwlM/6Ysidt8nzuO16/NSebweK3ceUhwzkwxPGXXmreZFb+PFdLYa5WbHsq2LjhtOIstAfSiMrIyAbl9KlGlPsYFuCqsnUvinuUuayy6awkurpsqdlUUfk6jm5alZ1jbolKGvClaUVYVtizjs2s3zkm68/sszlYIL3IugUavvrWpe3EoqZgU3shH7nd5tm6N5TiZOa9cieiwRAsJLC39Aj9/NwvIdP+pHJXlXJQXp4rBLwotlUrdTiMCbgD8aPxTzfz3SuCAHdSIxLV5a8INQOTW8qBO9sjxzgb7wYsJsxPSdzm2aAgDOPzX1/Roi2XI7nQgJF+HgkVrb91W3m5lJ5d3lu6J/U2K4xPLhXUOx4rfnKzQviWB/de2J6Mh1uuUSIevKMPZ5SRX/OjIbWSTdIxd5DrtdC5shJ0s5sAjneXHZnyDMyfehV1ZhLoj+Xz/Piyhs2XdvH4Q5Gytw+WnJmaLbDH+/pj/eXlaKK/u35X7PW0Xvq3JBeOElqdNowMc+3qh7XqLwUVU8IxCQkB2QFJNzj6LmiauQrC8IJ6q/KPK8cL73Uz+2A2leLJIiwqtlnA6VFtW8WEWWZTzxySahsqGwclSKvOxGDruiQwL7SArycnDdWR3QNJj664hWTbNx1zmnoISzt5MWFVXiEWVaiEQJ7RO4j1sOllZIkfnHEuzQ0ySYgTtGnqxd2GX0BIFENJF6651U0bLwIOHFIunusMt7Kew8EqfCBrXY9eMx4bJVx+sxc3VZ9HNU82KQYVeY9O46mvCe7oMX9bB9XSOflx+P1lk2RyaKdPR5icCOvdkZAeRkJsaMZJQ5KlECpmGG3RSZu0h4sUhqNL91eJoXbvIvwSdlRniRZRlPfiqmRbHCa9/siLsfYJSkTv93vv7LM6N/p7ITnR14/Wfwyfa3SlBf9r5312Dlzh+jn78tq7R9D69JZ80L+6plZkgJNeHrtcP1/1riXUUYlA67MSJpB3q3zUcqQMKLRVJFerWKcJI6h/O8AMDC7w6YXykzFcnP1U+/f7S2QfE5GiqtY9ky+pmFzWORNWnedTThaROcEPR4E8xVLy6O/u2nPYsIY9jJOSsjkLD3SZZlXQ3Ytv1HPaxNjICG2eize4fhl2d3xl9/1i8R1XIcEl4IS6jzB1zaryTOWRcQ11CZyddw8Kh5J84Q41NT26C9HwmvLiIOuwFJwi+GdNL8ns36T7ILHzNp+50kGYWX5Kuxc7BjT1YgkLCFpBz9j79Q5Hlh/u5amIffX3oqTsoLel8pFyDhxSLpvnpmpfvnruuPf1zb39b1zGherPj2slHStQ36FwiF1Rv5Kf/PQ5KAguY5GNSllVYJpmyadx6fIRhB7y98OGl6BauNy8qUEup/6MdmSJcMuyS8WCTd5x/WbKQnd4jKJGY0L1YGDFYgMVpoqxPxhqPRRnoOu43PQ2svHqPwRULLZ0pJXk4mrj2zvSP3+9vsrVix40fXncUJZ1H4vAQCCfN5Ue+B5hdobyNCl3SPNgowI4ZeFI5oVlK1tkMPK3kKzAhHYU2zkXXSZUBxGt6mc/m52RqlzfH3L7/DT6cuTsq8F62bOfMMkhF2EZGdSJ8X6Pu82KVfO/OOtY1J6pyvix8h4cUiadI/hNBNkS0svJi4n3hR5vriZ8WVje4qbXyu1kCaLim7nYb3pNSDc4smWWjfSjx3jBo/pf034l83DsTwbifhkct6JboqCYMdKxqjjRLk8+Ki5mXC+d3wbyZC0QyswB9IYUmGhBeLkN9CDD1ziqjQ4LrmhbEF/e6SU/XLamlebIxU1F2cQZbluMkqPzcLPW1kWk0ms9F5pxbi9V+eqYheSzfYsaIx2iiBwotL1x7QsSVaNDGvXZPl9DFRk/BiEZqMYuiN/aKrWlM+LxZGjLLDxwEALZtkYUDHlrpl1cJYxIxky2yULiOKDfhp++PLiR4TJRmjjdIZVvOSlfA8L+70HTv9OV1M1CS8WCSF+4Rp9AZ/tf+IFqaS1AmXjCcjYGy0Udclpnmxfl+jrJcEv115raVeacuyPVNcUkYbpTGs5kWSEmuEdUvutWMKSxcTNQkvFkl3h10FXkcb2RgwJEkyFB7UwtgHq8vw1eZ9tlbobH+hhT4f7nPhaVk4xTSCvIQQFbAJf6BeXCTKr8NNR2+r80ubZtmaeV5SDRJeLEKySwxnfF7MaF7sCBHGqxF1XdbursQvp63QvKtIX2CLJGN0ixdwM+xK6jL8gd3OCtNts9HFfYtdvX66oV7oJNL/cM2uw65c16zQ8fovz8TgLq3xt6tPSxsTdepvZesw5/UswJxNFbhlaOdEV8U36A39ohODqSR1tsw35jUvEULqBDAmYAdYWujbg9t+djYFdVl46VrQzNXrpxvq9zNheV4A3P32aleubVYgG97tJAzvdhIAYG/l8dh1Ulh6IeHFJC/eMAA7DhzFKTQgRdEb+0WFEjNmIzt2FxHhZckPP3KP14X0txXQo2kwtnVCLmcbBUIsSR3An6zsDNFum41SeQJJBHGalxR8vnYEMlYzmcpmIxJeTJKVEUDXwrxEV8NX6DrsCs4LpkKlhUvGIwmYjbQw2lZAjybZmXjz1rMAALnZJLyIol6BSpxjMmShlWrznExU1TTEHXc7VJpMzM6i1oCKTNBDTm6NRdsOOloPuwq7vJxMVHP6I6BM8W8W9sxUTulBPi+EbfR8OERXtV457IpoXrTQEl5EL3f2KW1w9iltrN08DRDx15WhESotcP3MDP5w57YZL3Wnj8SgNvOJOLeef2qh4/Wwm133Eh1fKFsBIenh8kLCC2EfR3xeTPiT2Iv6sb4Srj8hvOQFM7H2D6Mt14HgI9qsPM2ZyEJVa0Jw22HXTH9b8/vz8eCFPTD7/uHuVSjJad1MuSuyyPPVElwTiZ5WxJ7swkQ2Wr+M7yGzEWGbDq2aaH7nRpI6OwQkybLZqO5EQhBJajRBEM7SrbAZDhypVRzjDeJcnxeB0V5r/nLfbCTe33KyMnDHyJNdrE3yc/XA9thQVokRJxxUhdreBfOJHZn3g7vOxn9X7tL83o7mJYUtRQpoBCYs85/bB2PjntggwkPUbGRm9WvLbBSwYTaqjwgvkmLATGW7shd8fPdQvL54B/5vdHec9eSXiu/4SeriryHSAloTgtvRRma6B3UlY7IzA3jqqr7RzyJat0wLPiSS5EwwAo+2LXJ1BRSn8oilcloG/+nSiKThzM6tcPPZnXUnb1FrkBnNyxcby4XLqmnM82KNekbzQjhH77b5+PNP+3H36+FrXqxJL5pmowRFG908pFPcMUp+aR6RZ2YlkV0wU396tOPAL0n69bYTJaTIKWX9Mr6HhBfCVdzwedEKZRYhoNKamCFiNqIJJoHIfE2XiClQKwuv29sDaHWXC3oVxZd1tyopiZDPiwVpwOg9P1LLjxQSQTK4vh1tLntuCiteyGxEuIvoqvbdFdr2XycR2R5Ai6jZSH1Ne1UiTJLB9XkROS8xDrta8yZ/g0nqTWYR83cy/1zd8JOJIEn6G0ra0bzk52aheU4mZBlo1dT8ztTJAgkvhKu47QxpFjtmo1oyG3kO12zEGdnt+Ly4Hm2kUTuHEwWnLXYizXTPcTnDm9717Wh3MwISlv/2vOjfqQqZjQhXcXtiMIsds9HMVbsB0OrYS/hh0WLH4q6lUaTexrYPImjdNxCQ8MQVvRUmDepa5nHLjOvmxN+YbFH7e7u/KZiZgWBmaifD9ER4eeGFF9C5c2fk5ORgwIABWLhwoW75+fPnY8CAAcjJyUGXLl0wdepUL6pJuIDPFC+2NC+rSw8DoNVxIpERP6nIsqDZSGMyMpPd2UkkADcM6ohvfnNu7BhJL6YRkTEGn9zawnXdNBsZ+by4duuUwXXh5d1338V9992Hhx9+GKtXr8awYcNw0UUXobS0lFt++/btGDNmDIYNG4bVq1fjoYcewj333IMZM2a4XVXCBfxmNuL5vPQoMrfdg/p89vM53QsANKb+JuwjnufF+Fpak4U60u28ngUiVRNGSyCJHC9snoMP7jobn94zzNH7pg/Gjd+qaTYevLCHqau6mddOgoHPSwqbe5zC9RH2mWeewS233IJbb70VADBlyhR8/vnnePHFFzF58uS48lOnTkWHDh0wZcoUAEDPnj2xYsUKPP3007jqqqvcri7hMP4zG8WbIrJMjlJ6K6YbB3dEUX4OBnRsaal+hBLek1Y//8YuJhBtpCW8xJmNnJ04RBx2T2vfwtF7phOi87zZBYWbDrsw0Ly4eu8UwVXNS11dHVauXInRo5Wp1EePHo1FixZxz1m8eHFc+QsuuAArVqxAfX19XPna2lpUVVUp/hH+wW+aF97eRmbHifhoo9iRzIwAxvQp5uYsIczD01rwzD9CmheN0e6oKuTV6XlD63IUcu8MWs+xu80NdN3UfriZ5yVdcFV4OXDgAEKhEAoLlZtiFRYWorycn2isvLycW76hoQEHDhyIKz958mTk5+dH/7Vv3965H0DYxkjzkpPlrc84V3gxeQ3yS/AO9ZOWZZk76NuJNlLvNO30xKFpNnL2NmmLllA6vJtyE1Szr63bkTpu5XlJFzyZOeK2sJf1t7DnlecdB4BJkyahsrIy+m/XLm/yhRBiGPlCntXZvCOdHSSJ048kyZTfC40riYW3IhaKNtI4Xl1TryrnbANrRhtRR3IEtr1+eXZndDmpKV69eWBc8kGz7epqnhfoC8mR734+sJ1rdUh2XBVe2rRpg4yMjDgtS0VFRZx2JUJRURG3fGZmJlq3jp/ogsEgmjdvrvhH+AcjzYvX6tGMQPwQFpAa99cRheYc7+A9a96kYqdNqmvcNRtpQf3IGdjneGbnlvjq/0bi3B6FcWOP2eftrtlIEsrz8uSVfUyNTemEq8JLdnY2BgwYgNmzZyuOz549G0OGDOGeM3jw4LjyX3zxBQYOHIisrCzX6kq4g9Gmd14nUdIyG2VmBITrQitm7+BpW7nRRgLX0uqJ1bXxvnROQr3FXdj3MZgVy22i9rcz2w5ua15E8rxkZgTQu22+a/VIZlw3G02YMAH/+te/8Oqrr2LTpk24//77UVpainHjxgFoNPvceOON0fLjxo3Dzp07MWHCBGzatAmvvvoqXnnlFTzwwANuV5VwgRsHdwQAZGtE9HgtCEicaCO1z4PhNQwPEG7CzbAr0I+05Gi15sWrPklCsDOwzzGHScymXjj5S/Ni4PNC6WMNcT1U+uqrr8bBgwfx2GOPYe/evejduzc+/fRTdOzYOKnt3btXkfOlc+fO+PTTT3H//ffj+eefR0lJCZ599lkKk05SrjitLXqV5KOiqhY3vLI07ns/aF5q6kOmrqGeKGkK8harK2JZQ/fiutlIM8+Lw/dJU9jnGGQCANSbvZr2eXFZgND3eVF+edc5J+P5udvcrVCS4UkmrTvvvBN33nkn97tp06bFHRsxYgRWrVrlcq0IL5AkCd0K83DoaB33e6+TMfEy7JoVoGjSSSy86BI7bRJnXiDNS1LBPkY9zYvZVYa7ZiPJVKj0ry/oQcKLClJOEZ6g5W/gdTKmgBQvvUQGEVk12GlFINGUk1j4odLWzUbx1/IGkl2cQWE2YjQv6h3t/ZQSwTjPC3UOI0h4ITxBa+LwOtpIkqS4ic5sKCvlYEgsPE2ZSD8SFl6czvOicZwSkTmDlsOuetsHs++t+3letL+jIcYYEl4IT9DyN0iE2Uj0lpkZ/ILq82mg8Q4ZGpoXB9vAjea8ZWhnZAQknFrMpnKgjuMErMY0JzM2pd06rDMAYEyfIgA+izaS9Mc+2h7AGBJeiITixkuqt2JqdNhVfS8r/qcoy8PpJGaEOfjCi4DZSPD6TmvWJAn43SWnYvPjF+KUgmbR46R5cYZ6xjE3h9G89G3XAmv/MBrPX3c6ACvRRo5Uj4sEzjjE3puEF0NIeCG8QWPmcEM1m6knvAQ4Kee1ympchsaVxMLd20jgPLVPkxZuNW9WRkDRp8j86Ax1oVi0YDBTOaXl52ZFn7PftgfQuzp1DWNIeCE8QVNAcGGA0Bt0JE6otNnrqCedTDeXaEQc3GZxshs57vMSuyDbd2h+coa6htjokqkT32xWY+qm9qNxmxK978XufU73kxyqUfJBoy6RUNwwGxl58asHMa0VuabwIliOcB5Z1khS56AoYHStDq2aWL82c2kyDThDnXoTIw389LglaPezi/sUC12jSXYGXr35DAdrlVyQ8EIkFDcGFMMNz/guL3Foa16Un7M0HHsJ84j0B7XA27jTtPF5otFGRtey02eVWhjr1yFi1JpMMimKu6HS2hrgiRd2F7pGTlZGWpseSXghPIGdONhcDG6QzSSqUpPBGTQidVNPblorY/VxMhs5x9o/jDYs43q0kdNmI4W2xb37pCsjujWaTrq0aapbzuxE72bzcNZQzHdid9bz7UsHaNQlPIENlW7bItfRa6vVrGFZxju/GsQt25jnRbtuLKKaF62QasI8zXOMN1/lZth12WzkVJ+VFMIL9RsnKGiegzW/Px+f3z9ct5zZp+1282hd3+i+PYryUJKfg3+OHeB8pZIIEl4ITzgpLxj9mxUVeO+pmYlCkoB/XNtfcSwUljGoS2tc1Lsornxjnhd7Kxv10Sy3N0EhosiQ49pPhnLAv7J/W81zReAJR7nZ2to8lq5MKHQEtrZs3dN84ewoLZpkG76HZoURp5pnJMeplrdBLPudHhf3KcaiSaPQv0NLJ6qXtNCoS3hCj6LmePzyXnj5xoG65W4f3gUjTHjQF+blIBCQ0JSZXCJpwXmDAG9jRu3sv/xRZO3uSsVncth1lmd+3g8AMPWG07nfG4VKNwtmoqh5TlwZUZ8X3rQl6X4b4+oz2utfmdW8ULyRr3FCM5aVISn2WxK5ttF9SWHXCAkvhGeMHdwJ559aqDimflHNCgKR4p/cMyx6LJIWnDc5NOZ5UTt88q8tWpd0tz07zU9Ob4fv/ngRLuzdaA5c+tAoxfdcoZINQdYIQxWWXTglnZowJNK8JAyzwqJTzRPWGmC0zEYG1yNzYyMkvBC+IiMgmVghx17kToyzXnSXYO4cJ57nRTQHDfm8OA9rAihUaVF4zSKqGbGKVq4WoXMV2hZofCDcxrTZyKH20RrOtC5vZNYm2aUREl4IX2E29wWveEhH+uFNfFp5XkQ1KhRt5B2yrLUxo1K44LWcaIZdM8KzGp5gw9ZGUU+SXjzF/NO23z4SJOF+Fz3HKFSf+g0AEl4In2HWbMQVXqJmo3i4Pi9adREUpCjPi7dYDZW2IZMIO/saq/xjf5PZyFsSorGQgLCATx27UDKqJvWbRkh4IXyFeZ8X7fK87/YcrhFeuYiajchh1zu0duOVVGWcRnTxbObe5LvgNfYXRlbQ8nlhr68YQ4w0L9RtAJDwQiQY9YvYKHCIr5H13mPeS75hT6VwtJGoRoVCpb2FpxELKFauGjv22lG9MDg1d5DM6y2JCJWWICb4spoXQ58XMhsBIOGFSAQ6L7NZOUDvRed9s7eyhpMnRMvnJVaZvJxMzfuQ5sVbeC5GbBtodQlR2YU32SgO6TQ39yvNqBLqN15i9ml7qXlhhe0sAx860rw0QsIL4TnKJHXKN1HEYffhMT1j5+tNJJwvz+1REJ9hV1PzEns9snWkKnLY9Q5Z5vcREZ8BYYddjphj1umSRSsSSqJu4yl6Zro/XdUn7ljV8QbN8k6sV7SE1wwDjS+ZGxuh14fwFSJaDNEU67xvnrqqj7DDLms20qsXOey6TyRr7p0jT+FszKgSXtzweREsZ2ZioV7jLezzfu66/tG/e5U0x9VndIgrH9LytAWQKagiliSdPC9a1zYYA0nR24i2LpwgEoDVJHVcON+d1CwYN8FojS1s/ha9OUl0ICOs8+ef9sUtQzvj1OLmqK6NXxFnMG2gJUCITiFcHyYXHHbNpgUg7KGV3Vjr/dcTOjIDEupE7gkJ4bBIfWJojYHXndUB87fsx1UD2gncOfUh4YXwFQHJOEkdOznp+Q3wvjOzMmYnMb37ZNFSyHWyMgLo3TYfAF9gzVKZjezIBfee1xVvLi3V/N6kvKzZ50h28Razz1tPeBFNowCIh9kbXfvJK/tAlmUyG52AloxEQlG/h0JmI53zrcEfXBTCi859uhXlOVEJQhBeH1GHmlrtF/3a5aMgj7MvkrXLRaqjcZwmIS9RZkmOHddqWx2rkaFfCnsfretIGqoXvRQNJLjEIM0L4StEVjTKRF96L7rYPUVCpXmXmnHHYMzdvB+/PLuz2I0IR+BFiylMfBpCgZ3MuazDru4EYtDpWJNXdiatHT1Fw0yjhZ6Ttqn9zAS2NiKRxDwkvBAJRf3SZgQkFHB2BNY6R2+uEB1ftIYooyiiAR1bYUDHVmI3IRyDH22k1JLxBBiz6ns3KK+sif5NIfbeYiYK7ecD22HT3mrNa5lpO5FQacI8JPoTCSUuSV0AGDeii+45AY2ETupwZrtq+axMYydQwnt4E4dYqLTxtbWKCEcbGXy/lxFeCG9R+MrpOeAHJPz5p/0MHHYFo41gPtqIEIOEF8J3NMnORLfCZprfa2le8ptkKcsJm400fF5cDr8lrMFb9GYK+CfZMxuJlePdmz1WUx+yXgnCFpLOJ5bI4kjP50V0J3lJkrR9XshYZAsSXgjPsZPwSw27msrPtSi8aBzXmhCfvDI+oRXhHWotWHF+bpwPAq/t7ayAWZOTfrQRL8It9vczPz8NJ+UF8eL1p1uuC2ENZX6o2N9x3eLE57CO9GLGbKR1FdF8VQQf8nkhPKdpULvbiaxGlKHSMVqohBdRNzizextdd1Z8QivCW9751SDM3VyB7yqO4NHLemHHwaPR77T6kC3hhTnVzjwz+OTWWP7wedYvQFjGrKbDKM+L2D21F2skrtiDNC+E5/z9mtPQoygPL1x/uqUVhzLaKPa3Zc2LxuCicAKlocZXDOrSGpPG9MSrN5+B9q2aKFbCGQF+a13Wr8TwulrzlRu7ShPekqHhF6V25I581mvyDBNbgoj0Heo35iHhhfCcUwryMOu+4RjTp1izjKiwwDrsnt6xpeoaYohsD0CDi79hBc2crAxumQcu6I4Xrz8dpxRo+1MRqUtOljkHfCc0L4B2lBuZiuzhqvBy6NAhjB07Fvn5+cjPz8fYsWNx+PBh3XNuvvlmSJKk+Ddo0CA3q0kkEPXrG3mf9d7rkha5ceUB4LZhXXDPuadg5p1DzFVCYGNGGmb8DetAyU5SLMHMDFzUp5hjXoyhNdEo8rzoZnXmHaPe4weCmXyhNtK0nds0BQCM6HaS4jgPvURyCiSY3h6AEMNVn5frrrsOu3fvxqxZswAAv/rVrzB27Fh89NFHuuddeOGFeO2116Kfs7Oz3awmkUT8+oLuGHlicAGUq5fszAAmjO7OfGfvXqIRBUTiYVfCwcwM3VWtm3sK0WLavwSz+IuRiIzy9m2D8L81Zbj6jPYAnPN5EfG1Ii2MeVwTXjZt2oRZs2ZhyZIlOOusswAAL7/8MgYPHowtW7age/fumucGg0EUFRW5VTXCR7Rr1US47ICOLXHXOacojum98qKTlLbZiPK8JAtKs1FAfyVroSnF87zoRxsRiSOYqR9OX5Sfg9tHnBz97JjZSOMyPYuaC1+DiMc1s9HixYuRn58fFVwAYNCgQcjPz8eiRYt0z503bx4KCgrQrVs33HbbbaioqNAsW1tbi6qqKsU/Inm45oz2+NVw/aR0euhuDyB4Dc08L4zmJVfDj4LwB6yWTMs8EMGKLEF5xpIf1heKbU+t91/L3AOYy/OiZYrs0LoJ/nfX2Vg48RwyG1nANeGlvLwcBQUFcccLCgpQXl6ued5FF12EN998E1999RX++te/Yvny5Tj33HNRW1vLLT958uSoT01+fj7at2/v2G8g3CcrI4CHxvQUKssbZPS3mXFO8zL+3FNwanFz/PZisboS3sKuhHOyAroSip7AqxltxOZ50U/0QvgUVvNSH4pJJppZlfV2lTYRbaSX7K5f+xZob0L7TMQwLbw88sgjcQ616n8rVqwAwJ88jLb0vvrqq3HxxRejd+/euPTSS/HZZ59h69at+OSTT7jlJ02ahMrKyui/Xbt2mf1JRBLjpv8Ca4po3TQbn947DLcOs64lItyDbStDzYsVsxFpXpIetl/UNjBqFY221c2wa8psJOLzInw54gSmfV7Gjx+Pa665RrdMp06dsG7dOuzbty/uu/3796OwsFD4fsXFxejYsSO+++477vfBYBDBYFD4ekRqYXWD38ev6I3fffAtAO2JKTtTmTuE8C8ZAtFGEewKvHpzEfUS/8KagWsbjLdp0PN5ER0PJIkEX7cwLby0adMGbdq0MSw3ePBgVFZWYtmyZTjzzDMBAEuXLkVlZSWGDBEPZT148CB27dqF4mLtnCBE6mDWMbZptrVsvWMHdYwJLxpLL1Y1LBwaSSSELHW0kU5ZK7JLNmNyMLs7NTl7+wO2HeoajM1GTmheRKONSOw1j2s+Lz179sSFF16I2267DUuWLMGSJUtw22234ZJLLlFEGvXo0QPvv/8+AODIkSN44IEHsHjxYuzYsQPz5s3DpZdeijZt2uDKK690q6pEEvLnq/qiR1EefnfpqZpl1HNGu5a53HJaY0sGc4EMmoB8DbsSDhpoXvTQ1sKJXdNNMybhHAqzkQb6Pi/i7awnBBHWcTVJ3Ztvvok+ffpg9OjRGD16NPr27Ys33nhDUWbLli2orKwEAGRkZGD9+vW4/PLL0a1bN9x0003o1q0bFi9ejLy8PDerSvgMo9Xqz89oj1n3DUfbFnyBRM3kn/TBzDv4Gr/I2HLT4I6K46wpgsxG/obdRDMzIDme58XIjyYCd1dp03cj3IYVXjSjjZzI86ITbUTYw9Ukda1atcL06dN1y7AdJzc3F59//rmbVSJ8yk/6t8Wq0kM4v6e4P5QR7PBy7Zk6myme6IK/veRUSJKEaYt2AFBqW2hF7W/YySTTIBKE15RndGqJFTsP4fErenPPUZiNaC5Kemptmo1ENXGNGzMKlKPhxTS0qzThC565+jTDSDSzCG/MeGL4ysoIYFCXVjHhhZ0QKduur2HbqkkwAwM6tMT3FUe4ZXktedc5p+DsU9oowuNZghrH465N3SQpEFGc6GletPoJDxJ23YE2ZiR8g5HgYnYMEM6wq7gw31REViN/k5OVgUcv64XfXtwTbZoF8dtLeuLeUV25ZXn9IiBJuhOS0mHXHCTQ+IdJF/VA77bN8YshnaPHrOwkLqp5AQS3BxC+GhGBhBfClzjyMtu8CCu80OrJ/9w0pFM0D09eThbuP78btxxPmDASdMXNBDQN+ZnbR5yMj+8ehvwmWfj5wHYAgPvP5wu5ekKHcH+QxISXHsW0VYBZyGxE+BInVquiE4nW0EJOuqmK+XbNzmB9XrQnI16/bdWUNpb1I3+6qi/uP78bivP5Tv96QoeoGRGQhBY+T/+0L56ZvRU3DOpoXJgAQMILkcII+7xojC6swy4pXlIHnkxqFBEistJ+aewAHK+PJT/7+zWn4bt9RzC4S2vTdSTcR5IkTcEFcMZh1+g6EQqa5+Cpq/oKX5Mg4YVIIsyabuzqTUjzkprYTVLHo3fb5hjdqwh7Dh8H0JjN9fLT2lqpHuET9DRsZsxGItsDEOYh4YVIWcSjjWL0KonZnkl4SU14/i1G80tQ0GG3pEUuFv3mXDTPzbJYO8Iv6GpeTEQbUZI6dyDhhfAlXkZosBNX+1ZNMOu+YWjZJFthXijIo/2zkhl2Xxte3zKaXxQrbU5h1r+qRDBxIuFv9B12xZIWGl2HsA5FGxG+5PQOLW1fw2rkR4+i5ihsngNJkjD7/uH4aPxQtGhCTpfJyJX9G00395wbiyix0i9uGtxJ93sKh049nAiVlgCESfXiCqR5IXzJgxf2wEnNgrioTzHOe2Y+gMTk1+haSNtSJDN//mlf/OLsTuhdkh89xtW8aMxUhc2DmPvASDTR2QAUAK7Ty+BMpBxZgkkr2zQLYtehYy7XJj0hzQvhS5oGM3H3qK44paBZ9FhX5m8RaDdfIisjgL7tWih2BTfbL9SCi1rMeevWs3D1Ge2tVpFIQox8Xq7s3xZndmqFf1zXX2WWzsWNgykc2glI80L4nv/ddTbeX12G+8/jJx3TgkQXggc/VFocWZbx7LX9cc/bqwEAvUrySVBOMzINhJfBJ7fGzwc2CrR92+Vj0baDCGYGsODX51BfcQgSXgjf0699C/Rr38L0eQM72febIVIP7jYAJm2Sgzq3Yk4ln4Z0w8hsdCqTMXfK1afhhXnbcMOgDiS4OAgJL0TKMqzrSXjlpoHoWkB+K0SMTCdC4JlLUDBJ+qG1D9Yjl56KboV56N025mNV0DwHj1zWy6uqpQ0kvBApzaiehYmuAuEzAiaEFy3BhI1YItkl/dASXk4uaIYhp7TxuDbpCTnsEmnLUz/pAwB4/rrTE1wTwksyOKp7rcmIJ5jIoASG6Y5W+2cGaEr1CnrSRNpyzZkdsOWJC3Fx3+JEV4XwEN7EM/hk8f2HZLlxs8XL+pXg0n4ltPFiivKPa/sDAJ7+Wb+477RMj6Ih1IR9yGxEpDVBE5kyidRAvT3A7cO7aK6keWajnsWNPlTPnpjciNTk0n4luKBXEbIzA3jgvbWK7zQ1Lya2DSDsQcILQRBpRdz8IrhY/vSeYfjPil24+9xTHK8T4U+0Mulqm41I8+IVJLwQBJFWqB129bcLiKleTi1pTlEjBABt4UXLd4pwHnrSBEGkFWqHXUq9QZhF22xEnckrSHghCCKtUE88unoXioMmOORk8X3lsijayDPoSRMEkVaoHXZJ80KY4bZhndG2RS73O9K8eAcJLwRBpBXxmhftCYcULwTLrUM74+GLT9X8noQX7yDhhSCItCJOeKH5hhDEKDszmY28g540QRBpRZzZSKesTE4vBIO676ghzYt3kPBCEERaEZ/nhcxGhBjDuurvW0Sh0t5BeV4IgkgrzGheCAIAlj08Ctv3H8VZXfS3kaAkdd5BwgtBEGmFGZ8XshoRAFCQl4OCvBzDcrRhp3eQjosgiLSCJhjCLSTy/vYMEl4Igkgr1Gajlk20d4Umh12C8CeuCi9//OMfMWTIEDRp0gQtWrQQOkeWZTzyyCMoKSlBbm4uRo4ciQ0bNrhZTYIg0ghW83JJ32Jcc2b7uDIdWjUBAJzbo8CzehEEIY6rwktdXR1+9rOf4Y477hA+589//jOeeeYZPPfcc1i+fDmKiopw/vnno7q62sWaEgSRLgw9pTFipHlOJp677nQEM+NTvb83bjAevawXHr+it9fVI5KU/911dqKrkFZIsgd60WnTpuG+++7D4cOHdcvJsoySkhLcd999ePDBBwEAtbW1KCwsxJ/+9CfcfvvthveqqqpCfn4+Kisr0bx5cyeqTxBEirHn8HHk52ahaZBiFghrdPrNJ4rPO566OEE1SR3MzN++8nnZvn07ysvLMXr06OixYDCIESNGYNGiRdxzamtrUVVVpfhHEAShR0mLXBJcCFv0LI5Nrr3b0kLZa3z19paXlwMACgsLFccLCwuxc+dO7jmTJ0/Go48+6nrdCIIgCCLCR+PPxp7DNZi3tQJj+hQnujpph2nNyyOPPAJJknT/rVixwlal1OFmsixrhqBNmjQJlZWV0X+7du2ydW+CIAiCMCIzI4AOrZvgxsGd0KZZMNHVSTtMa17Gjx+Pa665RrdMp06dLFWmqKgIQKMGprg4JslWVFTEaWMiBINBBIPUcQiCIAgiXTAtvLRp0wZt2ujv72CVzp07o6ioCLNnz0b//v0BNEYszZ8/H3/6059cuSdBEARBEMmFqw67paWlWLNmDUpLSxEKhbBmzRqsWbMGR44ciZbp0aMH3n//fQCN5qL77rsPTz75JN5//318++23uPnmm9GkSRNcd911blaVIAiCIIgkwVWH3d///vf497//Hf0c0abMnTsXI0eOBABs2bIFlZWV0TITJ07E8ePHceedd+LQoUM466yz8MUXXyAvL8/NqhIEQRAEkSR4kufFSyjPC0EQBEEkH0mb54UgCIIgCMIIEl4IgiAIgkgqSHghCIIgCCKpIOGFIAiCIIikgoQXgiAIgiCSChJeCIIgCIJIKkh4IQiCIAgiqSDhhSAIgiCIpMLVDLuJIJJzr6qqKsE1IQiCIAhClMi8LZI7N+WEl+rqagBA+/btE1wTgiAIgiDMUl1djfz8fN0yKbc9QDgcxp49e5CXlwdJkhy9dlVVFdq3b49du3bR1gMuQs/ZO+hZewM9Z2+g5+wNbj1nWZZRXV2NkpISBAL6Xi0pp3kJBAJo166dq/do3rw5vRgeQM/ZO+hZewM9Z2+g5+wNbjxnI41LBHLYJQiCIAgiqSDhhSAIgiCIpIKEFxMEg0H84Q9/QDAYTHRVUhp6zt5Bz9ob6Dl7Az1nb/DDc045h12CIAiCIFIb0rwQBEEQBJFUkPBCEARBEERSQcILQRAEQRBJBQkvBEEQBEEkFSS8CPLCCy+gc+fOyMnJwYABA7Bw4cJEVympmDx5Ms444wzk5eWhoKAAV1xxBbZs2aIoI8syHnnkEZSUlCA3NxcjR47Ehg0bFGVqa2tx9913o02bNmjatCkuu+wy7N6928ufklRMnjwZkiThvvvuix6j5+wcZWVluOGGG9C6dWs0adIEp512GlauXBn9np61fRoaGvDb3/4WnTt3Rm5uLrp06YLHHnsM4XA4Woaes3kWLFiASy+9FCUlJZAkCR988IHie6ee6aFDhzB27Fjk5+cjPz8fY8eOxeHDh+3/AJkw5J133pGzsrLkl19+Wd64caN87733yk2bNpV37tyZ6KolDRdccIH82muvyd9++628Zs0a+eKLL5Y7dOggHzlyJFrmqaeekvPy8uQZM2bI69evl6+++mq5uLhYrqqqipYZN26c3LZtW3n27NnyqlWr5HPOOUfu16+f3NDQkIif5WuWLVsmd+rUSe7bt6987733Ro/Tc3aGH3/8Ue7YsaN88803y0uXLpW3b98uz5kzR/7++++jZehZ2+eJJ56QW7duLX/88cfy9u3b5ffee09u1qyZPGXKlGgZes7m+fTTT+WHH35YnjFjhgxAfv/99xXfO/VML7zwQrl3797yokWL5EWLFsm9e/eWL7nkEtv1J+FFgDPPPFMeN26c4liPHj3k3/zmNwmqUfJTUVEhA5Dnz58vy7Ish8NhuaioSH7qqaeiZWpqauT8/Hx56tSpsizL8uHDh+WsrCz5nXfeiZYpKyuTA4GAPGvWLG9/gM+prq6Wu3btKs+ePVseMWJEVHih5+wcDz74oDx06FDN7+lZO8PFF18s//KXv1Qc+8lPfiLfcMMNsizTc3YCtfDi1DPduHGjDEBesmRJtMzixYtlAPLmzZtt1ZnMRgbU1dVh5cqVGD16tOL46NGjsWjRogTVKvmprKwEALRq1QoAsH37dpSXlyueczAYxIgRI6LPeeXKlaivr1eUKSkpQe/evaktVNx11124+OKLcd555ymO03N2jg8//BADBw7Ez372MxQUFKB///54+eWXo9/Ts3aGoUOH4ssvv8TWrVsBAGvXrsXXX3+NMWPGAKDn7AZOPdPFixcjPz8fZ511VrTMoEGDkJ+fb/u5p9zGjE5z4MABhEIhFBYWKo4XFhaivLw8QbVKbmRZxoQJEzB06FD07t0bAKLPkvecd+7cGS2TnZ2Nli1bxpWhtojxzjvvYNWqVVi+fHncd/ScneOHH37Aiy++iAkTJuChhx7CsmXLcM899yAYDOLGG2+kZ+0QDz74ICorK9GjRw9kZGQgFArhj3/8I6699loA1KfdwKlnWl5ejoKCgrjrFxQU2H7uJLwIIkmS4rMsy3HHCDHGjx+PdevW4euvv477zspzpraIsWvXLtx777344osvkJOTo1mOnrN9wuEwBg4ciCeffBIA0L9/f2zYsAEvvvgibrzxxmg5etb2ePfddzF9+nS89dZb6NWrF9asWYP77rsPJSUluOmmm6Ll6Dk7jxPPlFfeiedOZiMD2rRpg4yMjDgpsaKiIk4qJYy5++678eGHH2Lu3Llo165d9HhRUREA6D7noqIi1NXV4dChQ5pl0p2VK1eioqICAwYMQGZmJjIzMzF//nw8++yzyMzMjD4nes72KS4uxqmnnqo41rNnT5SWlgKgPu0Uv/71r/Gb3/wG11xzDfr06YOxY8fi/vvvx+TJkwHQc3YDp55pUVER9u3bF3f9/fv3237uJLwYkJ2djQEDBmD27NmK47Nnz8aQIUMSVKvkQ5ZljB8/HjNnzsRXX32Fzp07K77v3LkzioqKFM+5rq4O8+fPjz7nAQMGICsrS1Fm7969+Pbbb6ktTjBq1CisX78ea9asif4bOHAgrr/+eqxZswZdunSh5+wQZ599dly4/9atW9GxY0cA1Ked4tixYwgElFNVRkZGNFSanrPzOPVMBw8ejMrKSixbtixaZunSpaisrLT/3G25+6YJkVDpV155Rd64caN83333yU2bNpV37NiR6KolDXfccYecn58vz5s3T967d2/037Fjx6JlnnrqKTk/P1+eOXOmvH79evnaa6/lhua1a9dOnjNnjrxq1Sr53HPPTetwRxHYaCNZpufsFMuWLZMzMzPlP/7xj/J3330nv/nmm3KTJk3k6dOnR8vQs7bPTTfdJLdt2zYaKj1z5ky5TZs28sSJE6Nl6Dmbp7q6Wl69erW8evVqGYD8zDPPyKtXr46mAHHqmV544YVy37595cWLF8uLFy+W+/TpQ6HSXvL888/LHTt2lLOzs+XTTz89GuJLiAGA+++1116LlgmHw/If/vAHuaioSA4Gg/Lw4cPl9evXK65z/Phxefz48XKrVq3k3Nxc+ZJLLpFLS0s9/jXJhVp4oefsHB999JHcu3dvORgMyj169JBfeuklxff0rO1TVVUl33vvvXKHDh3knJwcuUuXLvLDDz8s19bWRsvQczbP3LlzuWPyTTfdJMuyc8/04MGD8vXXXy/n5eXJeXl58vXXXy8fOnTIdv0lWZZle7obgiAIgiAI7yCfF4IgCIIgkgoSXgiCIAiCSCpIeCEIgiAIIqkg4YUgCIIgiKSChBeCIAiCIJIKEl4IgiAIgkgqSHghCIIgCCKpIOGFIAiCIIikgoQXgiAIgiCSChJeCIIgCIJIKkh4IQiCIAgiqSDhhSAIgiCIpOL/AUAZZJ1pLp9EAAAAAElFTkSuQmCC",
|
|
"text/plain": [
|
|
"<Figure size 640x480 with 1 Axes>"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Model: \"sequential\"\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" Layer (type) Output Shape Param # \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" simple_rnn (SimpleRNN) (None, 32) 1184 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense (Dense) (None, 8) 264 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" dense_1 (Dense) (None, 1) 9 \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
" \n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"=================================================================\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Total params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Trainable params: 1,457\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Non-trainable params: 0\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"_________________________________________________________________\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 1/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"2025-08-05 16:51:30.337261: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 1s - loss: 0.5641 - 803ms/epoch - 16ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 2/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4221 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 3/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.4073 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 4/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3955 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 5/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3934 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 6/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3879 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 7/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3849 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 8/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3850 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 9/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3834 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 10/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3845 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 11/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3831 - 307ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 12/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3826 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 13/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3829 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 14/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3834 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 15/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3817 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 16/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3784 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 17/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3813 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 18/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3806 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 19/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3799 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 20/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3795 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 21/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3809 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 22/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3771 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 23/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3785 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 24/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3798 - 315ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 25/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3782 - 358ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 26/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3771 - 365ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 27/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3760 - 326ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 28/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3761 - 302ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 29/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3790 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 30/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3769 - 344ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 31/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3768 - 333ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 32/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3765 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 33/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3763 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 34/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3778 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 35/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3750 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 36/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3750 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 37/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3747 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 38/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3733 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 39/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3767 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 40/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3745 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 41/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3761 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 42/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3744 - 300ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 43/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3727 - 307ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 44/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3752 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 45/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3734 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 46/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3740 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 47/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3725 - 329ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 48/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3741 - 307ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 49/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3709 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 50/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3741 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 51/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3736 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 52/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3722 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 53/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3721 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 54/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3707 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 55/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3704 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 56/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3705 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 57/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3713 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 58/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3689 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 59/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3680 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 60/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3684 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 61/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3688 - 307ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 62/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3694 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 63/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3676 - 306ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 64/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3649 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 65/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3674 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 66/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3674 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 67/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3671 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 68/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3676 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 69/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3653 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 70/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3648 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 71/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3635 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 72/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3623 - 302ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 73/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3637 - 300ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 74/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3621 - 332ms/epoch - 7ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 75/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3632 - 309ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 76/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3631 - 305ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 77/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3633 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 78/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3638 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 79/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3605 - 304ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 80/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3613 - 302ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 81/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3601 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 82/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3616 - 303ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 83/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3610 - 323ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 84/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3547 - 315ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 85/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3609 - 313ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 86/100\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"50/50 - 0s - loss: 0.3590 - 313ms/epoch - 6ms/step\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Epoch 87/100\n"
|
|
]
|
|
},
|
|
{
|
|
"ename": "KeyboardInterrupt",
|
|
"evalue": "",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m Traceback (most recent call last)",
|
|
"Cell \u001b[0;32mIn[1], line 58\u001b[0m\n\u001b[1;32m 55\u001b[0m model\u001b[38;5;241m.\u001b[39mcompile(loss\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmean_squared_error\u001b[39m\u001b[38;5;124m'\u001b[39m, optimizer\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mrmsprop\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m 56\u001b[0m model\u001b[38;5;241m.\u001b[39msummary()\n\u001b[0;32m---> 58\u001b[0m \u001b[43mmodel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfit\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtrainX\u001b[49m\u001b[43m,\u001b[49m\u001b[43mtrainY\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mepochs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mbatch_size\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m16\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mverbose\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m2\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[1;32m 59\u001b[0m trainPredict \u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(trainX)\n\u001b[1;32m 60\u001b[0m testPredict\u001b[38;5;241m=\u001b[39m model\u001b[38;5;241m.\u001b[39mpredict(testX)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/utils/traceback_utils.py:65\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 63\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 64\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m---> 65\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 66\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 67\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/keras/engine/training.py:1685\u001b[0m, in \u001b[0;36mModel.fit\u001b[0;34m(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\u001b[0m\n\u001b[1;32m 1677\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m tf\u001b[38;5;241m.\u001b[39mprofiler\u001b[38;5;241m.\u001b[39mexperimental\u001b[38;5;241m.\u001b[39mTrace(\n\u001b[1;32m 1678\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtrain\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m 1679\u001b[0m epoch_num\u001b[38;5;241m=\u001b[39mepoch,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 1682\u001b[0m _r\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m,\n\u001b[1;32m 1683\u001b[0m ):\n\u001b[1;32m 1684\u001b[0m callbacks\u001b[38;5;241m.\u001b[39mon_train_batch_begin(step)\n\u001b[0;32m-> 1685\u001b[0m tmp_logs \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mtrain_function\u001b[49m\u001b[43m(\u001b[49m\u001b[43miterator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 1686\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m data_handler\u001b[38;5;241m.\u001b[39mshould_sync:\n\u001b[1;32m 1687\u001b[0m context\u001b[38;5;241m.\u001b[39masync_wait()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py:150\u001b[0m, in \u001b[0;36mfilter_traceback.<locals>.error_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 148\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[1;32m 149\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 150\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 151\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 152\u001b[0m filtered_tb \u001b[38;5;241m=\u001b[39m _process_traceback_frames(e\u001b[38;5;241m.\u001b[39m__traceback__)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:894\u001b[0m, in \u001b[0;36mFunction.__call__\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 891\u001b[0m compiler \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mxla\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile \u001b[38;5;28;01melse\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnonXla\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 893\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m OptionalXlaContext(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_jit_compile):\n\u001b[0;32m--> 894\u001b[0m result \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 896\u001b[0m new_tracing_count \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mexperimental_get_tracing_count()\n\u001b[1;32m 897\u001b[0m without_tracing \u001b[38;5;241m=\u001b[39m (tracing_count \u001b[38;5;241m==\u001b[39m new_tracing_count)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py:926\u001b[0m, in \u001b[0;36mFunction._call\u001b[0;34m(self, *args, **kwds)\u001b[0m\n\u001b[1;32m 923\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m 924\u001b[0m \u001b[38;5;66;03m# In this case we have created variables on the first call, so we run the\u001b[39;00m\n\u001b[1;32m 925\u001b[0m \u001b[38;5;66;03m# defunned version which is guaranteed to never create variables.\u001b[39;00m\n\u001b[0;32m--> 926\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_no_variable_creation_fn\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwds\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# pylint: disable=not-callable\u001b[39;00m\n\u001b[1;32m 927\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_variable_creation_fn \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m 928\u001b[0m \u001b[38;5;66;03m# Release the lock early so that multiple threads can perform the call\u001b[39;00m\n\u001b[1;32m 929\u001b[0m \u001b[38;5;66;03m# in parallel.\u001b[39;00m\n\u001b[1;32m 930\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock\u001b[38;5;241m.\u001b[39mrelease()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compiler.py:143\u001b[0m, in \u001b[0;36mTracingCompiler.__call__\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m 140\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_lock:\n\u001b[1;32m 141\u001b[0m (concrete_function,\n\u001b[1;32m 142\u001b[0m filtered_flat_args) \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_maybe_define_function(args, kwargs)\n\u001b[0;32m--> 143\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mconcrete_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_call_flat\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 144\u001b[0m \u001b[43m \u001b[49m\u001b[43mfiltered_flat_args\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcaptured_inputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mconcrete_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcaptured_inputs\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/monomorphic_function.py:1757\u001b[0m, in \u001b[0;36mConcreteFunction._call_flat\u001b[0;34m(self, args, captured_inputs, cancellation_manager)\u001b[0m\n\u001b[1;32m 1753\u001b[0m possible_gradient_type \u001b[38;5;241m=\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPossibleTapeGradientTypes(args)\n\u001b[1;32m 1754\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (possible_gradient_type \u001b[38;5;241m==\u001b[39m gradients_util\u001b[38;5;241m.\u001b[39mPOSSIBLE_GRADIENT_TYPES_NONE\n\u001b[1;32m 1755\u001b[0m \u001b[38;5;129;01mand\u001b[39;00m executing_eagerly):\n\u001b[1;32m 1756\u001b[0m \u001b[38;5;66;03m# No tape is watching; skip to running the function.\u001b[39;00m\n\u001b[0;32m-> 1757\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_build_call_outputs(\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_inference_function\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcall\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 1758\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcancellation_manager\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcancellation_manager\u001b[49m\u001b[43m)\u001b[49m)\n\u001b[1;32m 1759\u001b[0m forward_backward \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_select_forward_and_backward_functions(\n\u001b[1;32m 1760\u001b[0m args,\n\u001b[1;32m 1761\u001b[0m possible_gradient_type,\n\u001b[1;32m 1762\u001b[0m executing_eagerly)\n\u001b[1;32m 1763\u001b[0m forward_function, args_with_tangents \u001b[38;5;241m=\u001b[39m forward_backward\u001b[38;5;241m.\u001b[39mforward()\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/polymorphic_function/monomorphic_function.py:381\u001b[0m, in \u001b[0;36m_EagerDefinedFunction.call\u001b[0;34m(self, ctx, args, cancellation_manager)\u001b[0m\n\u001b[1;32m 379\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m _InterpolateFunctionError(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m 380\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m cancellation_manager \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m--> 381\u001b[0m outputs \u001b[38;5;241m=\u001b[39m \u001b[43mexecute\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mexecute\u001b[49m\u001b[43m(\u001b[49m\n\u001b[1;32m 382\u001b[0m \u001b[43m \u001b[49m\u001b[38;5;28;43mstr\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43msignature\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mname\u001b[49m\u001b[43m)\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 383\u001b[0m \u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_num_outputs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 384\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 385\u001b[0m \u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 386\u001b[0m \u001b[43m \u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mctx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 387\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m 388\u001b[0m outputs \u001b[38;5;241m=\u001b[39m execute\u001b[38;5;241m.\u001b[39mexecute_with_cancellation(\n\u001b[1;32m 389\u001b[0m \u001b[38;5;28mstr\u001b[39m(\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39msignature\u001b[38;5;241m.\u001b[39mname),\n\u001b[1;32m 390\u001b[0m num_outputs\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_num_outputs,\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 393\u001b[0m ctx\u001b[38;5;241m=\u001b[39mctx,\n\u001b[1;32m 394\u001b[0m cancellation_manager\u001b[38;5;241m=\u001b[39mcancellation_manager)\n",
|
|
"File \u001b[0;32m~/miniforge3/envs/myenv/lib/python3.9/site-packages/tensorflow/python/eager/execute.py:52\u001b[0m, in \u001b[0;36mquick_execute\u001b[0;34m(op_name, num_outputs, inputs, attrs, ctx, name)\u001b[0m\n\u001b[1;32m 50\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m 51\u001b[0m ctx\u001b[38;5;241m.\u001b[39mensure_initialized()\n\u001b[0;32m---> 52\u001b[0m tensors \u001b[38;5;241m=\u001b[39m \u001b[43mpywrap_tfe\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mTFE_Py_Execute\u001b[49m\u001b[43m(\u001b[49m\u001b[43mctx\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_handle\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mdevice_name\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mop_name\u001b[49m\u001b[43m,\u001b[49m\n\u001b[1;32m 53\u001b[0m \u001b[43m \u001b[49m\u001b[43minputs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mattrs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mnum_outputs\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 54\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m core\u001b[38;5;241m.\u001b[39m_NotOkStatusException \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[1;32m 55\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m name \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
|
|
"\u001b[0;31mKeyboardInterrupt\u001b[0m: "
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%matplotlib inline\n",
|
|
"\n",
|
|
"# Start importing packages\n",
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"import tensorflow as tf\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras.models import Model, Sequential \n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"from tensorflow.keras import optimizers \n",
|
|
"from tensorflow.keras import regularizers \n",
|
|
"from tensorflow.keras.utils import to_categorical \n",
|
|
"\n",
|
|
"\n",
|
|
"\n",
|
|
"# convert into dataset matrix\n",
|
|
"def convertToMatrix(data, step):\n",
|
|
" X, Y =[], []\n",
|
|
" for i in range(len(data)-step):\n",
|
|
" d=i+step \n",
|
|
" X.append(data[i:d,])\n",
|
|
" Y.append(data[d,])\n",
|
|
" return np.array(X), np.array(Y)\n",
|
|
"\n",
|
|
"step = 4\n",
|
|
"N = 1000 \n",
|
|
"Tp = 800 \n",
|
|
"\n",
|
|
"t=np.arange(0,N)\n",
|
|
"x=np.sin(0.02*t)+2*np.random.rand(N)\n",
|
|
"df = pd.DataFrame(x)\n",
|
|
"df.head()\n",
|
|
"\n",
|
|
"plt.plot(df)\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"values=df.values\n",
|
|
"train,test = values[0:Tp,:], values[Tp:N,:]\n",
|
|
"\n",
|
|
"# add step elements into train and test\n",
|
|
"test = np.append(test,np.repeat(test[-1,],step))\n",
|
|
"train = np.append(train,np.repeat(train[-1,],step))\n",
|
|
" \n",
|
|
"trainX,trainY =convertToMatrix(train,step)\n",
|
|
"testX,testY =convertToMatrix(test,step)\n",
|
|
"trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))\n",
|
|
"testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))\n",
|
|
"\n",
|
|
"model = Sequential()\n",
|
|
"model.add(SimpleRNN(units=32, input_shape=(1,step), activation=\"relu\"))\n",
|
|
"model.add(Dense(8, activation=\"relu\")) \n",
|
|
"model.add(Dense(1))\n",
|
|
"model.compile(loss='mean_squared_error', optimizer='rmsprop')\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"model.fit(trainX,trainY, epochs=100, batch_size=16, verbose=2)\n",
|
|
"trainPredict = model.predict(trainX)\n",
|
|
"testPredict= model.predict(testX)\n",
|
|
"predicted=np.concatenate((trainPredict,testPredict),axis=0)\n",
|
|
"\n",
|
|
"trainScore = model.evaluate(trainX, trainY, verbose=0)\n",
|
|
"print(trainScore)\n",
|
|
"\n",
|
|
"index = df.index.values\n",
|
|
"plt.plot(index,df)\n",
|
|
"plt.plot(index,predicted)\n",
|
|
"plt.axvline(df.index[Tp], c=\"r\")\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "897a47c7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## An extrapolation example\n",
|
|
"\n",
|
|
"The following code provides an example of how recurrent neural\n",
|
|
"networks can be used to extrapolate to unknown values of physics data\n",
|
|
"sets. Specifically, the data sets used in this program come from\n",
|
|
"a quantum mechanical many-body calculation of energies as functions of the number of particles."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "6776ae2a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"\n",
|
|
"# For matrices and calculations\n",
|
|
"import numpy as np\n",
|
|
"# For machine learning (backend for keras)\n",
|
|
"import tensorflow as tf\n",
|
|
"# User-friendly machine learning library\n",
|
|
"# Front end for TensorFlow\n",
|
|
"import tensorflow.keras\n",
|
|
"# Different methods from Keras needed to create an RNN\n",
|
|
"# This is not necessary but it shortened function calls \n",
|
|
"# that need to be used in the code.\n",
|
|
"from tensorflow.keras import datasets, layers, models\n",
|
|
"from tensorflow.keras.layers import Input\n",
|
|
"from tensorflow.keras import regularizers\n",
|
|
"from tensorflow.keras.models import Model, Sequential\n",
|
|
"from tensorflow.keras.layers import Dense, SimpleRNN, LSTM, GRU\n",
|
|
"# For timing the code\n",
|
|
"from timeit import default_timer as timer\n",
|
|
"# For plotting\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"\n",
|
|
"\n",
|
|
"# The data set\n",
|
|
"datatype='VaryDimension'\n",
|
|
"X_tot = np.arange(2, 42, 2)\n",
|
|
"y_tot = np.array([-0.03077640549, -0.08336233266, -0.1446729567, -0.2116753732, -0.2830637392, -0.3581341341, -0.436462435, -0.5177783846,\n",
|
|
"\t-0.6019067271, -0.6887363571, -0.7782028952, -0.8702784034, -0.9649652536, -1.062292565, -1.16231451, \n",
|
|
"\t-1.265109911, -1.370782966, -1.479465113, -1.591317992, -1.70653767])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35227d36",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The way the recurrent neural networks are trained in this program\n",
|
|
"differs from how machine learning algorithms are usually trained.\n",
|
|
"Typically a machine learning algorithm is trained by learning the\n",
|
|
"relationship between the x data and the y data. In this program, the\n",
|
|
"recurrent neural network will be trained to recognize the relationship\n",
|
|
"in a sequence of y values. This is type of data formatting is\n",
|
|
"typically used time series forcasting, but it can also be used in any\n",
|
|
"extrapolation (time series forecasting is just a specific type of\n",
|
|
"extrapolation along the time axis). This method of data formatting\n",
|
|
"does not use the x data and assumes that the y data are evenly spaced.\n",
|
|
"\n",
|
|
"For a standard machine learning algorithm, the training data has the\n",
|
|
"form of (x,y) so the machine learning algorithm learns to assiciate a\n",
|
|
"y value with a given x value. This is useful when the test data has x\n",
|
|
"values within the same range as the training data. However, for this\n",
|
|
"application, the x values of the test data are outside of the x values\n",
|
|
"of the training data and the traditional method of training a machine\n",
|
|
"learning algorithm does not work as well. For this reason, the\n",
|
|
"recurrent neural network is trained on sequences of y values of the\n",
|
|
"form ((y1, y2), y3), so that the network is concerned with learning\n",
|
|
"the pattern of the y data and not the relation between the x and y\n",
|
|
"data. As long as the pattern of y data outside of the training region\n",
|
|
"stays relatively stable compared to what was inside the training\n",
|
|
"region, this method of training can produce accurate extrapolations to\n",
|
|
"y values far removed from the training data set."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "7dc577d1",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# FORMAT_DATA\n",
|
|
"def format_data(data, length_of_sequence = 2): \n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" data(a numpy array): the data that will be the inputs to the recurrent neural\n",
|
|
" network\n",
|
|
" length_of_sequence (an int): the number of elements in one iteration of the\n",
|
|
" sequence patter. For a function approximator use length_of_sequence = 2.\n",
|
|
" Returns:\n",
|
|
" rnn_input (a 3D numpy array): the input data for the recurrent neural network. Its\n",
|
|
" dimensions are length of data - length of sequence, length of sequence, \n",
|
|
" dimnsion of data\n",
|
|
" rnn_output (a numpy array): the training data for the neural network\n",
|
|
" Formats data to be used in a recurrent neural network.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" X, Y = [], []\n",
|
|
" for i in range(len(data)-length_of_sequence):\n",
|
|
" # Get the next length_of_sequence elements\n",
|
|
" a = data[i:i+length_of_sequence]\n",
|
|
" # Get the element that immediately follows that\n",
|
|
" b = data[i+length_of_sequence]\n",
|
|
" # Reshape so that each data point is contained in its own array\n",
|
|
" a = np.reshape (a, (len(a), 1))\n",
|
|
" X.append(a)\n",
|
|
" Y.append(b)\n",
|
|
" rnn_input = np.array(X)\n",
|
|
" rnn_output = np.array(Y)\n",
|
|
"\n",
|
|
" return rnn_input, rnn_output\n",
|
|
"\n",
|
|
"\n",
|
|
"# ## Defining the Recurrent Neural Network Using Keras\n",
|
|
"# \n",
|
|
"# The following method defines a simple recurrent neural network in keras consisting of one input layer, one hidden layer, and one output layer.\n",
|
|
"\n",
|
|
"def rnn(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with one hidden layer and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer\n",
|
|
" hidden_neurons = 200\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Define the hidden layer as a simple RNN layer with a set number of neurons and add it to \n",
|
|
" # the network immediately after the input layer\n",
|
|
" rnn = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\")(inp)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1978b6a1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Predicting New Points With A Trained Recurrent Neural Network"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "d4ad417a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def test_rnn (x1, y_test, plot_min, plot_max):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" x1 (a list or numpy array): The complete x component of the data set\n",
|
|
" y_test (a list or numpy array): The complete y component of the data set\n",
|
|
" plot_min (an int or float): the smallest x value used in the training data\n",
|
|
" plot_max (an int or float): the largest x valye used in the training data\n",
|
|
" Returns:\n",
|
|
" None.\n",
|
|
" Uses a trained recurrent neural network model to predict future points in the \n",
|
|
" series. Computes the MSE of the predicted data set from the true data set, saves\n",
|
|
" the predicted data set to a csv file, and plots the predicted and true data sets w\n",
|
|
" while also displaying the data range used for training.\n",
|
|
" \"\"\"\n",
|
|
" # Add the training data as the first dim points in the predicted data array as these\n",
|
|
" # are known values.\n",
|
|
" y_pred = y_test[:dim].tolist()\n",
|
|
" # Generate the first input to the trained recurrent neural network using the last two \n",
|
|
" # points of the training data. Based on how the network was trained this means that it\n",
|
|
" # will predict the first point in the data set after the training data. All of the \n",
|
|
" # brackets are necessary for Tensorflow.\n",
|
|
" next_input = np.array([[[y_test[dim-2]], [y_test[dim-1]]]])\n",
|
|
" # Save the very last point in the training data set. This will be used later.\n",
|
|
" last = [y_test[dim-1]]\n",
|
|
"\n",
|
|
" # Iterate until the complete data set is created.\n",
|
|
" for i in range (dim, len(y_test)):\n",
|
|
" # Predict the next point in the data set using the previous two points.\n",
|
|
" next = model.predict(next_input)\n",
|
|
" # Append just the number of the predicted data set\n",
|
|
" y_pred.append(next[0][0])\n",
|
|
" # Create the input that will be used to predict the next data point in the data set.\n",
|
|
" next_input = np.array([[last, next[0]]], dtype=np.float64)\n",
|
|
" last = next\n",
|
|
"\n",
|
|
" # Print the mean squared error between the known data set and the predicted data set.\n",
|
|
" print('MSE: ', np.square(np.subtract(y_test, y_pred)).mean())\n",
|
|
" # Save the predicted data set as a csv file for later use\n",
|
|
" name = datatype + 'Predicted'+str(dim)+'.csv'\n",
|
|
" np.savetxt(name, y_pred, delimiter=',')\n",
|
|
" # Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
" # for the training data.\n",
|
|
" fig, ax = plt.subplots()\n",
|
|
" ax.plot(x1, y_test, label=\"true\", linewidth=3)\n",
|
|
" ax.plot(x1, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
" ax.legend()\n",
|
|
" # Created a red region to represent the points used in the training data.\n",
|
|
" ax.axvspan(plot_min, plot_max, alpha=0.25, color='red')\n",
|
|
" plt.show()\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn(length_of_sequences = rnn_input.shape[1])\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e2cad4fc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Changing the size of the recurrent neural network and its parameters\n",
|
|
"can drastically change the results you get from the model. The below\n",
|
|
"code takes the simple recurrent neural network from above and adds a\n",
|
|
"second hidden layer, changes the number of neurons in the hidden\n",
|
|
"layer, and explicitly declares the activation function of the hidden\n",
|
|
"layers to be a sigmoid function. The loss function and optimizer can\n",
|
|
"also be changed but are kept the same as the above network. These\n",
|
|
"parameters can be tuned to provide the optimal result from the\n",
|
|
"network. For some ideas on how to improve the performance of a\n",
|
|
"[recurrent neural network](https://danijar.com/tips-for-training-recurrent-neural-networks)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c39f1516",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def rnn_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons in the input and output layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" # Number of neurons in the hidden layer, increased from the first network\n",
|
|
" hidden_neurons = 500\n",
|
|
" # Define the input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Create two hidden layers instead of one hidden layer. Explicitly set the activation\n",
|
|
" # function to be the sigmoid function (the default value is hyperbolic tangent)\n",
|
|
" rnn1 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=True, # This needs to be True if another hidden layer is to follow\n",
|
|
" stateful = stateful, activation = 'sigmoid',\n",
|
|
" name=\"RNN1\")(inp)\n",
|
|
" rnn2 = SimpleRNN(hidden_neurons, \n",
|
|
" return_sequences=False, activation = 'sigmoid',\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN2\")(rnn1)\n",
|
|
" # Define the output layer as a dense neural network layer (standard neural network layer)\n",
|
|
" #and add it to the network immediately after the hidden layer.\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn2)\n",
|
|
" # Create the machine learning model starting with the input layer and ending with the \n",
|
|
" # output layer\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the machine learning model using the mean squared error function as the loss \n",
|
|
" # function and an Adams optimizer.\n",
|
|
" model.compile(loss=\"mean_squared_error\", optimizer=\"adam\") \n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"model = rnn_2layers(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "842c7602",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Other Types of Recurrent Neural Networks\n",
|
|
"\n",
|
|
"Besides a simple recurrent neural network layer, there are two other\n",
|
|
"commonly used types of recurrent neural network layers: Long Short\n",
|
|
"Term Memory (LSTM) and Gated Recurrent Unit (GRU). For a short\n",
|
|
"introduction to these layers see <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>\n",
|
|
"and <https://medium.com/mindboard/lstm-vs-gru-experimental-comparison-955820c21e8b>.\n",
|
|
"\n",
|
|
"The first network created below is similar to the previous network,\n",
|
|
"but it replaces the SimpleRNN layers with LSTM layers. The second\n",
|
|
"network below has two hidden layers made up of GRUs, which are\n",
|
|
"preceeded by two dense (feeddorward) neural network layers. These\n",
|
|
"dense layers \"preprocess\" the data before it reaches the recurrent\n",
|
|
"layers. This architecture has been shown to improve the performance\n",
|
|
"of recurrent neural networks (see the link above and also\n",
|
|
"<https://arxiv.org/pdf/1807.02857.pdf>."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "6f0e9b62",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def lstm_2layers(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with two LSTM hidden layers and returns the model.\n",
|
|
" \"\"\"\n",
|
|
" # Number of neurons on the input/output layer and the number of neurons in the hidden layer\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input Layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden layers (in this case they are LSTM layers instead if SimpleRNN layers)\n",
|
|
" rnn= LSTM(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True, activation='tanh')(inp)\n",
|
|
" rnn1 = LSTM(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True, activation='tanh')(rnn)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn1)\n",
|
|
" # Define the midel\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the model\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"def dnn2_gru2(length_of_sequences, batch_size = None, stateful = False):\n",
|
|
" \"\"\"\n",
|
|
" Inputs:\n",
|
|
" length_of_sequences (an int): the number of y values in \"x data\". This is determined\n",
|
|
" when the data is formatted\n",
|
|
" batch_size (an int): Default value is None. See Keras documentation of SimpleRNN.\n",
|
|
" stateful (a boolean): Default value is False. See Keras documentation of SimpleRNN.\n",
|
|
" Returns:\n",
|
|
" model (a Keras model): The recurrent neural network that is built and compiled by this\n",
|
|
" method\n",
|
|
" Builds and compiles a recurrent neural network with four hidden layers (two dense followed by\n",
|
|
" two GRU layers) and returns the model.\n",
|
|
" \"\"\" \n",
|
|
" # Number of neurons on the input/output layers and hidden layers\n",
|
|
" in_out_neurons = 1\n",
|
|
" hidden_neurons = 250\n",
|
|
" # Input layer\n",
|
|
" inp = Input(batch_shape=(batch_size, \n",
|
|
" length_of_sequences, \n",
|
|
" in_out_neurons)) \n",
|
|
" # Hidden Dense (feedforward) layers\n",
|
|
" dnn = Dense(hidden_neurons/2, activation='relu', name='dnn')(inp)\n",
|
|
" dnn1 = Dense(hidden_neurons/2, activation='relu', name='dnn1')(dnn)\n",
|
|
" # Hidden GRU layers\n",
|
|
" rnn1 = GRU(hidden_neurons, \n",
|
|
" return_sequences=True,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN1\", use_bias=True)(dnn1)\n",
|
|
" rnn = GRU(hidden_neurons, \n",
|
|
" return_sequences=False,\n",
|
|
" stateful = stateful,\n",
|
|
" name=\"RNN\", use_bias=True)(rnn1)\n",
|
|
" # Output layer\n",
|
|
" dens = Dense(in_out_neurons,name=\"dense\")(rnn)\n",
|
|
" # Define the model\n",
|
|
" model = Model(inputs=[inp],outputs=[dens])\n",
|
|
" # Compile the mdoel\n",
|
|
" model.compile(loss='mean_squared_error', optimizer='adam') \n",
|
|
" # Return the model\n",
|
|
" return model\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"\n",
|
|
"# Generate the training data for the RNN, using a sequence of 2\n",
|
|
"rnn_input, rnn_training = format_data(y_train, 2)\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Change the method name to reflect which network you want to use\n",
|
|
"model = dnn2_gru2(length_of_sequences = 2)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(rnn_input, rnn_training, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict more points of the data set\n",
|
|
"test_rnn(X_tot, y_tot, X_tot[0], X_tot[dim-1])\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)\n",
|
|
"\n",
|
|
"\n",
|
|
"# ### Training Recurrent Neural Networks in the Standard Way (i.e. learning the relationship between the X and Y data)\n",
|
|
"# \n",
|
|
"# Finally, comparing the performace of a recurrent neural network using the standard data formatting to the performance of the network with time sequence data formatting shows the benefit of this type of data formatting with extrapolation.\n",
|
|
"\n",
|
|
"# Check to make sure the data set is complete\n",
|
|
"assert len(X_tot) == len(y_tot)\n",
|
|
"\n",
|
|
"# This is the number of points that will be used in as the training data\n",
|
|
"dim=12\n",
|
|
"\n",
|
|
"# Separate the training data from the whole data set\n",
|
|
"X_train = X_tot[:dim]\n",
|
|
"y_train = y_tot[:dim]\n",
|
|
"\n",
|
|
"# Reshape the data for Keras specifications\n",
|
|
"X_train = X_train.reshape((dim, 1))\n",
|
|
"y_train = y_train.reshape((dim, 1))\n",
|
|
"\n",
|
|
"\n",
|
|
"# Create a recurrent neural network in Keras and produce a summary of the \n",
|
|
"# machine learning model\n",
|
|
"# Set the sequence length to 1 for regular data formatting \n",
|
|
"model = rnn(length_of_sequences = 1)\n",
|
|
"model.summary()\n",
|
|
"\n",
|
|
"# Start the timer. Want to time training+testing\n",
|
|
"start = timer()\n",
|
|
"# Fit the model using the training data genenerated above using 150 training iterations and a 5%\n",
|
|
"# validation split. Setting verbose to True prints information about each training iteration.\n",
|
|
"hist = model.fit(X_train, y_train, batch_size=None, epochs=150, \n",
|
|
" verbose=True,validation_split=0.05)\n",
|
|
"\n",
|
|
"\n",
|
|
"# This section plots the training loss and the validation loss as a function of training iteration.\n",
|
|
"# This is not required for analyzing the couple cluster data but can help determine if the network is\n",
|
|
"# being overtrained.\n",
|
|
"for label in [\"loss\",\"val_loss\"]:\n",
|
|
" plt.plot(hist.history[label],label=label)\n",
|
|
"\n",
|
|
"plt.ylabel(\"loss\")\n",
|
|
"plt.xlabel(\"epoch\")\n",
|
|
"plt.title(\"The final validation loss: {}\".format(hist.history[\"val_loss\"][-1]))\n",
|
|
"plt.legend()\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Use the trained neural network to predict the remaining data points\n",
|
|
"X_pred = X_tot[dim:]\n",
|
|
"X_pred = X_pred.reshape((len(X_pred), 1))\n",
|
|
"y_model = model.predict(X_pred)\n",
|
|
"y_pred = np.concatenate((y_tot[:dim], y_model.flatten()))\n",
|
|
"\n",
|
|
"# Plot the known data set and the predicted data set. The red box represents the region that was used\n",
|
|
"# for the training data.\n",
|
|
"fig, ax = plt.subplots()\n",
|
|
"ax.plot(X_tot, y_tot, label=\"true\", linewidth=3)\n",
|
|
"ax.plot(X_tot, y_pred, 'g-.',label=\"predicted\", linewidth=4)\n",
|
|
"ax.legend()\n",
|
|
"# Created a red region to represent the points used in the training data.\n",
|
|
"ax.axvspan(X_tot[0], X_tot[dim], alpha=0.25, color='red')\n",
|
|
"plt.show()\n",
|
|
"\n",
|
|
"# Stop the timer and calculate the total time needed.\n",
|
|
"end = timer()\n",
|
|
"print('Time: ', end-start)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0752ba7f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"# Generative Models\n",
|
|
"\n",
|
|
"**Generative models** describe a class of statistical models that are a contrast\n",
|
|
"to **discriminative models**. Informally we say that generative models can\n",
|
|
"generate new data instances while discriminative models discriminate between\n",
|
|
"different kinds of data instances. A generative model could generate new photos\n",
|
|
"of animals that look like 'real' animals while a discriminative model could tell\n",
|
|
"a dog from a cat. More formally, given a data set $x$ and a set of labels /\n",
|
|
"targets $y$. Generative models capture the joint probability $p(x, y)$, or\n",
|
|
"just $p(x)$ if there are no labels, while discriminative models capture the\n",
|
|
"conditional probability $p(y | x)$. Discriminative models generally try to draw\n",
|
|
"boundaries in the data space (often high dimensional), while generative models\n",
|
|
"try to model how data is placed throughout the space.\n",
|
|
"\n",
|
|
"**Note**: this material is thanks to Linus Ekstrøm."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "784138f8",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Generative Adversarial Networks\n",
|
|
"\n",
|
|
"**Generative Adversarial Networks** are a type of unsupervised machine learning\n",
|
|
"algorithm proposed by [Goodfellow et. al](https://arxiv.org/pdf/1406.2661.pdf)\n",
|
|
"in 2014 (short and good article).\n",
|
|
"\n",
|
|
"The simplest formulation of\n",
|
|
"the model is based on a game theoretic approach, *zero sum game*, where we pit\n",
|
|
"two neural networks against one another. We define two rival networks, one\n",
|
|
"generator $g$, and one discriminator $d$. The generator directly produces\n",
|
|
"samples"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a42f89ee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto1\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" x = g(z; \\theta^{(g)})\n",
|
|
"\\label{_auto1} \\tag{1}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "abe7212f",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The discriminator attempts to distinguish between samples drawn from the\n",
|
|
"training data and samples drawn from the generator. In other words, it tries to\n",
|
|
"tell the difference between the fake data produced by $g$ and the actual data\n",
|
|
"samples we want to do prediction on. The discriminator outputs a probability\n",
|
|
"value given by"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0821eaee",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto2\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" d(x; \\theta^{(d)})\n",
|
|
"\\label{_auto2} \\tag{2}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "55e0ccf6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"indicating the probability that $x$ is a real training example rather than a\n",
|
|
"fake sample the generator has generated. The simplest way to formulate the\n",
|
|
"learning process in a generative adversarial network is a zero-sum game, in\n",
|
|
"which a function"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f37ece14",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto3\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto3} \\tag{3}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d6d6d5fa",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"determines the reward for the discriminator, while the generator gets the\n",
|
|
"conjugate reward"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3c74b45c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto4\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" -v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto4} \\tag{4}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "605ac8c9",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"During learning both of the networks maximize their own reward function, so that\n",
|
|
"the generator gets better and better at tricking the discriminator, while the\n",
|
|
"discriminator gets better and better at telling the difference between the fake\n",
|
|
"and real data. The generator and discriminator alternate on which one trains at\n",
|
|
"one time (i.e. for one epoch). In other words, we keep the generator constant\n",
|
|
"and train the discriminator, then we keep the discriminator constant to train\n",
|
|
"the generator and repeat. It is this back and forth dynamic which lets GANs\n",
|
|
"tackle otherwise intractable generative problems. As the generator improves with\n",
|
|
" training, the discriminator's performance gets worse because it cannot easily\n",
|
|
" tell the difference between real and fake. If the generator ends up succeeding\n",
|
|
" perfectly, the the discriminator will do no better than random guessing i.e.\n",
|
|
" 50\\%. This progression in the training poses a problem for the convergence\n",
|
|
" criteria for GANs. The discriminator feedback gets less meaningful over time,\n",
|
|
" if we continue training after this point then the generator is effectively\n",
|
|
" training on junk data which can undo the learning up to that point. Therefore,\n",
|
|
" we stop training when the discriminator starts outputting $1/2$ everywhere.\n",
|
|
"\n",
|
|
"At convergence we have"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfec7462",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto5\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" g^* = \\underset{g}{\\mathrm{argmin}}\\hspace{2pt}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto5} \\tag{5}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37c65d4c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The default choice for $v$ is"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c868a092",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto6\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" v(\\theta^{(g)}, \\theta^{(d)}) = \\mathbb{E}_{x\\sim p_\\mathrm{data}}\\log d(x)\n",
|
|
" + \\mathbb{E}_{x\\sim p_\\mathrm{model}}\n",
|
|
" \\log (1 - d(x))\n",
|
|
"\\label{_auto6} \\tag{6}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ad465af3",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The main motivation for the design of GANs is that the learning process requires\n",
|
|
"neither approximate inference (variational autoencoders for example) nor\n",
|
|
"approximation of a partition function. In the case where"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "27858a4e",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"<!-- Equation labels as ordinary links -->\n",
|
|
"<div id=\"_auto7\"></div>\n",
|
|
"\n",
|
|
"$$\n",
|
|
"\\begin{equation}\n",
|
|
" \\underset{d}{\\mathrm{max}}v(\\theta^{(g)}, \\theta^{(d)})\n",
|
|
"\\label{_auto7} \\tag{7}\n",
|
|
"\\end{equation}\n",
|
|
"$$"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "86006023",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"is convex in $\\theta^{(g)} then the procedure is guaranteed to converge and is\n",
|
|
"asymptotically consistent\n",
|
|
"( [Seth Lloyd on QuGANs](https://arxiv.org/pdf/1804.09139.pdf) ).\n",
|
|
"\n",
|
|
"This is in\n",
|
|
"general not the case and it is possible to get situations where the training\n",
|
|
"process never converges because the generator and discriminator chase one\n",
|
|
"another around in the parameter space indefinitely. A much deeper discussion on\n",
|
|
"the currently open research problem of GAN convergence is available\n",
|
|
"[here](https://www.deeplearningbook.org/contents/generative_models.html). To\n",
|
|
"anyone interested in learning more about GANs it is a highly recommended read.\n",
|
|
"Direct quote: \"In this best-performing formulation, the generator aims to\n",
|
|
"increase the log probability that the discriminator makes a mistake, rather than\n",
|
|
"aiming to decrease the log probability that the discriminator makes the correct\n",
|
|
"prediction.\" [Another interesting read](https://arxiv.org/abs/1701.00160)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2fee38bd",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"## Writing Our First Generative Adversarial Network\n",
|
|
"Let us now move on to actually implementing a GAN in tensorflow. We will study\n",
|
|
"the performance of our GAN on the MNIST dataset. This code is based on and\n",
|
|
"adapted from the\n",
|
|
"[google tutorial](https://www.tensorflow.org/tutorials/generative/dcgan)\n",
|
|
"\n",
|
|
"First we import our libraries"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "004a0b53",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import time\n",
|
|
"import numpy as np\n",
|
|
"import tensorflow as tf\n",
|
|
"import matplotlib.pyplot as plt\n",
|
|
"from tensorflow.keras import layers\n",
|
|
"from tensorflow.keras.utils import plot_model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af161",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define our hyperparameters and import our data the usual way"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "8cbaf16a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"BUFFER_SIZE = 60000\n",
|
|
"BATCH_SIZE = 256\n",
|
|
"EPOCHS = 30\n",
|
|
"\n",
|
|
"data = tf.keras.datasets.mnist.load_data()\n",
|
|
"(train_images, train_labels), (test_images, test_labels) = data\n",
|
|
"train_images = np.reshape(train_images, (train_images.shape[0],\n",
|
|
" 28,\n",
|
|
" 28,\n",
|
|
" 1)).astype('float32')\n",
|
|
"\n",
|
|
"# we normalize between -1 and 1\n",
|
|
"train_images = (train_images - 127.5) / 127.5\n",
|
|
"training_dataset = tf.data.Dataset.from_tensor_slices(\n",
|
|
" train_images).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "822b8cc7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"### MNIST and GANs\n",
|
|
"\n",
|
|
"Let's have a quick look"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "52b5965c",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plt.imshow(train_images[0], cmap='Greys')\n",
|
|
"plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "21c5199c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our two models. This is where the 'magic' happens. There are a\n",
|
|
"huge amount of possible formulations for both models. A lot of engineering and\n",
|
|
"trial and error can be done here to try to produce better performing models. For\n",
|
|
"more advanced GANs this is by far the step where you can 'make or break' a\n",
|
|
"model.\n",
|
|
"\n",
|
|
"We start with the generator. As stated in the introductory text the generator\n",
|
|
"$g$ upsamples from a random sample to the shape of what we want to predict. In\n",
|
|
"our case we are trying to predict MNIST images ($28\\times 28$ pixels)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "356759c7",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_model():\n",
|
|
" \"\"\"\n",
|
|
" The generator uses upsampling layers tf.keras.layers.Conv2DTranspose() to\n",
|
|
" produce an image from a random seed. We start with a Dense layer taking this\n",
|
|
" random sample as an input and subsequently upsample through multiple\n",
|
|
" convolutional layers.\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
"\n",
|
|
"\n",
|
|
" # adding our input layer. Dense means that every neuron is connected and\n",
|
|
" # the input shape is the shape of our random noise. The units need to match\n",
|
|
" # in some sense the upsampling strides to reach our desired output shape.\n",
|
|
" # we are using 100 random numbers as our seed\n",
|
|
" model.add(layers.Dense(units=7*7*BATCH_SIZE,\n",
|
|
" use_bias=False,\n",
|
|
" input_shape=(100, )))\n",
|
|
" # we normalize the output form the Dense layer\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" # and add an activation function to our 'layer'. LeakyReLU avoids vanishing\n",
|
|
" # gradient problem\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" model.add(layers.Reshape((7, 7, BATCH_SIZE)))\n",
|
|
" assert model.output_shape == (None, 7, 7, BATCH_SIZE)\n",
|
|
" # even though we just added four keras layers we think of everything above\n",
|
|
" # as 'one' layer\n",
|
|
"\n",
|
|
" # next we add our upscaling convolutional layers\n",
|
|
" model.add(layers.Conv2DTranspose(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(1, 1),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 7, 7, 128)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False))\n",
|
|
" model.add(layers.BatchNormalization())\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" assert model.output_shape == (None, 14, 14, 64)\n",
|
|
"\n",
|
|
" model.add(layers.Conv2DTranspose(filters=1,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" use_bias=False,\n",
|
|
" activation='tanh'))\n",
|
|
" assert model.output_shape == (None, 28, 28, 1)\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "854bcd6b",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"And there we have our 'simple' generator model. Now we move on to defining our\n",
|
|
"discriminator model $d$, which is a convolutional neural network based image\n",
|
|
"classifier."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "41473304",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_model():\n",
|
|
" \"\"\"\n",
|
|
" The discriminator is a convolutional neural network based image classifier\n",
|
|
" \"\"\"\n",
|
|
"\n",
|
|
" # we define our model\n",
|
|
" model = tf.keras.Sequential()\n",
|
|
" model.add(layers.Conv2D(filters=64,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same',\n",
|
|
" input_shape=[28, 28, 1]))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
"\n",
|
|
" model.add(layers.Conv2D(filters=128,\n",
|
|
" kernel_size=(5, 5),\n",
|
|
" strides=(2, 2),\n",
|
|
" padding='same'))\n",
|
|
" model.add(layers.LeakyReLU())\n",
|
|
" # adding a dropout layer as you do in conv-nets\n",
|
|
" model.add(layers.Dropout(0.3))\n",
|
|
"\n",
|
|
" model.add(layers.Flatten())\n",
|
|
" model.add(layers.Dense(1))\n",
|
|
"\n",
|
|
" return model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "353af567",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Let us take a look at our models."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "f899d4e3",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generator = generator_model()\n",
|
|
"plot_model(generator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "87ef384b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"discriminator = discriminator_model()\n",
|
|
"plot_model(discriminator, show_shapes=True, rankdir='LR')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b2bef82d",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we need a few helper objects we will use in training"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "e397847a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"cross_entropy = tf.keras.losses.BinaryCrossentropy(from_logits=True)\n",
|
|
"generator_optimizer = tf.keras.optimizers.Adam(1e-4)\n",
|
|
"discriminator_optimizer = tf.keras.optimizers.Adam(1e-4)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "db3396cc",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"The first object, *cross_entropy* is our loss function and the two others are\n",
|
|
"our optimizers. Notice we use the same learning rate for both $g$ and $d$. This\n",
|
|
"is because they need to improve their accuracy at approximately equal speeds to\n",
|
|
"get convergence (not necessarily exactly equal). Now we define our loss\n",
|
|
"functions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "931eaced",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generator_loss(fake_output):\n",
|
|
" loss = cross_entropy(tf.ones_like(fake_output), fake_output)\n",
|
|
"\n",
|
|
" return loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "0c4a44bb",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def discriminator_loss(real_output, fake_output):\n",
|
|
" real_loss = cross_entropy(tf.ones_like(real_output), real_output)\n",
|
|
" fake_loss = cross_entropy(tf.zeros_liks(fake_output), fake_output)\n",
|
|
" total_loss = real_loss + fake_loss\n",
|
|
"\n",
|
|
" return total_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fcf8f066",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a kind of seed to help us compare the learning process over\n",
|
|
"multiple training epochs."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "eea2bbee",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"noise_dimension = 100\n",
|
|
"n_examples_to_generate = 16\n",
|
|
"seed_images = tf.random.normal([n_examples_to_generate, noise_dimension])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94a6e341",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have everything we need to define our training step, which we will apply\n",
|
|
"for every step in our training loop. Notice the @tf.function flag signifying\n",
|
|
"that the function is tensorflow 'compiled'. Removing this flag doubles the\n",
|
|
"computation time."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "8d48470b",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"@tf.function\n",
|
|
"def train_step(images):\n",
|
|
" noise = tf.random.normal([BATCH_SIZE, noise_dimension])\n",
|
|
"\n",
|
|
" with tf.GradientTape() as gen_tape, tf.GradientTape() as disc_tape:\n",
|
|
" generated_images = generator(noise, training=True)\n",
|
|
"\n",
|
|
" real_output = discriminator(images, training=True)\n",
|
|
" fake_output = discriminator(generated_images, training=True)\n",
|
|
"\n",
|
|
" gen_loss = generator_loss(fake_output)\n",
|
|
" disc_loss = discriminator_loss(real_output, fake_output)\n",
|
|
"\n",
|
|
" gradients_of_generator = gen_tape.gradient(gen_loss,\n",
|
|
" generator.trainable_variables)\n",
|
|
" gradients_of_discriminator = disc_tape.gradient(disc_loss,\n",
|
|
" discriminator.trainable_variables)\n",
|
|
" generator_optimizer.apply_gradients(zip(gradients_of_generator,\n",
|
|
" generator.trainable_variables))\n",
|
|
" discriminator_optimizer.apply_gradients(zip(gradients_of_discriminator,\n",
|
|
" discriminator.trainable_variables))\n",
|
|
"\n",
|
|
" return gen_loss, disc_loss"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "62015b88",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Next we define a helper function to produce an output over our training epochs\n",
|
|
"to see the predictive progression of our generator model. **Note**: I am including\n",
|
|
"this code here, but comment it out in the training loop."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "b189ed96",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_and_save_images(model, epoch, test_input):\n",
|
|
" # we're making inferences here\n",
|
|
" predictions = model(test_input, training=False)\n",
|
|
"\n",
|
|
" fig = plt.figure(figsize=(4, 4))\n",
|
|
"\n",
|
|
" for i in range(predictions.shape[0]):\n",
|
|
" plt.subplot(4, 4, i+1)\n",
|
|
" plt.imshow(predictions[i, :, :, 0] * 127.5 + 127.5, cmap='gray')\n",
|
|
" plt.axis('off')\n",
|
|
"\n",
|
|
" plt.savefig(f'./images_from_seed_images/image_at_epoch_{str(epoch).zfill(3)}.png')\n",
|
|
" plt.close()\n",
|
|
" #plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ba2c82d5",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Setting up checkpoints to periodically save our model during training so that\n",
|
|
"everything is not lost even if the program were to somehow terminate while\n",
|
|
"training."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "a0e2fc8a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Setting up checkpoints to save model during training\n",
|
|
"checkpoint_dir = './training_checkpoints'\n",
|
|
"checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt')\n",
|
|
"checkpoint = tf.train.Checkpoint(generator_optimizer=generator_optimizer,\n",
|
|
" discriminator_optimizer=discriminator_optimizer,\n",
|
|
" generator=generator,\n",
|
|
" discriminator=discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d93a0f7",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we define our training loop"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "a1275556",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def train(dataset, epochs):\n",
|
|
" generator_loss_list = []\n",
|
|
" discriminator_loss_list = []\n",
|
|
"\n",
|
|
" for epoch in range(epochs):\n",
|
|
" start = time.time()\n",
|
|
"\n",
|
|
" for image_batch in dataset:\n",
|
|
" gen_loss, disc_loss = train_step(image_batch)\n",
|
|
" generator_loss_list.append(gen_loss.numpy())\n",
|
|
" discriminator_loss_list.append(disc_loss.numpy())\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epoch + 1, seed_images)\n",
|
|
"\n",
|
|
" if (epoch + 1) % 15 == 0:\n",
|
|
" checkpoint.save(file_prefix=checkpoint_prefix)\n",
|
|
"\n",
|
|
" print(f'Time for epoch {epoch} is {time.time() - start}')\n",
|
|
"\n",
|
|
" #generate_and_save_images(generator, epochs, seed_images)\n",
|
|
"\n",
|
|
" loss_file = './data/lossfile.txt'\n",
|
|
" with open(loss_file, 'w') as outfile:\n",
|
|
" outfile.write(str(generator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write(str(discriminator_loss_list))\n",
|
|
" outfile.write('\\n')\n",
|
|
" outfile.write('\\n')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6ff3a75a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"To train simply call this function. **Warning**: this might take a long time so\n",
|
|
"there is a folder of a pretrained network already included in the repository."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"id": "371ed41a",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"train(train_dataset, EPOCHS)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "654399f1",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now to avoid having to train and everything, which will take a while depending\n",
|
|
"on your computer setup we now load in the model which produced the above gif."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"id": "dec4b560",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))\n",
|
|
"restored_generator = checkpoint.generator\n",
|
|
"restored_discriminator = checkpoint.discriminator\n",
|
|
"\n",
|
|
"print(restored_generator)\n",
|
|
"print(restored_discriminator)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "296bfa5c",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We have successfully loaded in our latest model. Let us now play around a bit\n",
|
|
"and see what kind of things we can learn about this model. Our generator takes\n",
|
|
"an array of 100 numbers. One idea can be to try to systematically change our\n",
|
|
"input. Let us try and see what we get"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"id": "eecfbb1f",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def generate_latent_points(number=100, scale_means=1, scale_stds=1):\n",
|
|
" latent_dim = 100\n",
|
|
" means = scale_means * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" stds = scale_stds * tf.linspace(-1, 1, num=latent_dim)\n",
|
|
" latent_space_value_range = tf.random.normal([number, latent_dim],\n",
|
|
" means,\n",
|
|
" stds,\n",
|
|
" dtype=tf.float64)\n",
|
|
"\n",
|
|
" return latent_space_value_range\n",
|
|
"\n",
|
|
"def generate_images(latent_points):\n",
|
|
" # notice we set training to false because we are making inferences\n",
|
|
" generated_images = restored_generator.predict(latent_points)\n",
|
|
"\n",
|
|
" return generated_images"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"id": "333a593d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def plot_result(generated_images, number=100):\n",
|
|
" # obviously this assumes sqrt number is an int\n",
|
|
" fig, axs = plt.subplots(int(np.sqrt(number)), int(np.sqrt(number)),\n",
|
|
" figsize=(10, 10))\n",
|
|
"\n",
|
|
" for i in range(int(np.sqrt(number))):\n",
|
|
" for j in range(int(np.sqrt(number))):\n",
|
|
" axs[i, j].imshow(generated_images[i*j], cmap='Greys')\n",
|
|
" axs[i, j].axis('off')\n",
|
|
"\n",
|
|
" plt.show()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"id": "2f5f0154",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"generated_images = generate_images(generate_latent_points())\n",
|
|
"plot_result(generated_images)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff581bf2",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"We see that the generator generates images that look like MNIST\n",
|
|
"numbers: $1, 4, 7, 9$. Let's try to tweak it a bit more to see if we are able\n",
|
|
"to generate a similar plot where we generate every MNIST number. Let us now try\n",
|
|
"to 'move' a bit around in the latent space. **Note**: decrease the plot number if\n",
|
|
"these following cells take too long to run on your computer."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"id": "d3617ad8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 225\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=-5,\n",
|
|
" scale_stds=1))\n",
|
|
"plot_result(generated_images, number=plot_number)\n",
|
|
"\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=5))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a074f93",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Again, we have found something interesting. *Moving* around using our means\n",
|
|
"takes us from digit to digit, while *moving* around using our standard\n",
|
|
"deviations seem to increase the number of different digits! In the last image\n",
|
|
"above, we can barely make out every MNIST digit. Let us make on last plot using\n",
|
|
"this information by upping the standard deviation of our Gaussian noises."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"id": "4ef8937d",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 400\n",
|
|
"generated_images = generate_images(generate_latent_points(number=plot_number,\n",
|
|
" scale_means=1,\n",
|
|
" scale_stds=10))\n",
|
|
"plot_result(generated_images, number=plot_number)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "385a2d0a",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"A pretty cool result! We see that our generator indeed has learned a\n",
|
|
"distribution which qualitatively looks a whole lot like the MNIST dataset.\n",
|
|
"\n",
|
|
"Another interesting way to explore the latent space of our generator model is by\n",
|
|
"interpolating between the MNIST digits. This section is largely based on\n",
|
|
"[this excellent blogpost](https://machinelearningmastery.com/how-to-interpolate-and-perform-vector-arithmetic-with-faces-using-a-generative-adversarial-network/)\n",
|
|
"by Jason Brownlee.\n",
|
|
"\n",
|
|
"So let us start by defining a function to interpolate between two points in the\n",
|
|
"latent space."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"id": "57de87b8",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"def interpolation(point_1, point_2, n_steps=10):\n",
|
|
" ratios = np.linspace(0, 1, num=n_steps)\n",
|
|
" vectors = []\n",
|
|
" for i, ratio in enumerate(ratios):\n",
|
|
" vectors.append(((1.0 - ratio) * point_1 + ratio * point_2))\n",
|
|
"\n",
|
|
" return tf.stack(vectors)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cfb76bb6",
|
|
"metadata": {
|
|
"editable": true
|
|
},
|
|
"source": [
|
|
"Now we have all we need to do our interpolation analysis."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"id": "e25decef",
|
|
"metadata": {
|
|
"collapsed": false,
|
|
"editable": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"plot_number = 100\n",
|
|
"latent_points = generate_latent_points(number=plot_number)\n",
|
|
"results = None\n",
|
|
"for i in range(0, 2*np.sqrt(plot_number), 2):\n",
|
|
" interpolated = interpolation(latent_points[i], latent_points[i+1])\n",
|
|
" generated_images = generate_images(interpolated)\n",
|
|
"\n",
|
|
" if results is None:\n",
|
|
" results = generated_images\n",
|
|
" else:\n",
|
|
" results = tf.stack((results, generated_images))\n",
|
|
"\n",
|
|
"plot_results(results, plot_number)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.15"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
} |