diff --git a/doc/LectureNotes/_build/.doctrees/environment.pickle b/doc/LectureNotes/_build/.doctrees/environment.pickle index 53c6b7338..5a07beba8 100644 Binary files a/doc/LectureNotes/_build/.doctrees/environment.pickle and b/doc/LectureNotes/_build/.doctrees/environment.pickle differ diff --git a/doc/LectureNotes/_build/.doctrees/exercisesweek43.doctree b/doc/LectureNotes/_build/.doctrees/exercisesweek43.doctree index 38fbd834e..4b48c9592 100644 Binary files a/doc/LectureNotes/_build/.doctrees/exercisesweek43.doctree and b/doc/LectureNotes/_build/.doctrees/exercisesweek43.doctree differ diff --git a/doc/LectureNotes/_build/.doctrees/week43.doctree b/doc/LectureNotes/_build/.doctrees/week43.doctree index 9933a7f90..3fe163257 100644 Binary files a/doc/LectureNotes/_build/.doctrees/week43.doctree and b/doc/LectureNotes/_build/.doctrees/week43.doctree differ diff --git a/doc/LectureNotes/_build/html/_images/exercisesweek43_26_2.png b/doc/LectureNotes/_build/html/_images/exercisesweek43_26_2.png new file mode 100644 index 000000000..d6998a4a8 Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/exercisesweek43_26_2.png differ diff --git a/doc/LectureNotes/_build/html/_images/week43_30_2.png b/doc/LectureNotes/_build/html/_images/week43_30_2.png new file mode 100644 index 000000000..d6998a4a8 Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week43_30_2.png differ diff --git a/doc/LectureNotes/_build/html/_images/week43_72_1.png b/doc/LectureNotes/_build/html/_images/week43_72_1.png new file mode 100644 index 000000000..e70113e56 Binary files /dev/null and b/doc/LectureNotes/_build/html/_images/week43_72_1.png differ diff --git a/doc/LectureNotes/_build/html/_sources/exercisesweek43.ipynb b/doc/LectureNotes/_build/html/_sources/exercisesweek43.ipynb index 438de41c2..63211a3f0 100644 --- a/doc/LectureNotes/_build/html/_sources/exercisesweek43.ipynb +++ b/doc/LectureNotes/_build/html/_sources/exercisesweek43.ipynb @@ -2,10 +2,8 @@ "cells": [ { "cell_type": "markdown", - "id": "b2937d10", - "metadata": { - "editable": true - }, + "id": "ae832182", + "metadata": {}, "source": [ "\n", @@ -14,10 +12,8 @@ }, { "cell_type": "markdown", - "id": "3dd00d19", - "metadata": { - "editable": true - }, + "id": "77f844d9", + "metadata": {}, "source": [ "# Exercises weeks 43 and 44 \n", "**October 23-27, 2023**\n", @@ -29,10 +25,8 @@ }, { "cell_type": "markdown", - "id": "82a19a1d", - "metadata": { - "editable": true - }, + "id": "d5b983e3", + "metadata": {}, "source": [ "# Overarching aims of the exercises weeks 43 and 44\n", "\n", @@ -69,10 +63,8 @@ }, { "cell_type": "markdown", - "id": "f74f69af", - "metadata": { - "editable": true - }, + "id": "7f6cc12f", + "metadata": {}, "source": [ "## The AND and XOR Gates\n", "\n", @@ -107,10 +99,8 @@ }, { "cell_type": "markdown", - "id": "1b52d47a", - "metadata": { - "editable": true - }, + "id": "c0b07e46", + "metadata": {}, "source": [ "## Representing the Data Sets\n", "\n", @@ -119,10 +109,8 @@ }, { "cell_type": "markdown", - "id": "3e6910cb", - "metadata": { - "editable": true - }, + "id": "e0a9840a", + "metadata": {}, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n", @@ -134,10 +122,8 @@ }, { "cell_type": "markdown", - "id": "90a3b78a", - "metadata": { - "editable": true - }, + "id": "ff95d379", + "metadata": {}, "source": [ "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n", "\n", @@ -164,10 +150,159 @@ }, { "cell_type": "markdown", - "id": "d6a3ab1e", - "metadata": { - "editable": true - }, + "id": "36fa3466", + "metadata": {}, + "source": [ + "## Setting up dimensionalities by hand\n", + "\n", + "It can be useful to test the dimensionalities for the network. Let us assume we have performed an optimization for XOR gate and found that the weights for the hidden layer are given by" + ] + }, + { + "cell_type": "markdown", + "id": "e2e5e808", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n", + " 1 & 1 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "419e3904", + "metadata": {}, + "source": [ + "Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives" + ] + }, + { + "cell_type": "markdown", + "id": "88a9712a", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n", + " 1 & 1 \\\\\n", + "\t\t 1 & 1 \\\\\n", + "\t\t 2 & 2 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "036eee14", + "metadata": {}, + "source": [ + "Assume also that the bias vector for the hidden layer is" + ] + }, + { + "cell_type": "markdown", + "id": "3c771918", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n", + " -1\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "f5a4150d", + "metadata": {}, + "source": [ + "Adding it gives us the input to the activation function of the hidden layer" + ] + }, + { + "cell_type": "markdown", + "id": "e4eb7753", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n", + " 1 & 0 \\\\\n", + "\t\t 1 & 0 \\\\\n", + "\t\t 2 & 1 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b5442b32", + "metadata": {}, + "source": [ + "Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have" + ] + }, + { + "cell_type": "markdown", + "id": "64d99f85", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n", + " 1 & 0 \\\\\n", + "\t\t 1 & 0 \\\\\n", + "\t\t 2 & 1 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2aee4b6b", + "metadata": {}, + "source": [ + "Assume also that the bias of the output layer is zero and that the weights of the output layer are" + ] + }, + { + "cell_type": "markdown", + "id": "884541eb", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n", + " -2\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "3999a4d5", + "metadata": {}, + "source": [ + "and multiplying with $\\boldsymbol{a}_h$ gives the output" + ] + }, + { + "cell_type": "markdown", + "id": "a4896b4d", + "metadata": {}, + "source": [ + "$$\n", + "\\boldsymbol{a}_o=\\boldsymbol{w}_h^T\\begin{bmatrix} 0 & 0 \\\\\n", + " 1 & 0 \\\\\n", + "\t\t 1 & 0 \\\\\n", + "\t\t 2 & 1 \\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "2c74e7c0", + "metadata": {}, + "source": [ + "the wanted result." + ] + }, + { + "cell_type": "markdown", + "id": "249b9804", + "metadata": {}, "source": [ "## Setting up the Neural Network\n", "\n", @@ -177,11 +312,8 @@ { "cell_type": "code", "execution_count": 1, - "id": "152123b0", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "afdd3100", + "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", @@ -211,10 +343,6 @@ " probabilities = sigmoid(z_o)\n", " return probabilities\n", "\n", - "# we obtain a prediction by taking the class with the highest likelihood\n", - "def predict(X):\n", - " probabilities = feed_forward(X)\n", - " return np.argmax(probabilities, axis=1)\n", "\n", "# ensure the same random numbers appear every time\n", "np.random.seed(0)\n", @@ -232,7 +360,7 @@ "# Defining the neural network\n", "n_inputs, n_features = X.shape\n", "n_hidden_neurons = 2\n", - "n_categories = 2\n", + "n_categories = 1\n", "n_features = 2\n", "\n", "# we make the weights normally distributed using numpy.random.randn\n", @@ -246,29 +374,21 @@ "output_bias = np.zeros(n_categories) + 0.01\n", "\n", "probabilities = feed_forward(X)\n", - "print(probabilities)\n", - "\n", - "\n", - "predictions = predict(X)\n", - "print(predictions)" + "print(probabilities)" ] }, { "cell_type": "markdown", - "id": "73319f0a", - "metadata": { - "editable": true - }, + "id": "fb383164", + "metadata": {}, "source": [ "Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above." ] }, { "cell_type": "markdown", - "id": "a7e0c47a", - "metadata": { - "editable": true - }, + "id": "0d26394c", + "metadata": {}, "source": [ "## The Code using Scikit-Learn" ] @@ -276,11 +396,8 @@ { "cell_type": "code", "execution_count": 2, - "id": "dbbacc67", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "befc368c", + "metadata": {}, "outputs": [], "source": [ "# import necessary packages\n", @@ -304,10 +421,7 @@ "yAND = np.array( [ 0, 0 ,0, 1])\n", "\n", "# Defining the neural network\n", - "n_inputs, n_features = X.shape\n", "n_hidden_neurons = 2\n", - "n_categories = 2\n", - "n_features = 2\n", "\n", "eta_vals = np.logspace(-5, 1, 7)\n", "lmbd_vals = np.logspace(-5, 1, 7)\n", @@ -344,10 +458,8 @@ }, { "cell_type": "markdown", - "id": "1cac501a", - "metadata": { - "editable": true - }, + "id": "03895a9f", + "metadata": {}, "source": [ "## Building a neural network code\n", "\n", @@ -363,10 +475,8 @@ }, { "cell_type": "markdown", - "id": "dd153528", - "metadata": { - "editable": true - }, + "id": "6226915b", + "metadata": {}, "source": [ "### Learning rate methods\n", "\n", @@ -385,11 +495,8 @@ { "cell_type": "code", "execution_count": 3, - "id": "f55eea63", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "b6d5ea73", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -526,10 +633,8 @@ }, { "cell_type": "markdown", - "id": "1a9bcb3e", - "metadata": { - "editable": true - }, + "id": "edb42bef", + "metadata": {}, "source": [ "### Usage of the above learning rate schedulers\n", "\n", @@ -542,11 +647,8 @@ { "cell_type": "code", "execution_count": 4, - "id": "86013cb4", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "108c1209", + "metadata": {}, "outputs": [], "source": [ "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", @@ -555,10 +657,8 @@ }, { "cell_type": "markdown", - "id": "535331f6", - "metadata": { - "editable": true - }, + "id": "d1c423f0", + "metadata": {}, "source": [ "Here is a small example for how a segment of code using schedulers\n", "could look. Switching out the schedulers is simple." @@ -567,11 +667,8 @@ { "cell_type": "code", "execution_count": 5, - "id": "7e0f6b5a", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "63c8c749", + "metadata": {}, "outputs": [], "source": [ "weights = np.ones((3,3))\n", @@ -589,10 +686,8 @@ }, { "cell_type": "markdown", - "id": "f018ae57", - "metadata": { - "editable": true - }, + "id": "7cb80992", + "metadata": {}, "source": [ "### Cost functions\n", "\n", @@ -605,11 +700,8 @@ { "cell_type": "code", "execution_count": 6, - "id": "c13507bf", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "a181dacb", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -643,10 +735,8 @@ }, { "cell_type": "markdown", - "id": "6dab17bc", - "metadata": { - "editable": true - }, + "id": "26a7fe37", + "metadata": {}, "source": [ "Below we give a short example of how these cost function may be used\n", "to obtain results if you wish to test them out on your own using\n", @@ -656,11 +746,8 @@ { "cell_type": "code", "execution_count": 7, - "id": "a5dbba01", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "5894bd31", + "metadata": {}, "outputs": [], "source": [ "from autograd import grad\n", @@ -677,10 +764,8 @@ }, { "cell_type": "markdown", - "id": "b55d31d4", - "metadata": { - "editable": true - }, + "id": "71141fab", + "metadata": {}, "source": [ "### Activation functions\n", "\n", @@ -693,11 +778,8 @@ { "cell_type": "code", "execution_count": 8, - "id": "b3e045a6", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "433e0886", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -751,10 +833,8 @@ }, { "cell_type": "markdown", - "id": "c0189342", - "metadata": { - "editable": true - }, + "id": "14c9d538", + "metadata": {}, "source": [ "Below follows a short demonstration of how to use an activation\n", "function. The derivative of the activation function will be important\n", @@ -766,11 +846,8 @@ { "cell_type": "code", "execution_count": 9, - "id": "640aa861", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "4c3a44ef", + "metadata": {}, "outputs": [], "source": [ "z = np.array([[4, 5, 6]]).T\n", @@ -787,10 +864,8 @@ }, { "cell_type": "markdown", - "id": "1007ccdd", - "metadata": { - "editable": true - }, + "id": "aa7f604e", + "metadata": {}, "source": [ "### The Neural Network\n", "\n", @@ -811,11 +886,8 @@ { "cell_type": "code", "execution_count": 10, - "id": "9584a2da", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "585a227b", + "metadata": {}, "outputs": [], "source": [ "import math\n", @@ -1283,10 +1355,8 @@ }, { "cell_type": "markdown", - "id": "9ccd1fc1", - "metadata": { - "editable": true - }, + "id": "7de704af", + "metadata": {}, "source": [ "Before we make a model, we will quickly generate a dataset we can use\n", "for our linear regression problem as shown below" @@ -1295,11 +1365,8 @@ { "cell_type": "code", "execution_count": 11, - "id": "7f3a5b31", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "212b0a35", + "metadata": {}, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -1339,10 +1406,8 @@ }, { "cell_type": "markdown", - "id": "1ac05bb6", - "metadata": { - "editable": true - }, + "id": "1ff4ec7b", + "metadata": {}, "source": [ "Now that we have our dataset ready for the regression, we can create\n", "our regressor. Note that with the seed parameter, we can make sure our\n", @@ -1355,11 +1420,8 @@ { "cell_type": "code", "execution_count": 12, - "id": "0f857604", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "57d8eca3", + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -1370,10 +1432,8 @@ }, { "cell_type": "markdown", - "id": "eeff4315", - "metadata": { - "editable": true - }, + "id": "f1977bae", + "metadata": {}, "source": [ "We then fit our model with our training data using the scheduler of our choice." ] @@ -1381,11 +1441,8 @@ { "cell_type": "code", "execution_count": 13, - "id": "46246810", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "31ae2a1e", + "metadata": {}, "outputs": [], "source": [ "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -1396,10 +1453,8 @@ }, { "cell_type": "markdown", - "id": "8c6f9954", - "metadata": { - "editable": true - }, + "id": "96a64da4", + "metadata": {}, "source": [ "Due to the progress bar we can see the MSE (train_error) throughout\n", "the FFNN's training. Note that the fit() function has some optional\n", @@ -1412,11 +1467,8 @@ { "cell_type": "code", "execution_count": 14, - "id": "2661939c", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "246499fc", + "metadata": {}, "outputs": [], "source": [ "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -1426,10 +1478,8 @@ }, { "cell_type": "markdown", - "id": "74c5624c", - "metadata": { - "editable": true - }, + "id": "61d11e17", + "metadata": {}, "source": [ "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", "\n", @@ -1441,11 +1491,8 @@ { "cell_type": "code", "execution_count": 15, - "id": "4b8eb115", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "b5b91899", + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_breast_cancer\n", @@ -1467,11 +1514,8 @@ { "cell_type": "code", "execution_count": 16, - "id": "2c0f92bd", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "4bd01b3d", + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -1482,10 +1526,8 @@ }, { "cell_type": "markdown", - "id": "49201ae4", - "metadata": { - "editable": true - }, + "id": "3a138d66", + "metadata": {}, "source": [ "We will now make use of our validation data by passing it into our fit function as a keyword argument" ] @@ -1493,11 +1535,8 @@ { "cell_type": "code", "execution_count": 17, - "id": "55b5e426", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "faf1534e", + "metadata": {}, "outputs": [], "source": [ "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -1508,10 +1547,8 @@ }, { "cell_type": "markdown", - "id": "b51762fb", - "metadata": { - "editable": true - }, + "id": "33d2d346", + "metadata": {}, "source": [ "Finally, we will create a neural network with 2 hidden layers with activation functions." ] @@ -1519,11 +1556,8 @@ { "cell_type": "code", "execution_count": 18, - "id": "6b59e27d", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "a8f0a61c", + "metadata": {}, "outputs": [], "source": [ "input_nodes = X_train.shape[1]\n", @@ -1539,11 +1573,8 @@ { "cell_type": "code", "execution_count": 19, - "id": "72c87921", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "d651702d", + "metadata": {}, "outputs": [], "source": [ "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", @@ -1554,10 +1585,8 @@ }, { "cell_type": "markdown", - "id": "ed40d7d2", - "metadata": { - "editable": true - }, + "id": "200b64c0", + "metadata": {}, "source": [ "### Multiclass classification\n", "\n", @@ -1569,11 +1598,8 @@ { "cell_type": "code", "execution_count": 20, - "id": "315ef3fe", - "metadata": { - "collapsed": false, - "editable": true - }, + "id": "98b8d59b", + "metadata": {}, "outputs": [], "source": [ "from sklearn.datasets import load_digits\n", @@ -1603,9 +1629,66 @@ "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", "scores = multiclass.fit(X, target, scheduler, epochs=1000)" ] + }, + { + "cell_type": "markdown", + "id": "cadd7e47", + "metadata": {}, + "source": [ + "## Testing the XOR gate and other gates\n", + "\n", + "Let us now use our code to test the XOR gate." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "a40e5cfe", + "metadata": {}, + "outputs": [], + "source": [ + "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", + "\n", + "# The XOR gate\n", + "yXOR = np.array( [[ 0], [1] ,[1], [0]])\n", + "\n", + "input_nodes = X.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)\n", + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "b635db73", + "metadata": {}, + "source": [ + "Not bad, but the results depend strongly on the learning reate. Try different learning rates." + ] } ], - "metadata": {}, + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.10" + } + }, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/LectureNotes/_build/html/_sources/week43.ipynb b/doc/LectureNotes/_build/html/_sources/week43.ipynb index e80f0e0d1..8de3714cd 100644 --- a/doc/LectureNotes/_build/html/_sources/week43.ipynb +++ b/doc/LectureNotes/_build/html/_sources/week43.ipynb @@ -2,8 +2,10 @@ "cells": [ { "cell_type": "markdown", - "id": "944a1ec4", - "metadata": {}, + "id": "8d7cc066", + "metadata": { + "editable": true + }, "source": [ "\n", @@ -12,21 +14,25 @@ }, { "cell_type": "markdown", - "id": "571ef4ca", - "metadata": {}, + "id": "028dc8c6", + "metadata": { + "editable": true + }, "source": [ "# Week 43: Deep Learning: Constructing a Neural Network code and solving differential equations\n", "**Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and Facility for Rare Isotope Beams, Michigan State University\n", "\n", - "Date: **Oct 23, 2023**\n", + "Date: **Oct 25, 2023**\n", "\n", "Copyright 1999-2023, Morten Hjorth-Jensen. Released under CC Attribution-NonCommercial 4.0 license" ] }, { "cell_type": "markdown", - "id": "c53e0e9f", - "metadata": {}, + "id": "15ef2310", + "metadata": { + "editable": true + }, "source": [ "## Plans for week 43\n", "\n", @@ -67,8 +73,10 @@ }, { "cell_type": "markdown", - "id": "02ed9bb6", - "metadata": {}, + "id": "192aa783", + "metadata": { + "editable": true + }, "source": [ "## Using Automatic differentiation\n", "a\n", @@ -79,8 +87,10 @@ }, { "cell_type": "markdown", - "id": "05cfb0c9", - "metadata": {}, + "id": "22874f1a", + "metadata": { + "editable": true + }, "source": [ "## Back propagation and automatic differentiation\n", "\n", @@ -94,16 +104,20 @@ }, { "cell_type": "markdown", - "id": "21caf391", - "metadata": {}, + "id": "29c53621", + "metadata": { + "editable": true + }, "source": [ "## Material for exercises week 43 and week 44" ] }, { "cell_type": "markdown", - "id": "3c9b1e04", - "metadata": {}, + "id": "c3ab8226", + "metadata": { + "editable": true + }, "source": [ "## Writing our first neural network code, testing it for the OR and XOR gates\n", "\n", @@ -137,8 +151,10 @@ }, { "cell_type": "markdown", - "id": "4dabb457", - "metadata": {}, + "id": "d686af76", + "metadata": { + "editable": true + }, "source": [ "## The AND and XOR Gates\n", "\n", @@ -173,8 +189,10 @@ }, { "cell_type": "markdown", - "id": "a78cff46", - "metadata": {}, + "id": "9e6ae224", + "metadata": { + "editable": true + }, "source": [ "## Representing the Data Sets\n", "\n", @@ -183,8 +201,10 @@ }, { "cell_type": "markdown", - "id": "e4c2f9d7", - "metadata": {}, + "id": "ee5e7e34", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{X}=\\begin{bmatrix} 0 & 0 \\\\\n", @@ -196,16 +216,221 @@ }, { "cell_type": "markdown", - "id": "103736e3", - "metadata": {}, + "id": "da275174", + "metadata": { + "editable": true + }, "source": [ - "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate." + "while the vector of outputs is $\\boldsymbol{y}^T=[0,1,1,0]$ for the XOR gate, $\\boldsymbol{y}^T=[0,0,0,1]$ for the AND gate and $\\boldsymbol{y}^T=[0,1,1,1]$ for the OR gate.\n", + "\n", + "Your tasks here are\n", + "\n", + "1. Set up the design matrix with the inputs as discussed above and a vector containing the output, the so-called targets. Note that the design matrix is the same for all gates. You need just to define different outputs.\n", + "\n", + "2. Construct a neural network with only one hidden layer and two hidden nodes using the Sigmoid function as activation function.\n", + "\n", + "3. Set up the output layer with only one output node and use again the Sigmoid function as activation function for the output.\n", + "\n", + "4. Initialize the weights and biases and perform a feed forward pass and compare the outputs with the targets.\n", + "\n", + "5. Set up the cost function (cross entropy for classification of binary cases).\n", + "\n", + "6. Calculate the gradients needed for the back propagation part.\n", + "\n", + "7. Use the gradients to train the network in the back propagation part. Think of using automatic differentiation.\n", + "\n", + "8. Train the network and study your results and compare with results obtained either with **scikit-learn** or **TensorFlow**.\n", + "\n", + "Everything you develop here can be used directly into the code for the project." ] }, { "cell_type": "markdown", - "id": "a5e5d384", - "metadata": {}, + "id": "78669a54", + "metadata": { + "editable": true + }, + "source": [ + "## Setting up dimensionalities by hand\n", + "\n", + "It can be useful to test the dimensionalities for the network. Let us assume we have performed an optimization for XOR gate and found that the weights for the hidden layer are given by" + ] + }, + { + "cell_type": "markdown", + "id": "453ff3ab", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{W_h}=\\begin{bmatrix} 1 & 1 \\\\\n", + " 1 & 1 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "9d82c2f1", + "metadata": { + "editable": true + }, + "source": [ + "Multiplying $\\boldsymbol{X}$ and $\\boldsymbol{W}$ gives" + ] + }, + { + "cell_type": "markdown", + "id": "a22ab766", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{X}{W}_h=\\begin{bmatrix} 0 & 0 \\\\\n", + " 1 & 1 \\\\\n", + "\t\t 1 & 1 \\\\\n", + "\t\t 2 & 2 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "abfc4274", + "metadata": { + "editable": true + }, + "source": [ + "Assume also that the bias vector for the hidden layer is" + ] + }, + { + "cell_type": "markdown", + "id": "2b7b0970", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{b}_h=\\begin{bmatrix} 0 \\\\\n", + " -1\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "d1e78b4a", + "metadata": { + "editable": true + }, + "source": [ + "Adding it gives us the input to the activation function of the hidden layer" + ] + }, + { + "cell_type": "markdown", + "id": "0ee46428", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h=\\begin{bmatrix} 0 & -1 \\\\\n", + " 1 & 0 \\\\\n", + "\t\t 1 & 0 \\\\\n", + "\t\t 2 & 1 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "bac66371", + "metadata": { + "editable": true + }, + "source": [ + "Let us then assume that our activation function is the RELU function, which simply means that we take the max of $0$ and the elements of the input argument $\\boldsymbol{z}_h$, that is we have" + ] + }, + { + "cell_type": "markdown", + "id": "77606059", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{a}_h=\\mathrm{RELU}(\\boldsymbol{z}_h=\\boldsymbol{X}\\boldsymbol{W}_h+\\boldsymbol{b}_h)=\\begin{bmatrix} 0 & 0 \\\\\n", + " 1 & 0 \\\\\n", + "\t\t 1 & 0 \\\\\n", + "\t\t 2 & 1 \\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "b1a1f219", + "metadata": { + "editable": true + }, + "source": [ + "Assume also that the bias of the output layer is zero and that the weights of the output layer are" + ] + }, + { + "cell_type": "markdown", + "id": "a46fd0d7", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{w}_o=\\begin{bmatrix} 1 \\\\\n", + " -2\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "01e41f6f", + "metadata": { + "editable": true + }, + "source": [ + "and multiplying with $\\boldsymbol{a}_h$ gives the output" + ] + }, + { + "cell_type": "markdown", + "id": "371a2c47", + "metadata": { + "editable": true + }, + "source": [ + "$$\n", + "\\boldsymbol{a}_o=\\boldsymbol{w}_h^T\\begin{bmatrix} 0 & 0 \\\\\n", + " 1 & 0 \\\\\n", + "\t\t 1 & 0 \\\\\n", + "\t\t 2 & 1 \\end{bmatrix}=\\begin{bmatrix} 0 \\\\ 1 \\\\ 1 \\\\0\\end{bmatrix},\n", + "$$" + ] + }, + { + "cell_type": "markdown", + "id": "8ab62ed9", + "metadata": { + "editable": true + }, + "source": [ + "the wanted result." + ] + }, + { + "cell_type": "markdown", + "id": "ee45366d", + "metadata": { + "editable": true + }, "source": [ "## Setting up the Neural Network\n", "\n", @@ -215,21 +440,12 @@ { "cell_type": "code", "execution_count": 1, - "id": "09d0e5dd", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "[[0.80625657 0.36420967]\n", - " [0.90297441 0.30170017]\n", - " [0.89823921 0.28566769]\n", - " [0.93420126 0.25920793]]\n", - "[0 0 0 0]\n" - ] - } - ], + "id": "43b32c8a", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "%matplotlib inline\n", "\n", @@ -258,10 +474,6 @@ " probabilities = sigmoid(z_o)\n", " return probabilities\n", "\n", - "# we obtain a prediction by taking the class with the highest likelihood\n", - "def predict(X):\n", - " probabilities = feed_forward(X)\n", - " return np.argmax(probabilities, axis=1)\n", "\n", "# ensure the same random numbers appear every time\n", "np.random.seed(0)\n", @@ -279,7 +491,7 @@ "# Defining the neural network\n", "n_inputs, n_features = X.shape\n", "n_hidden_neurons = 2\n", - "n_categories = 2\n", + "n_categories = 1\n", "n_features = 2\n", "\n", "# we make the weights normally distributed using numpy.random.randn\n", @@ -293,25 +505,25 @@ "output_bias = np.zeros(n_categories) + 0.01\n", "\n", "probabilities = feed_forward(X)\n", - "print(probabilities)\n", - "\n", - "\n", - "predictions = predict(X)\n", - "print(predictions)" + "print(probabilities)" ] }, { "cell_type": "markdown", - "id": "d9f554db", - "metadata": {}, + "id": "ce3f8d7b", + "metadata": { + "editable": true + }, "source": [ "Not an impressive result, but this was our first forward pass with randomly assigned weights. Let us now add the full network with the back-propagation algorithm discussed above." ] }, { "cell_type": "markdown", - "id": "ab802e6b", - "metadata": {}, + "id": "0a60227f", + "metadata": { + "editable": true + }, "source": [ "## The Code using Scikit-Learn" ] @@ -319,254 +531,12 @@ { "cell_type": "code", "execution_count": 2, - "id": "9231d583", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Learning rate = 1e-05\n", - "Lambda = 1e-05\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1e-05\n", - "Lambda = 0.0001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1e-05\n", - "Lambda = 0.001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1e-05\n", - "Lambda = 0.01\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1e-05\n", - "Lambda = 0.1\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1e-05\n", - "Lambda = 1.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1e-05\n", - "Lambda = 10.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.0001\n", - "Lambda = 1e-05\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.0001\n", - "Lambda = 0.0001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.0001\n", - "Lambda = 0.001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.0001\n", - "Lambda = 0.01\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.0001\n", - "Lambda = 0.1\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.0001\n", - "Lambda = 1.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.0001\n", - "Lambda = 10.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.001\n", - "Lambda = 1e-05\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.001\n", - "Lambda = 0.0001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.001\n", - "Lambda = 0.001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.001\n", - "Lambda = 0.01\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.001\n", - "Lambda = 0.1\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.001\n", - "Lambda = 1.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.001\n", - "Lambda = 10.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.01\n", - "Lambda = 1e-05\n", - "Accuracy score on data set: 0.25\n", - "\n", - "Learning rate = 0.01\n", - "Lambda = 0.0001\n", - "Accuracy score on data set: 0.75\n", - "\n", - "Learning rate = 0.01\n", - "Lambda = 0.001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.01\n", - "Lambda = 0.01\n", - "Accuracy score on data set: 0.75\n", - "\n", - "Learning rate = 0.01\n", - "Lambda = 0.1\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.01\n", - "Lambda = 1.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.01\n", - "Lambda = 10.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.1\n", - "Lambda = 1e-05\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.1\n", - "Lambda = 0.0001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.1\n", - "Lambda = 0.001\n", - "Accuracy score on data set: 1.0\n", - "\n", - "Learning rate = 0.1\n", - "Lambda = 0.01\n", - "Accuracy score on data set: 1.0\n", - "\n", - "Learning rate = 0.1\n", - "Lambda = 0.1\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.1\n", - "Lambda = 1.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 0.1\n", - "Lambda = 10.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1.0\n", - "Lambda = 1e-05\n", - "Accuracy score on data set: 0.75\n", - "\n", - "Learning rate = 1.0\n", - "Lambda = 0.0001\n", - "Accuracy score on data set: 0.75\n", - "\n", - "Learning rate = 1.0\n", - "Lambda = 0.001\n", - "Accuracy score on data set: 0.75\n", - "\n", - "Learning rate = 1.0\n", - "Lambda = 0.01\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1.0\n", - "Lambda = 0.1\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1.0\n", - "Lambda = 1.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 1.0\n", - "Lambda = 10.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 10.0\n", - "Lambda = 1e-05\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 10.0\n", - "Lambda = 0.0001\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n", - "/Users/mhjensen/miniforge3/envs/myenv/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:692: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (100) reached and the optimization hasn't converged yet.\n", - " warnings.warn(\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 10.0\n", - "Lambda = 0.001\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 10.0\n", - "Lambda = 0.01\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 10.0\n", - "Lambda = 0.1\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 10.0\n", - "Lambda = 1.0\n", - "Accuracy score on data set: 0.5\n", - "\n", - "Learning rate = 10.0\n", - "Lambda = 10.0\n", - "Accuracy score on data set: 0.5\n", - "\n" - ] - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAyAAAANbCAYAAAC6lftqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAA9hAAAPYQGoP6dpAABrR0lEQVR4nO3dd3xUddbH8e9QJgMpQISQSOgEYpQSDEVpIqDriuIKPsqKC4tEmrCCCzYUUGwLigKCq1J0kQWlWigi7q4U6ShCCE1aME0gCSE9mecPJLvjBBhg7r2Z5PN+XvN6mN/cO/fk7BA5c8691+Z0Op0CAAAAABNUsDoAAAAAAOUHBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAICBuNcrAACuKlkdAACY6emnn9ayZcsuuU2dOnX0zTffXPOxPv30Ux0+fFhPP/30ZbddvHixnnvuOXXq1EkffPDBNR8bAIDSyubk6zkA5cjx48d1+vTp4uczZ85UXFycZsyYUbxmt9sVFRV1zce6/fbb1bZtW7322muX3bZv377KzMzUoUOH9NVXX6lu3brXfHwAAEojOiAAypV69eqpXr16xc+Dg4Nlt9vVqlUry2I6cuSIdu7cqffee09jxozRJ598oieffNKyeAAAMBLngABACQ4cOKDBgwerdevWat26tYYPH64TJ064bPOPf/xDv/vd79S8eXN16tRJEyZMUGZmpqTz3Y+TJ09q2bJlatasmRISEi56rCVLligwMFC33HKLfve732nJkiXKy8tz227Pnj0aNGiQbr75ZrVv316jRo1SYmJi8eunTp3Ss88+q1tvvVXR0dF6+OGHtWPHjuLXmzVrpunTp7u85/Tp09WsWbPi508//bT69++v8ePHKyYmRn/4wx9UUFCg06dPa+LEieratatuuukmtW3bVsOHD3f7ub788kvdf//9atmypW677TZNnjxZeXl5OnjwoJo1a6ZFixa5bJ+cnKwbbrjhsmNxAICygwIEAH7jyJEjeuihh3Tq1Cm99tprevnll3XixAn17dtXp06dknT+H9qvv/66Hn74Yc2ePVvDhw/XihUrNGnSJEnSjBkzVKtWLXXp0kWLFi1SSEhIiccqLCzUihUr9Pvf/152u13333+/Tp06pa+//tplu/j4ePXt21fZ2dl67bXX9OKLLyouLk4DBw5Ufn6+srKy9NBDD2nTpk168sknNWPGDPn7+2vQoEE6fPjwFf3827dv17FjxzR9+nQNHz5cFStW1ODBg7Vx40Y9+eSTmj17toYNG6ZNmzbphRdeKN5v4cKFGj16tG644QbNmDFDgwcP1oIFCzRhwgRFRESoZcuWWrFihcuxVqxYIYfDoTvvvPOKYgQA+C5GsADgN2bMmCGHw6F58+YpICBAknTLLbeoe/fu+uCDD/TUU09py5YtqlOnjh5++GFVqFBBbdu2VdWqVXXmzBlJUlRUlOx2u4KDgy853vXtt98qJSVFvXv3liS1atVKTZo00T//+U/9/ve/L95u5syZqlatmubMmSM/Pz9JUmhoqJ544gnt379fP/zwg06cOKHly5crMjJSkhQTE6P77rtP27ZtU+PGjT3++QsKCjRx4kTVr19f0vkuRZUqVfTUU08pJiZGktSuXTslJCRo4cKFkqSioiJNnz5dPXr00Msvv1z8Xrm5uVq2bJny8vLUu3dvvfDCCzpx4kTxOS7Lly/XXXfdpapVq3ocHwDAt9EBAYDf2Lx5s9q1ayeHw6GCggIVFBQoICBAMTEx2rRpkySpffv2Onr0qO6///7iE9nvuece9e/f/4qOtWTJEtWvX18NGzZURkaGMjIydNddd2nr1q0unYsdO3aoc+fOxcWHJLVo0ULffPONbrrpJm3fvl3h4eHFxYck+fn5adWqVXrooYeuKCaHw+Fynkzt2rX10UcfKSYmRj///LO+++47zZ8/Xzt37lR+fr6k812jX375Rd27d3d5rwEDBmjFihWy2+26++67VaVKleIuyO7du3X48GHdf//9VxQfAMC30QEBgN9IS0vTypUrtXLlSrfXgoODJUm///3vVVRUpAULFmjGjBl6++23VadOHT355JO6++67PTrO6dOn9e9//1v5+flq06aN2+uLFi3Ss88+WxzTddddd8mYL/X6lbjuuutks9lc1j777DO9+eabSkxMVPXq1RUZGSmHw+Fy/Av7XkxAQIB+97vf6bPPPtPjjz+uZcuWqX79+sVdFQBA+UABAgC/ERgYqFtvvVV//vOf3V6rVOm/vzZ79uypnj176uzZs9qwYYPef/99jRkzRjExMapdu/Zlj7NixQrl5+drxowZCgoKcnntnXfe0fLlyzV69Gg5HA4FBga6XD74gv/85z+KjIxUYGBgiSe679q1SwEBAYqIiJB0/pyT/5WVlXXZOLdv366nnnpK/fr106OPPqrQ0FBJ0t/+9rfik9wvxP/bGNPS0rR37161atVK/v7+6t27t5YtW6bdu3drzZo1euSRRy57fABA2cIIFgD8Rtu2bXXo0CHdcMMNat68uZo3b66bbrpJ8+bN09q1ayVJTzzxhB5//HFJ5wuWu+66S8OGDVNhYaFSUlIkSRUqXPpX7NKlS9WqVSv16NFD7dq1c3n07dtX6enpWrVqlaTz53OsX7/e5epY+/fv12OPPaYff/xRMTExOnHihPbv31/8el5enkaMGKFPPvlE0vkORFJSkksMO3fuvGw+du3apaKiIo0cObK4+CgsLCweRysqKlKjRo1Uo0YNrVu3zmXfzz//XLGxscrNzZUktWnTRg0aNNDkyZN15swZ3XfffZc9PgCgbKEAAYDfGDZsmI4fP67Bgwfr66+/1vr16zVixAh9+eWXxedYtG/fXmvXrtXrr7+u7777TmvWrNHbb7+tBg0aFG8TFBSkuLg4bd26VTk5OS7H2L17tw4cOHDRca1u3bqpWrVqxSd5Dxs2TGfOnFFsbKy++eYbrV69Wk888YRuvPFGde7cWffff7/q1q2roUOHasWKFVq/fr1GjhypnJyc4i7Dbbfdpi+//FILFizQd999p7Fjx+rYsWOXzUeLFi0kSS+++KI2b96sr776Sn/+858VHx8v6XwXpWLFihoxYoTWrFmjCRMmaOPGjfr444/11ltvqW/fvsWja5LUu3dvbd26VbfccovCwsKu5H8aAEAZQAECAL8RGRmpjz/+WDabTWPHjtXIkSOVmpqqd955R3fccYck6aGHHtK4ceP07bffasiQIXrhhRfUuHFjzZkzR5UrV5YkDRw4UL/88oseffRR7dmzx+UYS5YsUcWKFV2udPW/7Ha77rrrLn3//ffat2+foqKi9I9//ENFRUUaNWqUXnzxRbVq1Urvv/++7Ha7AgICNH/+fEVHR+vll1/WX/7yF+Xm5uof//hH8QnlzzzzjG6//XZNnjxZI0eOVJUqVTy64WG7du30wgsvaNeuXYqNjdWrr76q66+/vvju8RfGsB5++GG99tpr2r59uwYPHqw5c+Zo4MCBevrpp13e77bbbpMkTj4HgHLK5nQ6nVYHAQAoP95//3198MEHWr9+vex2u9XhAABMxknoAABTLFu2TAcOHNCCBQv02GOPUXwAQDlFAQIAMEV8fLwWLlyo7t27KzY21upwAAAWYQQLAAAAgGbOnKnvvvtO//jHPy66zZkzZzRp0iR9++23kqTf/e53euaZZ1S1alWPj8NJ6AAAAEA5N2/ePE2bNu2y240cOVInTpwo3n7jxo2aOHHiFR2LESwAAACgnEpOTtZzzz2nHTt2qGHDhpfcdteuXdq6datWrlypxo0bSzp/ifZBgwZp9OjRHt2EV6IDAgAAAJRbe/fuVbVq1fTZZ5+pZcuWl9x2+/btqlWrVnHxIZ2/ea/NZiu+JLsn6IAAAAAAPqxbt26XfH3dunUXfe3222/X7bff7tFxkpOT3W4ga7fbVb16dSUmJnr0HlI5K0BaD51qdQgAAAC4hJ2zRlkdwkUVJTW1OoSLqGvKUbKzs0u8hLqfn59yc3M9fp9yVYAAAAAAZc2lOhze5HA4lJeX57aem5vLVbAAAAAAeFdoaKhSUlJc1vLy8pSWlubxCegSBQgAAADgkaJS+n9madOmjZKSknTs2LHitS1btkiSWrdu7fH7UIAAAAAAcFNYWKjU1FTl5ORIklq2bKnWrVtr1KhR2r17tzZv3qzx48frvvvuowMCAAAA4NokJiaqY8eOWrlypSTJZrNpxowZCg8PV//+/fXEE0+oc+fOmjBhwhW9r83pdDoNiLdU4ipYAAAApVtpvgpWbmIjq0MokV/YT1aHcEXogAAAAAAwDQUIAAAAANNwHxAAAADAA0UqN2cuGIoOCAAAAADTUIAAAAAAMA0jWAAAAIAHzLzpX1lGBwQAAACAaShAAAAAAJiGESwAAADAA4Xl5/7dhqIDAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaRjBAgAAADxQyAiWV9ABAQAAAGAaChAAAAAApmEECwAAAPAAV8HyDjogAAAAAExDAQIAAADANIxgAQAAAB4odDKC5Q10QAAAAACYhgIEAAAAgGkYwQIAAAA8UGR1AGUEHRAAAAAApqEAAQAAAGAaRrAAAAAADxRyI0KvoAMCAAAAwDQUIAAAAABMwwgWAAAA4IFCJrC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgAe4EaF30AEBAAAAYBoKEAAAAACmYQQLAAAA8EChbFaHUCbQAQEAAABgGgoQAAAAAKZhBAsAAADwQBE3IvQKOiAAAAAATEMBAgAAAMA0jGBZ5Nao+hp2bwc1DAtW2tlsLV6/W3PXbLvo9g1q19DSCQPc1o8knVbviR8aGKnvIsfGIr/GI8fGI8fGIr/GI8fm4ipY3kEBYoEWjcI0dWgvfbXjgGZ+tlGtmtTR8Hs7qILNptmrt5a4T9O6tSRJsW9+qryCguL1nLyCErcv78ixsciv8cix8cixsciv8cgxfBUFiAUG391e+xNS9fy81ZKkTXHHVKlCBQ24s43mr9uh3PxCt32ahYfo5C/p2nEwwexwfRI5Nhb5NR45Nh45Nhb5NR45hq/iHBCTVa5UUTdHhOubXYdc1r/edVD+Druim9Qpcb9m4bW0PyHVjBB9Hjk2Fvk1Hjk2Hjk2Fvk1Hjm2RqFspfLha+iAmCy8ZjXZK1fS8ZQzLusnUtMkSfVDamjzvuNu+zUNr6UjSac0b8yDalY3RGezcvX55jjN+myTCoqKzAjdZ5BjY5Ff45Fj45FjY5Ff45Fj+DLLC5CCggJ99dVX2r59u37++Wfl5eWpSpUqCg0NVUxMjHr06KFKlSwP02sCq/hJkjJz8lzWs3597l/F7rZPcGBV1azmL6fTqWnLNijxTIbaNqunAXfEqHaNAI2bu9r4wH0IOTYW+TUeOTYeOTYW+TUeOYYvs/Rf9sePH1dsbKySk5MVFRWlkJAQVatWTbm5udq3b5+WLFmi6dOn64MPPtD1119vZaheY7P92iZzlnwnm5K+fDiXk6shby/RseTTSj6TKUnaefCk8gsKNbxXB81etVVHkk4bFbLPIcfGIr/GI8fGI8fGIr/GI8fWKHL63rhTaWTpOSATJ05UeHi41q9frwULFuitt97S66+/rrfeeksLFizQt99+q7CwML344otWhulVZ7NzJUn+DtdvJqr++jwzJ9dtn9z8Qm2NP178y+KC9XuOSJIiwmsaEarPIsfGIr/GI8fGI8fGIr/GI8fwZZYWIDt27NDYsWMVGBhY4utBQUEaM2aMtm27+PWsfU1CapoKCotUN6S6y3rdWuef/5R4ym2f+rVrqHenFm6/ZByVzzew0jJzDInVV5FjY5Ff45Fj45FjY5Ff45Fj+DJLC5CgoCClpKRccpuff/5ZDofDpIiMl1dQqF2HEnR7qyYu692jI5SRlaO9R5Pc9gmpHqDn/thN3aMjXNbviGmqzOxc7TuebGjMvoYcG4v8Go8cG48cG4v8Go8cW8Pqq11xFSwv6NOnj5555hmNHDlS7dq1U1hYmOx2u/Ly8pScnKytW7dqypQp6tOnj5Vhet0Hq7Zq1sjeen3Q3Vrx3V61bBSmP/WI0bTl65WbXyh/h12NwoJ1IjVdaZnZ2nEgQdv2n9DoPp3l8Kuso0mn1al5Qz10W7TeWvqtzma5t1nLO3JsLPJrPHJsPHJsLPJrPHIMX2VzOi9y9pIJnE6n3nnnHc2dO1dZWVlur/v7++vhhx/WX/7yF1WocO3NmtZDp17ze3hL15aNNaTnLapfu4ZS0s/pk39/r/nrdkqSbo4I1/ujH9D4D9fo881xkqQAh12De96i21o2Vs1q/kpITdeCb3Zq2cY9Vv4YpRo5Nhb5NR45Nh45Nhb5NV5ZzPHOWaOsDuGidh6vZ3UIJWpdz/2Sy6WZpQXIBfn5+dq3b5+Sk5OVnZ0th8Oh0NBQRUZGym53v4zc1SpNBQgAAADcleYCZNvxBlaHUKI29Y5aHcIVKRU32KhcubJatGhhdRgAAAAADGbpSegAAAAAypdS0QEBAAAASjtuROgddEAAAAAAmIYCBAAAAIBpGMECAAAAPOCLN/0rjeiAAAAAADANBQgAAAAA0zCCBQAAAHig0Ml3995AFgEAAACYhgIEAAAAgGkYwQIAAAA8UMR3915BFgEAAACYhgIEAAAAgGkYwQIAAAA8wI0IvYMOCAAAAADTUIAAAAAAMA0jWAAAAIAHuBGhd5BFAAAAAKahAAEAAABgGkawAAAAAA8UcRUsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOCBQr679wqyCAAAAMA0FCAAAAAATMMIFgAAAOABbkToHWQRAAAAgGkoQAAAAACYhhEsAAAAwANFfHfvFWQRAAAAgGkoQAAAAACYhhEsAAAAwAOFTpvVIZQJdEAAAAAAmIYCBAAAAIBpKEAAAAAAmIZzQAAAAAAPFPLdvVeQRQAAAKCcKioq0rRp09SpUye1bNlSAwcO1LFjxy66/YkTJzRkyBC1bdtWHTp00KRJk5SdnX1Fx6QAAQAAAMqpmTNnauHChZo0aZIWLVokm82m2NhY5eXluW179uxZ9e3bV+np6frggw/07rvvas+ePRo+fPgVHZMCBAAAAPBAkbNCqXxcrby8PM2ZM0cjRoxQly5dFBkZqalTpyo5OVlr1651237ZsmXKzMzUO++8oxYtWqh58+aaOnWqNm3apO3bt3t8XAoQAAAAoByKj4/XuXPn1L59++K1oKAgRUVFadu2bW7bHzlyRI0aNVJwcHDxWlhYmGrUqKGtW7d6fFxOQgcAAAB8WLdu3S75+rp160pcT0pKknS+iPhfISEhSkxMdNu+Vq1aSk1NVWFhoSpWrChJyszMVHp6uk6dOuVxvHRAAAAAAA8UqkKpfFytCyeP2+12l3U/Pz/l5ua6bX/33XcrPT1dr7zyis6dO6eMjAyNHz9eNputxHNGLoYOCAAAAODDLtbhuByHwyHp/LkgF/4sSbm5uapSpYrb9vXr19f06dP1wgsv6OOPP5bD4dAjjzyim266SQEBAR4flwIEAAAAKIcujF6lpKSoXr16xespKSmKjIwscZ8uXbroP//5j1JTUxUYGCiHw6Fbb71V999/v8fHZQQLAAAA8ECh01YqH1crMjJSAQEB2rJlS/FaRkaG4uLiFBMT47b9jh071K9fP+Xl5alWrVpyOBzaunWrzpw5o1tvvdXj49IBAQAAAMohu92ufv36acqUKQoODladOnU0efJkhYaGqkePHiosLNTp06eLOx2NGzfWwYMH9corr+jRRx/ViRMnNHbsWD300EOqW7eux8elAAEAAADKqZEjR6qgoEDjxo1TTk6O2rRpo9mzZ8tutyshIUHdunXTq6++qvvvv1/Vq1fXe++9p1dffVX33HOPatSooYceekhDhw69omPanE6n06Cfp9RpPXSq1SEAAADgEnbOGmV1CBf14UHPx4zM1D9ik9UhXBHOAQEAAABgGgoQAAAAAKbhHBAAAADAA4VOvrv3BrIIAAAAwDQUIAAAAABMwwgWAAAA4IEiXf1N//BfdEAAAAAAmIYCBAAAAIBpGMECAAAAPMBVsLyDLAIAAAAwDQUIAAAAANMwggUAAAB4oJDv7r2CLAIAAAAwDQUIAAAAANMwggUAAAB4oMjJjQi9gQ4IAAAAANNQgAAAAAAwDSNYAAAAgAe4CpZ3kEUAAAAApqEAAQAAAGAaRrAAAAAADxQ5+e7eG8giAAAAANNQgAAAAAAwDSNYAAAAgAcKxY0IvYEOCAAAAADTUIAAAAAAMA0jWAAAAIAHuAqWd5BFAAAAAKahAAEAAABgGkawAAAAAA9wFSzvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AGuguUdZBEAAACAaShAAAAAAJiGESwAAADAA4WMYHkFWQQAAABgGgoQAAAAAKZhBAsAAADwQBE3IvQKOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BFgEAAACYhgIEAAAAgGkYwQIAAAA8UOTkKljeQAcEAAAAgGkoQAAAAACYhhEsAAAAwAOFfHfvFRQgFrk1qr6G3dtBDcOClXY2W4vX79bcNdsuun2D2jW0dMIAt/UjSafVe+KHBkbqu8ixsciv8cix8cixsciv8cgxfBEFiAVaNArT1KG99NWOA5r52Ua1alJHw+/toAo2m2av3lriPk3r1pIkxb75qfIKCorXc/IKSty+vCPHxiK/xiPHxiPHxiK/xiPH8FUUIBYYfHd77U9I1fPzVkuSNsUdU6UKFTTgzjaav26HcvML3fZpFh6ik7+ka8fBBLPD9Unk2Fjk13jk2Hjk2Fjk13jk2HxcBcs7GGQzWeVKFXVzRLi+2XXIZf3rXQfl77ArukmdEvdrFl5L+xNSzQjR55FjY5Ff45Fj45FjY5Ff45Fj+DI6ICYLr1lN9sqVdDzljMv6idQ0SVL9kBravO+4235Nw2vpSNIpzRvzoJrVDdHZrFx9vjlOsz7bpIKiIjNC9xnk2Fjk13jk2Hjk2Fjk13jkGL6MAsRkgVX8JEmZOXku61m/PvevYnfbJziwqmpW85fT6dS0ZRuUeCZDbZvV04A7YlS7RoDGzV1tfOA+hBwbi/wajxwbjxwbi/wajxxbo4jhIa+gADGZzfbr7KDTWeLrJX35cC4nV0PeXqJjyaeVfCZTkrTz4EnlFxRqeK8Omr1qq44knTYqZJ9Djo1Ffo1Hjo1Hjo1Ffo1HjuHLKONMdjY7V5Lk73D9ZqLqr88zc3Ld9snNL9TW+OPFvywuWL/niCQpIrymEaH6LHJsLPJrPHJsPHJsLPJrPHIMX2Z5B+SRRx75bxV/GR999JHB0RgvITVNBYVFqhtS3WW9bq3zz39KPOW2T/3aNRTTtK5Wb4vXuf9ptToqn/+fLy0zx7B4fRE5Nhb5NR45Nh45Nhb5NR45tkYhV8HyCss7ILfccou2bdumU6dOqU6dOpd8lAV5BYXadShBt7dq4rLePTpCGVk52ns0yW2fkOoBeu6P3dQ9OsJl/Y6YpsrMztW+48mGxuxryLGxyK/xyLHxyLGxyK/xyDF8meUdkGHDhqlq1aqaNm2a/v73vys8PNzqkAz3waqtmjWyt14fdLdWfLdXLRuF6U89YjRt+Xrl5hfK32FXo7BgnUhNV1pmtnYcSNC2/Sc0uk9nOfwq62jSaXVq3lAP3Ratt5Z+q7NZ7m3W8o4cG4v8Go8cG48cG4v8Go8cw1fZnM6LnL1kskGDBql69eqaMmWKYcdoPXSqYe99pbq2bKwhPW9R/do1lJJ+Tp/8+3vNX7dTknRzRLjeH/2Axn+4Rp9vjpMkBTjsGtzzFt3WsrFqVvNXQmq6FnyzU8s27rHyxyjVyLGxyK/xyLHxyLGxyK/xymKOd84aZXUIF/WXXX2tDqFEb0f/0+oQrkipKUCSk5MVFxenrl27GnaM0lSAAAAAwB0FyJXztQLE8hGsC2rXrq3atWtbHQYAAAAAA5WaAgQAAAAozYqcll+/qUwgiwAAAABMQwECAAAAwDSMYAEAAAAeKBQ3IvQGOiAAAAAATEMBAgAAAMA0jGABAAAAHihyMoLlDXRAAAAAAJiGAgQAAACAaRjBAgAAADzAjQi9gywCAAAAMA0FCAAAAADTMIIFAAAAeKCIGxF6BR0QAAAAAKahAAEAAABgGkawAAAAAA8UciNCr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHWQRAAAAgGkoQAAAAACYhhEsAAAAwANFXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHigSIxgeQMdEAAAAACmoQABAAAAYBpGsAAAAAAPcBUs76ADAgAAAJRTRUVFmjZtmjp16qSWLVtq4MCBOnbs2EW3T01N1ejRo9WuXTu1a9dOf/nLX5SUlHRFx6QAAQAAAMqpmTNnauHChZo0aZIWLVokm82m2NhY5eXllbj9qFGjlJiYqLlz52ru3LlKSkrSsGHDruiYFCAAAACAB4qcFUrl42rl5eVpzpw5GjFihLp06aLIyEhNnTpVycnJWrt2rdv2GRkZ2rZtm2JjYxUVFaWoqCg99thj2rt3r86cOePxcSlAAAAAgHIoPj5e586dU/v27YvXgoKCFBUVpW3btrlt7+fnp6pVq2r58uXKzMxUZmamVqxYoQYNGqhatWoeH5eT0AEAAIBy6MK5G2FhYS7rISEhSkxMdNvez89PL7/8sl588UXFxMTIZrOpVq1amj9/vipU8LyvQQECAAAAeKC0XgWrW7dul3x93bp1Ja5nZ2dLkux2u8u6n5+f0tPT3bZ3Op3av3+/oqOjNWjQIBUWFmrq1KkaPny4/vnPfyogIMCjeClAAAAAgHLI4XBIOn8uyIU/S1Jubq6qVKnitv2XX36pBQsW6F//+ldxsfHuu++qa9euWrJkifr37+/RcSlAAAAAAB92sQ7H5VwYvUpJSVG9evWK11NSUhQZGem2/Y4dO9SwYUOXTke1atXUsGFDHT161OPjchI6AAAA4IEi2Url42pFRkYqICBAW7ZsKV7LyMhQXFycYmJi3LYPCwvTsWPHlJubW7yWnZ2thIQE1a9f3+PjUoAAAAAA5ZDdble/fv00ZcoUrVu3TvHx8Ro1apRCQ0PVo0cPFRYWKjU1VTk5OZKk++67T5L0xBNPKD4+vnh7u92u+++/3+PjUoAAAAAA5dTIkSPVp08fjRs3Tn379lXFihU1e/Zs2e12JSYmqmPHjlq5cqWk81fHWrBggZxOp/r3768///nPqly5sv75z38qKCjI42PanE6n06gfqLRpPXSq1SEAAADgEnbOGmV1CBf1h43DrQ6hRMs6vGN1CFeEDggAAAAA01CAAAAAADANl+EFAAAAPFBab0Toa+iAAAAAADANBQgAAAAA0zCCBQAAAHiAESzvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AFGsLyDDggAAAAA01CAAAAAADANI1gAAACAB4rECJY30AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAHmAEyzvogAAAAAAwDQUIAAAAANMwggUAAAB4gBEs76ADAgAAAMA0FCAAAAAATMMIFgAAAOABRrC8gw4IAAAAANNQgAAAAAAwDSNYAAAAgAecjGB5BR0QAAAAAKahAAEAAABgGkawAAAAAA8UiREsb6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaRjBAgAAADzAjQi9gw4IAAAAANNQgAAAAAAwDSNYAAAAgAe4CpZ30AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BBwQAAACAaShAAAAAAJiGESwAAADAA06n1RGUDXRAAAAAAJiGAgQAAACAaRjBAgAAADxQJK6C5Q10QAAAAACYhgIEAAAAgGkYwQIAAAA84ORGhF5BBwQAAACAaShAAAAAAJiGESyL3BpVX8Pu7aCGYcFKO5utxet3a+6abRfdvkHtGlo6YYDb+pGk0+o98UMDI/Vd5NhY5Nd45Nh45NhY5Nd45NhcRYxgeQUFiAVaNArT1KG99NWOA5r52Ua1alJHw+/toAo2m2av3lriPk3r1pIkxb75qfIKCorXc/IKSty+vCPHxiK/xiPHxiPHxiK/xiPH8FUUIBYYfHd77U9I1fPzVkuSNsUdU6UKFTTgzjaav26HcvML3fZpFh6ik7+ka8fBBLPD9Unk2Fjk13jk2Hjk2Fjk13jkGL6Kc0BMVrlSRd0cEa5vdh1yWf9610H5O+yKblKnxP2ahdfS/oRUM0L0eeTYWOTXeOTYeOTYWOTXeOTYGk5n6Xz4GjogJguvWU32ypV0POWMy/qJ1DRJUv2QGtq877jbfk3Da+lI0inNG/OgmtUN0dmsXH2+OU6zPtukgqIiM0L3GeTYWOTXeOTYeOTYWOTXeOQYvszyAuTIkSP64osvlJ6erk6dOqlLly4ur2dmZurll1/Wq6++alGE3hVYxU+SlJmT57Ke9etz/yp2t32CA6uqZjV/OZ1OTVu2QYlnMtS2WT0NuCNGtWsEaNzc1cYH7kPIsbHIr/HIsfHIsbHIr/HIMXyZpQXIjh079Oijj6p27dpyOp36+OOP1b17d73xxhuy28//xcnJydHy5cvLTAFis/169YSL9MtK+vLhXE6uhry9RMeSTyv5TKYkaefBk8ovKNTwXh00e9VWHUk6bVTIPoccG4v8Go8cG48cG4v8Go8cW4MbEXqHpeeAvPHGG+rTp4/WrFmjr776Sm+++aY2btyoIUOGKD8/38rQDHM2O1eS5O9w/Wai6q/PM3Ny3fbJzS/U1vjjxb8sLli/54gkKSK8phGh+ixybCzyazxybDxybCzyazxyDF9maQGyf/9+9evXr/j5XXfdpffff1+7du3S2LFjLYzMOAmpaSooLFLdkOou63VrnX/+U+Ipt33q166h3p1auP2ScVQ+38BKy8wxJFZfRY6NRX6NR46NR46NRX6NR47hyywtQAICAnTmjOvJUzfffLMmT56sNWvWlJmxq/+VV1CoXYcSdHurJi7r3aMjlJGVo71Hk9z2CakeoOf+2E3doyNc1u+IaarM7FztO55saMy+hhwbi/wajxwbjxwbi/wajxxbw+m0lcqHr7H0HJAuXbroxRdf1IQJExQVFaXKlStLkrp3765nn31WkyZNUmJiopUhGuKDVVs1a2RvvT7obq34bq9aNgrTn3rEaNry9crNL5S/w65GYcE6kZqutMxs7TiQoG37T2h0n85y+FXW0aTT6tS8oR66LVpvLf1WZ7Pc26zlHTk2Fvk1Hjk2Hjk2Fvk1HjmGr7I5ndZdPTg9PV2jRo3Sd999p7///e/q3Lmzy+sLFizQK6+8osLCQu3bt++aj9d66NRrfg9v6dqysYb0vEX1a9dQSvo5ffLv7zV/3U5J0s0R4Xp/9AMa/+Eafb45TpIU4LBrcM9bdFvLxqpZzV8Jqela8M1OLdu4x8ofo1Qjx8Yiv8Yjx8Yjx8Yiv8YrizneOWuU1SFcVPPPxlsdQol+vHei1SFcEUsLkAuOHz+uGjVqKDAw0O21I0eO6KuvvtLgwYOv+TilqQABAACAu9JcgNy4YoLVIZRob68JVodwRSy/D4gk1atX76KvNWzY0CvFBwAAAADrWXoSOgAAAIDypVR0QAAAAIDSzvoTF8oGOiAAAAAATEMBAgAAAMA0jGABAAAAHvDFm/6VRnRAAAAAAJiGAgQAAACAaRjBAgAAADzACJZ30AEBAAAAYBoKEAAAAACmYQQLAAAA8AD3IfQOOiAAAAAATEMBAgAAAMA0jGABAAAAHuAqWN5BBwQAAACAaShAAAAAAJiGESwAAADAE1wGyyvogAAAAAAwDQUIAAAAANMwggUAAAB4gKtgeQcdEAAAAACmoQABAAAAyqmioiJNmzZNnTp1UsuWLTVw4EAdO3asxG2nT5+uZs2alfh45plnPD4mBQgAAADgAaezdD6uxcyZM7Vw4UJNmjRJixYtks1mU2xsrPLy8ty2HThwoDZs2ODyeOKJJ+RwONS/f3+Pj0kBAgAAAJRDeXl5mjNnjkaMGKEuXbooMjJSU6dOVXJystauXeu2vb+/v2rVqlX8yM7O1t///nc9/fTTioyM9Pi4FCAAAABAORQfH69z586pffv2xWtBQUGKiorStm3bLrv/a6+9poiICD344INXdFyuggUAAAB4oLReBatbt26XfH3dunUlriclJUmSwsLCXNZDQkKUmJh4yff88ccftW7dOn344YeqUOHKehp0QAAAAIByKDs7W5Jkt9td1v38/JSbm3vJfefNm6eWLVu6dE88RQcEAAAA8GEX63BcjsPhkHT+XJALf5ak3NxcValS5aL7ZWVlae3atRo/fvxVHZcOCAAAAOAJp610Pq7ShdGrlJQUl/WUlBSFhoZedL/169erqKhIPXr0uKrjUoAAAAAA5VBkZKQCAgK0ZcuW4rWMjAzFxcUpJibmovvt2LFDN954o4KCgq7quIxgAQAAAOWQ3W5Xv379NGXKFAUHB6tOnTqaPHmyQkND1aNHDxUWFur06dMKDAx0GdGKj49X06ZNr/q4dEAAAAAAD1h9w0EjbkQ4cuRI9enTR+PGjVPfvn1VsWJFzZ49W3a7XYmJierYsaNWrlzpss8vv/yi6tWrX/UxbU7ntYbtO1oPnWp1CAAAALiEnbNGWR3CRTX65ytWh1Cin/o+a3UIV4QOCAAAAADTcA4IAAAA4IlyMzdkLDogAAAAAExDAQIAAADANIxgAQAAAB5wXsNN//BfdEAAAAAAmIYCBAAAAIBpGMECAAAAPMFVsLyCDggAAAAA01CAAAAAADANI1gAAACAB7gKlnfQAQEAAABgGgoQAAAAAKZhBAsAAADwBFfB8go6IAAAAABMU646INf9fZPVIZRp6SsjrA6hzCtYUcvqEMq0Sr1SrQ6hzOMzDAAoVwUIAAAAcPW4CpY3MIIFAAAAwDQUIAAAAABMwwgWAAAA4AmuguUVdEAAAAAAmIYCBAAAAIBpGMECAAAAPMEIllfQAQEAAABgGgoQAAAAAKZhBAsAAADwhJMbEXoDHRAAAAAApqEAAQAAAGAaRrAAAAAADzi5CpZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8AQjWF5BBwQAAACAaShAAAAAAJiGESwAAADAE9yI0CvogAAAAAAwDQUIAAAAANMwggUAAAB4wMZVsLyCDggAAAAA01CAAAAAADANI1gAAACAJxjB8go6IAAAAABMQwECAAAAwDSMYAEAAACe4EaEXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMATXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANI1gAAACAJ5w2qyMoE+iAAAAAADANBQgAAAAA0zCCBQAAAHjAxlWwvIIOCAAAAADTUIAAAAAAMA0jWAAAAIAnGMHyCjogAAAAAExDB8QkMXe20p9fekj1osKVnpqhL/7+lRa+tvyi21eqXEl9nuypHn+6TbXqXqdfEk7rmwXrtfC15SrILyjebtHP7ys4tLrb/v8XFqszyWne/0FKqfY1IzQ0oocaBoToTN45LT2xVR/+9J8St727TmuNb97nou81cfen+vLnXZKkVV2f0XV+gW7b3PXNKzqVl+md4H3UrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt/AZ9gafI6NRX6NR47hiyhATBB1S1O9uOIp/WfRJs19fqFu6hipP0/qqwoVKmjBK0tL3GfoWwPU409d9PGkJTqw7ZCatG6kR8Y/oJD6tfTmoFmSpOoh1RQcWl2zRs/Tvu8OuOyfceqs4T9XadG8ej290foRrU38UbMOrlWrGg00NKKHKsimuT/92237jSn7NfC7WS5rNpv07I33y7+Snzamns9lsD1A1/kFauq+L/Vj2nGX7dPyswz7eXxBi0Zhmjq0l77acUAzP9uoVk3qaPi9HVTBZtPs1VtL3Kdp3VqSpNg3P1VewX+L6Jy8ghK3L0/4DFuDz7GxyK/xyDF8FQWICR554QEd/v6oXu8/XZK0fc33qlS5oh586j4tfvML5eXkuWwfWCNAPQf30AdPf6xPp3wmSdr1zR5J0mN/e0Szn/5Y6b9kqEl0Q0nShqVblHL8FxN/otIltsntOpCRqAk/fipJ2vzLQVWyVdCfGnXRgqMblFvk+ks1Lf+c0tLPuaw9WP9WNQiopUGb31Va/vnXmgaFSZL+lbxXSTlpxv8gPmTw3e21PyFVz89bLUnaFHdMlSpU0IA722j+uh3KzS9026dZeIhO/pKuHQcTzA631OMzbA0+x8Yiv8Yjx/BVV3QOSGpqqiZMmKBHH31UTz31lObNm6ft27crOzvbqPh8XmV7JbW47UZtWLbFZf3bxZtVNbCKmneKdNvHv1pVffH3tfrus+0u6wkHfpYkhTUKkSQ1btVAZ89kluvio7KtoloHN9K/kve6rK9L2iP/Sn5qVaPBZd/jOnuAhkT00JLjW7Q3/b+/kJsGhikjP5t/uP1G5UoVdXNEuL7Zdchl/etdB+XvsCu6SZ0S92sWXkv7E1LNCNGn8Bm2Bp9jY5Ff45Fj+LIr6oA8++yz2rBhgyIiIpSQkKDPP/9cTqdTFSpUUKNGjXTTTTepefPmat68uSIjI1W5cmWj4vYZYY1qy+5XWSd/LR4u+PlQkiQpvOn12rF2t8trSUdTNH34B27v1fEP7ZSfV6CEA4mSpMYtGyjzzDmNX/xXRXdrrgoVK2jzFzv07uh5Op2UZswPVMrUqRose4VKOp51ymU94dfn9fxrasupQyXtWuyxiO4qchbp3YNrXdabBl2vs/nZer3Vw2pzXWNVsNm0IXW/psZ/qVO55WfE7bfCa1aTvXIlHU8547J+IjVNklQ/pIY27zvutl/T8Fo6knRK88Y8qGZ1Q3Q2K1efb47TrM82qaCoyIzQSyU+w9bgc2ws8ms8cmwNbkToHVdUgOzatUtjxozRwIEDJUlZWVnau3evfvzxR/3444/atm2bli1bJkmy2+3avXv3pd5OkpSbm6uDBw+qSZMmcjgc2rdvn+bPn6/k5GRFRESof//+Cg0NvYofrXTwr+4vSTqX4dolyjp7/nnVoCoevU/H+9up+yOdtWzaSmWmnR+vaNyqgWqGX6eVH3ytJW99oXo3hKv/xAf1xr8namjrscrJyvXiT1I6BVZ2SJLOFeS4rGcVnh9r86/kd8n9a9j99fvro/Xx0Q3K/M17NA0MU4gjSMsTtumfxzaqgX8tDY7orr+3jVW/TdOVU5jvxZ/EdwRWOZ/TzN+MDmb9+ty/it1tn+DAqqpZzV9Op1PTlm1Q4pkMtW1WTwPuiFHtGgEaN3e18YGXUnyGrcHn2Fjk13jkGL7sigoQPz8/RUVFFT+vWrWq2rRpozZt2hSvpaWlaffu3dqzZ89l3+/w4cMaMGCAUlNTdf3112vSpEkaNmyYwsPD1bhxY3399ddaunSpFixYoMaNG19JqKVGhQq2839wllwyFxVdvpTu1Lu9np4/Urv/E6fZT39cvD5l4Ezl5eTp8PdHJUl7NsTr2N4TemvDJHX/Uxd98e5X1xx/aWfThfyW/HrRRfJ+wX3hbVTBZtPCo5vcXntpz2LlFhbowNnzHafvzxzVT5nJ+qD9EN19fWstObHFbZ/ywGa73Gfafe1cTq6GvL1Ex5JPK/nM+Ssv7Tx4UvkFhRreq4Nmr9qqI0mnjQq5VOMzbA0+x8Yiv8Yjx/BlV3QOSPfu3RUXF3fJbapXr67OnTtr2LBhl32/v/3tb4qOjtby5ct18803a+jQobrnnnv0+eef6+2339aqVavUoUMHvfrqq1cSZqlyoVtRNaiqy3rVwPOdj3Ppl74STe9RPfXcwlHauyFez9/7uvL/5yoV+zYfKC4+Lti7ab8y086pccv6Xoi+9Lvwje9vvyWuWvH8Nz/nCi7dBbo99CZt+eVQ8Um7/+vHtBPF/3C7YHfacZ3Nz1ZEoO925a7V2ezzOfV3uH67VvXX55k57jnPzS/U1vjjxf/Bu2D9niOSpIjwmkaE6hP4DFuDz7GxyK/xyLFFnLbS+fAxV1SA9O7dW6tWrdKhQ5eeR/bU1q1b9cQTTygyMlJPPfWUcnNz1bdv3+KqvlKlShoyZIh27NjhleNZ4efDySosKNT1TVz/Y3/h+fG4i1+FYvi0gRryRn+tX7xZz939inLO/Xe8wr9aVd35566qd0O4236V7JWU/kv5mO9OyDqtgqJChVe9zmX9wvOfMlMuum+IX5CaBV2vr5N+dHstoJJD99S5WQ39Q9xeq1yhYrm+hGlCapoKCotUN6S6y3rdWuef/5R4ym2f+rVrqHenFm7/oXRUPt+ETcvMcdunvOAzbA0+x8Yiv8Yjx/BlV1SA/N///Z/27NmjBx54QM8884xWrlypY8eOXfXBHQ6HcnLOf9hr1qyp//u//5Ofn+u3gBkZGQoMdL+Jlq/Iz83X7m/3qeMf2rmsd+7TXmfPZCp+a8nF3MBX/qj7Hr9Li6d+oZf7TnXpfEhSQV6BRr4zSA89dZ/L+q292shR1U8//Nv1ijplVV5Rgb4/c1RdQ290We8WepMy8rMVl37iovtGVa8rSfrhjPtnOL+oUGOj7lX/Rp1d1ruE3CBHRbt2nP7JC9H7pryCQu06lKDbWzVxWe8eHaGMrBztPZrktk9I9QA998du6h4d4bJ+R0xTZWbnat/xZENjLs34DFuDz7GxyK/xyDF82RWdAzJp0iTt27dPe/fu1apVq7Rs2TLZbDb5+/srKipKN910k8aOHevx+3Xs2FEvvfSSJk2apMaNG+vFF18sfs3pdGrr1q2aOHGiunfvfiVhljoLXl6i19c+r+cXjdbqud8o6tZmeuCv9+qDpz9WXk6eqgZWUf2ocP18OFnpv2SoccsGenBsL+3fdkjffrJJN7Rz/UVxLC5BWWez9cnkz9Tv+T46k5Ku7at3qWGL+vrT+P/T5i92aNc6929Ey6o5h/+lGW0G6tVWffVZwg61qF5P/Rp20oz9a5RbVCD/in5qGBCihKzTLmMqTQJqK7cwXyez3eddc4vy9Y8j6zWoye06lZepzakH1SQwVLFNumlDSry2nTps5o9Y6nywaqtmjeyt1wfdrRXf7VXLRmH6U48YTVu+Xrn5hfJ32NUoLFgnUtOVlpmtHQcStG3/CY3u01kOv8o6mnRanZo31EO3Reutpd/qbDm4YMKl8Bm2Bp9jY5Ff45FjC3AVLK+wOZ2XOcPxIoqKinT48GHt3btXe/bsUVxcnOLj47Vz506P3+P06dMaMmSI6tatqzfeeMPltS+//FJPPvmkOnXqpKlTpyogIOBqwnTRo8ID1/weV6vDfW31pwn/p/Bm1+vUydP6bOZqLX7zC0lSiy5ReuNfEzX5z+/oqw//rf4TH1S/5/tc9L2e7Dpeu/8TJ5vNpnuG3qGeQ+7Q9Y1DlXHqrL755wZ9NP4Tt5sbmiF9ZcTlNzLIbSFRio3opvr+tZSak6FPj2/WgqMbJEmtgxvq3baxmvjjYn158r+fz7FR96pr7Rt1179KPsfIJpt612un3nXbqU7VYKXnZ+mrn3/Qe4e+drsxnFkKVtSy5Lgl6dqysYb0vEX1a9dQSvo5ffLv7zV/3fn83hwRrvdHP6DxH67R55vPnzcW4LBrcM9bdFvLxqpZzV8Jqela8M1OLdt4+QtWmKVSL+uujc9n2Bpl8XNcmpBf45XFHO+cNcrqEC6q0VtvWh1CiX56YrTVIVyRqy5ASuJ0Ov97VYYrkJaWpurVq7usnT59WikpKYqMdL9R39WysgApD6wsQMqL0vaPt7LGygKkvOAzDOByKECunK8VIFc0gnU5V1N8SHIrPiQpODhYwcHB1xgRAAAA4CWMYHnFFZ2EDgAAAADXggIEAAAAgGm8OoIFAAAAlFU2RrC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4gBsRegcdEAAAAACmoQABAAAAYBoKEAAAAMATTlvpfFyDoqIiTZs2TZ06dVLLli01cOBAHTt27KLb5+fn64033lCnTp3UqlUr9evXT/v27buiY1KAAAAAAOXUzJkztXDhQk2aNEmLFi2SzWZTbGys8vLyStx+woQJWrx4sV566SUtWbJE1atXV2xsrM6ePevxMSlAAAAAgHIoLy9Pc+bM0YgRI9SlSxdFRkZq6tSpSk5O1tq1a922P3HihBYvXqxXX31Vt912mxo3bqxXXnlFdrtde/bs8fi4FCAAAACAJ5yl9HGV4uPjde7cObVv3754LSgoSFFRUdq2bZvb9hs2bFBQUJA6d+7ssv0333yjW265xePjchleAAAAwId169btkq+vW7euxPWkpCRJUlhYmMt6SEiIEhMT3bY/evSo6tatq6+++krvvfeekpOTFRUVpaefflqNGzf2OF46IAAAAEA5lJ2dLUmy2+0u635+fsrNzXXbPjMzU8ePH9fMmTM1evRozZo1S5UqVdIf//hHnTp1yuPj0gEBAAAAPFBab0R4sQ7H5TgcDknnzwW58GdJys3NVZUqVdy2r1y5ss6ePaupU6cWdzymTp2qLl26aNmyZRo0aJBHx6UDAgAAAJRDF0avUlJSXNZTUlIUGhrqtn1oaKgqVarkMm7lcDhUt25dJSQkeHxcChAAAACgHIqMjFRAQIC2bNlSvJaRkaG4uDjFxMS4bR8TE6OCggL9+OOPxWs5OTk6ceKE6tev7/FxGcECAAAAPFFKR7Cult1uV79+/TRlyhQFBwerTp06mjx5skJDQ9WjRw8VFhbq9OnTCgwMlMPhUExMjG699VY99dRTevHFF1W9enVNmzZNFStWVK9evTw+Lh0QAAAAoJwaOXKk+vTpo3Hjxqlv376qWLGiZs+eLbvdrsTERHXs2FErV64s3n769Olq27atHn/8cfXp00eZmZn66KOPFBwc7PExbU6ns4zVchfXo8IDVodQpqWvjLA6hDKvYEUtq0Mo0yr1SrU6hDKPzzCAy9k5a5TVIVxUxKtTrQ6hRAefKb05KwkjWAAAAIAHSutVsHwNI1gAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABbkToHXRAAAAAAJiGAgQAAACAaShAAAAAAJiGAgQAAACAaShAAAAAAJiGq2ABAAAAnuAqWF5BBwQAAACAaShAAAAAAJiGESwAAADAA9yI0DvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsryhXBcipwbdaHULZtsLqAMq+7eNnWR0CcE1iVgy1OgQAgMUYwQIAAABgmnLVAQEAAACuGiNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMAD3IjQO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AGuguUddEAAAAAAmIYCBAAAAIBpGMECAAAAPMEIllfQAQEAAABgGgoQAAAAAKZhBAsAAADwBCNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMAD3IjQO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANI1gAAACAB7gRoXfQAQEAAABgGgoQAAAAAKZhBAsAAADwBCNYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMADXAXLO+iAAAAAADANBQgAAAAA0zCCBQAAAHiCESyvoAMCAAAAwDQUIAAAAABMwwgWAAAA4AlGsLyCDggAAAAA01CAAAAAADANBQgAAAAA03AOCAAAAOABm9UBlBF0QAAAAACYhgIEAAAAgGkYwbLIrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt9Fjq2XmCL1+rM0Y5LUNtrqaMomcnxt+D1hLPJrPHJsMi7D6xUUIBZo0ShMU4f20lc7DmjmZxvVqkkdDb+3gyrYbJq9emuJ+zStW0uSFPvmp8orKChez8krKHH78o4cW+9kkhQ7RjqbaRO/sY1Bjq8NvyeMRX6NR47hqyhALDD47vban5Cq5+etliRtijumShUqaMCdbTR/3Q7l5he67dMsPEQnf0nXjoMJZofrk8ixdYqKpOWrpb/NsjqSsoscewe/J4xFfo1HjuGrOAfEZJUrVdTNEeH6Ztchl/Wvdx2Uv8Ou6CZ1StyvWXgt7U9INSNEn0eOrbX/sDRxqnTfndLrz1kdTdlEjq8dvyeMRX6NR46tYXOWzoevoQNisvCa1WSvXEnHU864rJ9ITZMk1Q+poc37jrvt1zS8lo4kndK8MQ+qWd0Qnc3K1eeb4zTrs00qKCoyI3SfQY6tFVZbWvOxFBoibd1ldTRlEzm+dvyeMBb5NR45hi8rtQXIPffco/fee09hYWFWh+JVgVX8JEmZOXku61m/PvevYnfbJziwqmpW85fT6dS0ZRuUeCZDbZvV04A7YlS7RoDGzV1tfOA+hBxbq3qQpCCroyjbyPG14/eEsciv8cgxfJmlBcjy5csv+tqxY8e0atUqBQcHS5Luu+8+c4IymM326y1snCX3y0r68uFcTq6GvL1Ex5JPK/lMpiRp58GTyi8o1PBeHTR71VYdSTptVMg+hxwDuBx+TxiL/BqPHFvEB8edSiNLC5CJEycqJydHkuQs4S/Q3/72N0nn/5KVlQLkbHauJMnf4frNRNVfn2fm5Lrtk5tfqK3x7m3U9XuOaHivDooIr8kvjP9BjgFcDr8njEV+jUeO4cssPQl96dKlioqKUrt27fSf//xH8fHxxY8qVapo7dq1io+P1759+6wM06sSUtNUUFikuiHVXdbr1jr//KfEU2771K9dQ707tXD7JeOofL5+TMvMMSRWX0WOAVwOvyeMRX6NR47hyywtQBo2bKhFixapRYsW6tWrl1auXGllOKbIKyjUrkMJur1VE5f17tERysjK0d6jSW77hFQP0HN/7Kbu0REu63fENFVmdq72HU82NGZfQ44BXA6/J4xFfo1Hji3iLKUPH2P5SeiVKlXS6NGj1alTJz311FNat26dJkyYYHVYhvpg1VbNGtlbrw+6Wyu+26uWjcL0px4xmrZ8vXLzC+XvsKtRWLBOpKYrLTNbOw4kaNv+Exrdp7McfpV1NOm0OjVvqIdui9ZbS7/V2Sz3Nmt5R44BXA6/J4xFfo1HjuGrbM6STr6wSEZGhiZOnKjt27fr1KlTWrVqlerWreu19289dKrX3utadW3ZWEN63qL6tWsoJf2cPvn395q/bqck6eaIcL0/+gGN/3CNPt8cJ0kKcNg1uOctuq1lY9Ws5q+E1HQt+Ganlm3cY+WPUaqVxRxvH+9bd57bukvq/4RNH77lVNtoq6Mpm3wtxzETh1odgouy+HuiNCG/xiuLOd45a5TVIVxUy5Gl59+S/+uHaaU3ZyUpVQXIBcuXL9fSpUs1ZcoUhYSEeO19S1MBAlwNXytAgN8qbQUIgNKnNBcgrUaUzn9Lfj+99OasJJaPYJXkvvvuKzNXvQIAAADwX5aehA4AAACgfCmVHRAAAACg1Cl1Jy74JjogAAAAAExDAQIAAADANIxgAQAAAB6wMYLlFXRAAAAAAJiGAgQAAAAop4qKijRt2jR16tRJLVu21MCBA3Xs2LGLbr9s2TI1a9bM7XGpfX6LESwAAADAE2VwBGvmzJlauHChXn31VdWuXVuTJ09WbGysvvjiC9ntdrft9+/fr7Zt2+rNN990WQ8ODvb4mHRAAAAAgHIoLy9Pc+bM0YgRI9SlSxdFRkZq6tSpSk5O1tq1a0vc58CBA4qMjFStWrVcHhUrVvT4uBQgAAAAQDkUHx+vc+fOqX379sVrQUFBioqK0rZt20rcZ//+/WrSpMk1HZcCBAAAAPCAzVk6H1crKSlJkhQWFuayHhISosTERLftT58+rV9++UXbtm1Tz5491bFjRw0fPlxHjhy5ouNyDggAAADgw7p163bJ19etW1fienZ2tiS5nevh5+en9PR0t+0PHDggSapYsaJef/11ZWVlaebMmfrjH/+ozz//XDVr1vQoXgoQAAAAoBxyOBySzp8LcuHPkpSbm6sqVaq4bd++fXtt3bpV1apVK15755131LVrVy1dulSPPfaYR8elAAEAAAA8UUqvgnWxDsflXBi9SklJUb169YrXU1JSFBkZWeI+/1t8SFLVqlUVHh6u5ORkj4/LOSAAAABAORQZGamAgABt2bKleC0jI0NxcXGKiYlx237BggVq166dcnJyitcyMzN19OjRKzoxnQIEAAAAKIfsdrv69eunKVOmaN26dYqPj9eoUaMUGhqqHj16qLCwUKmpqcUFR9euXeV0OjV27FgdPHhQP/74o0aMGKHg4GD94Q9/8Pi4FCAAAACAJ5yl9HENRo4cqT59+mjcuHHq27evKlasqNmzZ8tutysxMVEdO3bUypUrJZ0f2frwww917tw59e3bVwMGDFBgYKA++ugjl3NILodzQAAAAIByqmLFihozZozGjBnj9lp4eLj279/vsnbDDTdo9uzZ13RMOiAAAAAATEMHBAAAAPDAtdz0D/9FBwQAAACAaShAAAAAAJiGESwAAADAE4xgeQUdEAAAAACmoQABAAAAYBpGsAAAAAAP2JzMYHkDHRAAAAAApqEAAQAAAGAaRrAAAAAATzCB5RV0QAAAAACYhgIEAAAAgGkYwQIAAAA8YGMEyyvogAAAAAAwDQUIAAAAANMwggUAAAB4ghEsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABroLlHXRAAAAAAJiGAgQAAACAaRjBAgAAADzBCJZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAnmAEyyvogAAAAAAwTbnqgFTqlWp1CMA1af99H6tDKNMKVtSyOgQAAMq8clWAAAAAAFeLq2B5ByNYAAAAAExDAQIAAADANIxgAQAAAJ5wMoPlDXRAAAAAAJiGAgQAAACAaRjBAgAAADzAVbC8gw4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAHrAVWR1B2UAHBAAAAIBpKEAAAAAAmIYRLAAAAMATXAXLK+iAAAAAADANBQgAAAAA0zCCBQAAAHjAxgiWV9ABAQAAAGAaChAAAAAApmEECwAAAPCEkxksb6ADAgAAAMA0FCAAAAAATMMIFgAAAOABroLlHXRAAAAAAJiGAgQAAACAaRjBAgAAADzBCJZX0AEBAAAAYBoKEAAAAACmYQQLAAAA8ABXwfIOOiAAAAAATEMBAgAAAMA0jGABAAAAnnAyg+UNdEAAAAAAmIYCBAAAAIBpGMECAAAAPMBVsLyDDggAAAAA01CAAAAAADANI1gAAACAJxjB8go6IAAAAABMQwECAAAAwDSMYAEAAAAe4CpY3kEHBAAAAIBpKEAAAAAAmIYRLAAAAMATRcxgeQMdEAAAAACmoQNikvY1IzQ0oocaBoToTN45LT2xVR/+9J8St727TmuNb97nou81cfen+vLnXZKkVV2f0XV+gW7b3PXNKzqVl+md4H0A+TUeOTbfrVH1NezeDmoYFqy0s9lavH635q7ZdtHtG9SuoaUTBritH0k6rd4TPzQwUt9Fjo1Ffo1HjuGLKEBM0Lx6Pb3R+hGtTfxRsw6uVasaDTQ0oocqyKa5P/3bbfuNKfs18LtZLms2m/TsjffLv5KfNqYekCQF2wN0nV+gpu77Uj+mHXfZPi0/y7Cfp7Qhv8Yjx+Zr0ShMU4f20lc7DmjmZxvVqkkdDb+3gyrYbJq9emuJ+zStW0uSFPvmp8orKChez8krKHH78o4cG4v8Go8cW4AJLK+gADFBbJPbdSAjURN+/FSStPmXg6pkq6A/NeqiBUc3KLfI9S99Wv45paWfc1l7sP6tahBQS4M2v6u0/POvNQ0KkyT9K3mvknLSjP9BSinyazxybL7Bd7fX/oRUPT9vtSRpU9wxVapQQQPubKP563YoN7/QbZ9m4SE6+Uu6dhxMMDtcn0SOjUV+jUeO4as4B8RglW0V1Tq4kf6VvNdlfV3SHvlX8lOrGg0u+x7X2QM0JKKHlhzfor3p//2F0TQwTBn52eX6H27k13jk2HyVK1XUzRHh+mbXIZf1r3cdlL/DrugmdUrcr1l4Le1PSDUjRJ9Hjo1Ffo1HjuHL6IAYrE7VYNkrVNLxrFMu6wm/Pq/nX1NbTh0qaddij0V0V5GzSO8eXOuy3jToep3Nz9brrR5Wm+saq4LNpg2p+zU1/kudyj3r3R+klCK/xiPH5guvWU32ypV0POWMy/qJ1DRJUv2QGtq877jbfk3Da+lI0inNG/OgmtUN0dmsXH2+OU6zPtukgqIiM0L3GeTYWOTXeOTYGtyI0DssLUAWL16se++9V3a7vXht8+bNmjNnjpKSkhQREaGhQ4eqSZMmFkZ5bQIrOyRJ5wpyXNazCvMkSf6V/C65fw27v35/fbQ+PrpBmb95j6aBYQpxBGl5wjb989hGNfCvpcER3fX3trHqt2m6cgrzvfiTlE7k13jk2HyBVc7nNDMnz2U969fn/lXsbvsEB1ZVzWr+cjqdmrZsgxLPZKhts3oacEeMatcI0Li5q40P3IeQY2ORX+ORY/gySwuQ559/Xl27dtV1110nSdqwYYNiY2PVoUMHdezYUXv27FHv3r01d+5ctW7d2spQr5pNtvN/uEjFXOS8dCl9X3gbVbDZtPDoJrfXXtqzWLmFBTpwNlGS9P2Zo/opM1kftB+iu69vrSUntlxT7L6A/BqPHJvPZruQ85JzW9KXlOdycjXk7SU6lnxayWfOXz1s58GTyi8o1PBeHTR71VYdSTptVMg+hxwbi/wajxzDl1l6DojzN39pZs6cqT/96U/64IMPNHbsWH300Ud66KGHNGXKFIsivHYXvvH97bfEVSue/2biXEHuJfe/PfQmbfnlUPFJu//rx7QTxf9wu2B32nGdzc9WRGDotYTtM8iv8cix+c5mn8+pv8P1G8yqvz7PzHHPeW5+obbGHy/+R8UF6/cckSRFhNc0IlSfRY6NRX6NR44t4nSWzoePKVUnoR87dky9evVyWXvwwQcVFxdnUUTXLiHrtAqKChVe9TqX9QvPf8pMuei+IX5BahZ0vb5O+tHttYBKDt1T52Y19A9xe61yhYrl5hKm5Nd45Nh8CalpKigsUt2Q6i7rdWudf/5T4im3ferXrqHenVq4/WPEUfl8ozstM8dtn/KMHBuL/BqPHMOXWVqAFLcPf9WgQQNlZbn+o+PMmTMKDHS/SZmvyCsq0Pdnjqpr6I0u691Cb1JGfrbi0k9cdN+o6nUlST+cOeb2Wn5RocZG3av+jTq7rHcJuUGOinbtOP2TF6Iv/civ8cix+fIKCrXrUIJub+V6/lv36AhlZOVo79Ekt31CqgfouT92U/foCJf1O2KaKjM7V/uOJxsas68hx8Yiv8Yjx/Bllp4D4nQ61a1bNzVs2FCNGzeW3W7X5MmTNX/+fFWuXFk7d+7UxIkT1aVLFyvDvGZzDv9LM9oM1Kut+uqzhB1qUb2e+jXspBn71yi3qED+Ff3UMCBECVmnXcZUmgTUVm5hvk5mu89j5hbl6x9H1mtQk9t1Ki9Tm1MPqklgqGKbdNOGlHhtO3XYzB/RUuTXeOTYfB+s2qpZI3vr9UF3a8V3e9WyUZj+1CNG05avV25+ofwddjUKC9aJ1HSlZWZrx4EEbdt/QqP7dJbDr7KOJp1Wp+YN9dBt0Xpr6bc6m3XpUbnyiBwbi/wajxybj6tgeYfN+dsTMUz0888/a//+/Tpw4EDx/z969Ki2b98uh8Oh6OhoNWvWTLNmzVKNGjWu+XhtVz/rhaivzm0hUYqN6Kb6/rWUmpOhT49v1oKjGyRJrYMb6t22sZr442J9eXJn8T5jo+5V19o36q5/vVrie9pkU+967dS7bjvVqRqs9PwsffXzD3rv0NduN4Yr68iv8cpDjgtW1DL9mJfStWVjDel5i+rXrqGU9HP65N/fa/668/m9OSJc749+QOM/XKPPN58fUw1w2DW45y26rWVj1azmr4TUdC34ZqeWbdxj5Y9RqpFjY5Ff45XFHO+cNcrqEC6q652vWx1Cif615imrQ7gilhYgJcnPz1flypUlSfv371fTpk3dRrWulpUFCIDSr7QVIABQHlGAXDlfK0BK1UnokoqLD0lq1qyZ14oPAAAA4Jo4S+njGhQVFWnatGnq1KmTWrZsqYEDB+rYMfdzN0vy+eefq1mzZkpISLiiY5a6AgQAAACAOWbOnKmFCxdq0qRJWrRokWw2m2JjY5WXl3fJ/U6ePKmJEyde1TEpQAAAAIByKC8vT3PmzNGIESPUpUsXRUZGaurUqUpOTtbatWsvul9RUZHGjBmjG2+88aLbXAoFCAAAAOABm9NZKh9XKz4+XufOnVP79u2L14KCghQVFaVt27ZddL93331X+fn5Gjx48FUd19LL8AIAAAC4Nt26dbvk6+vWrStxPSnp/P1iwsLCXNZDQkKUmJhY4j67d+/WnDlztHjxYiUnX929Y+iAAAAAAOVQdna2JMlut7us+/n5KTfX/b4wWVlZ+utf/6q//vWvatCgwVUflw4IAAAA4IkiqwMo2cU6HJfjcDgknT8X5MKfJSk3N1dVqlRx237SpElq0KCBHnrooasL9FcUIAAAAEA5dGH0KiUlRfXq1SteT0lJUWRkpNv2S5Yskd1uV3R0tCSpsLBQktSzZ0/de++9evHFFz06LgUIAAAAUA5FRkYqICBAW7ZsKS5AMjIyFBcXp379+rlt/9VXX7k8/+GHHzRmzBi99957aty4scfHpQABAAAAPHAtV5wqjex2u/r166cpU6YoODhYderU0eTJkxUaGqoePXqosLBQp0+fVmBgoBwOh+rXr++y/4WT2K+//npdd911Hh+Xk9ABAACAcmrkyJHq06ePxo0bp759+6pixYqaPXu27Ha7EhMT1bFjR61cudKrx7Q5nWWslLuEtquftToEAKVYwYpaVocAAOXezlmjrA7horrd/qrVIZRo3TfPWB3CFWEECwAAAPBEufna3liMYAEAAAAwDQUIAAAAANMwggUAAAB4ovycOm0oOiAAAAAATEMBAgAAAMA0jGABAAAAHrAxgeUVdEAAAAAAmIYCBAAAAIBpGMECAAAAPMFVsLyCDggAAAAA01CAAAAAADANI1gAAACAB2xFVkdQNtABAQAAAGAaChAAAAAApmEECwAAAPAEV8HyCjogAAAAAExDAQIAAADANIxgAQAAAJ5gAssr6IAAAAAAMA0FCAAAAADTMIIFAAAAeMDGVbC8gg4IAAAAANNQgAAAAAAwDSNYAAAAgCcYwfIKOiAAAAAATEMBAgAAAMA0jGABAAAAniiyOoCygQ4IAAAAANNQgAAAAAAwDSNYAAAAgAe4EaF30AEBAAAAYBoKEAAAAACmYQQLAAAA8AQjWF5BBwQAAACAaShAAAAAAJiGESwAAADAE4xgeUW5KkAKVtSyOgQAAACgXGMECwAAAIBpylUHBAAAALhqRVYHUDbQAQEAAABgGgoQAAAAAKZhBAsAAADwgI2rYHkFHRAAAAAApqEAAQAAAGAaRrAAAAAATzCC5RV0QAAAAACYhgIEAAAAgGkYwQIAAAA8wQiWV9ABAQAAAGAaChAAAAAApmEECwAAAPAEI1heQQcEAAAAgGkoQAAAAACYhhEsAAAAwBNFVgdQNtABAQAAAGAaChAAAAAApmEECwAAAPCAjatgeQUdEAAAAACmoQABAAAAYBpGsAAAAABPMILlFXRAAAAAAJiGAgQAAACAaRjBAgAAADxRxAiWN9ABAQAAAGAaChAAAAAApmEECwAAAPAEV8HyCjogAAAAAExDAQIAAADANIxgAQAAAJ5gBMsr6IAAAAAAMA0FCAAAAADTMIIFAAAAeIIRLK+gAwIAAADANBQgAAAAAEzDCBYAAADgiSJGsLyBDggAAAAA01CAAAAAADANI1gAAACAJ5xFVkdQJtABAQAAAGAaChAAAAAApmEECwAAAPAENyL0CjogAAAAAExDAQIAAADANIxgAQAAAJ7gRoReQQFikVuj6mvYvR3UMCxYaWeztXj9bs1ds+2i2zeoXUNLJwxwWz+SdFq9J35oYKS+ixwbi/wajxwbjxwbi/wajxzDF1GAWKBFozBNHdpLX+04oJmfbVSrJnU0/N4OqmCzafbqrSXu07RuLUlS7JufKq+goHg9J6+gxO3LO3JsLPJrPHJsPHJsLPJrPHIMX0UBYoHBd7fX/oRUPT9vtSRpU9wxVapQQQPubKP563YoN7/QbZ9m4SE6+Uu6dhxMMDtcn0SOjUV+jUeOjUeOjUV+jUeOLcBVsLyCk9BNVrlSRd0cEa5vdh1yWf9610H5O+yKblKnxP2ahdfS/oRUM0L0eeTYWOTXeOTYeOTYWOTXeOQYvowOiMnCa1aTvXIlHU8547J+IjVNklQ/pIY27zvutl/T8Fo6knRK88Y8qGZ1Q3Q2K1efb47TrM82qaCoyIzQfQY5Nhb5NR45Nh45Nhb5NR45hi+zvAD54YcftGXLFj322GOSpM2bN2vevHlKSEhQvXr1NHDgQMXExFgcpfcEVvGTJGXm5LmsZ/363L+K3W2f4MCqqlnNX06nU9OWbVDimQy1bVZPA+6IUe0aARo3d7XxgfsQcmws8ms8cmw8cmws8ms8cmwRRrC8wtICZPXq1Ro9erRuvfVWPfbYY/rXv/6lYcOGqXPnzurSpYsOHDig/v37a8aMGeratauVoXqNzWY7/4eLfIBL+vLhXE6uhry9RMeSTyv5TKYkaefBk8ovKNTwXh00e9VWHUk6bVTIPoccG4v8Go8cG48cG4v8Go8cw1uKioo0Y8YMffrpp8rIyNDNN9+s8ePHq379+iVuv2fPHk2ePFm7d++Wn5+f7rjjDv31r39VUFCQx8e09ByQGTNm6PHHH9cHH3wgSZo1a5aGDBmiv//97xozZozef/99Pf7445o2bZqVYXrV2excSZK/w/Wbiaq/Ps/MyXXbJze/UFvjjxf/srhg/Z4jkqSI8JpGhOqzyLGxyK/xyLHxyLGxyK/xyDG8ZebMmVq4cKEmTZqkRYsWyWazKTY2Vnl5eW7bpqSk6M9//rPq1aunZcuWaebMmdq5c6eeeuqpKzqmpQXI8ePHdc899xQ/T0hI0J133umyTc+ePXX48GGzQzNMQmqaCgqLVDekust63Vrnn/+UeMptn/q1a6h3pxZuv2Qclc83sNIycwyJ1VeRY2ORX+ORY+ORY2ORX+ORY4s4naXzcZXy8vI0Z84cjRgxQl26dFFkZKSmTp2q5ORkrV271m37kydPqlOnTho/frwaNGig1q1b64EHHtB33313Rce1tACpW7eu/vOf/xQ/v+GGGxQfH++yze7du1W7dm2zQzNMXkGhdh1K0O2tmrisd4+OUEZWjvYeTXLbJ6R6gJ77Yzd1j45wWb8jpqkys3O173iyoTH7GnJsLPJrPHJsPHJsLPJrPHIMb4iPj9e5c+fUvn374rWgoCBFRUVp2zb3G1pGR0frzTffVKVK54vWQ4cOadmyZerQocMVHdfSc0BiY2P13HPPKSkpST179tSwYcP09NNPKzc3VxEREfrhhx/0zjvv6PHHH7cyTK/7YNVWzRrZW68Pulsrvturlo3C9KceMZq2fL1y8wvl77CrUViwTqSmKy0zWzsOJGjb/hMa3aezHH6VdTTptDo1b6iHbovWW0u/1dks9zZreUeOjUV+jUeOjUeOjUV+jUeOcUG3bt0u+fq6detKXE9KOl+ohoWFuayHhIQoMTHxku9555136ujRo6pTp45mzpx5BdFKNqfT2tP5V6xYoWnTpunkyZOy2Wz633D8/f01aNAgDR061CvHaj10qlfexxu6tmysIT1vUf3aNZSSfk6f/Pt7zV+3U5J0c0S43h/9gMZ/uEafb46TJAU47Brc8xbd1rKxalbzV0JquhZ8s1PLNu6x8sco1cixsciv8cix8cixsciv8cpijnfOGmV1CBd1V9hwq0MoUV5U/CVfv1gBsmLFCo0dO1b79u1ThQr/HYwaO3asUlJSNG/evIu+548//qicnBxNmTJFp06d0ooVK+Tv7+9RvJYXIBf89NNPOnr0qDIzM1W5cmWFhoYqKipKfn5+XjtGaSpAAAAA4I4C5MqtSnznqvZbs2aNRo4cqR9++EEOh6N4/S9/+Yvy8vI0a9asy75HSkqKunTpoldffVX33XefR8e1/D4gFzRq1EiNGjWyOgwAAACgXLgwepWSkqJ69eoVr6ekpCgyMtJt+8OHDyshIUFdunQpXgsJCVG1atWUnOz5OUSWnoQOAAAA+Ayrr3bl5atgRUZGKiAgQFu2bCley8jIUFxcXIk3Al+/fr3+8pe/KDPzv5dyPn78uM6cOaPGjRt7fFwKEAAAAKAcstvt6tevn6ZMmaJ169YpPj5eo0aNUmhoqHr06KHCwkKlpqYqJ+f8JZp79eqlwMBAjRkzRgcPHtT27ds1cuRItWjR4opuGk4BAgAAAJRTI0eOVJ8+fTRu3Dj17dtXFStW1OzZs2W325WYmKiOHTtq5cqVkqQaNWroo48+UlFRkfr27avhw4crKipKs2fPVsWKFT0+Zqk5Cd0MnIQOAABQupXqk9Bre+fKrN62KvnyJ4uXJnRAAAAAAJiGAgQAAACAaUrNZXgBAACAUq2o3Jy5YCg6IAAAAABMQwECAAAAwDSMYAEAAAAecDqLrA6hTKADAgAAAMA0FCAAAAAATMMIFgAAAOAJroLlFXRAAAAAAJiGAgQAAACAaRjBAgAAADzhZATLG+iAAAAAADANBQgAAAAA0zCCBQAAAHiiiBsRegMdEAAAAACmoQABAAAAYBpGsAAAAABPcBUsr6ADAgAAAMA0FCAAAAAATMMIFgAAAOABJ1fB8go6IAAAAABMQwECAAAAwDSMYAEAAACe4CpYXkEHBAAAAIBpKEAAAAAAmIYRLAAAAMATRYxgeQMdEAAAAACmoQABAAAAYBpGsAAAAABPOLkRoTfQAQEAAABgGgoQAAAAAKZhBAsAAADwgJOrYHkFHRAAAAAApqEAAQAAAGAaRrAAAAAAT3AVLK+gAwIAAADANBQgAAAAAEzDCBYAAADgAa6C5R10QAAAAACYhgIEAAAAgGkYwQIAAAA8wVWwvIIOCAAAAADTUIAAAAAAMI3N6XRyOj8AAAAAU9ABAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaChAAAAAApqEAAQAAAGAaCpBSpqioSNOmTVOnTp3UsmVLDRw4UMeOHbM6rDJr5syZeuSRR6wOo0xJS0vTCy+8oM6dO6t169bq27evtm/fbnVYZcqpU6c0ZswYtW/fXtHR0Xrsscd06NAhq8Mqk44cOaLo6GgtXbrU6lDKlJMnT6pZs2Zuj08//dTq0MqU5cuX6/e//72aN2+uu+++W6tWrbI6JEASBUipM3PmTC1cuFCTJk3SokWLZLPZFBsbq7y8PKtDK3PmzZunadOmWR1GmTN69Gj98MMPevPNN7V48WLdeOONevTRR3X48GGrQyszhg4dqhMnTuj999/X4sWL5XA4NGDAAGVnZ1sdWpmSn5+vv/71r8rKyrI6lDJn//798vPz0/r167Vhw4bixz333GN1aGXGihUr9Oyzz+rBBx/UF198od///vcaPXq0du3aZXVoAAVIaZKXl6c5c+ZoxIgR6tKliyIjIzV16lQlJydr7dq1VodXZiQnJ2vQoEF6++231bBhQ6vDKVOOHTumjRs3avz48YqJiVGjRo303HPPqXbt2vriiy+sDq9MOHPmjMLDw/XSSy+pefPmaty4sYYNG6bU1FQdPHjQ6vDKlOnTp8vf39/qMMqkAwcOqGHDhgoJCVGtWrWKHw6Hw+rQygSn06m3335b/fv3V//+/VW/fn0NHz5ct956q7Zu3Wp1eAAFSGkSHx+vc+fOqX379sVrQUFBioqK0rZt2yyMrGzZu3evqlWrps8++0wtW7a0OpwypUaNGnrvvfd00003Fa/ZbDY5nU6lp6dbGFnZUaNGDb355puKiIiQJP3yyy+aPXu2QkND1aRJE4ujKzu2bdumRYsW6fXXX7c6lDJp//79fF4N9NNPP+nkyZNuHaXZs2dr8ODBFkUF/FclqwPAfyUlJUmSwsLCXNZDQkKUmJhoRUhl0u23367bb7/d6jDKpKCgIHXp0sVlbdWqVTp+/Lg6duxoUVRl1/PPP69PPvlEdrtds2bNUtWqVa0OqUzIyMjQ2LFjNW7cOLffx/COAwcOqFatWvrjH/+oo0ePqn79+ho2bJg6depkdWhlwtGjRyVJWVlZevTRRxUXF6fw8HANHTqU//6hVKADUopcmN+22+0u635+fsrNzbUiJOCa7NixQ88++6y6devGf/QM0L9/fy1ZskT33nuvhg8frr1791odUpkwYcIEtWrVivMRDJKXl6ejR48qMzNTTzzxhN577z01b95csbGx+u6776wOr0zIzMyUJD311FPq2bOn5syZow4dOmjYsGHkGKUCHZBS5MLsa15ensscbG5urqpUqWJVWMBV+frrr/XXv/5VLVu21Jtvvml1OGXShRGWl156Sd9//73mz5+vV1991eKofNvy5cu1fft2ff7551aHUmbZ7XZt27ZNlSpVKv7C7aabbtLhw4c1e/Zs3XLLLRZH6PsqV64sSXr00Uf1hz/8QZJ0ww03KC4uTnPnziXHsBwdkFLkQqs/JSXFZT0lJUWhoaFWhARclfnz52vEiBHq3Lmz3n//fU4s9aJTp07piy++UGFhYfFahQoV1LhxY7ffHbhyS5Ys0alTp3TbbbcpOjpa0dHRkqTx48fr7rvvtji6sqNq1apu3f6mTZsqOTnZoojKlgv/ZmjatKnLepMmTZSQkGBFSIALCpBSJDIyUgEBAdqyZUvxWkZGhuLi4hQTE2NhZIDnFixYoJdeekkPP/yw3nrrLbd/ZODapKSk6Mknn3S5kk1+fr7i4uLUuHFjCyMrG6ZMmaKVK1dq+fLlxQ9JGjlypN577z1rgysj4uPjFR0d7XZ/oD179nBiupdERUXJ399fP/zwg8v6gQMHVK9ePYuiAv6LEaxSxG63q1+/fpoyZYqCg4NVp04dTZ48WaGhoerRo4fV4QGXdeTIEb3yyivq0aOHBg8erFOnThW/5nA4FBgYaGF0ZUNkZKQ6duyoiRMnatKkSQoKCtK7776rjIwMDRgwwOrwfF7t2rVLXL/uuutUp04dk6Mpm5o2baqIiAhNnDhR48ePV40aNfTJJ5/o+++/1+LFi60Or0xwOBwaNGiQ3nnnHdWuXVstWrTQl19+qY0bN2revHlWhwdQgJQ2I0eOVEFBgcaNG6ecnBy1adNGs2fP5ltk+IQ1a9YoPz9fa9eudbt3zR/+8Ae99tprFkVWdthsNr311lt644039MQTT+js2bOKiYnRxx9/rOuvv97q8IDLqlChgt59911NmTJFTzzxhDIyMhQVFaW5c+eqWbNmVodXZgwbNkxVqlQpvp9Y48aNNX36dLVr187q0ADZnE6n0+ogAAAAAJQPnAMCAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAAAAwDQUIAAAAABMQwECAD4mNzdXUVFRio6O1ksvvWR1OAAAXBEKEADwMTabTR9++KFatGih+fPn68iRI1aHBACAxyhAAMDH2O12tWnTRoMGDZIk7d271+KIAADwHAUIAPioRo0aSZL27dtncSQAAHiOAgQAfNT7778vSYqPj7c4EgAAPEcBAgA+aMOGDfrnP/+patWqKS4uzupwAADwGAUIAPiYjIwMPfvss+rWrZv69u2r06dPKzk52eqwAADwCAUIAPiYiRMnqqCgQJMmTVJUVJQkxrAAAL6DAgQAfMjq1av1xRdf6OWXX1ZwcHBxAcKJ6AAAX0EBAgA+IjU1VePHj9eDDz6orl27SpLq1q2roKAgzgMBAPgMChAA8BHPP/+8qlWrpqefftpl/YYbbmAECwDgMyhAAMAHfPrpp/r222/1t7/9TVWrVnV5LSoqSsePH1dmZqZF0QEA4Dmb0+l0Wh0EAAAAgPKBDggAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADANBQgAAAAA01CAAAAAADDN/wOpUE87FNLINAAAAABJRU5ErkJggg==\n", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], + "id": "0574c4c7", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], "source": [ "# import necessary packages\n", "import numpy as np\n", @@ -589,10 +559,7 @@ "yAND = np.array( [ 0, 0 ,0, 1])\n", "\n", "# Defining the neural network\n", - "n_inputs, n_features = X.shape\n", "n_hidden_neurons = 2\n", - "n_categories = 2\n", - "n_features = 2\n", "\n", "eta_vals = np.logspace(-5, 1, 7)\n", "lmbd_vals = np.logspace(-5, 1, 7)\n", @@ -629,24 +596,1328 @@ }, { "cell_type": "markdown", - "id": "1c0ad443", - "metadata": {}, + "id": "c3db2190", + "metadata": { + "editable": true + }, "source": [ - "How do we interpret these results?" + "## Building a neural network code\n", + "\n", + "Here we present a flexible object oriented codebase\n", + "for a feed forward neural network, along with a demonstration of how\n", + "to use it. Before we get into the details of the neural network, we\n", + "will first present some implementations of various schedulers, cost\n", + "functions and activation functions that can be used together with the\n", + "neural network.\n", + "\n", + "The codes here were developed by Eric Reber and Gregor Kajda during spring 2023." ] }, { "cell_type": "markdown", - "id": "1cea859c", - "metadata": {}, + "id": "3b6e8500", + "metadata": { + "editable": true + }, + "source": [ + "### Learning rate methods\n", + "\n", + "The code below shows object oriented implementations of the Constant,\n", + "Momentum, Adagrad, AdagradMomentum, RMS prop and Adam schedulers. All\n", + "of the classes belong to the shared abstract Scheduler class, and\n", + "share the update_change() and reset() methods allowing for any of the\n", + "schedulers to be seamlessly used during the training stage, as will\n", + "later be shown in the fit() method of the neural\n", + "network. Update_change() only has one parameter, the gradient\n", + "($δ^l_ja^{l−1}_k$), and returns the change which will be subtracted\n", + "from the weights. The reset() function takes no parameters, and resets\n", + "the desired variables. For Constant and Momentum, reset does nothing." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "ee32fb88", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "class Scheduler:\n", + " \"\"\"\n", + " Abstract class for Schedulers\n", + " \"\"\"\n", + "\n", + " def __init__(self, eta):\n", + " self.eta = eta\n", + "\n", + " # should be overwritten\n", + " def update_change(self, gradient):\n", + " raise NotImplementedError\n", + "\n", + " # overwritten if needed\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Constant(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + "\n", + " def update_change(self, gradient):\n", + " return self.eta * gradient\n", + " \n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Momentum(Scheduler):\n", + " def __init__(self, eta: float, momentum: float):\n", + " super().__init__(eta)\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " self.change = self.momentum * self.change + self.eta * gradient\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " pass\n", + "\n", + "\n", + "class Adagrad(Scheduler):\n", + " def __init__(self, eta):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " return self.eta * gradient * G_t_inverse\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class AdagradMomentum(Scheduler):\n", + " def __init__(self, eta, momentum):\n", + " super().__init__(eta)\n", + " self.G_t = None\n", + " self.momentum = momentum\n", + " self.change = 0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " if self.G_t is None:\n", + " self.G_t = np.zeros((gradient.shape[0], gradient.shape[0]))\n", + "\n", + " self.G_t += gradient @ gradient.T\n", + "\n", + " G_t_inverse = 1 / (\n", + " delta + np.sqrt(np.reshape(np.diagonal(self.G_t), (self.G_t.shape[0], 1)))\n", + " )\n", + " self.change = self.change * self.momentum + self.eta * gradient * G_t_inverse\n", + " return self.change\n", + "\n", + " def reset(self):\n", + " self.G_t = None\n", + "\n", + "\n", + "class RMS_prop(Scheduler):\n", + " def __init__(self, eta, rho):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.second = 0.0\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + " self.second = self.rho * self.second + (1 - self.rho) * gradient * gradient\n", + " return self.eta * gradient / (np.sqrt(self.second + delta))\n", + "\n", + " def reset(self):\n", + " self.second = 0.0\n", + "\n", + "\n", + "class Adam(Scheduler):\n", + " def __init__(self, eta, rho, rho2):\n", + " super().__init__(eta)\n", + " self.rho = rho\n", + " self.rho2 = rho2\n", + " self.moment = 0\n", + " self.second = 0\n", + " self.n_epochs = 1\n", + "\n", + " def update_change(self, gradient):\n", + " delta = 1e-8 # avoid division ny zero\n", + "\n", + " self.moment = self.rho * self.moment + (1 - self.rho) * gradient\n", + " self.second = self.rho2 * self.second + (1 - self.rho2) * gradient * gradient\n", + "\n", + " moment_corrected = self.moment / (1 - self.rho**self.n_epochs)\n", + " second_corrected = self.second / (1 - self.rho2**self.n_epochs)\n", + "\n", + " return self.eta * moment_corrected / (np.sqrt(second_corrected + delta))\n", + "\n", + " def reset(self):\n", + " self.n_epochs += 1\n", + " self.moment = 0\n", + " self.second = 0" + ] + }, + { + "cell_type": "markdown", + "id": "6477d18f", + "metadata": { + "editable": true + }, + "source": [ + "### Usage of the above learning rate schedulers\n", + "\n", + "To initalize a scheduler, simply create the object and pass in the\n", + "necessary parameters such as the learning rate and the momentum as\n", + "shown below. As the Scheduler class is an abstract class it should not\n", + "called directly, and will raise an error upon usage." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "5a396095", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "momentum_scheduler = Momentum(eta=1e-3, momentum=0.9)\n", + "adam_scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)" + ] + }, + { + "cell_type": "markdown", + "id": "28bd1ab9", + "metadata": { + "editable": true + }, + "source": [ + "Here is a small example for how a segment of code using schedulers\n", + "could look. Switching out the schedulers is simple." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "fb5d04e9", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "weights = np.ones((3,3))\n", + "print(f\"Before scheduler:\\n{weights=}\")\n", + "\n", + "epochs = 10\n", + "for e in range(epochs):\n", + " gradient = np.random.rand(3, 3)\n", + " change = adam_scheduler.update_change(gradient)\n", + " weights = weights - change\n", + " adam_scheduler.reset()\n", + "\n", + "print(f\"\\nAfter scheduler:\\n{weights=}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f1229684", + "metadata": { + "editable": true + }, + "source": [ + "### Cost functions\n", + "\n", + "Here we discuss cost functions that can be used when creating the\n", + "neural network. Every cost function takes the target vector as its\n", + "parameter, and returns a function valued only at $x$ such that it may\n", + "easily be differentiated." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "f4a50c76", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "\n", + "def CostOLS(target):\n", + " \n", + " def func(X):\n", + " return (1.0 / target.shape[0]) * np.sum((target - X) ** 2)\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostLogReg(target):\n", + "\n", + " def func(X):\n", + " \n", + " return -(1.0 / target.shape[0]) * np.sum(\n", + " (target * np.log(X + 10e-10)) + ((1 - target) * np.log(1 - X + 10e-10))\n", + " )\n", + "\n", + " return func\n", + "\n", + "\n", + "def CostCrossEntropy(target):\n", + " \n", + " def func(X):\n", + " return -(1.0 / target.size) * np.sum(target * np.log(X + 10e-10))\n", + "\n", + " return func" + ] + }, + { + "cell_type": "markdown", + "id": "3d646164", + "metadata": { + "editable": true + }, + "source": [ + "Below we give a short example of how these cost function may be used\n", + "to obtain results if you wish to test them out on your own using\n", + "AutoGrad's automatics differentiation." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "b0057b55", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from autograd import grad\n", + "\n", + "target = np.array([[1, 2, 3]]).T\n", + "a = np.array([[4, 5, 6]]).T\n", + "\n", + "cost_func = CostCrossEntropy\n", + "cost_func_derivative = grad(cost_func(target))\n", + "\n", + "valued_at_a = cost_func_derivative(a)\n", + "print(f\"Derivative of cost function {cost_func.__name__} valued at a:\\n{valued_at_a}\")" + ] + }, + { + "cell_type": "markdown", + "id": "37f67196", + "metadata": { + "editable": true + }, + "source": [ + "### Activation functions\n", + "\n", + "Finally, before we look at the neural network, we will look at the\n", + "activation functions which can be specified between the hidden layers\n", + "and as the output function. Each function can be valued for any given\n", + "vector or matrix X, and can be differentiated via derivate()." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "ae3c2dad", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from autograd import elementwise_grad\n", + "\n", + "def identity(X):\n", + " return X\n", + "\n", + "\n", + "def sigmoid(X):\n", + " try:\n", + " return 1.0 / (1 + np.exp(-X))\n", + " except FloatingPointError:\n", + " return np.where(X > np.zeros(X.shape), np.ones(X.shape), np.zeros(X.shape))\n", + "\n", + "\n", + "def softmax(X):\n", + " X = X - np.max(X, axis=-1, keepdims=True)\n", + " delta = 10e-10\n", + " return np.exp(X) / (np.sum(np.exp(X), axis=-1, keepdims=True) + delta)\n", + "\n", + "\n", + "def RELU(X):\n", + " return np.where(X > np.zeros(X.shape), X, np.zeros(X.shape))\n", + "\n", + "\n", + "def LRELU(X):\n", + " delta = 10e-4\n", + " return np.where(X > np.zeros(X.shape), X, delta * X)\n", + "\n", + "\n", + "def derivate(func):\n", + " if func.__name__ == \"RELU\":\n", + "\n", + " def func(X):\n", + " return np.where(X > 0, 1, 0)\n", + "\n", + " return func\n", + "\n", + " elif func.__name__ == \"LRELU\":\n", + "\n", + " def func(X):\n", + " delta = 10e-4\n", + " return np.where(X > 0, 1, delta)\n", + "\n", + " return func\n", + "\n", + " else:\n", + " return elementwise_grad(func)" + ] + }, + { + "cell_type": "markdown", + "id": "5be7f268", + "metadata": { + "editable": true + }, + "source": [ + "Below follows a short demonstration of how to use an activation\n", + "function. The derivative of the activation function will be important\n", + "when calculating the output delta term during backpropagation. Note\n", + "that derivate() can also be used for cost functions for a more\n", + "generalized approach." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "5296db7b", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "z = np.array([[4, 5, 6]]).T\n", + "print(f\"Input to activation function:\\n{z}\")\n", + "\n", + "act_func = sigmoid\n", + "a = act_func(z)\n", + "print(f\"\\nOutput from {act_func.__name__} activation function:\\n{a}\")\n", + "\n", + "act_func_derivative = derivate(act_func)\n", + "valued_at_z = act_func_derivative(a)\n", + "print(f\"\\nDerivative of {act_func.__name__} activation function valued at z:\\n{valued_at_z}\")" + ] + }, + { + "cell_type": "markdown", + "id": "fa62814a", + "metadata": { + "editable": true + }, + "source": [ + "### The Neural Network\n", + "\n", + "Now that we have gotten a good understanding of the implementation of\n", + "some important components, we can take a look at an object oriented\n", + "implementation of a feed forward neural network. The feed forward\n", + "neural network has been implemented as a class named FFNN, which can\n", + "be initiated as a regressor or classifier dependant on the choice of\n", + "cost function. The FFNN can have any number of input nodes, hidden\n", + "layers with any amount of hidden nodes, and any amount of output nodes\n", + "meaning it can perform multiclass classification as well as binary\n", + "classification and regression problems. Although there is a lot of\n", + "code present, it makes for an easy to use and generalizeable interface\n", + "for creating many types of neural networks as will be demonstrated\n", + "below." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "08db6527", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import math\n", + "import autograd.numpy as np\n", + "import sys\n", + "import warnings\n", + "from autograd import grad, elementwise_grad\n", + "from random import random, seed\n", + "from copy import deepcopy, copy\n", + "from typing import Tuple, Callable\n", + "from sklearn.utils import resample\n", + "\n", + "warnings.simplefilter(\"error\")\n", + "\n", + "\n", + "class FFNN:\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Feed Forward Neural Network with interface enabling flexible design of a\n", + " nerual networks architecture and the specification of activation function\n", + " in the hidden layers and output layer respectively. This model can be used\n", + " for both regression and classification problems, depending on the output function.\n", + "\n", + " Attributes:\n", + " ------------\n", + " I dimensions (tuple[int]): A list of positive integers, which specifies the\n", + " number of nodes in each of the networks layers. The first integer in the array\n", + " defines the number of nodes in the input layer, the second integer defines number\n", + " of nodes in the first hidden layer and so on until the last number, which\n", + " specifies the number of nodes in the output layer.\n", + " II hidden_func (Callable): The activation function for the hidden layers\n", + " III output_func (Callable): The activation function for the output layer\n", + " IV cost_func (Callable): Our cost function\n", + " V seed (int): Sets random seed, makes results reproducible\n", + " \"\"\"\n", + "\n", + " def __init__(\n", + " self,\n", + " dimensions: tuple[int],\n", + " hidden_func: Callable = sigmoid,\n", + " output_func: Callable = lambda x: x,\n", + " cost_func: Callable = CostOLS,\n", + " seed: int = None,\n", + " ):\n", + " self.dimensions = dimensions\n", + " self.hidden_func = hidden_func\n", + " self.output_func = output_func\n", + " self.cost_func = cost_func\n", + " self.seed = seed\n", + " self.weights = list()\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + " self.classification = None\n", + "\n", + " self.reset_weights()\n", + " self._set_classification()\n", + "\n", + " def fit(\n", + " self,\n", + " X: np.ndarray,\n", + " t: np.ndarray,\n", + " scheduler: Scheduler,\n", + " batches: int = 1,\n", + " epochs: int = 100,\n", + " lam: float = 0,\n", + " X_val: np.ndarray = None,\n", + " t_val: np.ndarray = None,\n", + " ):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " This function performs the training the neural network by performing the feedforward and backpropagation\n", + " algorithm to update the networks weights.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray) : training data\n", + " II t (np.ndarray) : target data\n", + " III scheduler (Scheduler) : specified scheduler (algorithm for optimization of gradient descent)\n", + " IV scheduler_args (list[int]) : list of all arguments necessary for scheduler\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " V batches (int) : number of batches the datasets are split into, default equal to 1\n", + " VI epochs (int) : number of iterations used to train the network, default equal to 100\n", + " VII lam (float) : regularization hyperparameter lambda\n", + " VIII X_val (np.ndarray) : validation set\n", + " IX t_val (np.ndarray) : validation target set\n", + "\n", + " Returns:\n", + " ------------\n", + " I scores (dict) : A dictionary containing the performance metrics of the model.\n", + " The number of the metrics depends on the parameters passed to the fit-function.\n", + "\n", + " \"\"\"\n", + "\n", + " # setup \n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " val_set = False\n", + " if X_val is not None and t_val is not None:\n", + " val_set = True\n", + "\n", + " # creating arrays for score metrics\n", + " train_errors = np.empty(epochs)\n", + " train_errors.fill(np.nan)\n", + " val_errors = np.empty(epochs)\n", + " val_errors.fill(np.nan)\n", + "\n", + " train_accs = np.empty(epochs)\n", + " train_accs.fill(np.nan)\n", + " val_accs = np.empty(epochs)\n", + " val_accs.fill(np.nan)\n", + "\n", + " self.schedulers_weight = list()\n", + " self.schedulers_bias = list()\n", + "\n", + " batch_size = X.shape[0] // batches\n", + "\n", + " X, t = resample(X, t)\n", + "\n", + " # this function returns a function valued only at X\n", + " cost_function_train = self.cost_func(t)\n", + " if val_set:\n", + " cost_function_val = self.cost_func(t_val)\n", + "\n", + " # create schedulers for each weight matrix\n", + " for i in range(len(self.weights)):\n", + " self.schedulers_weight.append(copy(scheduler))\n", + " self.schedulers_bias.append(copy(scheduler))\n", + "\n", + " print(f\"{scheduler.__class__.__name__}: Eta={scheduler.eta}, Lambda={lam}\")\n", + "\n", + " try:\n", + " for e in range(epochs):\n", + " for i in range(batches):\n", + " # allows for minibatch gradient descent\n", + " if i == batches - 1:\n", + " # If the for loop has reached the last batch, take all thats left\n", + " X_batch = X[i * batch_size :, :]\n", + " t_batch = t[i * batch_size :, :]\n", + " else:\n", + " X_batch = X[i * batch_size : (i + 1) * batch_size, :]\n", + " t_batch = t[i * batch_size : (i + 1) * batch_size, :]\n", + "\n", + " self._feedforward(X_batch)\n", + " self._backpropagate(X_batch, t_batch, lam)\n", + "\n", + " # reset schedulers for each epoch (some schedulers pass in this call)\n", + " for scheduler in self.schedulers_weight:\n", + " scheduler.reset()\n", + "\n", + " for scheduler in self.schedulers_bias:\n", + " scheduler.reset()\n", + "\n", + " # computing performance metrics\n", + " pred_train = self.predict(X)\n", + " train_error = cost_function_train(pred_train)\n", + "\n", + " train_errors[e] = train_error\n", + " if val_set:\n", + " \n", + " pred_val = self.predict(X_val)\n", + " val_error = cost_function_val(pred_val)\n", + " val_errors[e] = val_error\n", + "\n", + " if self.classification:\n", + " train_acc = self._accuracy(self.predict(X), t)\n", + " train_accs[e] = train_acc\n", + " if val_set:\n", + " val_acc = self._accuracy(pred_val, t_val)\n", + " val_accs[e] = val_acc\n", + "\n", + " # printing progress bar\n", + " progression = e / epochs\n", + " print_length = self._progress_bar(\n", + " progression,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " except KeyboardInterrupt:\n", + " # allows for stopping training at any point and seeing the result\n", + " pass\n", + "\n", + " # visualization of training progression (similiar to tensorflow progression bar)\n", + " sys.stdout.write(\"\\r\" + \" \" * print_length)\n", + " sys.stdout.flush()\n", + " self._progress_bar(\n", + " 1,\n", + " train_error=train_errors[e],\n", + " train_acc=train_accs[e],\n", + " val_error=val_errors[e],\n", + " val_acc=val_accs[e],\n", + " )\n", + " sys.stdout.write(\"\")\n", + "\n", + " # return performance metrics for the entire run\n", + " scores = dict()\n", + "\n", + " scores[\"train_errors\"] = train_errors\n", + "\n", + " if val_set:\n", + " scores[\"val_errors\"] = val_errors\n", + "\n", + " if self.classification:\n", + " scores[\"train_accs\"] = train_accs\n", + "\n", + " if val_set:\n", + " scores[\"val_accs\"] = val_accs\n", + "\n", + " return scores\n", + "\n", + " def predict(self, X: np.ndarray, *, threshold=0.5):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs prediction after training of the network has been finished.\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Optional Parameters:\n", + " ------------\n", + " II threshold (float) : sets minimal value for a prediction to be predicted as the positive class\n", + " in classification problems\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " This vector is thresholded if regression=False, meaning that classification results\n", + " in a vector of 1s and 0s, while regressions in an array of decimal numbers\n", + "\n", + " \"\"\"\n", + "\n", + " predict = self._feedforward(X)\n", + "\n", + " if self.classification:\n", + " return np.where(predict > threshold, 1, 0)\n", + " else:\n", + " return predict\n", + "\n", + " def reset_weights(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Resets/Reinitializes the weights in order to train the network for a new problem.\n", + "\n", + " \"\"\"\n", + " if self.seed is not None:\n", + " np.random.seed(self.seed)\n", + "\n", + " self.weights = list()\n", + " for i in range(len(self.dimensions) - 1):\n", + " weight_array = np.random.randn(\n", + " self.dimensions[i] + 1, self.dimensions[i + 1]\n", + " )\n", + " weight_array[0, :] = np.random.randn(self.dimensions[i + 1]) * 0.01\n", + "\n", + " self.weights.append(weight_array)\n", + "\n", + " def _feedforward(self, X: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates the activation of each layer starting at the input and ending at the output.\n", + " Each following activation is calculated from a weighted sum of each of the preceeding\n", + " activations (except in the case of the input layer).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each\n", + "\n", + " Returns:\n", + " ------------\n", + " I z (np.ndarray): A prediction vector (row) for each row in our design matrix\n", + " \"\"\"\n", + "\n", + " # reset matrices\n", + " self.a_matrices = list()\n", + " self.z_matrices = list()\n", + "\n", + " # if X is just a vector, make it into a matrix\n", + " if len(X.shape) == 1:\n", + " X = X.reshape((1, X.shape[0]))\n", + "\n", + " # Add a coloumn of zeros as the first coloumn of the design matrix, in order\n", + " # to add bias to our data\n", + " bias = np.ones((X.shape[0], 1)) * 0.01\n", + " X = np.hstack([bias, X])\n", + "\n", + " # a^0, the nodes in the input layer (one a^0 for each row in X - where the\n", + " # exponent indicates layer number).\n", + " a = X\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(a)\n", + "\n", + " # The feed forward algorithm\n", + " for i in range(len(self.weights)):\n", + " if i < len(self.weights) - 1:\n", + " z = a @ self.weights[i]\n", + " self.z_matrices.append(z)\n", + " a = self.hidden_func(z)\n", + " # bias column again added to the data here\n", + " bias = np.ones((a.shape[0], 1)) * 0.01\n", + " a = np.hstack([bias, a])\n", + " self.a_matrices.append(a)\n", + " else:\n", + " try:\n", + " # a^L, the nodes in our output layers\n", + " z = a @ self.weights[i]\n", + " a = self.output_func(z)\n", + " self.a_matrices.append(a)\n", + " self.z_matrices.append(z)\n", + " except Exception as OverflowError:\n", + " print(\n", + " \"OverflowError in fit() in FFNN\\nHOW TO DEBUG ERROR: Consider lowering your learning rate or scheduler specific parameters such as momentum, or check if your input values need scaling\"\n", + " )\n", + "\n", + " # this will be a^L\n", + " return a\n", + "\n", + " def _backpropagate(self, X, t, lam):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Performs the backpropagation algorithm. In other words, this method\n", + " calculates the gradient of all the layers starting at the\n", + " output layer, and moving from right to left accumulates the gradient until\n", + " the input layer is reached. Each layers respective weights are updated while\n", + " the algorithm propagates backwards from the output layer (auto-differentation in reverse mode).\n", + "\n", + " Parameters:\n", + " ------------\n", + " I X (np.ndarray): The design matrix, with n rows of p features each.\n", + " II t (np.ndarray): The target vector, with n rows of p targets.\n", + " III lam (float32): regularization parameter used to punish the weights in case of overfitting\n", + "\n", + " Returns:\n", + " ------------\n", + " No return value.\n", + "\n", + " \"\"\"\n", + " out_derivative = derivate(self.output_func)\n", + " hidden_derivative = derivate(self.hidden_func)\n", + "\n", + " for i in range(len(self.weights) - 1, -1, -1):\n", + " # delta terms for output\n", + " if i == len(self.weights) - 1:\n", + " # for multi-class classification\n", + " if (\n", + " self.output_func.__name__ == \"softmax\"\n", + " ):\n", + " delta_matrix = self.a_matrices[i + 1] - t\n", + " # for single class classification\n", + " else:\n", + " cost_func_derivative = grad(self.cost_func(t))\n", + " delta_matrix = out_derivative(\n", + " self.z_matrices[i + 1]\n", + " ) * cost_func_derivative(self.a_matrices[i + 1])\n", + "\n", + " # delta terms for hidden layer\n", + " else:\n", + " delta_matrix = (\n", + " self.weights[i + 1][1:, :] @ delta_matrix.T\n", + " ).T * hidden_derivative(self.z_matrices[i + 1])\n", + "\n", + " # calculate gradient\n", + " gradient_weights = self.a_matrices[i][:, 1:].T @ delta_matrix\n", + " gradient_bias = np.sum(delta_matrix, axis=0).reshape(\n", + " 1, delta_matrix.shape[1]\n", + " )\n", + "\n", + " # regularization term\n", + " gradient_weights += self.weights[i][1:, :] * lam\n", + "\n", + " # use scheduler\n", + " update_matrix = np.vstack(\n", + " [\n", + " self.schedulers_bias[i].update_change(gradient_bias),\n", + " self.schedulers_weight[i].update_change(gradient_weights),\n", + " ]\n", + " )\n", + "\n", + " # update weights and bias\n", + " self.weights[i] -= update_matrix\n", + "\n", + " def _accuracy(self, prediction: np.ndarray, target: np.ndarray):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Calculates accuracy of given prediction to target\n", + "\n", + " Parameters:\n", + " ------------\n", + " I prediction (np.ndarray): vector of predicitons output network\n", + " (1s and 0s in case of classification, and real numbers in case of regression)\n", + " II target (np.ndarray): vector of true values (What the network ideally should predict)\n", + "\n", + " Returns:\n", + " ------------\n", + " A floating point number representing the percentage of correctly classified instances.\n", + " \"\"\"\n", + " assert prediction.size == target.size\n", + " return np.average((target == prediction))\n", + " def _set_classification(self):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Decides if FFNN acts as classifier (True) og regressor (False),\n", + " sets self.classification during init()\n", + " \"\"\"\n", + " self.classification = False\n", + " if (\n", + " self.cost_func.__name__ == \"CostLogReg\"\n", + " or self.cost_func.__name__ == \"CostCrossEntropy\"\n", + " ):\n", + " self.classification = True\n", + "\n", + " def _progress_bar(self, progression, **kwargs):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Displays progress of training\n", + " \"\"\"\n", + " print_length = 40\n", + " num_equals = int(progression * print_length)\n", + " num_not = print_length - num_equals\n", + " arrow = \">\" if num_equals > 0 else \"\"\n", + " bar = \"[\" + \"=\" * (num_equals - 1) + arrow + \"-\" * num_not + \"]\"\n", + " perc_print = self._format(progression * 100, decimals=5)\n", + " line = f\" {bar} {perc_print}% \"\n", + "\n", + " for key in kwargs:\n", + " if not np.isnan(kwargs[key]):\n", + " value = self._format(kwargs[key], decimals=4)\n", + " line += f\"| {key}: {value} \"\n", + " sys.stdout.write(\"\\r\" + line)\n", + " sys.stdout.flush()\n", + " return len(line)\n", + "\n", + " def _format(self, value, decimals=4):\n", + " \"\"\"\n", + " Description:\n", + " ------------\n", + " Formats decimal numbers for progress bar\n", + " \"\"\"\n", + " if value > 0:\n", + " v = value\n", + " elif value < 0:\n", + " v = -10 * value\n", + " else:\n", + " v = 1\n", + " n = 1 + math.floor(math.log10(v))\n", + " if n >= decimals - 1:\n", + " return str(round(value))\n", + " return f\"{value:.{decimals-n-1}f}\"" + ] + }, + { + "cell_type": "markdown", + "id": "464f4548", + "metadata": { + "editable": true + }, + "source": [ + "Before we make a model, we will quickly generate a dataset we can use\n", + "for our linear regression problem as shown below" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "ea3d3c51", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "import autograd.numpy as np\n", + "from sklearn.model_selection import train_test_split\n", + "\n", + "def SkrankeFunction(x, y):\n", + " return np.ravel(0 + 1*x + 2*y + 3*x**2 + 4*x*y + 5*y**2)\n", + "\n", + "def create_X(x, y, n):\n", + " if len(x.shape) > 1:\n", + " x = np.ravel(x)\n", + " y = np.ravel(y)\n", + "\n", + " N = len(x)\n", + " l = int((n + 1) * (n + 2) / 2) # Number of elements in beta\n", + " X = np.ones((N, l))\n", + "\n", + " for i in range(1, n + 1):\n", + " q = int((i) * (i + 1) / 2)\n", + " for k in range(i + 1):\n", + " X[:, q + k] = (x ** (i - k)) * (y**k)\n", + "\n", + " return X\n", + "\n", + "step=0.5\n", + "x = np.arange(0, 1, step)\n", + "y = np.arange(0, 1, step)\n", + "x, y = np.meshgrid(x, y)\n", + "target = SkrankeFunction(x, y)\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "poly_degree=3\n", + "X = create_X(x, y, poly_degree)\n", + "\n", + "X_train, X_test, t_train, t_test = train_test_split(X, target)" + ] + }, + { + "cell_type": "markdown", + "id": "a6b49446", + "metadata": { + "editable": true + }, + "source": [ + "Now that we have our dataset ready for the regression, we can create\n", + "our regressor. Note that with the seed parameter, we can make sure our\n", + "results stay the same every time we run the neural network. For\n", + "inititialization, we simply specify the dimensions (we wish the amount\n", + "of input nodes to be equal to the datapoints, and the output to\n", + "predict one value)." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "4f0f50b6", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "linear_regression = FFNN((input_nodes, output_nodes), output_func=identity, cost_func=CostOLS, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "8ae3f1e4", + "metadata": { + "editable": true + }, + "source": [ + "We then fit our model with our training data using the scheduler of our choice." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "4d206fa2", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Constant(eta=1e-3)\n", + "scores = linear_regression.fit(X_train, t_train, scheduler)" + ] + }, + { + "cell_type": "markdown", + "id": "27ffcb75", + "metadata": { + "editable": true + }, + "source": [ + "Due to the progress bar we can see the MSE (train_error) throughout\n", + "the FFNN's training. Note that the fit() function has some optional\n", + "parameters with defualt arguments. For example, the regularization\n", + "hyperparameter can be left ignored if not needed, and equally the FFNN\n", + "will by default run for 100 epochs. These can easily be changed, such\n", + "as for example:" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "8cf19432", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "linear_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scores = linear_regression.fit(X_train, t_train, scheduler, lam=1e-4, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "97abfa15", + "metadata": { + "editable": true + }, + "source": [ + "We see that given more epochs to train on, the regressor reaches a lower MSE.\n", + "\n", + "Let us then switch to a binary classification. We use a binary\n", + "classification dataset, and follow a similar setup to the regression\n", + "case." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "5aa3cb16", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import load_breast_cancer\n", + "from sklearn.preprocessing import MinMaxScaler\n", + "\n", + "wisconsin = load_breast_cancer()\n", + "X = wisconsin.data\n", + "target = wisconsin.target\n", + "target = target.reshape(target.shape[0], 1)\n", + "\n", + "X_train, X_val, t_train, t_val = train_test_split(X, target)\n", + "\n", + "scaler = MinMaxScaler()\n", + "scaler.fit(X_train)\n", + "X_train = scaler.transform(X_train)\n", + "X_val = scaler.transform(X_val)" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "793cb2e9", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "markdown", + "id": "7b47a35a", + "metadata": { + "editable": true + }, + "source": [ + "We will now make use of our validation data by passing it into our fit function as a keyword argument" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "e34123b4", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-3, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "413243c9", + "metadata": { + "editable": true + }, + "source": [ + "Finally, we will create a neural network with 2 hidden layers with activation functions." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "aa7b9e8f", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "input_nodes = X_train.shape[1]\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 1\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "neural_network = FFNN(dims, hidden_func=RELU, output_func=sigmoid, cost_func=CostLogReg, seed=2023)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "c66ee57f", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "neural_network.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = neural_network.fit(X_train, t_train, scheduler, epochs=1000, X_val=X_val, t_val=t_val)" + ] + }, + { + "cell_type": "markdown", + "id": "aca7922f", + "metadata": { + "editable": true + }, + "source": [ + "### Multiclass classification\n", + "\n", + "Finally, we will demonstrate the use case of multiclass classification\n", + "using our FFNN with the famous MNIST dataset, which contain images of\n", + "digits between the range of 0 to 9." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "1503f15a", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "from sklearn.datasets import load_digits\n", + "\n", + "def onehot(target: np.ndarray):\n", + " onehot = np.zeros((target.size, target.max() + 1))\n", + " onehot[np.arange(target.size), target] = 1\n", + " return onehot\n", + "\n", + "digits = load_digits()\n", + "\n", + "X = digits.data\n", + "target = digits.target\n", + "target = onehot(target)\n", + "\n", + "input_nodes = 64\n", + "hidden_nodes1 = 100\n", + "hidden_nodes2 = 30\n", + "output_nodes = 10\n", + "\n", + "dims = (input_nodes, hidden_nodes1, hidden_nodes2, output_nodes)\n", + "\n", + "multiclass = FFNN(dims, hidden_func=LRELU, output_func=softmax, cost_func=CostCrossEntropy)\n", + "\n", + "multiclass.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "\n", + "scheduler = Adam(eta=1e-4, rho=0.9, rho2=0.999)\n", + "scores = multiclass.fit(X, target, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "838cf72d", + "metadata": { + "editable": true + }, + "source": [ + "## Testing the XOR gate and other gates\n", + "\n", + "Let us now use our code to test the XOR gate." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "a4444105", + "metadata": { + "collapsed": false, + "editable": true + }, + "outputs": [], + "source": [ + "X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)\n", + "\n", + "# The XOR gate\n", + "yXOR = np.array( [[ 0], [1] ,[1], [0]])\n", + "\n", + "input_nodes = X.shape[1]\n", + "output_nodes = 1\n", + "\n", + "logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)\n", + "logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights\n", + "scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)\n", + "scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)" + ] + }, + { + "cell_type": "markdown", + "id": "8bbdf09c", + "metadata": { + "editable": true + }, + "source": [ + "Not bad, but the results depend strongly on the learning reate. Try different learning rates." + ] + }, + { + "cell_type": "markdown", + "id": "c5cffe73", + "metadata": { + "editable": true + }, "source": [ "## Lecture Thursday October 26" ] }, { "cell_type": "markdown", - "id": "bbdb3e49", - "metadata": {}, + "id": "e88dc6ab", + "metadata": { + "editable": true + }, "source": [ "## Developing a code for doing neural networks with back propagation\n", "\n", @@ -672,8 +1943,10 @@ }, { "cell_type": "markdown", - "id": "45d25695", - "metadata": {}, + "id": "8ec227a0", + "metadata": { + "editable": true + }, "source": [ "## Collect and pre-process data\n", "\n", @@ -719,9 +1992,12 @@ }, { "cell_type": "code", - "execution_count": 3, - "id": "87c0ee64", - "metadata": {}, + "execution_count": 22, + "id": "5336138a", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# import necessary packages\n", @@ -770,8 +2046,10 @@ }, { "cell_type": "markdown", - "id": "4e780f35", - "metadata": {}, + "id": "da4f9c76", + "metadata": { + "editable": true + }, "source": [ "## Train and test datasets\n", "\n", @@ -788,9 +2066,12 @@ }, { "cell_type": "code", - "execution_count": 4, - "id": "bb737921", - "metadata": {}, + "execution_count": 23, + "id": "3d4a38d4", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.model_selection import train_test_split\n", @@ -824,8 +2105,10 @@ }, { "cell_type": "markdown", - "id": "ffb0a087", - "metadata": {}, + "id": "bc7c43bb", + "metadata": { + "editable": true + }, "source": [ "## Define model and architecture\n", "\n", @@ -866,8 +2149,10 @@ }, { "cell_type": "markdown", - "id": "fed0ab83", - "metadata": {}, + "id": "60a15786", + "metadata": { + "editable": true + }, "source": [ "## Layers\n", "\n", @@ -904,8 +2189,10 @@ }, { "cell_type": "markdown", - "id": "6f3ab464", - "metadata": {}, + "id": "d9b2c952", + "metadata": { + "editable": true + }, "source": [ "## Weights and biases\n", "\n", @@ -922,9 +2209,12 @@ }, { "cell_type": "code", - "execution_count": 5, - "id": "6745d89e", - "metadata": {}, + "execution_count": 24, + "id": "a6dfcaaf", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# building our neural network\n", @@ -946,8 +2236,10 @@ }, { "cell_type": "markdown", - "id": "c06d038e", - "metadata": {}, + "id": "623ec89c", + "metadata": { + "editable": true + }, "source": [ "## Feed-forward pass\n", "\n", @@ -972,8 +2264,10 @@ }, { "cell_type": "markdown", - "id": "4f0e93ce", - "metadata": {}, + "id": "391b8f09", + "metadata": { + "editable": true + }, "source": [ "## Matrix multiplications\n", "\n", @@ -1006,9 +2300,12 @@ }, { "cell_type": "code", - "execution_count": 6, - "id": "3c39783f", - "metadata": {}, + "execution_count": 25, + "id": "a3a7a56e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# setup the feed-forward pass, subscript h = hidden layer\n", @@ -1050,8 +2347,10 @@ }, { "cell_type": "markdown", - "id": "c956d7fe", - "metadata": {}, + "id": "6a467f9c", + "metadata": { + "editable": true + }, "source": [ "## Choose cost function and optimizer\n", "\n", @@ -1079,8 +2378,10 @@ }, { "cell_type": "markdown", - "id": "e939a07d", - "metadata": {}, + "id": "824f2150", + "metadata": { + "editable": true + }, "source": [ "## Optimizing the cost function\n", "\n", @@ -1115,8 +2416,10 @@ }, { "cell_type": "markdown", - "id": "ccdee2fd", - "metadata": {}, + "id": "ad801a4f", + "metadata": { + "editable": true + }, "source": [ "## Regularization\n", "\n", @@ -1147,8 +2450,10 @@ }, { "cell_type": "markdown", - "id": "1bf8ea1a", - "metadata": {}, + "id": "adbd5bf5", + "metadata": { + "editable": true + }, "source": [ "## Matrix multiplication\n", "\n", @@ -1185,9 +2490,12 @@ }, { "cell_type": "code", - "execution_count": 7, - "id": "addc476d", - "metadata": {}, + "execution_count": 26, + "id": "03850373", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# to categorical turns our integer vector into a onehot representation\n", @@ -1262,8 +2570,10 @@ }, { "cell_type": "markdown", - "id": "435d5f8e", - "metadata": {}, + "id": "5c4ce702", + "metadata": { + "editable": true + }, "source": [ "## Improving performance\n", "\n", @@ -1281,8 +2591,10 @@ }, { "cell_type": "markdown", - "id": "4d67793b", - "metadata": {}, + "id": "f54c055f", + "metadata": { + "editable": true + }, "source": [ "## Full object-oriented implementation\n", "\n", @@ -1292,9 +2604,12 @@ }, { "cell_type": "code", - "execution_count": 8, - "id": "e189ad94", - "metadata": {}, + "execution_count": 27, + "id": "05fadefb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "class NeuralNetwork:\n", @@ -1400,8 +2715,10 @@ }, { "cell_type": "markdown", - "id": "131559a9", - "metadata": {}, + "id": "6d6bba32", + "metadata": { + "editable": true + }, "source": [ "## Evaluate model performance on test data\n", "\n", @@ -1416,9 +2733,12 @@ }, { "cell_type": "code", - "execution_count": 9, - "id": "6014b15e", - "metadata": {}, + "execution_count": 28, + "id": "5605c90f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "epochs = 100\n", @@ -1441,8 +2761,10 @@ }, { "cell_type": "markdown", - "id": "a215be5c", - "metadata": {}, + "id": "f22bb889", + "metadata": { + "editable": true + }, "source": [ "## Adjust hyperparameters\n", "\n", @@ -1452,9 +2774,12 @@ }, { "cell_type": "code", - "execution_count": 10, - "id": "3c371733", - "metadata": {}, + "execution_count": 29, + "id": "a097c9e1", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "eta_vals = np.logspace(-5, 1, 7)\n", @@ -1481,17 +2806,22 @@ }, { "cell_type": "markdown", - "id": "ed44fd1b", - "metadata": {}, + "id": "bc02f298", + "metadata": { + "editable": true + }, "source": [ "## Visualization" ] }, { "cell_type": "code", - "execution_count": 11, - "id": "38c2057e", - "metadata": {}, + "execution_count": 30, + "id": "970ba03e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# visual representation of grid search\n", @@ -1531,8 +2861,10 @@ }, { "cell_type": "markdown", - "id": "647e5296", - "metadata": {}, + "id": "e4ff0700", + "metadata": { + "editable": true + }, "source": [ "## scikit-learn implementation\n", "\n", @@ -1551,9 +2883,12 @@ }, { "cell_type": "code", - "execution_count": 12, - "id": "277402f6", - "metadata": {}, + "execution_count": 31, + "id": "8bc77d1c", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from sklearn.neural_network import MLPClassifier\n", @@ -1576,17 +2911,22 @@ }, { "cell_type": "markdown", - "id": "cfbf38a6", - "metadata": {}, + "id": "76d38bcf", + "metadata": { + "editable": true + }, "source": [ "## Visualization" ] }, { "cell_type": "code", - "execution_count": 13, - "id": "920c188a", - "metadata": {}, + "execution_count": 32, + "id": "d22894cb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# optional\n", @@ -1627,8 +2967,10 @@ }, { "cell_type": "markdown", - "id": "97323847", - "metadata": {}, + "id": "d4efa31d", + "metadata": { + "editable": true + }, "source": [ "## Building neural networks in Tensorflow and Keras\n", "\n", @@ -1643,8 +2985,10 @@ }, { "cell_type": "markdown", - "id": "c2a7c615", - "metadata": {}, + "id": "d6caddb7", + "metadata": { + "editable": true + }, "source": [ "## Tensorflow\n", "\n", @@ -1675,9 +3019,12 @@ }, { "cell_type": "code", - "execution_count": 14, - "id": "bba703cc", - "metadata": {}, + "execution_count": 33, + "id": "96f112a6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "pip3 install tensorflow" @@ -1685,8 +3032,10 @@ }, { "cell_type": "markdown", - "id": "a4068463", - "metadata": {}, + "id": "2dee5b42", + "metadata": { + "editable": true + }, "source": [ "and/or if you use **anaconda**, just write (or install from the graphical user interface)\n", "(current release of CPU-only TensorFlow)" @@ -1694,9 +3043,12 @@ }, { "cell_type": "code", - "execution_count": 15, - "id": "d6bd0dd0", - "metadata": {}, + "execution_count": 34, + "id": "29e13f69", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "conda create -n tf tensorflow\n", @@ -1705,17 +3057,22 @@ }, { "cell_type": "markdown", - "id": "acc51b3f", - "metadata": {}, + "id": "4695192d", + "metadata": { + "editable": true + }, "source": [ "To install the current release of GPU TensorFlow" ] }, { "cell_type": "code", - "execution_count": 16, - "id": "a0672dbc", - "metadata": {}, + "execution_count": 35, + "id": "573fade6", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "conda create -n tf-gpu tensorflow-gpu\n", @@ -1724,8 +3081,10 @@ }, { "cell_type": "markdown", - "id": "63f07738", - "metadata": {}, + "id": "0d8ced47", + "metadata": { + "editable": true + }, "source": [ "## Using Keras\n", "\n", @@ -1736,9 +3095,12 @@ }, { "cell_type": "code", - "execution_count": 17, - "id": "9a9dc1f6", - "metadata": {}, + "execution_count": 36, + "id": "28886e64", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "conda install keras" @@ -1746,8 +3108,10 @@ }, { "cell_type": "markdown", - "id": "69b5c338", - "metadata": {}, + "id": "298e5186", + "metadata": { + "editable": true + }, "source": [ "You can look up the [instructions here](https://keras.io/) for more information.\n", "\n", @@ -1756,8 +3120,10 @@ }, { "cell_type": "markdown", - "id": "4a78e3e1", - "metadata": {}, + "id": "ea131902", + "metadata": { + "editable": true + }, "source": [ "## Collect and pre-process data\n", "\n", @@ -1766,9 +3132,12 @@ }, { "cell_type": "code", - "execution_count": 18, - "id": "08e4f938", - "metadata": {}, + "execution_count": 37, + "id": "686215bb", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# import necessary packages\n", @@ -1818,9 +3187,12 @@ }, { "cell_type": "code", - "execution_count": 19, - "id": "07f3f725", - "metadata": {}, + "execution_count": 38, + "id": "fa9efbca", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "from tensorflow.keras.layers import Input\n", @@ -1844,9 +3216,12 @@ }, { "cell_type": "code", - "execution_count": 20, - "id": "646f3c64", - "metadata": {}, + "execution_count": 39, + "id": "5a6a0547", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -1871,9 +3246,12 @@ }, { "cell_type": "code", - "execution_count": 21, - "id": "02e48d62", - "metadata": {}, + "execution_count": 40, + "id": "9df608ec", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "DNN_keras = np.zeros((len(eta_vals), len(lmbd_vals)), dtype=object)\n", @@ -1895,9 +3273,12 @@ }, { "cell_type": "code", - "execution_count": 22, - "id": "b870683f", - "metadata": {}, + "execution_count": 41, + "id": "8d20fdf7", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# optional\n", @@ -1935,17 +3316,22 @@ }, { "cell_type": "markdown", - "id": "854ae0f4", - "metadata": {}, + "id": "bd641a2a", + "metadata": { + "editable": true + }, "source": [ "## The Breast Cancer Data, now with Keras" ] }, { "cell_type": "code", - "execution_count": 23, - "id": "4ccfdbf2", - "metadata": {}, + "execution_count": 42, + "id": "a6d8bf75", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "\n", @@ -2118,8 +3504,10 @@ }, { "cell_type": "markdown", - "id": "9b5849c3", - "metadata": {}, + "id": "505033e6", + "metadata": { + "editable": true + }, "source": [ "## Fine-tuning neural network hyperparameters\n", "\n", @@ -2144,8 +3532,10 @@ }, { "cell_type": "markdown", - "id": "d5e48e8e", - "metadata": {}, + "id": "d138fcf8", + "metadata": { + "editable": true + }, "source": [ "## Hidden layers\n", "\n", @@ -2165,8 +3555,10 @@ }, { "cell_type": "markdown", - "id": "58942375", - "metadata": {}, + "id": "df986885", + "metadata": { + "editable": true + }, "source": [ "## Which activation function should I use?\n", "\n", @@ -2194,8 +3586,10 @@ }, { "cell_type": "markdown", - "id": "04de1245", - "metadata": {}, + "id": "67ccb46b", + "metadata": { + "editable": true + }, "source": [ "## Is the Logistic activation function (Sigmoid) our choice?\n", "\n", @@ -2224,8 +3618,10 @@ }, { "cell_type": "markdown", - "id": "a00515d3", - "metadata": {}, + "id": "a043bdea", + "metadata": { + "editable": true + }, "source": [ "## The derivative of the Logistic funtion\n", "\n", @@ -2260,8 +3656,10 @@ }, { "cell_type": "markdown", - "id": "a31e0434", - "metadata": {}, + "id": "ac370685", + "metadata": { + "editable": true + }, "source": [ "## The RELU function family\n", "\n", @@ -2283,8 +3681,10 @@ }, { "cell_type": "markdown", - "id": "200106bb", - "metadata": {}, + "id": "68411972", + "metadata": { + "editable": true + }, "source": [ "$$\n", "ELU(z) = \\left\\{\\begin{array}{cc} \\alpha\\left( \\exp{(z)}-1\\right) & z < 0,\\\\ z & z \\ge 0.\\end{array}\\right.\n", @@ -2293,8 +3693,10 @@ }, { "cell_type": "markdown", - "id": "ec25493e", - "metadata": {}, + "id": "68d05dd2", + "metadata": { + "editable": true + }, "source": [ "## Which activation function should we use?\n", "\n", @@ -2314,8 +3716,10 @@ }, { "cell_type": "markdown", - "id": "3c158df0", - "metadata": {}, + "id": "55c89a89", + "metadata": { + "editable": true + }, "source": [ "## More on activation functions, output layers\n", "\n", @@ -2332,8 +3736,10 @@ }, { "cell_type": "markdown", - "id": "f3320b0b", - "metadata": {}, + "id": "20f0b6de", + "metadata": { + "editable": true + }, "source": [ "## Batch Normalization\n", "\n", @@ -2352,8 +3758,10 @@ }, { "cell_type": "markdown", - "id": "05cf40b3", - "metadata": {}, + "id": "8f0a4e60", + "metadata": { + "editable": true + }, "source": [ "## Dropout\n", "\n", @@ -2368,8 +3776,10 @@ }, { "cell_type": "markdown", - "id": "0d29278d", - "metadata": {}, + "id": "ee9f6778", + "metadata": { + "editable": true + }, "source": [ "## Gradient Clipping\n", "\n", @@ -2385,8 +3795,10 @@ }, { "cell_type": "markdown", - "id": "55e322d2", - "metadata": {}, + "id": "8ec34edc", + "metadata": { + "editable": true + }, "source": [ "## A very nice website on Neural Networks\n", "\n", @@ -2395,8 +3807,10 @@ }, { "cell_type": "markdown", - "id": "c5f566a8", - "metadata": {}, + "id": "4cb4f14f", + "metadata": { + "editable": true + }, "source": [ "## A top-down perspective on Neural networks\n", "\n", @@ -2438,8 +3852,10 @@ }, { "cell_type": "markdown", - "id": "baa5a3ba", - "metadata": {}, + "id": "3ceba030", + "metadata": { + "editable": true + }, "source": [ "## Limitations of supervised learning with deep networks\n", "\n", @@ -2466,8 +3882,10 @@ }, { "cell_type": "markdown", - "id": "9398e7b0", - "metadata": {}, + "id": "fad4c7c0", + "metadata": { + "editable": true + }, "source": [ "## Solving ODEs with Deep Learning\n", "\n", @@ -2491,8 +3909,10 @@ }, { "cell_type": "markdown", - "id": "38ca5c29", - "metadata": {}, + "id": "92edec9a", + "metadata": { + "editable": true + }, "source": [ "## Ordinary Differential Equations\n", "\n", @@ -2503,8 +3923,10 @@ }, { "cell_type": "markdown", - "id": "0c60c696", - "metadata": {}, + "id": "07153151", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2518,8 +3940,10 @@ }, { "cell_type": "markdown", - "id": "f2583e1e", - "metadata": {}, + "id": "31a707ec", + "metadata": { + "editable": true + }, "source": [ "where $g(x)$ is the function to find, and $g^{(n)}(x)$ is the $n$-th derivative of $g(x)$.\n", "\n", @@ -2532,8 +3956,10 @@ }, { "cell_type": "markdown", - "id": "39729d0e", - "metadata": {}, + "id": "d0cbacbc", + "metadata": { + "editable": true + }, "source": [ "## The trial solution\n", "\n", @@ -2542,8 +3968,10 @@ }, { "cell_type": "markdown", - "id": "161e15ef", - "metadata": {}, + "id": "88e6378a", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2558,8 +3986,10 @@ }, { "cell_type": "markdown", - "id": "f9cf300b", - "metadata": {}, + "id": "a85abfe5", + "metadata": { + "editable": true + }, "source": [ "where $h_1(x)$ is a function that makes $g_t(x)$ satisfy a given set\n", "of conditions, $N(x,P)$ a neural network with weights and biases\n", @@ -2577,8 +4007,10 @@ }, { "cell_type": "markdown", - "id": "733ed455", - "metadata": {}, + "id": "28e312e0", + "metadata": { + "editable": true + }, "source": [ "## Minimization process\n", "\n", @@ -2592,8 +4024,10 @@ }, { "cell_type": "markdown", - "id": "f53e9f95", - "metadata": {}, + "id": "7e562a52", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C\\left(x, P\\right) = \\big(f\\left(x, \\, g(x), \\, g'(x), \\, g''(x), \\, \\dots \\, , \\, g^{(n)}(x)\\right)\\big)^2\n", @@ -2602,8 +4036,10 @@ }, { "cell_type": "markdown", - "id": "34517bb4", - "metadata": {}, + "id": "ca872cb5", + "metadata": { + "editable": true + }, "source": [ "If $N$ inputs are given as a vector $\\boldsymbol{x}$ with elements $x_i$ for $i = 1,\\dots,N$,\n", "the cost function becomes" @@ -2611,8 +4047,10 @@ }, { "cell_type": "markdown", - "id": "e02c971a", - "metadata": {}, + "id": "3f5aaf97", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2626,8 +4064,10 @@ }, { "cell_type": "markdown", - "id": "8556f292", - "metadata": {}, + "id": "ea91ec87", + "metadata": { + "editable": true + }, "source": [ "The neural net should then find the parameters $P$ that minimizes the cost function in\n", "([3](#cost)) for a set of $N$ training samples $x_i$." @@ -2635,8 +4075,10 @@ }, { "cell_type": "markdown", - "id": "cca6ee92", - "metadata": {}, + "id": "23d4791e", + "metadata": { + "editable": true + }, "source": [ "## Minimizing the cost function using gradient descent and automatic differentiation\n", "\n", @@ -2649,8 +4091,10 @@ }, { "cell_type": "markdown", - "id": "5c2b6a47", - "metadata": {}, + "id": "645d3042", + "metadata": { + "editable": true + }, "source": [ "## Example: Exponential decay\n", "\n", @@ -2659,8 +4103,10 @@ }, { "cell_type": "markdown", - "id": "10ceefea", - "metadata": {}, + "id": "1957cac1", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2674,8 +4120,10 @@ }, { "cell_type": "markdown", - "id": "7a6beee0", - "metadata": {}, + "id": "7ba994c2", + "metadata": { + "editable": true + }, "source": [ "with $g(0) = g_0$ for some chosen initial value $g_0$.\n", "\n", @@ -2684,8 +4132,10 @@ }, { "cell_type": "markdown", - "id": "53982347", - "metadata": {}, + "id": "0b4d1e86", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2700,16 +4150,20 @@ }, { "cell_type": "markdown", - "id": "ce561b16", - "metadata": {}, + "id": "7c000d4f", + "metadata": { + "editable": true + }, "source": [ "Having an analytical solution at hand, it is possible to use it to compare how well a neural network finds a solution of ([4](#solve_expdec))." ] }, { "cell_type": "markdown", - "id": "b5d5314b", - "metadata": {}, + "id": "5c6e9896", + "metadata": { + "editable": true + }, "source": [ "## The function to solve for\n", "\n", @@ -2718,8 +4172,10 @@ }, { "cell_type": "markdown", - "id": "c4d67b46", - "metadata": {}, + "id": "c610b5a3", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2733,8 +4189,10 @@ }, { "cell_type": "markdown", - "id": "fe827114", - "metadata": {}, + "id": "7b156ba0", + "metadata": { + "editable": true + }, "source": [ "where $g(0) = g_0$ with $\\gamma$ and $g_0$ being some chosen values.\n", "\n", @@ -2743,8 +4201,10 @@ }, { "cell_type": "markdown", - "id": "8ffca9fa", - "metadata": {}, + "id": "6332b4fb", + "metadata": { + "editable": true + }, "source": [ "## The trial solution\n", "To begin with, a trial solution $g_t(t)$ must be chosen. A general trial solution for ordinary differential equations could be" @@ -2752,8 +4212,10 @@ }, { "cell_type": "markdown", - "id": "9ec1e42a", - "metadata": {}, + "id": "dadb48a5", + "metadata": { + "editable": true + }, "source": [ "$$\n", "g_t(x, P) = h_1(x) + h_2(x, N(x, P))\n", @@ -2762,16 +4224,20 @@ }, { "cell_type": "markdown", - "id": "a002d557", - "metadata": {}, + "id": "14979de9", + "metadata": { + "editable": true + }, "source": [ "with $h_1(x)$ ensuring that $g_t(x)$ satisfies some conditions and $h_2(x,N(x, P))$ an expression involving $x$ and the output from the neural network $N(x,P)$ with $P $ being the collection of the weights and biases for each layer. For now, it is assumed that the network consists of one input layer, one hidden layer, and one output layer." ] }, { "cell_type": "markdown", - "id": "760f494e", - "metadata": {}, + "id": "b15978ac", + "metadata": { + "editable": true + }, "source": [ "## Setup of Network\n", "\n", @@ -2788,8 +4254,10 @@ }, { "cell_type": "markdown", - "id": "5f15f368", - "metadata": {}, + "id": "ab70d7dc", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2803,8 +4271,10 @@ }, { "cell_type": "markdown", - "id": "c91d6d7c", - "metadata": {}, + "id": "3158ad88", + "metadata": { + "editable": true + }, "source": [ "## Reformulating the problem\n", "\n", @@ -2820,8 +4290,10 @@ }, { "cell_type": "markdown", - "id": "bd6ed5a3", - "metadata": {}, + "id": "424ea4cd", + "metadata": { + "editable": true + }, "source": [ "$$\n", "g_t(x, P) = g_0 + x \\cdot N(x, P)\n", @@ -2830,16 +4302,20 @@ }, { "cell_type": "markdown", - "id": "06a70c9a", - "metadata": {}, + "id": "9875f210", + "metadata": { + "editable": true + }, "source": [ "has been chosen such that it already solves the condition $g(0) = g_0$. What remains, is to find $P$ such that" ] }, { "cell_type": "markdown", - "id": "6d20f780", - "metadata": {}, + "id": "c317c081", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2853,16 +4329,20 @@ }, { "cell_type": "markdown", - "id": "9546abd9", - "metadata": {}, + "id": "850d7062", + "metadata": { + "editable": true + }, "source": [ "is fulfilled as *best as possible*." ] }, { "cell_type": "markdown", - "id": "e4073ebd", - "metadata": {}, + "id": "05b6c21e", + "metadata": { + "editable": true + }, "source": [ "## More technicalities\n", "\n", @@ -2875,8 +4355,10 @@ }, { "cell_type": "markdown", - "id": "2dfb903c", - "metadata": {}, + "id": "52090d6f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\min_{P}\\Big\\{ \\big(g_t'(x, P) - ( -\\gamma g_t(x, P) \\big)^2 \\Big\\}\n", @@ -2885,8 +4367,10 @@ }, { "cell_type": "markdown", - "id": "b1890129", - "metadata": {}, + "id": "f4f56b6e", + "metadata": { + "editable": true + }, "source": [ "(the notation $\\min_{P}\\{ f(x, P) \\}$ means that we desire to find $P$ that yields the minimum of $f(x, P)$)\n", "\n", @@ -2895,8 +4379,10 @@ }, { "cell_type": "markdown", - "id": "a787ccbe", - "metadata": {}, + "id": "e76317e2", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\min_{P_{\\text{hidden} }, \\ P_{\\text{output} }}\\Big\\{ \\big(g_t'(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) - ( -\\gamma g_t(x, \\{ P_{\\text{hidden} }, P_{\\text{output} }\\}) \\big)^2 \\Big\\}\n", @@ -2905,16 +4391,20 @@ }, { "cell_type": "markdown", - "id": "f0a4fa04", - "metadata": {}, + "id": "cf1ff27d", + "metadata": { + "editable": true + }, "source": [ "for an input value $x$." ] }, { "cell_type": "markdown", - "id": "94e4bd68", - "metadata": {}, + "id": "c6558ca7", + "metadata": { + "editable": true + }, "source": [ "## More details\n", "\n", @@ -2923,8 +4413,10 @@ }, { "cell_type": "markdown", - "id": "1c418fe5", - "metadata": {}, + "id": "eb10b3cf", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -2938,16 +4430,20 @@ }, { "cell_type": "markdown", - "id": "92fee060", - "metadata": {}, + "id": "95040138", + "metadata": { + "editable": true + }, "source": [ "Letting $\\boldsymbol{x}$ be a vector with elements $x_i$ and $C(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2$ denote the cost function, the minimization problem that our network must solve, becomes" ] }, { "cell_type": "markdown", - "id": "81b76e58", - "metadata": {}, + "id": "efbb37df", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\min_{P} C(\\boldsymbol{x}, P)\n", @@ -2956,8 +4452,10 @@ }, { "cell_type": "markdown", - "id": "2d3f26a5", - "metadata": {}, + "id": "ce339463", + "metadata": { + "editable": true + }, "source": [ "In terms of $P_{\\text{hidden} }$ and $P_{\\text{output} }$, this could also be expressed as\n", "\n", @@ -2968,8 +4466,10 @@ }, { "cell_type": "markdown", - "id": "872160e6", - "metadata": {}, + "id": "51d1021d", + "metadata": { + "editable": true + }, "source": [ "## A possible implementation of a neural network\n", "\n", @@ -2982,8 +4482,10 @@ }, { "cell_type": "markdown", - "id": "f7c201c9", - "metadata": {}, + "id": "5471c13f", + "metadata": { + "editable": true + }, "source": [ "## Technicalities\n", "\n", @@ -2992,8 +4494,10 @@ }, { "cell_type": "markdown", - "id": "67b3b2c5", - "metadata": {}, + "id": "ca1015ba", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -3012,8 +4516,10 @@ }, { "cell_type": "markdown", - "id": "4b0c1a9f", - "metadata": {}, + "id": "325048d9", + "metadata": { + "editable": true + }, "source": [ "## Final technicalities I\n", "\n", @@ -3022,8 +4528,10 @@ }, { "cell_type": "markdown", - "id": "7d711054", - "metadata": {}, + "id": "5c6b185f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -3043,8 +4551,10 @@ }, { "cell_type": "markdown", - "id": "959f1555", - "metadata": {}, + "id": "eddea0e3", + "metadata": { + "editable": true + }, "source": [ "## Final technicalities II\n", "\n", @@ -3057,8 +4567,10 @@ }, { "cell_type": "markdown", - "id": "0ee5702b", - "metadata": {}, + "id": "39eeb7a1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "f(z) = \\frac{1}{1 + \\exp{(-z)}}\n", @@ -3067,8 +4579,10 @@ }, { "cell_type": "markdown", - "id": "86386ed5", - "metadata": {}, + "id": "25477c00", + "metadata": { + "editable": true + }, "source": [ "It is possible to use other activations functions for the hidden layer also.\n", "\n", @@ -3089,8 +4603,10 @@ }, { "cell_type": "markdown", - "id": "792a7767", - "metadata": {}, + "id": "dbdce765", + "metadata": { + "editable": true + }, "source": [ "## Final technicalities III\n", "\n", @@ -3099,8 +4615,10 @@ }, { "cell_type": "markdown", - "id": "debfdf7d", - "metadata": {}, + "id": "3b833783", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -3118,8 +4636,10 @@ }, { "cell_type": "markdown", - "id": "cec57344", - "metadata": {}, + "id": "98103cac", + "metadata": { + "editable": true + }, "source": [ "## Final technicalities IV\n", "\n", @@ -3128,8 +4648,10 @@ }, { "cell_type": "markdown", - "id": "fc406bc4", - "metadata": {}, + "id": "01de1e28", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{z}_{1}^{\\text{output}} =\n", @@ -3145,16 +4667,20 @@ }, { "cell_type": "markdown", - "id": "8280357c", - "metadata": {}, + "id": "3d50cf6a", + "metadata": { + "editable": true + }, "source": [ "In this case we seek a continuous range of values since we are approximating a function. This means that after computing $\\boldsymbol{z}_{1}^{\\text{output}}$ the neural network has finished its feed forward step, and $\\boldsymbol{z}_{1}^{\\text{output}}$ is the final output of the network." ] }, { "cell_type": "markdown", - "id": "e682cf5d", - "metadata": {}, + "id": "221269b0", + "metadata": { + "editable": true + }, "source": [ "## Back propagation\n", "\n", @@ -3165,8 +4691,10 @@ }, { "cell_type": "markdown", - "id": "86c630c3", - "metadata": {}, + "id": "b736af88", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C(\\boldsymbol{x}, P) = \\frac{1}{N} \\sum_i \\big(g_t'(x_i, P) - ( -\\gamma g_t(x_i, P) \\big)^2\n", @@ -3175,8 +4703,10 @@ }, { "cell_type": "markdown", - "id": "64bc4c3a", - "metadata": {}, + "id": "ea505169", + "metadata": { + "editable": true + }, "source": [ "In order to minimize the cost function, an optimization method must be chosen.\n", "\n", @@ -3185,8 +4715,10 @@ }, { "cell_type": "markdown", - "id": "329b6929", - "metadata": {}, + "id": "515470a3", + "metadata": { + "editable": true + }, "source": [ "## Gradient descent\n", "\n", @@ -3200,8 +4732,10 @@ }, { "cell_type": "markdown", - "id": "ffdb4b35", - "metadata": {}, + "id": "f9eaeace", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\boldsymbol{\\omega}_{\\text{new} } = \\boldsymbol{\\omega} - \\lambda \\nabla_{\\boldsymbol{\\omega}} C(\\boldsymbol{x}, \\boldsymbol{\\omega})\n", @@ -3210,8 +4744,10 @@ }, { "cell_type": "markdown", - "id": "972f3fe7", - "metadata": {}, + "id": "e6488f9b", + "metadata": { + "editable": true + }, "source": [ "for a number of iterations or until $ \\big|\\big| \\boldsymbol{\\omega}_{\\text{new} } - \\boldsymbol{\\omega} \\big|\\big|$ becomes smaller than some given tolerance.\n", "\n", @@ -3230,8 +4766,10 @@ }, { "cell_type": "markdown", - "id": "26f47c23", - "metadata": {}, + "id": "f2bc476e", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -3243,17 +4781,22 @@ }, { "cell_type": "markdown", - "id": "46e2f864", - "metadata": {}, + "id": "2d7b7d19", + "metadata": { + "editable": true + }, "source": [ "## The code for solving the ODE" ] }, { "cell_type": "code", - "execution_count": 24, - "id": "5c9ef3e2", - "metadata": {}, + "execution_count": 43, + "id": "4aad6e10", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3404,8 +4947,10 @@ }, { "cell_type": "markdown", - "id": "9541ca3b", - "metadata": {}, + "id": "f8c418c1", + "metadata": { + "editable": true + }, "source": [ "## The network with one input layer, specified number of hidden layers, and one output layer\n", "\n", @@ -3416,9 +4961,12 @@ }, { "cell_type": "code", - "execution_count": 25, - "id": "cb1f580a", - "metadata": {}, + "execution_count": 44, + "id": "96758ada", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3583,8 +5131,10 @@ }, { "cell_type": "markdown", - "id": "1ce3dba0", - "metadata": {}, + "id": "c62fedb8", + "metadata": { + "editable": true + }, "source": [ "## Example: Population growth\n", "\n", @@ -3594,8 +5144,10 @@ }, { "cell_type": "markdown", - "id": "bd8d79a2", - "metadata": {}, + "id": "1f962431", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -3609,8 +5161,10 @@ }, { "cell_type": "markdown", - "id": "3a2583e3", - "metadata": {}, + "id": "71b1d885", + "metadata": { + "editable": true + }, "source": [ "where $g(t)$ is the population density at time $t$, $\\alpha > 0$ the growth rate and $A > 0$ is the maximum population number in the environment.\n", "Also, at $t = 0$ the population has the size $g(0) = g_0$, where $g_0$ is some chosen constant.\n", @@ -3623,8 +5177,10 @@ }, { "cell_type": "markdown", - "id": "bda6ba67", - "metadata": {}, + "id": "0b96a028", + "metadata": { + "editable": true + }, "source": [ "## Setting up the problem\n", "\n", @@ -3634,8 +5190,10 @@ }, { "cell_type": "markdown", - "id": "1483f8e7", - "metadata": {}, + "id": "311ae3d3", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -3649,8 +5207,10 @@ }, { "cell_type": "markdown", - "id": "82aa5d57", - "metadata": {}, + "id": "98cb3d44", + "metadata": { + "editable": true + }, "source": [ "where $g(0) = g_0$.\n", "\n", @@ -3659,8 +5219,10 @@ }, { "cell_type": "markdown", - "id": "ba1d1722", - "metadata": {}, + "id": "3e8e134d", + "metadata": { + "editable": true + }, "source": [ "## The trial solution\n", "\n", @@ -3684,8 +5246,10 @@ }, { "cell_type": "markdown", - "id": "58ccc1eb", - "metadata": {}, + "id": "a1844ff9", + "metadata": { + "editable": true + }, "source": [ "## The program using Autograd\n", "\n", @@ -3694,9 +5258,12 @@ }, { "cell_type": "code", - "execution_count": 26, - "id": "0a4c20d1", - "metadata": {}, + "execution_count": 45, + "id": "4410b99e", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -3866,8 +5433,10 @@ }, { "cell_type": "markdown", - "id": "97450de8", - "metadata": {}, + "id": "f0ab0e0f", + "metadata": { + "editable": true + }, "source": [ "## Using forward Euler to solve the ODE\n", "\n", @@ -3884,8 +5453,10 @@ }, { "cell_type": "markdown", - "id": "89c1f1e5", - "metadata": {}, + "id": "333b0a8d", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -3897,8 +5468,10 @@ }, { "cell_type": "markdown", - "id": "fa3644af", - "metadata": {}, + "id": "5b358fbf", + "metadata": { + "editable": true + }, "source": [ "along with the condition that $g(0) = g_0$.\n", "\n", @@ -3909,8 +5482,10 @@ }, { "cell_type": "markdown", - "id": "107bef62", - "metadata": {}, + "id": "c5d60864", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -3923,16 +5498,20 @@ }, { "cell_type": "markdown", - "id": "65796f1d", - "metadata": {}, + "id": "bcce6eb8", + "metadata": { + "editable": true + }, "source": [ "Now, if $g_i = g(t_i)$ then" ] }, { "cell_type": "markdown", - "id": "fdb684ca", - "metadata": {}, + "id": "e6497fd7", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -3951,8 +5530,10 @@ }, { "cell_type": "markdown", - "id": "405eb53d", - "metadata": {}, + "id": "31b650fc", + "metadata": { + "editable": true + }, "source": [ "for $i \\geq 1$ and $g_0 = g(t_0) = g(0) = g_0$.\n", "\n", @@ -3962,9 +5543,12 @@ }, { "cell_type": "code", - "execution_count": 27, - "id": "a903f1cf", - "metadata": {}, + "execution_count": 46, + "id": "6e267e72", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Assume that all function definitions from the example program using Autograd\n", @@ -4036,8 +5620,10 @@ }, { "cell_type": "markdown", - "id": "f79a730a", - "metadata": {}, + "id": "8b054a85", + "metadata": { + "editable": true + }, "source": [ "## Example: Solving the one dimensional Poisson equation\n", "\n", @@ -4046,8 +5632,10 @@ }, { "cell_type": "markdown", - "id": "28015c61", - "metadata": {}, + "id": "f637dd9d", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -4061,8 +5649,10 @@ }, { "cell_type": "markdown", - "id": "54ab04bb", - "metadata": {}, + "id": "ac088e86", + "metadata": { + "editable": true + }, "source": [ "where $f(x)$ is a given function for $x \\in (0,1)$.\n", "\n", @@ -4071,8 +5661,10 @@ }, { "cell_type": "markdown", - "id": "9cc20be6", - "metadata": {}, + "id": "14afd7fa", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -4084,8 +5676,10 @@ }, { "cell_type": "markdown", - "id": "7af7511b", - "metadata": {}, + "id": "0a2680eb", + "metadata": { + "editable": true + }, "source": [ "This equation can be solved numerically using programs where e.g Autograd and TensorFlow are used.\n", "The results from the networks can then be compared to the analytical solution.\n", @@ -4094,8 +5688,10 @@ }, { "cell_type": "markdown", - "id": "96856fc2", - "metadata": {}, + "id": "7e30575f", + "metadata": { + "editable": true + }, "source": [ "## The specific equation to solve for\n", "\n", @@ -4104,8 +5700,10 @@ }, { "cell_type": "markdown", - "id": "1cf407bd", - "metadata": {}, + "id": "3b30e08a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "-g''(x) = f(x),\\qquad x \\in (0,1)\n", @@ -4114,16 +5712,20 @@ }, { "cell_type": "markdown", - "id": "493a79a2", - "metadata": {}, + "id": "be7d958b", + "metadata": { + "editable": true + }, "source": [ "where $f(x)$ is a given function, along with the chosen conditions" ] }, { "cell_type": "markdown", - "id": "8b735b17", - "metadata": {}, + "id": "d6a52e39", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -4137,8 +5739,10 @@ }, { "cell_type": "markdown", - "id": "39ddedfe", - "metadata": {}, + "id": "488d0a14", + "metadata": { + "editable": true + }, "source": [ "In this example, we consider the case when $f(x) = (3x + x^2)\\exp(x)$.\n", "\n", @@ -4147,8 +5751,10 @@ }, { "cell_type": "markdown", - "id": "46bba0f0", - "metadata": {}, + "id": "59834674", + "metadata": { + "editable": true + }, "source": [ "$$\n", "g_t(x) = x \\cdot (1-x) \\cdot N(P,x)\n", @@ -4157,16 +5763,20 @@ }, { "cell_type": "markdown", - "id": "1ba4c8b3", - "metadata": {}, + "id": "af640638", + "metadata": { + "editable": true + }, "source": [ "The analytical solution for this problem is" ] }, { "cell_type": "markdown", - "id": "e8faf0e1", - "metadata": {}, + "id": "8d83ea23", + "metadata": { + "editable": true + }, "source": [ "$$\n", "g(x) = x(1 - x)\\exp(x)\n", @@ -4175,17 +5785,22 @@ }, { "cell_type": "markdown", - "id": "e4ce498c", - "metadata": {}, + "id": "4d4e4ea8", + "metadata": { + "editable": true + }, "source": [ "## Solving the equation using Autograd" ] }, { "cell_type": "code", - "execution_count": 28, - "id": "1d1cb692", - "metadata": {}, + "execution_count": 47, + "id": "ad4cc32f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4342,8 +5957,10 @@ }, { "cell_type": "markdown", - "id": "16aa2193", - "metadata": {}, + "id": "ceb283ce", + "metadata": { + "editable": true + }, "source": [ "## Comparing with a numerical scheme\n", "\n", @@ -4362,8 +5979,10 @@ }, { "cell_type": "markdown", - "id": "b3ef1c2b", - "metadata": {}, + "id": "a68e0e56", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -4377,16 +5996,20 @@ }, { "cell_type": "markdown", - "id": "0e1c476f", - "metadata": {}, + "id": "8441e4e6", + "metadata": { + "editable": true + }, "source": [ "If $x_i = i \\Delta x = x_{i-1} + \\Delta x$ and $g_i = g(x_i)$ for $i = 1,\\dots N_x - 2$ with $N_x$ being the number of values for $x$, ([15](#approx)) becomes" ] }, { "cell_type": "markdown", - "id": "7b2ad33d", - "metadata": {}, + "id": "1d343e68", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -4398,16 +6021,20 @@ }, { "cell_type": "markdown", - "id": "1c57f992", - "metadata": {}, + "id": "1c566531", + "metadata": { + "editable": true + }, "source": [ "Since we know from our problem that" ] }, { "cell_type": "markdown", - "id": "99e9dc9f", - "metadata": {}, + "id": "7a06b755", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -4419,8 +6046,10 @@ }, { "cell_type": "markdown", - "id": "ecc62734", - "metadata": {}, + "id": "2e88c504", + "metadata": { + "editable": true + }, "source": [ "along with the conditions $g(0) = g(1) = 0$,\n", "the following scheme can be used to find an approximate solution for $g(x)$ numerically:" @@ -4428,8 +6057,10 @@ }, { "cell_type": "markdown", - "id": "2d467ec2", - "metadata": {}, + "id": "7299b0d3", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -4446,8 +6077,10 @@ }, { "cell_type": "markdown", - "id": "2de58f3a", - "metadata": {}, + "id": "c849f0cc", + "metadata": { + "editable": true + }, "source": [ "for $i = 1, \\dots, N_x - 2$ where $g_0 = g_{N_x - 1} = 0$ and $f(x_i) = (3x_i + x_i^2)\\exp(x_i)$, which is given for our specific problem.\n", "\n", @@ -4456,8 +6089,10 @@ }, { "cell_type": "markdown", - "id": "0182bb77", - "metadata": {}, + "id": "f19b2ed7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{aligned}\n", @@ -4491,16 +6126,20 @@ }, { "cell_type": "markdown", - "id": "7ef6f3b0", - "metadata": {}, + "id": "96581350", + "metadata": { + "editable": true + }, "source": [ "which makes it possible to solve for the vector $\\boldsymbol{g}$." ] }, { "cell_type": "markdown", - "id": "6414011b", - "metadata": {}, + "id": "2c5aa09d", + "metadata": { + "editable": true + }, "source": [ "## Setting up the code\n", "\n", @@ -4509,9 +6148,12 @@ }, { "cell_type": "code", - "execution_count": 29, - "id": "dc3a8d10", - "metadata": {}, + "execution_count": 48, + "id": "9b979206", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -4708,8 +6350,10 @@ }, { "cell_type": "markdown", - "id": "4b05b79c", - "metadata": {}, + "id": "ec7cab35", + "metadata": { + "editable": true + }, "source": [ "## Partial Differential Equations\n", "\n", @@ -4723,8 +6367,10 @@ }, { "cell_type": "markdown", - "id": "0f1b8d34", - "metadata": {}, + "id": "7366658f", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -4738,16 +6384,20 @@ }, { "cell_type": "markdown", - "id": "358a4063", - "metadata": {}, + "id": "b6557284", + "metadata": { + "editable": true + }, "source": [ "where $f$ is an expression involving all kinds of possible mixed derivatives of $g(x_1,\\dots,x_N)$ up to an order $n$. In order for the solution to be unique, some additional conditions must also be given." ] }, { "cell_type": "markdown", - "id": "e4009b2b", - "metadata": {}, + "id": "df2d7d07", + "metadata": { + "editable": true + }, "source": [ "## Type of problem\n", "\n", @@ -4759,8 +6409,10 @@ }, { "cell_type": "markdown", - "id": "c5a9ee8a", - "metadata": {}, + "id": "aad0506a", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -4771,8 +6423,10 @@ }, { "cell_type": "markdown", - "id": "b300ac7d", - "metadata": {}, + "id": "9e5776d1", + "metadata": { + "editable": true + }, "source": [ "where $h_1(x_1,\\dots,x_N)$ is a function that ensures $g_t(x_1,\\dots,x_N)$ satisfies some given conditions.\n", "The neural network $N(x_1,\\dots,x_N,P)$ has weights and biases described by $P$ and $h_2(x_1,\\dots,x_N,N(x_1,\\dots,x_N,P))$ is an expression using the output from the neural network in some way.\n", @@ -4782,8 +6436,10 @@ }, { "cell_type": "markdown", - "id": "ee3d4dba", - "metadata": {}, + "id": "7fad71c5", + "metadata": { + "editable": true + }, "source": [ "## Network requirements\n", "\n", @@ -4800,8 +6456,10 @@ }, { "cell_type": "markdown", - "id": "73ef05ff", - "metadata": {}, + "id": "d98bd52b", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C\\left(x_1, \\dots, x_N, P\\right) = \\left( f\\left(x_1, \\, \\dots \\, , x_N, \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_1}, \\dots , \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_N}, \\frac{\\partial g(x_1,\\dots,x_N) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(x_1,\\dots,x_N) }{\\partial x_N^n} \\right) \\right)^2\n", @@ -4810,8 +6468,10 @@ }, { "cell_type": "markdown", - "id": "bc7a6597", - "metadata": {}, + "id": "3f6cb731", + "metadata": { + "editable": true + }, "source": [ "## More details\n", "\n", @@ -4820,8 +6480,10 @@ }, { "cell_type": "markdown", - "id": "a158dec6", - "metadata": {}, + "id": "0efeb6a7", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C\\left(\\boldsymbol{x}, P\\right) = f\\left( \\left( \\boldsymbol{x}, \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x}) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x}) }{\\partial x_N^n} \\right) \\right)^2\n", @@ -4830,16 +6492,20 @@ }, { "cell_type": "markdown", - "id": "6cbfa1b1", - "metadata": {}, + "id": "b3367fd8", + "metadata": { + "editable": true + }, "source": [ "If we also have $M$ different sets of values for $x_1, \\dots, x_N$, that is $\\boldsymbol{x}_i = \\big(x_1^{(i)}, \\dots, x_N^{(i)}\\big)$ for $i = 1,\\dots,M$ being the rows in matrix $X$, the cost function can be generalized into" ] }, { "cell_type": "markdown", - "id": "a3af3492", - "metadata": {}, + "id": "f5cdf037", + "metadata": { + "editable": true + }, "source": [ "$$\n", "C\\left(X, P \\right) = \\sum_{i=1}^M f\\left( \\left( \\boldsymbol{x}_i, \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_1}, \\dots , \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_N}, \\frac{\\partial g(\\boldsymbol{x}_i) }{\\partial x_1\\partial x_2}, \\, \\dots \\, , \\frac{\\partial^n g(\\boldsymbol{x}_i) }{\\partial x_N^n} \\right) \\right)^2.\n", @@ -4848,8 +6514,10 @@ }, { "cell_type": "markdown", - "id": "ccd61199", - "metadata": {}, + "id": "39090b0c", + "metadata": { + "editable": true + }, "source": [ "## Example: The diffusion equation\n", "\n", @@ -4858,8 +6526,10 @@ }, { "cell_type": "markdown", - "id": "dc36a5cb", - "metadata": {}, + "id": "306f52ce", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial g(x,t)}{\\partial t} = \\frac{\\partial^2 g(x,t)}{\\partial x^2}\n", @@ -4868,16 +6538,20 @@ }, { "cell_type": "markdown", - "id": "a8b26da0", - "metadata": {}, + "id": "4dddd0e4", + "metadata": { + "editable": true + }, "source": [ "where a possible choice of conditions are" ] }, { "cell_type": "markdown", - "id": "41406843", - "metadata": {}, + "id": "19ba57a3", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -4890,16 +6564,20 @@ }, { "cell_type": "markdown", - "id": "f45dd83f", - "metadata": {}, + "id": "3f4b369e", + "metadata": { + "editable": true + }, "source": [ "with $u(x)$ being some given function." ] }, { "cell_type": "markdown", - "id": "e30bdb98", - "metadata": {}, + "id": "01054c66", + "metadata": { + "editable": true + }, "source": [ "## Defining the problem\n", "\n", @@ -4908,8 +6586,10 @@ }, { "cell_type": "markdown", - "id": "f9cdb4e5", - "metadata": {}, + "id": "48250dfa", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -4923,16 +6603,20 @@ }, { "cell_type": "markdown", - "id": "16dc73b3", - "metadata": {}, + "id": "830538cd", + "metadata": { + "editable": true + }, "source": [ "and" ] }, { "cell_type": "markdown", - "id": "ee4dbc98", - "metadata": {}, + "id": "1e05f3d1", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -4945,8 +6629,10 @@ }, { "cell_type": "markdown", - "id": "13748484", - "metadata": {}, + "id": "278b7644", + "metadata": { + "editable": true + }, "source": [ "with $u(x) = \\sin(\\pi x)$.\n", "\n", @@ -4957,8 +6643,10 @@ }, { "cell_type": "markdown", - "id": "4b26b938", - "metadata": {}, + "id": "08bf068d", + "metadata": { + "editable": true + }, "source": [ "## Setting up the network using Autograd\n", "\n", @@ -4973,9 +6661,12 @@ }, { "cell_type": "code", - "execution_count": 30, - "id": "9e3bba2f", - "metadata": {}, + "execution_count": 49, + "id": "fb15b704", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "def sigmoid(z):\n", @@ -5026,8 +6717,10 @@ }, { "cell_type": "markdown", - "id": "c4fffb7e", - "metadata": {}, + "id": "ca43eaaa", + "metadata": { + "editable": true + }, "source": [ "## Setting up the network using Autograd; The trial solution\n", "\n", @@ -5054,8 +6747,10 @@ }, { "cell_type": "markdown", - "id": "3ee8799d", - "metadata": {}, + "id": "8cb99e50", + "metadata": { + "editable": true + }, "source": [ "## Why the jacobian?\n", "\n", @@ -5080,9 +6775,12 @@ }, { "cell_type": "code", - "execution_count": 31, - "id": "8a2377ff", - "metadata": {}, + "execution_count": 50, + "id": "2c311458", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "# Set up the trial function:\n", @@ -5125,8 +6823,10 @@ }, { "cell_type": "markdown", - "id": "8dbd62e1", - "metadata": {}, + "id": "6ce6fcdb", + "metadata": { + "editable": true + }, "source": [ "## Setting up the network using Autograd; The full program\n", "\n", @@ -5148,9 +6848,12 @@ }, { "cell_type": "code", - "execution_count": 32, - "id": "61b77dd3", - "metadata": {}, + "execution_count": 51, + "id": "975ad28f", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -5380,8 +7083,10 @@ }, { "cell_type": "markdown", - "id": "41169c76", - "metadata": {}, + "id": "a85b7bc6", + "metadata": { + "editable": true + }, "source": [ "## Example: Solving the wave equation with Neural Networks\n", "\n", @@ -5390,8 +7095,10 @@ }, { "cell_type": "markdown", - "id": "3bc273de", - "metadata": {}, + "id": "913742fb", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\frac{\\partial^2 g(x,t)}{\\partial t^2} = c^2\\frac{\\partial^2 g(x,t)}{\\partial x^2}\n", @@ -5400,8 +7107,10 @@ }, { "cell_type": "markdown", - "id": "37896dc3", - "metadata": {}, + "id": "9b936d66", + "metadata": { + "editable": true + }, "source": [ "with $c$ being the specified wave speed.\n", "\n", @@ -5410,8 +7119,10 @@ }, { "cell_type": "markdown", - "id": "09477c51", - "metadata": {}, + "id": "a0cd719f", + "metadata": { + "editable": true + }, "source": [ "$$\n", "\\begin{align*}\n", @@ -5425,16 +7136,20 @@ }, { "cell_type": "markdown", - "id": "93579057", - "metadata": {}, + "id": "8a8ddb73", + "metadata": { + "editable": true + }, "source": [ "where $\\frac{\\partial g(x,t)}{\\partial t} \\Big |_{t = 0}$ means the derivative of $g(x,t)$ with respect to $t$ is evaluated at $t = 0$, and $u(x)$ and $v(x)$ being given functions." ] }, { "cell_type": "markdown", - "id": "dcbb82f6", - "metadata": {}, + "id": "460bb8e2", + "metadata": { + "editable": true + }, "source": [ "## The problem to solve for\n", "\n", @@ -5443,8 +7158,10 @@ }, { "cell_type": "markdown", - "id": "f91e973a", - "metadata": {}, + "id": "619814ea", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -5458,8 +7175,10 @@ }, { "cell_type": "markdown", - "id": "e13ab50c", - "metadata": {}, + "id": "833fef22", + "metadata": { + "editable": true + }, "source": [ "where $c$ is the given wave speed.\n", "The chosen conditions for this equation are" @@ -5467,8 +7186,10 @@ }, { "cell_type": "markdown", - "id": "fca02e71", - "metadata": {}, + "id": "72df2582", + "metadata": { + "editable": true + }, "source": [ "\n", "
\n", @@ -5485,16 +7206,20 @@ }, { "cell_type": "markdown", - "id": "4bb32ee3", - "metadata": {}, + "id": "efc2be56", + "metadata": { + "editable": true + }, "source": [ "In this example, let $c = 1$ and $u(x) = \\sin(\\pi x)$ and $v(x) = -\\pi\\sin(\\pi x)$." ] }, { "cell_type": "markdown", - "id": "feea59f2", - "metadata": {}, + "id": "8bee39ce", + "metadata": { + "editable": true + }, "source": [ "## The trial solution\n", "Setting up the network is done in similar matter as for the example of solving the diffusion equation.\n", @@ -5517,8 +7242,10 @@ }, { "cell_type": "markdown", - "id": "880753aa", - "metadata": {}, + "id": "e133bda5", + "metadata": { + "editable": true + }, "source": [ "## The analytical solution\n", "\n", @@ -5531,17 +7258,22 @@ }, { "cell_type": "markdown", - "id": "42a69cb7", - "metadata": {}, + "id": "8cc804aa", + "metadata": { + "editable": true + }, "source": [ "## Solving the wave equation - the full program using Autograd" ] }, { "cell_type": "code", - "execution_count": 33, - "id": "1b321e14", - "metadata": {}, + "execution_count": 52, + "id": "8e3fb51d", + "metadata": { + "collapsed": false, + "editable": true + }, "outputs": [], "source": [ "import autograd.numpy as np\n", @@ -5768,8 +7500,10 @@ }, { "cell_type": "markdown", - "id": "c81c7c8a", - "metadata": {}, + "id": "a77decf5", + "metadata": { + "editable": true + }, "source": [ "## Resources on differential equations and deep learning\n", "\n", @@ -5783,25 +7517,7 @@ ] } ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.10" - } - }, + "metadata": {}, "nbformat": 4, "nbformat_minor": 5 } diff --git a/doc/LectureNotes/_build/html/exercisesweek43.html b/doc/LectureNotes/_build/html/exercisesweek43.html index 25a596f47..120ef889f 100644 --- a/doc/LectureNotes/_build/html/exercisesweek43.html +++ b/doc/LectureNotes/_build/html/exercisesweek43.html @@ -451,6 +451,11 @@ const thebe_selector_output = ".output, .cell_output" Representing the Data Sets +
  • + + Setting up dimensionalities by hand + +
  • Setting up the Neural Network @@ -498,6 +503,11 @@ const thebe_selector_output = ".output, .cell_output"
    • +
  • + + Testing the XOR gate and other gates + +
    • @@ -540,6 +550,11 @@ const thebe_selector_output = ".output, .cell_output" Representing the Data Sets +
  • + + Setting up dimensionalities by hand + +
  • Setting up the Neural Network @@ -587,6 +602,11 @@ const thebe_selector_output = ".output, .cell_output"
    • +
  • + + Testing the XOR gate and other gates + +
    • @@ -683,6 +703,60 @@ inputs \(x_1\) and

    Everything you develop here can be used directly into the code for the project.

    +
    +

    Setting up dimensionalities by hand¶

    +

    It can be useful to test the dimensionalities for the network. Let us assume we have performed an optimization for XOR gate and found that the weights for the hidden layer are given by

    +
    +\[\begin{split} +\boldsymbol{W_h}=\begin{bmatrix} 1 & 1 \\ + 1 & 1 \end{bmatrix}, +\end{split}\]
    +

    Multiplying \(\boldsymbol{X}\) and \(\boldsymbol{W}\) gives

    +
    +\[\begin{split} +\boldsymbol{X}{W}_h=\begin{bmatrix} 0 & 0 \\ + 1 & 1 \\ + 1 & 1 \\ + 2 & 2 \end{bmatrix}, +\end{split}\]
    +

    Assume also that the bias vector for the hidden layer is

    +
    +\[\begin{split} +\boldsymbol{b}_h=\begin{bmatrix} 0 \\ + -1\end{bmatrix}, +\end{split}\]
    +

    Adding it gives us the input to the activation function of the hidden layer

    +
    +\[\begin{split} +\boldsymbol{z}_h=\boldsymbol{X}\boldsymbol{W}_h+\boldsymbol{b}_h=\begin{bmatrix} 0 & -1 \\ + 1 & 0 \\ + 1 & 0 \\ + 2 & 1 \end{bmatrix}, +\end{split}\]
    +

    Let us then assume that our activation function is the RELU function, which simply means that we take the max of \(0\) and the elements of the input argument \(\boldsymbol{z}_h\), that is we have

    +
    +\[\begin{split} +\boldsymbol{a}_h=\mathrm{RELU}(\boldsymbol{z}_h=\boldsymbol{X}\boldsymbol{W}_h+\boldsymbol{b}_h)=\begin{bmatrix} 0 & 0 \\ + 1 & 0 \\ + 1 & 0 \\ + 2 & 1 \end{bmatrix}, +\end{split}\]
    +

    Assume also that the bias of the output layer is zero and that the weights of the output layer are

    +
    +\[\begin{split} +\boldsymbol{w}_o=\begin{bmatrix} 1 \\ + -2\end{bmatrix}, +\end{split}\]
    +

    and multiplying with \(\boldsymbol{a}_h\) gives the output

    +
    +\[\begin{split} +\boldsymbol{a}_o=\boldsymbol{w}_h^T\begin{bmatrix} 0 & 0 \\ + 1 & 0 \\ + 1 & 0 \\ + 2 & 1 \end{bmatrix}=\begin{bmatrix} 0 \\ 1 \\ 1 \\0\end{bmatrix}, +\end{split}\]
    +

    the wanted result.

    +

    Setting up the Neural Network¶

    We define first our design matrix and the various output vectors for the different gates.

    @@ -715,10 +789,6 @@ inputs \(x_1\) and probabilities = sigmoid(z_o) return probabilities -# we obtain a prediction by taking the class with the highest likelihood -def predict(X): - probabilities = feed_forward(X) - return np.argmax(probabilities, axis=1) # ensure the same random numbers appear every time np.random.seed(0) @@ -736,7 +806,7 @@ inputs \(x_1\) and # Defining the neural network n_inputs, n_features = X.shape n_hidden_neurons = 2 -n_categories = 2 +n_categories = 1 n_features = 2 # we make the weights normally distributed using numpy.random.randn @@ -751,19 +821,14 @@ inputs \(x_1\) and probabilities = feed_forward(X) print(probabilities) - - -predictions = predict(X) -print(predictions)
    -
    [[0.80625657 0.36420967]
- [0.90297441 0.30170017]
- [0.89823921 0.28566769]
- [0.93420126 0.25920793]]
-[0 0 0 0]
+
    [[0.61238907]
+ [0.61939429]
+ [0.73482109]
+ [0.70115106]]
    @@ -795,10 +860,7 @@ inputs \(x_1\) and yAND = np.array( [ 0, 0 ,0, 1]) # Defining the neural network -n_inputs, n_features = X.shape n_hidden_neurons = 2 -n_categories = 2 -n_features = 2 eta_vals = np.logspace(-5, 1, 7) lmbd_vals = np.logspace(-5, 1, 7) @@ -1054,7 +1116,7 @@ Accuracy score on data set: 0.5 warnings.warn(
    -_images/exercisesweek43_11_2.png +_images/exercisesweek43_26_2.png @@ -8394,9 +8456,8 @@ case.

    Adam: Eta=0.0001, Lambda=0
-
    -
    -
      [----------------------------------------] 0.000% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.4 | val_acc: 0.497 
+
+  [----------------------------------------] 0.000% | train_error: 11.3 | train_acc: 0.453 | val_error: 10.4 | val_acc: 0.497 
 
      [----------------------------------------] 0.1000% | train_error: 11.4 | train_acc: 0.448 | val_error: 10.4 | val_acc: 0.497 
@@ -14455,6 +14516,3039 @@ digits between the range of 0 to 9.
    +
    +

    Testing the XOR gate and other gates¶

    +

    Let us now use our code to test the XOR gate.

    +
    +
    +
    X = np.array([ [0, 0], [0, 1], [1, 0],[1, 1]],dtype=np.float64)
+
+# The XOR gate
+yXOR = np.array( [[ 0], [1] ,[1], [0]])
+
+input_nodes = X.shape[1]
+output_nodes = 1
+
+logistic_regression = FFNN((input_nodes, output_nodes), output_func=sigmoid, cost_func=CostLogReg, seed=2023)
+logistic_regression.reset_weights() # reset weights such that previous runs or reruns don't affect the weights
+scheduler = Adam(eta=1e-1, rho=0.9, rho2=0.999)
+scores = logistic_regression.fit(X, yXOR, scheduler, epochs=1000)
+
    +
    +
    +
    +
    Adam: Eta=0.1, Lambda=0
+
+  [----------------------------------------] 0.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.1000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.2000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.3000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.4000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.5000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.6000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.7000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.8000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 0.9000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 1.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 2.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 2.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 2.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 2.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [----------------------------------------] 2.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 2.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 2.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 2.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 2.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 2.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 3.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [>---------------------------------------] 4.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 5.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 6.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 7.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 7.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 7.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 7.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=>--------------------------------------] 7.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 7.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 7.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 7.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 7.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 7.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 8.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.000% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.100% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.200% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.300% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.400% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.500% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.600% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.700% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.800% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==>-------------------------------------] 9.900% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 10.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 11.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 12.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 12.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 12.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 12.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===>------------------------------------] 12.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 12.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 12.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 12.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 12.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 12.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 13.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====>-----------------------------------] 14.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 15.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 16.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 17.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 17.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 17.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 17.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====>----------------------------------] 17.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 17.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 17.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 17.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 17.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 17.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 18.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======>---------------------------------] 19.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 20.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 21.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 22.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 22.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 22.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 22.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======>--------------------------------] 22.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 22.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 22.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 22.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 22.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 22.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 23.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========>-------------------------------] 24.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 25.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 26.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 27.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 27.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 27.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 27.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========>------------------------------] 27.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 27.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 27.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 27.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 27.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 27.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 28.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========>-----------------------------] 29.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 30.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 31.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 32.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 32.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 32.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 32.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========>----------------------------] 32.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 32.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 32.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 32.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 32.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 32.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 33.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============>---------------------------] 34.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 35.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 36.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 37.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 37.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 37.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 37.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============>--------------------------] 37.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 37.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 37.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 37.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 37.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 37.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 38.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==============>-------------------------] 39.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 40.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 41.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 42.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 42.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 42.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 42.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===============>------------------------] 42.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 42.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 42.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 42.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 42.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 42.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 43.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [================>-----------------------] 44.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 45.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 46.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 47.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 47.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 47.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 47.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=================>----------------------] 47.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 47.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 47.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 47.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 47.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 47.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 48.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.10% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.30% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.50% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.70% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==================>---------------------] 49.90% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 50.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 50.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 50.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 50.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 50.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 50.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 50.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 50.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 50.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 50.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 51.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 51.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 51.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 51.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 51.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 51.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 51.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 51.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 51.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 51.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 52.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 52.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 52.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===================>--------------------] 52.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================>--------------------] 52.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 52.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 52.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 52.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 52.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 52.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 53.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 53.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 53.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 53.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 53.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 53.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 53.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 53.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 53.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 53.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 54.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 54.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 54.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 54.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 54.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 54.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 54.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 54.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================>-------------------] 54.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [====================>-------------------] 54.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 55.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 55.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 55.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 55.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 55.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 55.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 55.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 55.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 55.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 55.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 56.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 56.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 56.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 56.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 56.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 56.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 56.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 56.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 56.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 56.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 57.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 57.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 57.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=====================>------------------] 57.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================>------------------] 57.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 57.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 57.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 57.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 57.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 57.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 58.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 58.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 58.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 58.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 58.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 58.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 58.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 58.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 58.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 58.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 59.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 59.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 59.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 59.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 59.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 59.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 59.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 59.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================>-----------------] 59.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [======================>-----------------] 59.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 60.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 60.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 60.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 60.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 60.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 60.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 60.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 60.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 60.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 60.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 61.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 61.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 61.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 61.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 61.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 61.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 61.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 61.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 61.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 61.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 62.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 62.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 62.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=======================>----------------] 62.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=======================>----------------] 62.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 62.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 62.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 62.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 62.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 62.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 63.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 63.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 63.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 63.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 63.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 63.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 63.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 63.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 63.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 63.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 64.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 64.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 64.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 64.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 64.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 64.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 64.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 64.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [========================>---------------] 64.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [========================>---------------] 64.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 65.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 65.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 65.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 65.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 65.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 65.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 65.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 65.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 65.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 65.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 66.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 66.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 66.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 66.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 66.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 66.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 66.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 66.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 66.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 66.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 67.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 67.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 67.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=========================>--------------] 67.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=========================>--------------] 67.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 67.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 67.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 67.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 67.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 67.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 68.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 68.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 68.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 68.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 68.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 68.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 68.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 68.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 68.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 68.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 69.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 69.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 69.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 69.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 69.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 69.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 69.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 69.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==========================>-------------] 69.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [==========================>-------------] 69.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 70.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 70.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 70.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 70.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 70.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 70.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 70.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 70.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 70.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 70.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 71.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 71.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 71.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 71.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 71.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 71.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 71.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 71.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 71.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 71.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 72.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 72.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 72.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [===========================>------------] 72.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===========================>------------] 72.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 72.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 72.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 72.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 72.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 72.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 73.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 73.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 73.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 73.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 73.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 73.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 73.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 73.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 73.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 73.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 74.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 74.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 74.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 74.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 74.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 74.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 74.60% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 74.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [============================>-----------] 74.80% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [============================>-----------] 74.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 75.00% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============================>----------] 75.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 75.20% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============================>----------] 75.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 75.40% | train_error: 10.4 | train_acc: 0.500 
+
    +
    +
      [=============================>----------] 75.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 75.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 75.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 75.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 75.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 76.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 77.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 77.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 77.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 77.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=============================>----------] 77.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 77.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 77.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 77.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 77.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 77.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 78.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==============================>---------] 79.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 80.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 81.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 82.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 82.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 82.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 82.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===============================>--------] 82.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 82.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 82.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 82.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 82.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 82.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 83.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [================================>-------] 84.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 85.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 86.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 87.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 87.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 87.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 87.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=================================>------] 87.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 87.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 87.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 87.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 87.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 87.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 88.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [==================================>-----] 89.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 90.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 91.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 92.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 92.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 92.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 92.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [===================================>----] 92.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 92.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 92.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 92.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 92.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 92.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 93.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [====================================>---] 94.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 95.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 96.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 97.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 97.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 97.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 97.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [=====================================>--] 97.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 97.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 97.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 97.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 97.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 97.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 98.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.00% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.10% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.20% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.30% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.40% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.50% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.60% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.70% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.80% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
      [======================================>-] 99.90% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
                                                                                                       
+
    +
    +
      [=======================================>] 100.0% | train_error: -0.0000000010 | train_acc: 1.00 
+
    +
    +
    +
    +

    Not bad, but the results depend strongly on the learning reate. Try different learning rates.

    +